A method and system for detecting biological posture

Through the combination of multi-scale feature extraction and structure-guided learning modules, multiple mutual contact biological target detection problems are solved, the detection process is simplified, and the efficiency and accuracy of biological posture detection are improved.

CN118212650BActive Publication Date: 2024-11-05TECH & ENG CENT FOR SPACE UTILIZATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410261160.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-07
Publication Date
2024-11-05
Estimated Expiration
2044-03-07

AI Technical Summary

Technical Problem

The existing biological attitude detection technology has low detection accuracy in multiple biological target scenarios that are in contact with each other, and the top-down attitude estimation method requires complex pre-processing and bottom-up methods require cumbersome post-processing, which affects efficiency.

Method used

Multi-scale feature extraction unit is used to obtain the features of the image under different downsampling steps, combining the biological attitude grouping module, multi-scale feature sampling and fusion module, structure guidance learning module and biological attitude estimation module to simplify the detection process and improve the detection accuracy.

Benefits of technology

High-precision biological attitude detection in any scenario is realized, which reduces detection limitations, simplifies the detection process, and improves efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118212650B_ABST
    Figure CN118212650B_ABST
Patent Text Reader

Abstract

The present invention provides a biological posture detection method and system, in which the first image features corresponding to the image to be detected at the scale of at least two downsampling steps are obtained. The image to be detected includes at least one detection target, and any detection target includes at least one limb area. A primary feature corresponding to any detection target corresponds to a limb area in at least one limb area. Based on the number of any first image feature and at least one primary feature corresponding to any detection target, the position of the sampling point for each first image feature is determined. Based on the position of the sampling point, each first image feature is sampled to obtain a second image feature corresponding to each first image feature. Based on the fusion of each second image feature, a one-hop feature corresponding to any detection target is determined. Based on the one-hop feature corresponding to any detection target, the posture of any detection target is determined, which can reduce the limitations of detection and improve the accuracy of detection results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of information processing and analysis of biological experimental observation images, and in particular to a biological posture detection method and system. Background Art

[0002] The development of deep learning has made significant progress in biological posture detection. Based on deep learning, the posture of organisms can be detected.

[0003] However, deep learning-based detection of biological gestures has limitations. For example, deep learning-based detection of biological gestures can only achieve high-precision detection results in scenarios involving a single biological target or multiple biological targets that are not in contact with each other. In scenarios involving multiple biological targets that are in contact with each other, the accuracy of the detection results will be affected because the multiple biological targets in contact with each other may block each other. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a biological posture detection method and system, which can be applicable to biological posture detection in any scenario, can reduce the limitations of detection, and improve the accuracy of detection results.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] On the one hand, the present invention provides a method for detecting biological posture, in which the first image features corresponding to the image to be detected at the scale of at least two downsampling steps are obtained. The image to be detected includes at least one detection target, and each first image feature includes the image feature of at least one detection target. The first-level features corresponding to each of the detection targets in at least one detection target are the same. For any detection target in at least one detection target, any detection target includes at least one limb area. A first-level feature corresponding to any detection target corresponds to a limb area in at least one limb area, and different first-level features corresponding to any detection target correspond to different limb areas. Based on any first image feature and the number of at least one first-level feature corresponding to any detection target, the position of the sampling point for each first image feature is determined. For any first image feature in each first image feature, based on the position of the sampling point, any first image feature is sampled to obtain a second image feature corresponding to any first image feature. Based on the fusion of the second image features corresponding to each first image feature, a one-hop feature corresponding to any detection target is determined. Based on the one-hop feature corresponding to any detection target, the posture of any detection target is determined.

[0007] Based on the above technical solution, the present invention can also be improved as follows.

[0008] Further, the one-hop feature corresponding to any detection target is enhanced to obtain an enhanced one-hop feature. The enhanced one-hop feature has a stronger structural perception capability for any detection target than the one-hop feature. Based on the enhanced one-hop feature corresponding to any detection target, the posture of any detection target is determined.

[0009] Further, the dimension of the one-jump feature is transformed from [P, C, H, W] to [P, H×W, C]. Where P is the number of primary features corresponding to any detection target, C is the number of channels of each first image feature, H is the length of the feature map corresponding to the one-jump feature, and W is the width of the feature map corresponding to the one-jump feature. The one-jump feature of dimension [P, H×W, C] is sampled to determine at least one primary feature corresponding to any detection target. The dimension of at least one primary feature corresponding to any detection target is [P, C]. Based on the target encoding vector and at least one primary feature corresponding to any detection target, an enhanced one-jump feature is obtained. Wherein the dimension of the target encoding vector is [P, C].

[0010] Further, based on at least one primary feature corresponding to any detection target, a value vector corresponding to any detection target is determined. Based on adding at least one primary feature corresponding to any detection target to the target encoding vector, a query vector and a key vector corresponding to any detection target are determined. Based on the value vector, the query vector and the key vector, an enhanced one-hop feature is determined.

[0011] Further, based on at least one primary feature corresponding to any detection target and the enhanced one-jump feature corresponding to any detection target, at least one key point corresponding to each primary feature corresponding to any detection target is determined. Among them, any key point of the at least one key point corresponding to any primary feature is located in the limb area corresponding to any primary feature. Based on each key point corresponding to any detection target, the posture of any detection target is determined.

[0012] Further, determine the center point position corresponding to the center point of any detected target. For any primary feature, determine the first offset of a jump landing point relative to the center point position, and the jump landing point is located in the limb area corresponding to any primary feature. For any primary feature, determine the second offset of a second jump landing point relative to the first landing point, and the position of the second jump landing point is the same as the position of a key point corresponding to any primary feature. Based on the first offset and the second offset corresponding to each primary feature, determine the posture of any detected target.

[0013] Further, based on the number of channels of the target first image feature in each first image feature, the number of channels of each first image feature is unified.

[0014] Furthermore, the number of channels of the target first image feature is the smallest among the first image features.

[0015] Furthermore, based on the weights of the second image features respectively corresponding to the first image features, a weighted sum is performed on the second image features to determine a one-hop feature corresponding to any detection target.

[0016] On the other hand, the present invention provides a biological posture detection system, including a multi-scale feature extraction unit and a posture detection branch unit; the posture detection branch unit includes a biological posture grouping module, a multi-scale feature sampling fusion module, a structure-guided learning module and a biological posture estimation module.

[0017] The multi-scale feature extraction unit is used to obtain the first image features corresponding to the image to be detected at the scales of at least two downsampling steps. The image to be detected includes at least one detection target. Each first image feature includes the image features of at least one detection target. The primary features corresponding to each detection target in at least one detection target are the same. For any detection target in at least one detection target, any detection target includes at least one limb area. A primary feature corresponding to any detection target corresponds to a limb area in at least one limb area, and different primary features corresponding to any detection target correspond to different limb areas.

[0018] The biological posture grouping module is used to determine the position of the sampling point for each first image feature based on the number of at least one primary feature corresponding to any first image feature and any detection target.

[0019] For any first image feature among the first image features, the multi-scale feature sampling and fusion module is used to sample any first image feature based on the position of the sampling point to obtain a second image feature corresponding to any first image feature.

[0020] The multi-scale feature sampling and fusion module is also used to determine a one-hop feature corresponding to any detection target based on fusing the second image features corresponding to each first image feature.

[0021] The structure guided learning module is used to enhance the one-hop feature corresponding to any detection target to obtain an enhanced one-hop feature. The enhanced one-hop feature has a stronger structural perception capability for any detection target than the one-hop feature.

[0022] The biological posture estimation module is used to determine the posture of any detection target based on at least one primary feature corresponding to any detection target and an enhanced one-hop feature corresponding to any detection target.

[0023] The beneficial effect of the present invention is that the biological posture detection method and system provided by the present invention can not only avoid the pre-processing process of the detector for target detection that needs to be designed separately in the current top-down posture estimation method, but also avoid the complex and cumbersome post-processing process of key point grouping that needs to be designed in the current bottom-up posture estimation method, thereby simplifying the detection process and improving detection efficiency.

[0024] Moreover, the biological posture detection method and system provided by the present invention can be applicable to biological posture detection in any scenario, for example, suitable for posture detection of a single biological target, posture detection of multiple non-contact biological targets, and posture detection of multiple contact biological targets, which can reduce the limitations of detection and improve the accuracy of detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the structure of the biological posture detection system provided by the present invention;

[0026] Figure 2 A schematic diagram of the structure of the fruit fly provided by the present invention;

[0027] Figure 3 A schematic diagram of the structure of the nematode provided by the present invention;

[0028] Figure 4 A schematic diagram of the workflow of the multi-scale feature sampling and fusion module provided by the present invention;

[0029] Figure 5 A schematic diagram of the workflow of the structure-guided learning module provided by the present invention;

[0030] Figure 6 A schematic flow chart of the biological posture detection method provided by the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. And, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the differences. At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0032] Posture estimation (also referred to as posture detection in the embodiments of the present application) is a prerequisite task for deep learning behavior analysis and posture tracking of organisms. It can help extract the spatial features of biological behavior, thereby achieving more accurate biological behavior analysis.

[0033] In one existing technology, biological posture can be detected based on a series of behavior analysis tools such as Deepposekit, Deeplabcut, SLEAP, etc.

[0034] Among them, DeepPoseKit is a biological behavior analysis tool based on fruit fly, zebra and locust datasets. It adopts a top-down posture estimation framework, uses an efficient deep learning model StackedDenseNet, and a fast peak detection algorithm based on a graphics processor. The tool is developed based on the Keras and TensorFlow frameworks and uses StackedDenseNet, which is an improvement on the basic network architecture DenseNet. At the same time, DeepPoseKit also provides some visualization and interactive tools to help users better understand and analyze the behavior of organisms.

[0035] However, the DeepPoseKit-based tool is only applicable to detecting a single biological target in a scene including the single biological target, and is not applicable to detecting multiple biological targets in a scene including multiple biological targets.

[0036] Deeplabcut is a biological behavior analysis tool based on deep learning for two-dimensional key point tracking. The tool uses the bottom-up Openpose method in posture estimation to directly predict key points from the global information of the image, and then associates them to obtain the entire posture.

[0037] However, Deeplabcut only supports image-level pose detection for fixed species such as mice and macaques. Moreover, Deeplabcut can only achieve high-precision detection results in scenarios that include a small number of biological targets. As the number of biological targets in the scene increases, the occlusion between the creatures intensifies, and the accuracy and efficiency of pose detection based on Deeplabcut will decrease.

[0038] SLEAP is a biological behavior analysis tool for multi-organism posture estimation and tracking. It is based on seven data sets including flies, bees, mice and gerbils. It adopts top-down and bottom-up deep learning methods in posture estimation algorithms and can directly infer the posture information of organisms from image pixels.

[0039] SLEAP uses a small and lightweight model, UNet. Although the inference speed is improved, the accuracy is reduced, which is about 90% of Deeplabcut. In addition, this analysis tool requires users to configure more training parameters and model parameters, and even select a suitable posture estimation algorithm from more than 30 algorithms provided, which is cumbersome for non-computer science biology users.

[0040] It can be seen that the existing detection of biological posture can only achieve high-precision detection results in the scenario including a single biological target, or a small number of biological targets (for example, 1-3) that are not in contact with each other. In the scenario including multiple biological targets in contact with each other, the accuracy of the detection results will be affected because the multiple biological targets in contact with each other may block each other.

[0041] Furthermore, organisms include people and animals. Existing posture detection for animals is only applicable to arthropods such as fruit flies and locusts, and mammals such as mice and macaques. Currently, there is no posture detection method for linear organisms (e.g., nematodes) that lack bone structures and have variable shapes.

[0042] In addition, the above-mentioned biological posture detection based on a series of behavioral analysis tools such as Deepposekit, Deeplabcut, SLEAP, etc. all need to be based on top-down posture estimation methods or bottom-up posture estimation methods. However, both the top-down posture estimation method and the bottom-up posture estimation method have certain limitations when detecting biological posture.

[0043] For example, the top-down pose estimation method requires training a separate detector to detect biological targets and perform pose estimation based on the detection results. It can be seen that the accuracy of the estimation results obtained based on the top-down pose estimation method is limited by the accuracy of the detector detection. If the detector cannot accurately locate the biological target, the result of the pose estimation will also be affected. In addition, when it is necessary to estimate the poses of multiple biological targets at the same time, the top-down pose estimation method may face speed challenges because it requires independent detection and pose estimation for each biological target. As the number of biological targets increases, the speed of pose estimation will also decrease, affecting the efficiency of pose estimation.

[0044] For another example, although the bottom-up posture estimation method can directly estimate key points in the image to be detected, the bottom-up posture estimation method brings a new challenge: how to determine the identity of the estimated joints? Among them, according to the method of determining the identity of the estimated key points, it can be divided into the key point grouping method and the center regression method. The key point grouping method is to first detect the non-identity key points of all target objects, and then group the key points. However, the key point grouping method requires a specially designed grouping strategy, and the process of designing a grouping strategy is complicated and cumbersome. In addition, most of them are based on heat map regression for key point detection, which introduces large quantization errors, especially for the key points of the small area of ​​the leg of a single fruit fly in the ultra-large field of view in the spatial fruit fly dataset. It is impossible to accurately identify the key points.

[0045] In view of the limitations of the above-mentioned existing biological posture detection methods and other problems, the present invention provides a biological posture detection method and system, which can not only avoid the pre-processing-detector target detection process that needs to be separately designed in the current top-down posture estimation method, but also avoid the complex and cumbersome post-processing-key point grouping process that needs to be designed in the current bottom-up posture estimation method, thereby simplifying the detection process and improving detection efficiency.

[0046] Moreover, the biological posture detection method and system provided by the present invention can be applicable to biological posture detection in any scenario, for example, suitable for posture detection of a single biological target, posture detection of multiple non-contact biological targets, and posture detection of multiple contact biological targets, which can reduce the limitations of detection and improve the accuracy of detection results.

[0047] In some embodiments, the biological posture detection method provided by the present invention can be implemented based on the biological posture detection system provided by the present invention. Figure 1 The biological posture detection system provided by the present invention (also referred to as the system for short) includes a multi-scale feature extraction unit, a center heat map branch unit, a key point heat map branch unit and a posture detection branch unit.

[0048] Among them, the multi-scale feature extraction unit is used to obtain the image features corresponding to the input image at the scales of at least two downsampling steps. The posture detection branch unit can be used to detect the biological target (also referred to as the detection target in the embodiment of the present application) included in the input image, and output the posture of the corresponding detection target. The center heat map branch unit can process the input image and output the center heat map corresponding to the input image. The key point heat map branch unit can process the input image and output the key point heat map corresponding to the input image. In addition, the center heat map branch unit and the key point heat map branch unit can also be used to train the posture detection branch unit to improve the detection accuracy of the posture detection branch unit.

[0049] The following describes in detail the training process of the center heat map branch unit and the key point heat map branch unit for the posture detection branch unit.

[0050] In some embodiments, in combination Figure 1 , the image data set can be input into the multi-scale feature extraction unit, and the output of the multi-scale feature extraction unit is used as the input of the center heat map branch unit, the key point heat map branch unit and the posture detection branch unit. Among them, each image in the image data set includes at least one detection target. For the sake of ease of description, the training process of the posture detection branch unit is described in detail below, taking the image data set including only one image (also referred to as a training image in the embodiment of the present application) as an example.

[0051] After the training image is input into the multi-scale feature extraction unit, the multi-scale feature extraction unit can obtain the first image features corresponding to the training image at the scales of at least two downsampling steps. The training image includes at least one detection target, and the types of the detection targets included in the same training image are the same. For example, the types of the detection targets included in the same training image are all nematodes. Each first image feature includes the image features of at least one detection target. For the sake of ease of description, the following takes the example that the training image only includes one detection target to continue describing the training process of the posture detection branch unit.

[0052] In some embodiments, the detection target included in the training image includes at least one limb region, a primary feature corresponding to the detection target corresponds to a limb region of the detection target, and different primary features corresponding to the detection target correspond to different limb regions.

[0053] For example, see Figure 2 If the detection target included in the training image is a fruit fly, the limb regions included in the fruit fly may be a head region 210, a body region 220, a left forelimb region 231, a left middle limb region 232, a left hind limb region 233, a right forelimb region 241, a right middle limb region 242, a right hind limb region 243, a left wing region 250, and a right wing region 260.

[0054] For example, see Figure 3 If the detection target included in the training image is a nematode, the limb regions included in the nematode may be a head region 310, a front region 320, a middle region 330, a rear region 340 and a tail region 350.

[0055] After the first image feature is input to the center point heat map branch unit, the center point heat map branch unit can predict the center point of each detection target in the input training image and output the center point heat map corresponding to the training image. The center point heat map corresponding to the training image is marked with the center point of each detection target in the training image.

[0056] After the first image feature is input to the key point heat map branch unit, the key point heat map branch unit can predict the key points of each detection target in the input training image and output the key point heat map corresponding to the training image. The key point heat map corresponding to the training image is marked with the key points of each detection target in the training image.

[0057] In some embodiments, in order to improve the performance of the center point heat map branch unit and the key point heat map branch unit (for example, the accuracy of the output results, the output speed, etc.), the true center point position and key point position of each detection target included in the training image can be obtained.

[0058] Based on the true center point position of each detected target, a standard center point heat map corresponding to the training image can be generated. The true center point position of each detected target is marked in the standard center point heat map. Based on the difference between the standard center point heat map and the center point heat map output by the center point heat map branch unit, the model parameters of the center point heat map branch unit are continuously adjusted to improve the performance of the center point heat map branch unit.

[0059] Based on the real key point positions of each detection target, a standard key point heat map corresponding to the training image can be generated. The real key point positions of each detection target are marked in the standard key point heat map. Based on the gap between the standard key point heat map and the key point heat map output by the key point heat map branch unit, the model parameters of the key point heat map branch unit are continuously adjusted to improve the performance of the key point heat map branch unit.

[0060] In some embodiments, see Figure 1 The posture detection branch unit includes a biological posture grouping module, a multi-scale feature sampling and fusion module, a structure-guided learning module and a biological posture estimation module. The training image may include at least one detection target.

[0061] In some embodiments, after the output (multiple first image features) of the multi-scale feature extraction unit is used as the input of the posture detection branch unit, the biological posture grouping module can determine the position of the sampling point for each first image feature based on any first image feature and the number of primary features corresponding to the detection target. Afterwards, for any first image feature among the first image features, the multi-scale feature sampling and fusion module can sample each first image feature based on the position of the acquired sampling point to obtain the second image features corresponding to each first image feature. Among them, one first image feature corresponds to one second image feature, and different first image features correspond to different second image features. Afterwards, the multi-scale feature sampling and fusion module can also determine the one-hop feature corresponding to the detection target based on the fusion of each second image feature.

[0062] After the multi-scale feature extraction unit obtains each first image feature, in order to facilitate subsequent calculations, the number of channels of each first image feature may be compressed to unify the first image features.

[0063] In some embodiments, the multi-scale feature extraction unit may compress the number of channels of each first image feature based on the number of channels of the target first image feature to unify the number of channels of each first image feature to the number of channels of the first image feature, wherein the target first image feature has the smallest number of channels among the first image features.

[0064] For example, see Figure 4After the training image is input to the multi-scale feature extraction unit, the multi-scale feature extraction unit can obtain the first image features corresponding to the three different scales of the detection target at the downsampling step of 1 / 4, 1 / 8, and 1 / 16. Afterwards, the number of channels of each first image feature can be compressed to unify the number of channels of each first image feature.

[0065] In some embodiments, after the multi-scale feature sampling and fusion module determines the one-hop feature corresponding to the detection target, the structure-guided learning module can enhance the one-hop feature corresponding to the detection target to obtain an enhanced one-hop feature. The enhanced one-hop feature has a stronger structural perception capability for the detection target than the one-hop feature.

[0066] Specifically, the structure-guided learning module can learn the structural relationship between each feature corresponding to the detection target based on the multi-head self-attention mechanism, thereby enhancing the ability to perceive the structure of the detection target. Afterwards, the structure-guided learning module can enhance the input content. For example, the structure-guided learning module can output an enhanced one-hop feature based on the input one-hop feature.

[0067] Combine the following Figure 5 The process of outputting one-hop features from the structure-guided learning module is described in detail.

[0068] See also Figure 5 After the one-hop feature is input into the structure-guided learning module, the structure-guided learning module can transform the dimension of the one-hop feature from [P, C, H, W] to [P, H×W, C] through a flattening operation, where P is the number of primary features corresponding to the detection target, C is the number of channels of each first image feature, H is the length of the feature map corresponding to the one-hop feature, and W is the width of the feature map corresponding to the one-hop feature.

[0069] Afterwards, the structure-guided learning module can sample the one-hop feature of dimension [P, H×W, C] to determine at least one first-level feature corresponding to the detection target. The dimension of at least one first-level feature corresponding to the detection target is [P, C]. Afterwards, the structure-guided learning module can initialize a set of learnable parameters to obtain a corresponding encoding vector (which may also be referred to as a target encoding vector in the embodiment of the present application). Among them, the dimension of the target encoding vector is [P, C]. Based on the at least one first-level feature and the target encoding vector, the structure-guided learning module can determine the enhanced one-hop feature.

[0070] In some embodiments, see Figure 5 , the structure-guided learning module can input the at least one primary feature and the target encoding vector into the feature enhancement module to obtain the enhanced one-hop feature.

[0071] Among them, the feature enhancement module includes at least a first computing layer, a second computing layer, a first linear layer, a second linear layer, a multi-head self-attention computing layer and a feedforward propagation layer. After the at least one primary feature and the target coding vector are input into the feature enhancement module, the first computing layer can determine Q (query vector) based on the sum of the at least one primary feature and the target coding vector, and the second computing layer can determine K (key vector) based on the sum of the at least one primary feature and the target coding vector. The first linear layer can determine V (value vector) based on the at least one primary feature. Afterwards, the Q (query vector) determined by the first computing layer, the K (key vector) determined by the second computing layer, and the V (value vector) determined by the first linear layer are used as inputs to the multi-head self-attention computing layer, and the output results are sequentially passed through the second linear layer and the feedforward propagation layer, and the enhanced one-hop feature can be output.

[0072] Among them, the calculation formula of the multi-head self-attention calculation layer for the input Q (query vector), K (key vector) and V (value vector) is as follows:

[0073]

[0074] It should be noted that the at least one primary feature and the target coding vector can be passed through multiple feature enhancement modules in sequence to continuously enhance the one-hop feature. This application does not limit the number of times the one-hop feature is enhanced.

[0075] In some embodiments, the dimension of the one-hop feature output by the feature enhancement module is [P, C]. To facilitate subsequent calculations, the structure-guided learning module may also transform the dimension of the one-hop feature output by the feature enhancement module to obtain an enhanced one-hop feature with a dimension of [P, C, H, W].

[0076] In some embodiments, after the structure-guided learning module determines the enhanced one-jump feature, the biological posture estimation module can determine at least one key point corresponding to each primary feature corresponding to the detection target based on the enhanced one-jump feature corresponding to the detection target, wherein any key point corresponding to any primary feature is located in the limb area corresponding to the any primary feature.

[0077] For example, see Figure 2 In the case where the primary feature of the fruit fly includes the head region 210, the key points corresponding to the head region 210 may be the left eye key point 211, the right eye key point 212, the mouth key point 213, and the neck key point 214, etc. In the case where the primary feature of the fruit fly includes the body region, the key points corresponding to the body region may be the back key point, the tail key point, the abdomen key point, etc.

[0078] In some embodiments, the posture detection branch unit can be trained based on the gap between the key point positions determined in the key point heat map described in the aforementioned embodiments and the key point positions determined by the biological posture estimation module until the training results meet the target conditions to improve the performance of the posture detection branch unit (for example, detection accuracy, detection efficiency, etc.).

[0079] In some embodiments, the posture detection branch unit can also be trained based on the difference between the center point position determined in the center point heat map described in the aforementioned embodiments and the center point position determined by the biological posture estimation module until the training result reaches the target condition to improve the performance of the posture detection branch unit (for example, detection accuracy, detection efficiency, etc.).

[0080] In some embodiments, the loss functions corresponding to the center heat map branch unit, the key point heat map branch unit, and the posture detection branch unit can also be determined respectively. The total loss function can be calculated based on the branch functions corresponding to each branch unit. Based on the total loss function, the system can continue to be trained to adjust the system parameters of the system, thereby improving the performance of the system (for example, detection accuracy, detection efficiency, etc.).

[0081] The method for calculating the total loss function is described in detail below.

[0082] In some embodiments, the center point heat map branch unit can calculate the loss function L by using the FocalLoss with penalty term hm for:

[0083]

[0084] Among them, N represents the number of positive samples of the center point, and P C Represents the predicted confidence score of each pixel and the corresponding true value respectively. α and β are hyperparameters and can be set to 2 and 4.

[0085] The posture detection branch unit is used to predict the offset of key points, and the loss function L can be calculated using L1Loss hm-hp for:

[0086]

[0087] in, and Represent the true value and predicted value of the offset from the center point to each key point respectively.

[0088] The key point heat map branch unit can use FocalLoss with penalty to calculate the loss function L hp for:

[0089]

[0090] Among them, K represents the number of positive samples of key points, and P C Represents the predicted confidence score of each pixel and the corresponding true value respectively. α and β are hyperparameters and can be set to 2 and 4.

[0091] The total loss is calculated by weighted summation of the three loss functions. The total loss is:

[0092] L total =λ hm L hm +λ hm-hp L hm-hp +λ hp L hp ;

[0093] Among them, λ hm , hm-hp and λ hp These are weight parameters. People skilled in the art can set λ based on actual scenarios and requirements. hm , hm-hp and λ hp For example, hm , hm-hp and λ hp The values ​​of are set to 1.

[0094] In some embodiments, after training the system, an image to be detected can be input into the system, and the multi-scale feature extraction unit in the system can extract features from the image to be detected and output a corresponding feature map. Afterwards, the posture detection branch unit in the present application can perform posture detection on each detection target included in the feature map to output the posture of each detection target included in the image to be detected.

[0095] In summary, the posture detection branch unit provided by the present invention includes a biological posture grouping module, a multi-scale feature sampling and fusion module, a structure-guided learning module and a biological posture estimation module.

[0096] The biological posture grouping module is used to model the internal structural relationship of the detection target by obtaining the number of at least one primary feature corresponding to the detection target.

[0097] The multi-scale feature sampling and fusion module is used for adaptive multi-scale feature sampling and fusion of detection targets of different types and sizes. It can improve the representation capability of each primary feature and provide strong feature support for the subsequent posture recognition of different detection targets.

[0098] The structure-guided learning module is used to guide the network to learn the implicit structural features of the detection target, obtain the enhanced one-hop features, and further improve the accuracy of posture detection of detection targets with flexible structures and mutually occluded detection targets.

[0099] The biological posture estimation module is used to determine the posture of the detection target based on the enhanced one-hop feature and the first-level features corresponding to the detection target.

[0100] In some embodiments, the present invention also provides a method for detecting a biological posture, see Figure 6 The method comprises the following steps S601-S606:

[0101] S601: For any detection target included in the image to be detected, determine the number of at least one primary feature corresponding to any detection target.

[0102] The image to be detected includes at least one detection target. Any detection target includes at least one limb region. A primary feature corresponding to any detection target corresponds to a limb region in at least one limb region, and different primary features corresponding to any detection target correspond to different limb regions. For examples of primary features corresponding to detection targets, please refer to the description in the aforementioned embodiment, which will not be repeated here.

[0103] S602: Obtain first image features corresponding to the image to be detected at at least two downsampling step sizes.

[0104] The first image feature includes at least one image feature of a detection target.

[0105] In some embodiments, the number of channels of each first image feature may be unified based on the number of channels of a target first image feature among the first image features.

[0106] In some embodiments, the number of channels of the target first image feature is the smallest among the first image features.

[0107] S603: Determine the position of a sampling point for each first image feature based on the number of at least one primary feature corresponding to any first image feature and any detection target.

[0108] S604: For any first image feature among the first image features, based on the position of the sampling point, sample any first image feature to obtain a second image feature corresponding to the any first image feature.

[0109] The second image feature includes at least one primary feature corresponding to any detection target.

[0110] S605: Determine a one-hop feature corresponding to any detection target based on fusing the second image features corresponding to each of the first image features.

[0111] In some embodiments, based on the weights of the second image features, a weighted sum may be performed on the second image features to determine a one-hop feature corresponding to any detection target.

[0112] S606: Determine the posture of any detection target based on a one-hop feature corresponding to any detection target.

[0113] In some embodiments, the one-jump feature corresponding to any detection target may be enhanced to obtain an enhanced one-jump feature. The enhanced one-jump feature has a stronger structural perception capability for any detection target than the one-jump feature. Based on the enhanced one-jump feature corresponding to any detection target, the posture of any detection target may be determined.

[0114] In some embodiments, the dimension of the one-jump feature can be transformed from [P, C, H, W] to [P, H×W, C]. Where P is the number of primary features corresponding to any detection target, C is the number of channels of each first image feature, H is the length of the feature map corresponding to the one-jump feature, and W is the width of the feature map corresponding to the one-jump feature. The one-jump feature of dimension [P, H×W, C] is sampled to determine at least one primary feature corresponding to any detection target. The dimension of at least one primary feature corresponding to any detection target is [P, C]. Based on the target encoding vector and at least one primary feature corresponding to any detection target, an enhanced one-jump feature is obtained. Wherein the dimension of the target encoding vector is [P, C].

[0115] In some embodiments, based on at least one primary feature corresponding to any detection target, a value vector corresponding to any detection target is determined. Based on adding at least one primary feature corresponding to any detection target to the target encoding vector, a query vector and a key vector corresponding to any detection target are determined. Based on the value vector, the query vector and the key vector, an enhanced one-hop feature is determined.

[0116] In some embodiments, at least one key point corresponding to each primary feature corresponding to any detection target may be determined based on the enhanced first-hop feature corresponding to any detection target. Any key point among the at least one key point corresponding to any primary feature is located in the limb area corresponding to any primary feature. The posture of any detection target is determined based on each primary feature corresponding to any detection target and each key point corresponding to any detection target.

[0117] In some embodiments, the center point position corresponding to the center point of any detection target can be determined. For any primary feature, a first offset of a jump landing point relative to the center point position is determined, and the jump landing point is located in the limb area corresponding to any primary feature. For any primary feature, a second offset of a second jump landing point relative to the first landing point is determined, and the position of the second jump landing point is the same as the position of a key point corresponding to any primary feature. Based on the first offset and the second offset corresponding to each primary feature, the posture of any detection target is determined.

[0118] The biological posture detection method provided by the present invention comprehensively considers the characteristics of biological anatomy and the flexible posture changes between the grouped parts of its limb regions. The total offset of each key point on the detection target relative to the center point of the detection target is determined by calculating two offsets (the first offset and the second offset), that is, the primary feature of the detection target is first determined based on the first-jump landing point, and then the key points of the detection target under the primary feature are determined based on the second-jump landing point. The posture of the detection target can be determined based on the total offset of each key point on the detection target relative to the center point of the detection target. Compared with directly determining the key points of the detection target, the present invention has a stronger structural perception ability for the detection target, and the posture of the detection target obtained is more in line with the actual posture of the detection target.

[0119] It can be seen that the biological posture detection method and system provided by the present invention can not only avoid the pre-processing-detector for target detection that needs to be designed separately in the current top-down posture estimation method, but also avoid the complex and cumbersome post-processing-key point grouping process that needs to be designed in the current bottom-up posture estimation method, thereby simplifying the detection process and improving detection efficiency.

[0120] Moreover, the biological posture detection method and system provided by the present invention can be applicable to biological posture detection in any scenario, for example, suitable for the detection of a single biological target, the detection of multiple non-contact biological targets, and the detection of multiple contact biological targets, which can reduce the limitations of detection and improve the accuracy of detection results.

[0121] In some schemes, multiple embodiments of the present application can be combined, and the combined scheme can be implemented. Optionally, some operations in the process of each method embodiment are optionally combined, and / or the order of some operations is optionally changed. In addition, the execution order between the steps of each process is only exemplary and does not constitute a restriction on the execution order between the steps. There can also be other execution orders between the steps. It is not intended to indicate that the execution order is the only order in which these operations can be performed. A person of ordinary skill in the art will think of a variety of ways to reorder the operations described herein. In addition, it should be noted that the process details involved in a certain embodiment of this article are also applicable to other embodiments in a similar manner, or different embodiments can be used in combination.

[0122] In addition, some steps in the method embodiment may be equivalently replaced by other possible steps. Alternatively, some steps in the method embodiment may be optional and may be deleted in certain usage scenarios. Alternatively, other possible steps may be added to the method embodiment.

[0123] Furthermore, the various method embodiments may be implemented separately or in combination.

[0124] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0125] In the several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0128] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for detecting biological posture, characterized in that: include: Obtain first image features corresponding to the image to be detected at at least two downsampling step sizes; wherein the image to be detected includes at least one detection target; each of the first image features includes the image features of the at least one detection target; the primary features corresponding to each of the at least one detection target are the same; for any of the at least one detection target, the any one detection target includes at least one limb region; one of the primary features corresponding to any one detection target corresponds to a limb region in the at least one limb region, and different primary features corresponding to any one of the detection targets correspond to different limb regions; Determining the position of the sampling point for each of the first image features based on the number of at least one primary feature corresponding to any of the first image features and any of the detection targets; For any first image feature among the first image features, based on the position of the sampling point, sampling the any first image feature to obtain a second image feature corresponding to the any first image feature; the second image feature includes at least one primary feature corresponding to the any detection target; Determine a one-hop feature corresponding to any detection target based on fusing the second image features respectively corresponding to the first image features; Performing enhancement processing on the one-hop feature corresponding to any one of the detection targets to obtain an enhanced one-hop feature; wherein the enhanced one-hop feature has a stronger structural perception capability for the any one of the detection targets than the one-hop feature has for the any one of the detection targets; Determining the posture of any detection target based on the enhanced one-hop feature corresponding to any detection target; The step of performing enhancement processing on the one-hop feature corresponding to any one of the detection targets to obtain the enhanced one-hop feature includes: The dimension of the one-jump feature is transformed from [P, C, H, W] to [P, H×W, C]; wherein P is the number of primary features corresponding to any detection target; C is the number of channels of each of the first image features; H is the length of the feature map corresponding to the one-jump feature; and W is the width of the feature map corresponding to the one-jump feature; Sampling the one-hop feature of dimension [P, H×W, C] to determine at least one primary feature corresponding to any detection target; the dimension of the at least one primary feature corresponding to any detection target is [P, C]; Based on the target encoding vector and at least one primary feature corresponding to any one of the detection targets, an enhanced one-hop feature is obtained; the dimension of the target encoding vector is [P, C].

2. The method according to claim 1, characterized in that The step of obtaining an enhanced one-hop feature based on the target encoding vector and at least one primary feature corresponding to any one of the detected targets includes: Determine a value vector corresponding to any one of the detection targets based on at least one primary feature corresponding to any one of the detection targets; Determine a query vector and a key vector corresponding to any detection target based on adding at least one primary feature corresponding to any detection target to the target encoding vector; An enhanced one-hop feature is determined based on the value vector, the query vector and the key vector.

3. The method according to claim 2, characterized in that The determining the posture of any detection target based on the enhanced one-hop feature corresponding to any detection target includes: Based on at least one primary feature corresponding to any one of the detection targets and the enhanced first-hop feature corresponding to any one of the detection targets, determine at least one key point corresponding to each of the primary features corresponding to any one of the detection targets; wherein any key point among the at least one key point corresponding to any one of the primary features is located in a limb area corresponding to any one of the primary features; Based on the key points corresponding to any one of the detection targets, the posture of the any one of the detection targets is determined.

4. The method according to claim 3, characterized in that The determining the posture of any detection target based on each of the key points corresponding to any detection target includes: Determine the center point position corresponding to the center point of any detection target; For any of the first-level features, determine a first offset of a first-jump landing point relative to the center point position, wherein the first-jump landing point is located in a limb region corresponding to any of the first-level features; for any of the first-level features, determine a second offset of a second-jump landing point relative to the first-jump landing point; the position of the second-jump landing point is the same as the position of a key point corresponding to any of the first-level features; Based on the first offset and the second offset corresponding to each of the primary features, the posture of any one of the detection targets is determined.

5. The method according to claim 4, characterized in that The step of obtaining first image features corresponding to the image to be detected at at least two downsampling step sizes also includes: Based on the number of channels of the target first image feature in each of the first image features, the number of channels of each of the first image features is unified.

6. The method according to claim 5, characterized in that The number of channels of the target first image feature is the smallest among the first image features.

7. The method according to claim 6, characterized in that The step of determining a one-hop feature corresponding to any detection target based on fusing the second image features respectively corresponding to the first image features includes: Based on the weights of the second image features, weighted sum is performed on the second image features to determine a one-hop feature corresponding to any detection target.

8. A biological posture detection system, characterized in that: include: A multi-scale feature extraction unit and a posture detection branch unit; the posture detection branch unit includes a biological posture grouping module, a multi-scale feature sampling fusion module, a structure-guided learning module and a biological posture estimation module; The multi-scale feature extraction unit is used to obtain first image features corresponding to the image to be detected at the scales of at least two downsampling steps; wherein the image to be detected includes at least one detection target; each of the first image features includes the image features of the at least one detection target; the primary features corresponding to each of the at least one detection target are the same; for any of the at least one detection target, the any one detection target includes at least one limb area; one of the primary features corresponding to any one of the detection targets corresponds to a limb area in the at least one limb area, and different primary features corresponding to any one of the detection targets correspond to different limb areas; The biological posture grouping module is used to determine the position of the sampling point for each of the first image features based on the number of at least one primary feature corresponding to any of the first image features and any of the detection targets; For any first image feature among the first image features, the multi-scale feature sampling and fusion module is used to sample the any first image feature based on the position of the sampling point to obtain a second image feature corresponding to the any first image feature; The multi-scale feature sampling and fusion module is further used to determine a one-hop feature corresponding to any detection target based on fusing the second image features respectively corresponding to each of the first image features; The structure-guided learning module is used to enhance the one-hop feature corresponding to any detection target to obtain an enhanced one-hop feature; the enhanced one-hop feature has a stronger structural perception ability for any detection target than the one-hop feature has for any detection target; The biological posture estimation module is used to determine the posture of any detection target based on at least one primary feature corresponding to any detection target and an enhanced one-hop feature corresponding to any detection target; The structure-guided learning module is used to enhance the one-hop feature corresponding to any detection target, and the enhanced one-hop feature includes: The structure-guided learning module transforms the dimension of the one-jump feature from [P, C, H, W] to [P, H×W, C]; wherein P is the number of primary features corresponding to any detection target; C is the number of channels of each of the first image features; H is the length of the feature map corresponding to the one-jump feature; and W is the width of the feature map corresponding to the one-jump feature. The structure-guided learning module samples the one-hop feature of dimension [P, H×W, C] to determine at least one primary feature corresponding to any detection target; the dimension of the at least one primary feature corresponding to any detection target is [P, C]; The structure-guided learning module obtains an enhanced one-hop feature based on a target encoding vector and at least one primary feature corresponding to any one of the detection targets; the dimension of the target encoding vector is [P, C].