A behavioral posture recognition analysis method and system

Abstract

The application discloses a behavior gesture recognition analysis method and system, and belongs to the field of computer vision, which comprises the following steps: performing target detection on a preprocessed image to obtain a cropped image of a single target object; performing high-dimensional gesture key point region extraction on the cropped image of the single target object to obtain a high-dimensional heat map; splicing the high-dimensional heat map and the cropped image of the single target object to obtain a first spliced image; inputting the first spliced image into a multi-level multi-scale feature extraction module; in each multi-scale feature extraction module, a multi-scale convolution unit performs multi-scale feature extraction on a feature image, and a spatial attention unit adaptively allocates weights to each region in the feature image extracted by the multi-scale convolution unit; inputting the output of the multi-level multi-scale feature extraction module into a key point classification head to obtain a key point classification result, and obtaining a behavior gesture recognition result according to the key point classification result; and the application can improve the accuracy of the gesture recognition result.

Description

Technical Field

[0001] This invention relates to the technical field of attitude estimation, and more particularly to an attitude estimation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, the development of deep learning technology has driven progress in pose recognition. By using deep neural networks, features can be extracted from image or video data, and the locations of key points on the human body can be identified, thereby inferring the human pose.

[0003] Currently, deep learning-based algorithms have made significant progress in the field of human pose estimation, but some challenges remain to be addressed. On the one hand, existing technologies have limitations in handling different scales and spatial locations, lacking effective fusion of multi-scale and spatial aspects, which affects the accuracy and robustness of pose estimation. On the other hand, human movements and poses are diverse, and the imaging environment can be very complex, leading to problems such as self-occlusion and mutual occlusion, further increasing the challenges of pose estimation. Summary of the Invention

[0004] According to one aspect of this disclosure, a pose estimation method is provided, comprising the following steps:

[0005] Obtain the original image containing the target object, and then annotate the original image;

[0006] Preprocess the original image;

[0007] The target detection model is used to detect targets in the preprocessed image to determine the location information of the target in the image. Based on the location information, the image is cropped to obtain a cropped image of a single target.

[0008] The heatmap feature extraction module is used to extract high-dimensional pose key point regions from the cropped image of a single target object to obtain a high-dimensional heatmap;

[0009] The high-dimensional heat map and the cropped image of a single target object are stitched together to obtain the first stitched image;

[0010] The first stitched image is input into the multi-level multi-scale feature extraction module, which includes multiple layers of multi-scale feature extraction modules connected in sequence. Each layer of multi-scale feature extraction module includes a connected multi-scale convolutional unit and a spatial attention unit. In each layer of multi-scale feature extraction module, the multi-scale convolutional unit performs multi-scale feature extraction on the feature image, and the spatial attention unit performs adaptive weight allocation on each region in the feature image extracted by the multi-scale convolutional unit of this layer.

[0011] The output of the multi-level, multi-scale feature extraction module is input into the key point classification head to obtain the key point classification result, and the behavior pose recognition result is obtained based on the key point classification result.

[0012] During the training of the heatmap feature extraction module and the multi-level multi-scale feature extraction module, the heatmap representations from each multi-level multi-scale feature extraction module are input into the multi-dimensional heatmap prediction head, and then the outputs of each multi-dimensional heatmap prediction head are fused to obtain the multi-dimensional heatmap. The heatmap feature extraction module and the multi-level multi-scale feature extraction module are trained using high-dimensional heatmap loss, multi-dimensional heatmap loss, and key point classification loss.

[0013] Specifically, the step of obtaining the original image containing the target object and annotating the original image includes:

[0014] The original images or videos of human behavior and postures are captured by a camera in an electronic device, and images of the target object are obtained from the original images or videos.

[0015] If it is the original video, the original video is split into individual images frame by frame, and images that meet the resolution requirements are selected as valid images; if it is the original image, clear images are directly selected as valid images.

[0016] The target objects in the valid image are labeled with key points and categories. Based on the key points and a Gaussian kernel function with a fixed standard deviation, an initial heatmap of the target key points is generated.

[0017] Specifically, the preprocessing of the original image includes:

[0018] The original image is decomposed by wavelet transform, and then the details at different scales are denoised in the wavelet domain.

[0019] Image enhancement is achieved through adaptive histogram equalization.

[0020] Preferably, the target detection model is one of YOLOV7, YOLOV8, EfficientDet, RetinaNet, or Faster R-CNN.

[0021] Furthermore, the heatmap feature extraction module consists of a multi-layer downsampling unit and a high-dimensional heatmap prediction head, with the enhanced image X_img as input and the predicted high-dimensional heatmap H_h as output;

[0022] The multi-layer downsampling unit comprises three sequentially connected downsampling convolutional structures. Each downsampling convolutional structure includes a first convolutional layer, a first max-pooling layer, and a second convolutional layer. The first convolutional layer has a kernel size of k1*k1 and a stride of 1. The first max-pooling layer has a kernel size of 2 and a stride of 2. The second convolutional layer has a kernel size of k2*k2 and a stride of 1.

[0023] The high-dimensional heatmap prediction head includes a first prediction head, a second prediction head, and a third prediction head. The output of the high-dimensional heatmap prediction head is the result of merging the outputs of the first, second, and third prediction heads. The first prediction head receives the output of the first downsampled convolutional structure, the second prediction head receives the output of the second downsampled convolutional structure, and the third prediction head receives the output of the third downsampled convolutional structure. The first prediction head includes a fourth convolutional layer and a first upsampled layer. The kernel size of the fourth convolutional layer is 1*1 and the stride is 1. The first upsampled layer is upsampled by 2 times. The second prediction head includes a fifth convolutional layer and a second upsampled layer. The kernel size of the fifth convolutional layer is 1*1 and the stride is 1. The second upsampled layer is upsampled by 4 times. The third prediction head includes a sixth convolutional layer and a third upsampled layer. The kernel size of the sixth convolutional layer is 1*1 and the stride is 1. The third upsampled layer is upsampled by 8 times.

[0024] Furthermore, the multi-level multi-scale feature extraction module includes four layers of multi-scale feature extraction modules connected in sequence;

[0025] In the 4-layer multi-scale feature extraction module, each multi-scale convolutional unit includes three parallel dilated convolutional structures. These three parallel dilated convolutional structures are used to extract features of different scales from the input feature image.

[0026] The spatial attention unit is used to obtain the weight matrix. The spatial attention unit includes three parallel weight matrix acquisition units, each with the same structure. The specific operation of each weight matrix acquisition unit is as follows:

[0027] Assuming the input feature image has dimensions C1*H1*W1, global pooling is performed on the input feature image along the horizontal axis to obtain pooling matrix one; then, a window of size 3 is used to perform a moving average on pooling matrix one along the vertical axis with a stride of 3 to obtain an attention matrix of dimensions C1*1*W1.

[0028] Global pooling is performed on the input feature image along the vertical axis to obtain pooling matrix 2; then, a window of size 3 is used to perform a moving average on pooling matrix 2 in the horizontal direction with a step size of 3 to obtain an attention matrix of dimension C1*H1*1.

[0029] Matrix multiplication is used to multiply the attention matrix of dimension C1*H1*1 and the attention matrix of dimension C*1*W to obtain the total attention matrix of dimension C1*H1*W1.

[0030] The values ​​of the total attention matrix are scaled using the Sigmoid function to obtain a weight matrix with weights constrained between 0 and 1.

[0031] The weight matrices obtained by the three parallel weight matrix acquisition units are the first weight matrix, the second weight matrix, and the third weight matrix, respectively. The outputs of the three parallel dilated convolutional structures in the same layer are multiplied by the three parallel weight matrices (connected through the residual structure, as shown in the figure). The results of the three multiplications are added together, and the sum is input into the next layer multi-scale feature extraction module.

[0032] Furthermore, the dilated convolutional structure includes a dilated convolutional layer, a second max pooling layer, and a batch normalization (BN) layer. The dilated convolutional layers in the three parallel dilated convolutional structures are a first dilated convolutional layer, a second dilated convolutional layer, and a third dilated convolutional layer, respectively. The parameters of the first dilated convolutional layer are: Kernel_size = 5*5, padding = 2, stride = 1, and dilation = 2. The parameters of the second dilated convolutional layer are: Kernel_size = 4*4, padding = 1, stride = 1, and dilation = 2. The parameters of the third dilated convolutional layer are: Kernel_size = 3*3, padding = 1, stride = 2, and dilation = 1.

[0033] Furthermore, the high-dimensional heatmap loss, the multi-dimensional heatmap loss, and the keypoint classification loss are specifically loss1, loss2, and loss3, as follows:

[0034] loss1 = MSE(H_h, Heat_1)

[0035] Where H_h represents the predicted high-dimensional heat map, Heat_1 represents the actual high-dimensional heat map, and MSE represents the mean squared error loss.

[0036] loss2 = MSE(Heatmap, Heat_2)

[0037]

[0038] sum = 1 / loss min1 +1 / loss min2 +1 / loss min3 +1 / loss min4

[0039] Where, loss mini Let xi represent the heatmap loss of the i-th layer, which is the predicted heatmap and the previous prediction loss of the corresponding label of the i-th layer multi-scale convolutional unit and spatial attention unit in the multi-level multi-scale feature extraction module. xi represents the predicted heatmap of the i-th layer multi-scale feature extraction module, and ai represents the weight parameter corresponding to xi, where i is 1, 2, 3, or 4. Heatmap represents the predicted multi-dimensional heatmap; Heat_2 represents the true multi-dimensional heatmap.

[0040] loss3 = CES(pred, label)

[0041] Where pred is the predicted category of each heatmap location, label is the category of the actual heatmap location, and CES is the cross-entropy loss.

[0042] The present invention also discloses a behavior posture recognition system, which is used to perform behavior posture recognition on images or videos, and the method used is one of the behavior posture recognition methods described above.

[0043] After adopting the above solution, the beneficial effects of the present invention are as follows:

[0044] (1) By performing target detection and high-dimensional heatmap feature enhancement on the target area, the target area can be effectively selected, avoiding interference from occlusion information. Furthermore, due to image preprocessing operations, the interference from image noise information can be reduced.

[0045] (2) The pose key point regions are extracted from the image through the heat map feature extraction module and the multi-level multi-scale feature extraction module, and multi-level progressive feature extraction is performed. The spatial attention module is used to realize the effective region determination and weight adaptive allocation of key point information. Then, the multi-scale features are weighted and fused to obtain the multi-dimensional key point heat map and key point classification results. The accuracy of the model is improved by combining multiple loss functions. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the process of the present invention;

[0047] Figure 2 This is a schematic diagram illustrating the thermal properties of the present invention;

[0048] Figure 3 This is a schematic diagram of the heatmap feature extraction module;

[0049] Figure 4 This is a schematic diagram of a multi-level, multi-scale feature extraction module;

[0050] Figure 5 This is a schematic diagram illustrating the process of obtaining the weight matrix for a spatial attention unit. Detailed Implementation

[0051] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0052] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0053] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0054] It should be understood that the term "and / or" used in this document is merely a description of the same field in the related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0055] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0056] Figure 1 A flowchart illustrating an exemplary embodiment of the attitude estimation method of this disclosure is shown. Figure 1 As shown,

[0057] The attitude estimation method of the exemplary embodiments of this disclosure includes:

[0058] S100. Obtain the original image containing the target object, and perform annotation processing on the original image; this step specifically includes:

[0059] S110. Capture raw images or raw videos of human behavior and posture using a camera in an electronic device, wherein the raw images or raw videos contain images of target objects; the target objects may be human or animal objects with key point location information that require posture estimation.

[0060] S120. If it is the original video, the original video is split into individual images frame by frame, and images that meet the resolution requirements are selected as valid images; if it is the original image, clear images are directly selected as valid images; clear images can be selected according to specific pixel requirements or according to the top 50% of clarity. For example, here the resolution requirement is at least 1024*1024 or 2048*2048, depending on the device.

[0061] The target objects in the valid image are labeled with key points and categories. Based on the key points and a Gaussian kernel function with a fixed standard deviation, an initial heatmap of the target key points is generated.

[0062] In practical applications, the labeling of key points requires professional personnel, and the fixed value of the standard deviation needs to be confirmed based on the actual image data containing the target object and the imaging pixel information. For example, Figure 2 A schematic diagram of image annotations for this specific embodiment is shown, such as... Figure 2 As shown, with coordinates (3, 3) as the keypoint, the coordinate distance scores between all neighboring pixels and the keypoint are calculated. A higher coordinate distance score indicates a closer distance, while a lower score indicates a farther distance. When the coordinate distance score is less than 0.5, it is set to 0. Based on the keypoint locations of the target object, a Gaussian kernel function with a standard deviation of 2 is selected to generate an initial heatmap of the keypoints in the image, and their categories are labeled according to their keypoint information.

[0063] The formula for calculating the coordinate distance fraction is as follows:

[0064]

[0065] Where yi and zi are the coordinates to be calculated, respectively. These represent the coordinates of key points in the image, and σ is the standard deviation. In this specific implementation, σ = 2. In actual needs, it can be customized according to the actual situation.

[0066] Step 200. Preprocess the original image, which includes denoising and enhancement. The original image is decomposed using wavelet transform, and then denoising is performed on details at different scales in the wavelet domain. Image enhancement is then achieved through adaptive histogram equalization. Specifically, the image is divided into small blocks (preferably 4 or 9 blocks), and histogram equalization is performed within each block to enhance local contrast in the processed image.

[0067] Step 300. Utilize an object detection model to detect objects in the preprocessed image, determining the location information of the target objects within the image. Based on this location information, crop the image to obtain a cropped image of a single target object. The object detection model can be one of YOLOv7, YOLOv8, EfficientDet, RetinaNet, or Faster R-CNN, but is not limited to these; any common object detection model is acceptable. The object detection model is an existing technology; the annotation of key points and the training and prediction processes of the object detection model will not be elaborated upon here. Furthermore, in practical applications, the selection of the pre-trained model should be determined based on the actual equipment and response speed requirements, taking into account the specific circumstances; no specific explanation will be provided here.

[0068] Step 400. Use the heatmap feature extraction module to extract high-dimensional pose key point regions from the cropped image of a single target object, obtaining a high-dimensional heatmap; the heatmap feature extraction module is as follows: Figure 3 As shown, the heatmap feature extraction module consists of a multi-layer downsampling unit and a high-dimensional heatmap prediction head. The input is the enhanced image X_img, and the output is the predicted high-dimensional heatmap H_h.

[0069] The multi-layer downsampling unit comprises three sequentially connected downsampling convolutional structures. Each downsampling convolutional structure includes a first convolutional layer, a first max-pooling layer, and a second convolutional layer. The first convolutional layer has a kernel size of k1*k1, a stride of 1, c1 input channels, and c2 output channels. The first max-pooling layer has a kernel size of 2 and a stride of 2. The second convolutional layer has a kernel size of k2*k2, a stride of 1, c2 input channels, and c2 output channels.

[0070] The kernel k1 of each downsampling convolutional structure remains constant, while c1 and c2 vary with the number of layers. The first downsampling convolutional module uses c1 and c2 of 3 (when the input data is a 3-channel image) or 1 (when the input image is a grayscale image) and 64, respectively. The second downsampling convolutional module uses c1 and c2 of 64 and 128, respectively. The third downsampling convolutional module uses c1 and c2 of 128 and 256, respectively.

[0071] The high-dimensional heatmap prediction head includes a first prediction head, a second prediction head, and a third prediction head. The output of the high-dimensional heatmap prediction head is the result of merging the outputs of the first, second, and third prediction heads; merging here means adding and then averaging. The first prediction head receives the output of the first downsampling convolutional structure, the second prediction head receives the output of the second downsampling convolutional structure, and the third prediction head receives the output of the third downsampling convolutional structure. The first prediction head includes a fourth convolutional layer and a first upsampling layer; the kernel size of the fourth convolutional layer is 1. *1. The stride is 1, the number of input channels is c2, the number of output channels is 1, and the first upsampling layer is upsampled by 2x. The second prediction head includes a fifth convolutional layer and a second upsampling layer. The kernel size of the fifth convolutional layer is 1*1, the stride is 1, the number of input channels is c3, the number of output channels is 1, and the second upsampling layer is upsampled by 4x. The third prediction head includes a sixth convolutional layer and a third upsampling layer. The kernel size of the sixth convolutional layer is 1*1, the stride is 1, the number of input channels is c4, the number of output channels is 1, and the third upsampling layer is upsampled by 8x.

[0072] Specifically, the heatmap feature extraction module takes an image X_img as input, with dimensions C*H*W. After passing through the first downsampling convolutional structure, it obtains a feature image X_1 with dimensions c2*H / 2*W / 2. After passing through the second downsampling convolutional structure, it obtains a feature image X_2 with dimensions c3*H / 4*W / 4. After passing through the second downsampling convolutional structure, it obtains a feature image X_3 with dimensions c4*H / 8*W / 8.

[0073] The inputs to the first, second, and third prediction heads of the high-dimensional heatmap prediction head are feature images X_1, X_2, and X_3, respectively. The feature channels are compressed using 1x1 convolution operations, reducing the number of channels from c2, c3, and c4 to 1. Then, the features are scaled up using different upsampling methods (2x, 4x, and 8x) to the size of image X_img, at which point the dimension is 1*H*W. The three sampling results are then numerically merged to obtain the high-dimensional heatmap estimate H_h, using the following formula:

[0074] H_h=(H1+H2+H3) / 3

[0075] H1, H2, and H2 represent the three sampling results of the first, second, and third prediction heads, respectively.

[0076] Step 500. Stitch together the high-dimensional heat map and the cropped image of a single target object to obtain the first stitched image;

[0077] Step 600. Input the first stitched image into the multi-level, multi-scale feature extraction module, such as... Figure 4 As shown, the multi-level, multi-scale feature extraction module includes multiple interconnected multi-level multi-scale feature extraction modules. Each multi-level multi-scale feature extraction module includes interconnected multi-scale convolutional units and spatial attention units. In each multi-level multi-scale feature extraction module, the multi-scale convolutional unit performs multi-scale feature extraction on the feature image (the input of the first multi-scale feature extraction module is the first stitched image), and the spatial attention unit adaptively assigns weights to each region in the feature map extracted by the multi-scale convolutional unit of this layer. An example of its multi-level, multi-scale feature extraction module is shown below. Figure 4 As shown in the diagram, only a portion of the layers are displayed; the remaining overlapping layers are not shown, but they can still be deduced from the text. Figure 4 The complete structure of the multi-layer, multi-scale feature extraction module is obtained. In practice, those skilled in the art can customize the padding size, and an upsampling module can be added to correct the dimensionality when the calculated output scales are inconsistent.

[0078] The multi-level, multi-scale feature extraction module comprises four sequentially connected multi-scale feature extraction modules. Specifically, each module includes four multi-scale convolutional units and four spatial attention units, arranged alternately with multi-scale convolutional units followed by spatial attention units. One multi-scale convolutional unit and one spatial attention unit constitute one layer of the multi-scale feature extraction module, as shown in the figure. Within the same layer, the spatial attention unit calculates spatial attention weights for the image features extracted by the multi-scale convolutional units, and then uses these attention weights to optimize the extracted image features. Through residual connections, the unoptimized and optimized image features are added together, and the result is input to the next layer.

[0079] In the 4-layer multi-scale feature extraction module, each multi-scale convolutional unit includes three parallel dilated convolutional structures. These three parallel dilated convolutional structures are used to extract features of different scales from the input feature image.

[0080] The spatial attention unit is used to obtain the weight matrix. The spatial attention unit includes three parallel weight matrix acquisition units, each with the same structure, such as... Figure 5 As shown, the specific operation for obtaining each weight matrix unit is as follows:

[0081] Assuming the input feature image has dimensions C1*H1*W1, global pooling is performed on the input feature image along the horizontal axis to obtain pooling matrix one; then, a window of size 3 is used to perform a moving average on pooling matrix one along the vertical axis with a stride of 3 to obtain an attention matrix of dimensions C1*1*W1.

[0082] Global pooling is performed on the input feature image along the vertical axis to obtain pooling matrix 2; then, a window of size 3 is used to perform a moving average on pooling matrix 2 in the horizontal direction with a step size of 3 to obtain an attention matrix of dimension C1*H1*1.

[0083] Matrix multiplication is used to multiply the attention matrix of dimension C1*H1*1 and the attention matrix of dimension C*1*W to obtain the total attention matrix of dimension C1*H1*W1.

[0084] The values ​​of the total attention matrix are scaled using the Sigmoid function to obtain a weight matrix with weights constrained between 0 and 1.

[0085] The weight matrices obtained by the three parallel weight matrix acquisition units are the first weight matrix, the second weight matrix, and the third weight matrix, respectively. The outputs of the three parallel dilated convolutional structures in the same layer are multiplied by the three parallel weight matrices (connected through the residual structure, as shown in the figure). The results of the three multiplications are added together and the sum is input into the next layer multi-scale feature extraction module. Here, if the sizes of the three multiplication results are different, the other two are scaled according to the largest size or according to a fixed uniform size.

[0086] The dilated convolutional structure includes a dilated convolutional layer, a second max-pooling layer, and a batch normalization (BN) layer. The dilated convolutional layers in the three parallel structures are designated as a first dilated convolutional layer, a second dilated convolutional layer, and a third dilated convolutional layer, respectively. The parameters of the first dilated convolutional layer are: Kernel_size 5*5, padding 2, stride 1, and dilation 2. The parameters of the second dilated convolutional layer are: Kernel_size 4*4, padding 1, stride 1, and dilation 2. The parameters of the third dilated convolutional layer are: Kernel_size 3*3, padding 1, stride 2, and dilation 1. The parameters of the max-pooling layers in these three dilated convolutional structures are all: Kernel_size 3*3 and stride 2. In the four multi-scale convolutional units, the number of input channels and output channels of the dilated convolutional structures in each multi-scale convolutional unit are different. The input image of the multi-level, multi-scale feature extraction module is the first stitched image. Therefore, in the dilated convolution structure of the multi-scale convolution unit in the first layer, the input channels are 4 (when the input image is a 3-channel image) or 2 (when the input image is a grayscale image), and the output channels are 64. In the dilated convolution structure of the multi-scale convolution unit in the second layer, the input channels are 64, and the output channels are 128. In the dilated convolution structure of the multi-scale convolution unit in the third layer, the input channels are 128, and the output channels are 256. In the dilated convolution structure of the multi-scale convolution unit in the fourth layer, the input channels are 256, and the output channels are 256.

[0087] Dilated convolution is an existing technique. The formula for calculating the output feature size after dilated convolution is:

[0088]

[0089] Where `out_size` is the height or width of the output feature, `size` is the height or width of the input feature, `p` is the padding size, `k` is the kernel size, `d` is the dilation size, and `s` is the stride size. When the output feature sizes are inconsistent after the input features pass through three parallel dilated convolutional structures in the same layer, the output features of the three parallel dilated convolutional structures in the same layer are sampled and padded according to the largest output feature size of the three output features, specifically using the nearest neighbor interpolation method.

[0090] For the features fused from the i-th layer, i.e., the input features of the (i+1)-th layer, the following is true:

[0091]

[0092] Where, x i+1Xi_1 is the output of the i-th multi-scale feature extraction module and the input of the (i+1)-th multi-scale feature extraction module. Xi_1, Xi_2 and Xi_3 are the outputs of three parallel dilated convolutional structures, and wi_1, wi_2 and wi_3 are the first weight matrix, the second weight matrix and the third weight matrix, respectively.

[0093] Taking the input features of the second layer as an example:

[0094]

[0095] S700. Input the output of the multi-level, multi-scale feature extraction module into the key point classification head to obtain the key point classification result, and obtain the behavior pose recognition result based on the key point classification result.

[0096] Weighted fusion of multi-scale features yields a low-dimensional keypoint heatmap and keypoint classification results. By updating the fused features with multi-layer weights, the features are predicted separately. The prediction results of different layers are weighted according to the overall prediction performance of the prediction results, resulting in a weighted fusion of multi-scale features and a low-dimensional keypoint heatmap. The classification of keypoints is predicted by merging multi-layer prediction features.

[0097] During the training of the heatmap feature extraction module and the multi-level multi-scale feature extraction module, the heatmap representations from each multi-level multi-scale feature extraction module are input into the multi-dimensional heatmap prediction head, and then the outputs of each multi-dimensional heatmap prediction head are fused to obtain the multi-dimensional heatmap. The heatmap feature extraction module and the multi-level multi-scale feature extraction module are trained using high-dimensional heatmap loss, multi-dimensional heatmap loss, and key point classification loss.

[0098] For prediction results of different layers and different dimensions, it is necessary to perform upsampling to change the dimensions to obtain the same dimensions as the original input image.

[0099] The high-dimensional heatmap loss, the multi-dimensional heatmap loss, and the keypoint classification loss are specifically loss1, loss2, and loss3, as follows:

[0100] loss1 = MSE(H_h, Heat_1)

[0101] Where H_h represents the predicted high-dimensional heat map, Heat_1 represents the actual high-dimensional heat map, and MSE represents the mean squared error loss.

[0102] loss2 = MSE(Heatmap, Heat_2)

[0103] Heat_2 represents the actual multidimensional heatmap; Heatmap represents the predicted multidimensional heatmap, which is the result of fusing the outputs of various multidimensional heatmap prediction heads. Specifically, the formula for fusing multidimensional heatmaps is as follows:

[0104]

[0105]

[0106] sum = 1 / loss min1 +1 / loss min2 +1 / loss min3 +1 / loss min4

[0107] Where, loss mini Let xi represent the heatmap loss of the i-th layer, which is the prediction heatmap of the i-th layer multi-scale convolutional unit and spatial attention unit in the multi-level multi-scale feature extraction module and the previous prediction loss of the corresponding label. Let ai represent the prediction heatmap of the i-th layer multi-scale feature extraction module, and let i represent the weight parameter corresponding to xi, where i is 1, 2, 3 or 4. The default initial values ​​of a1, a2, a3 and a4 are 0.25, 0.25, 0.25 and 0.25 respectively.

[0108] loss3 = CES(pred, label)

[0109] Where pred is the predicted category of each heatmap location, label is the category of the actual heatmap location, and CES is the cross-entropy loss.

[0110] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A behavior pose recognition and analysis method, characterized in that, Includes the following steps: Obtain the original image containing the target object, and then annotate the original image; Preprocess the original image; The target detection model is used to detect targets in the preprocessed image to determine the location information of the target in the image. Based on the location information, the image is cropped to obtain a cropped image of a single target. The heatmap feature extraction module is used to extract high-dimensional pose key point regions from the cropped image of a single target object to obtain a high-dimensional heatmap; The high-dimensional heat map and the cropped image of a single target object are stitched together to obtain the first stitched image; The first stitched image is input into the multi-level multi-scale feature extraction module, which includes multiple layers of multi-scale feature extraction modules connected in sequence. Each layer of multi-scale feature extraction module includes a connected multi-scale convolutional unit and a spatial attention unit. In each layer of multi-scale feature extraction module, the multi-scale convolutional unit performs multi-scale feature extraction on the feature image, and the spatial attention unit performs adaptive weight allocation on each region in the feature image extracted by the multi-scale convolutional unit of this layer. The output of the multi-level, multi-scale feature extraction module is input into the key point classification head to obtain the key point classification result, and the behavior pose recognition result is obtained based on the key point classification result. During the training of the heatmap feature extraction module and the multi-level multi-scale feature extraction module, the heatmap representations from each multi-level multi-scale feature extraction module are input into the multi-dimensional heatmap prediction head, and then the outputs of each multi-dimensional heatmap prediction head are fused to obtain the multi-dimensional heatmap. The heatmap feature extraction module and the multi-level multi-scale feature extraction module are trained using high-dimensional heatmap loss, multi-dimensional heatmap loss, and key point classification loss.

2. The behavioral pose recognition and analysis method according to claim 1, characterized in that, The step of acquiring the original image containing the target object and annotating the original image includes: The original images or videos of human behavior and postures are captured by a camera in an electronic device, and images of the target object are obtained from the original images or videos. If it is the original video, the original video is split into individual images frame by frame, and images that meet the resolution requirements are selected as valid images; if it is the original image, clear images are directly selected as valid images. The target objects in the valid image are labeled with key points and categories. Based on the key points and a Gaussian kernel function with a fixed standard deviation, an initial heat map of the target key points is generated.

3. The behavioral pose recognition and analysis method according to claim 1, characterized in that, The preprocessing of the original image includes: The original image is decomposed by wavelet transform, and then the details at different scales are denoised in the wavelet domain. Image enhancement is achieved through adaptive histogram equalization.

4. The behavior pose recognition and analysis method according to claim 1, characterized in that, The target detection model used is one of YOLOV7, YOLOV8, EfficientDet, RetinaNet, or Faster R-CNN.

5. The behavioral pose recognition and analysis method according to claim 1, characterized in that, The heatmap feature extraction module consists of a multi-layer downsampling unit and a high-dimensional heatmap prediction head. The input is the enhanced image X_img, and the output is the predicted high-dimensional heatmap H_h. The multi-layer downsampling unit comprises three sequentially connected downsampling convolutional structures. Each downsampling convolutional structure includes a first convolutional layer, a first max-pooling layer, and a second convolutional layer. The first convolutional layer has a kernel size of k1*k1 and a stride of 1. The first max-pooling layer has a kernel size of 2 and a stride of 2. The second convolutional layer has a kernel size of k2*k2 and a stride of 1. The high-dimensional heatmap prediction head includes a first prediction head, a second prediction head, and a third prediction head. The output of the high-dimensional heatmap prediction head is the result of merging the outputs of the first, second, and third prediction heads. The first prediction head receives the output of the first downsampled convolutional structure, the second prediction head receives the output of the second downsampled convolutional structure, and the third prediction head receives the output of the third downsampled convolutional structure. The first prediction head includes a fourth convolutional layer and a first upsampled layer. The kernel size of the fourth convolutional layer is 1*1 and the stride is 1. The first upsampled layer is 2x upsampled. The second prediction head includes a fifth convolutional layer and a second upsampled layer. The kernel size of the fifth convolutional layer is 1*1 and the stride is 1. The second upsampled layer is 4x upsampled. The third prediction head includes a sixth convolutional layer and a third upsampled layer. The kernel size of the sixth convolutional layer is 1*1 and the stride is 1. The third upsampled layer is 8x upsampled.

6. The behavior pose recognition and analysis method according to claim 1, characterized in that, The multi-level, multi-scale feature extraction module includes four layers of multi-scale feature extraction modules connected in sequence. In the 4-layer multi-scale feature extraction module, each multi-scale convolutional unit includes three parallel dilated convolutional structures. These three parallel dilated convolutional structures are used to extract features of different scales from the input feature image. The spatial attention unit is used to obtain the weight matrix. The spatial attention unit includes three parallel weight matrix acquisition units, each with the same structure. The specific operation of each weight matrix acquisition unit is as follows: Assuming the input feature image has dimensions C1*H1*W1, global pooling is performed on the input feature image along the horizontal axis to obtain pooling matrix one; then, a window of size 3 is used to perform a moving average on pooling matrix one along the vertical axis with a stride of 3 to obtain an attention matrix of dimensions C1*1*W1. Global pooling is performed on the input feature image along the vertical axis to obtain pooling matrix 2; then, a window of size 3 is used to perform a moving average on pooling matrix 2 in the horizontal direction with a step size of 3 to obtain an attention matrix of dimension C1*H1*1. Matrix multiplication is used to multiply the attention matrix of dimension C1*H1*1 and the attention matrix of dimension C*1*W to obtain the total attention matrix of dimension C1*H1*W1. The values ​​of the total attention matrix are scaled using the Sigmoid function to obtain a weight matrix with weights constrained between 0 and 1. The weight matrices obtained by the three parallel weight matrix acquisition units are the first weight matrix, the second weight matrix, and the third weight matrix, respectively. The outputs of the three parallel dilated convolutional structures in the same layer are multiplied by the three parallel weight matrices, and the results of the three multiplications are added together. The result of the addition is then input into the next layer multi-scale feature extraction module.

7. The behavioral pose recognition and analysis method according to claim 6, characterized in that, The dilated convolutional structure includes a dilated convolutional layer, a second max pooling layer, and a batch normalization (BN) layer. The dilated convolutional layers in the three parallel dilated convolutional structures are a first dilated convolutional layer, a second dilated convolutional layer, and a third dilated convolutional layer, respectively. The parameters of the first dilated convolutional layer are: Kernel_size = 5*5, padding = 2, stride = 1, and dilation = 2. The parameters of the second dilated convolutional layer are: Kernel_size = 4*4, padding = 1, stride = 1, and dilation = 2. The parameters of the third dilated convolutional layer are: Kernel_size = 3*3, padding = 1, stride = 2, and dilation = 1.

8. The behavior pose recognition and analysis method according to claim 1, characterized in that, The high-dimensional heatmap loss, the multi-dimensional heatmap loss, and the keypoint classification loss are specifically loss1, loss2, and loss3, as follows: loss1 = MSE(H_h, Heat_1) Where H_h represents the predicted high-dimensional heat map, Heat_1 represents the actual high-dimensional heat map, and MSE represents the mean squared error loss. loss2 = MSE(Heatmap, Heat_2) sum=1 / loss min1 +1 / loss min2 +1 / loss min3 +1 / loss min4 Where, loss mini Let xi represent the heatmap loss of the i-th layer, which is the predicted heatmap and corresponding label loss of the i-th layer multi-scale convolutional unit and spatial attention unit in the multi-level multi-scale feature extraction module, and ai represent the weight parameter corresponding to xi, where i is 1, 2, 3 or 4; Heatmap represents the predicted multi-dimensional heatmap; Heat_2 represents the true multi-dimensional heatmap. loss3 = CES(pred, label) Where pred is the predicted category of each heatmap location, label is the category of the actual heatmap location, and CES is the cross-entropy loss.

9. A behavior posture recognition system, characterized in that, The system is used to perform behavior and pose recognition on images or videos, and the method used is a behavior and pose recognition method as described in any one of claims 1-8.

Images