Behavior posture recognition analysis method and system

Abstract

The invention discloses a behavior posture recognition analysis method and system, and belongs to the field of computer vision, and the method comprises the steps: carrying out the target detection of a preprocessed image, and obtaining a cut image of a single target object; performing high-dimensional attitude key point region extraction on the cut image of the single target object to obtain a high-dimensional thermodynamic diagram; splicing the high-dimensional thermodynamic diagram and the cut image of the single target object to obtain a first spliced image; the first spliced image is input into a multi-level multi-scale feature extraction module, in each layer of multi-scale feature extraction module, a multi-scale convolution unit performs multi-scale feature extraction on the feature image, and a space attention unit performs weight adaptive distribution on each region in the feature image extracted by the multi-scale convolution unit; inputting the output of the multi-level and multi-scale feature extraction module into a key point classification head to obtain a key point classification result, and obtaining a behavior posture recognition result according to the key point classification result; the accuracy of a posture recognition result can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of attitude estimation, and in particular to an attitude estimation method, device, electronic equipment and storage medium. Background Art

[0002] In recent years, the development of deep learning technology has promoted the progress of posture recognition. By using deep neural networks, features can be extracted from image or video data, and the key points of the human body can be identified to infer the posture of the human body.

[0003] At present, deep learning-based algorithms have made significant research progress in the field of human posture estimation, but there are still some challenges that need to be solved. On the one hand, the existing technology has certain deficiencies in dealing with different scales and spatial positions, and lacks effective fusion of multiple scales and spatial positions, which affects the accuracy and robustness of posture estimation. In addition, human body movements and postures are diverse, and the imaging environment may be very complex, so there are problems such as self-occlusion and mutual occlusion, which further increases the challenge of posture estimation. Summary of the invention

[0004] According to one aspect of the present disclosure, a posture estimation method is provided, comprising the following steps:

[0005] Obtain the original image containing the target object and perform annotation processing on the original image;

[0006] Preprocess the original image;

[0007] The target detection model is used to perform target detection on the preprocessed image, determine the position information of the target object in the image, and crop the image according to the position information to obtain a cropped image of a single target object;

[0008] The heat map feature extraction module is used to extract the high-dimensional posture key point area of ​​the cropped image of a single target object to obtain a high-dimensional heat map;

[0009] Splicing the high-dimensional heat map and the cropped image of a single target object to obtain a first spliced ​​image;

[0010] Inputting the first spliced ​​image into a multi-level multi-scale feature extraction module, the multi-level multi-scale feature extraction module includes multiple layers of multi-scale feature extraction modules connected in sequence, each layer of multi-scale feature extraction modules includes connected multi-scale convolution units and spatial attention units; in each layer of multi-scale feature extraction modules, the multi-scale convolution unit extracts multi-scale features from the feature image, and the spatial attention unit adaptively allocates weights to each region in the feature map extracted by the multi-scale convolution unit of this layer;

[0011] The output of the multi-level and multi-scale feature extraction module is input into the key point classification head to obtain the key point classification result, and the behavior posture recognition result is obtained according to the key point classification result;

[0012] During the training process of the heat map feature extraction module and the multi-level multi-scale feature extraction module, the heat map representations in each layer of the multi-scale feature extraction module are respectively input into the multi-dimensional heat map prediction head, and then the results output by each multi-dimensional heat map prediction head are fused to obtain a multi-dimensional heat map; the heat map feature extraction module and the multi-level multi-scale feature extraction module are trained using high-dimensional heat map loss, multi-dimensional heat map loss, and key point classification loss.

[0013] Specifically, the obtaining of the original image containing the target object and the labeling of the original image include:

[0014] Capturing an original image or original video of a human body's behavior posture through a camera in an electronic device, wherein the original image or original video contains an image of a target object;

[0015] If it is an original video, split the original video into individual images by frame, and select the images that meet the resolution requirements as valid images; if it is an original image, directly select the clear image as the valid image;

[0016] The joint point positions and categories of the target objects in the valid image are annotated, and the initial heat map of the target joint points is generated based on the key points and the Gaussian kernel function with a fixed standard deviation.

[0017] Specifically, the preprocessing of the original image includes:

[0018] The original image is decomposed by wavelet transform, and then the details at different scales are denoised in the wavelet domain;

[0019] Image enhancement via adaptive histogram equalization.

[0020] Preferably, the target detection model adopts one of YOLOV7, YOLOV8, EfficientDet, RetinaNet, and Faster R-CNN.

[0021] Furthermore, the heat map feature extraction module is composed of a multi-layer downsampling unit and a high-dimensional heat map prediction head, the input is the enhanced image X_img, and the output is the predicted high-dimensional heat map H_h;

[0022] Among them, the multi-layer downsampling unit includes three layers of downsampling convolution structures connected in sequence; the downsampling convolution structure includes a first convolution layer, a first maximum pooling layer and a second convolution layer, the convolution kernel size of the first convolution layer is k1*k1, and the step size is 1; the convolution kernel size of the first maximum pooling layer is 2, and the step size is 2; the convolution kernel size of the second convolution layer is k2*k2, and the step size is 1;

[0023] The high-dimensional heat map prediction head includes a first prediction head, a second prediction head and a third prediction head. The output of the high-dimensional heat map prediction head is the result of merging the outputs of the first prediction head, the second prediction head and the third prediction head. The first prediction head receives the output of the first layer of downsampling convolution structure, the second prediction head receives the output of the second layer of downsampling convolution structure, and the third prediction head receives the output of the third layer of downsampling convolution structure. The first prediction head includes a fourth convolution layer and a first upsampling layer. The convolution kernel size of the fourth convolution layer is 1*1 and the step size is 1. The first upsampling layer is 2 times upsampling. The second prediction head includes a fifth convolution layer and a second upsampling layer. The convolution kernel size of the fifth convolution layer is 1*1 and the step size is 1. The second upsampling layer is 4 times upsampling. The third prediction head includes a sixth convolution layer and a third upsampling layer. The convolution kernel size of the sixth convolution layer is 1*1 and the step size is 1. The third upsampling layer is 8 times upsampling.

[0024] Further, the multi-level multi-scale feature extraction module includes 4 layers of multi-scale feature extraction modules connected in sequence;

[0025] In the 4-layer multi-scale feature extraction module, the multi-scale convolution unit of each layer includes three parallel dilated convolution structures, which are used to extract features of different scales from the input feature image;

[0026] The spatial attention unit is used to obtain a weight matrix. The spatial attention unit includes three parallel weight matrix acquisition units. The structure of each weight matrix acquisition unit is the same. The specific operation of each weight matrix acquisition unit is as follows:

[0027] Assuming that the dimension of the input feature image is C1*H1*W1, perform global pooling on the input feature image on the horizontal axis to obtain the pooling matrix 1; then use a window of size 3 to perform sliding average processing on the pooling matrix 1 in the vertical direction with a step size of 3 to obtain an attention matrix with a dimension of C1*1*W1;

[0028] Perform global pooling on the input feature image on the vertical axis to obtain pooling matrix 2; then use a window of size 3 to perform sliding average processing on pooling matrix 2 in the horizontal direction with a step size of 3 to obtain an attention matrix with a dimension of C1*H1*1;

[0029] Use matrix multiplication to multiply the attention matrix with dimension C1*H1*1 and the attention matrix with dimension C*1*W to obtain the total attention matrix with dimension C1*H1*W1;

[0030] The Sigmoid function is used to scale the value of the total attention matrix to obtain a weight matrix with a weight constraint between 0 and 1;

[0031] The weight matrices acquired by the three parallel weight matrix acquisition units are respectively the first weight matrix, the second weight matrix and the third weight matrix; the outputs of the three parallel atrous convolution structures of the same layer are respectively multiplied by the three parallel weight matrices (connected by residual structures, as shown in the figure), the three multiplication results are added, and the added results are input into the multi-scale feature extraction module of the next layer.

[0032] Furthermore, the atrous convolution structure includes a atrous convolution layer, a second maximum pooling layer and a BN layer, and the atrous convolution layers in the three parallel atrous convolution structures are respectively the first atrous convolution layer, the second atrous convolution layer and the third atrous convolution layer; the parameters of the first atrous convolution layer are: Kernel_size is 5*5, padding is 2, stride is 1, and dilation is 2; the parameters of the second atrous convolution layer are: Kernel_size is 4*4, padding is 1, stride is 1, and dilation is 2; the parameters of the second atrous convolution layer are: Kernel_size is 3*3, padding is 1, stride is 2, and dilation is 1.

[0033] Furthermore, the high-dimensional heat map loss, the multi-dimensional heat map loss and the key point classification loss are specifically: loss1, loss2 and loss3, respectively as follows:

[0034] loss1 = MSE(H_h, Heat_1)

[0035] Among them, H_h represents the predicted high-dimensional heat map, Heat_1 represents the real high-dimensional heat map, and MSE represents the mean square error loss;

[0036] loss2 = MSE(Heatmap, Heat_2)

[0037]

[0038] sum = 1 / loss min1 +1 / loss min2 +1 / loss min3 +1 / loss min4

[0039] Among them, loss mini represents the heat map loss of the i-th layer, that is, the predicted heat map of the i-th layer multi-scale convolution unit and spatial attention unit in the multi-level multi-scale feature extraction module and the previous round prediction loss of the corresponding label, xi represents the predicted heat map of the i-th layer multi-scale feature extraction module in the multi-level multi-scale feature extraction module, ai represents the weight parameter corresponding to xi, and i is 1, 2, 3 or 4; Heatmap represents the predicted multi-dimensional heat map; Heat_2 represents the real multi-dimensional heat map;

[0040] loss3 = CES(pred, label)

[0041] Among them, pred is the predicted category of each heat map location, label is the category of the actual heat map location, and CES is the cross entropy loss.

[0042] The present invention also discloses a behavior posture recognition system, wherein the system is used to perform behavior posture recognition on an image or a video, and the method adopted is the above-mentioned behavior posture recognition method.

[0043] After adopting the above scheme, the beneficial effects of the present invention are as follows:

[0044] (1) By performing target detection and high-dimensional heat map feature enhancement on the target area, the target area can be effectively selected to avoid the interference of occlusion information, and the interference of image noise information can be reduced due to image preprocessing operations;

[0045] (2) Through the heat map feature extraction module and the multi-level multi-scale feature extraction module, the posture key point area is extracted from the image, and multi-level progressive feature extraction is performed. The spatial attention module is used to realize the effective area judgment and weight adaptive allocation of key point information, and then the multi-scale features are weightedly fused to obtain the multi-dimensional key point heat map and key point classification results. The accuracy of the model is improved by combining multiple loss functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the process of the present invention;

[0047] Figure 2 It is a schematic diagram of the thermodynamic diagram of the present invention;

[0048] Figure 3 It is a schematic diagram of the heat map feature extraction module;

[0049] Figure 4 It is a schematic diagram of a multi-level and multi-scale feature extraction module;

[0050] Figure 5 Schematic diagram of the process of obtaining the weight matrix of the spatial attention unit. DETAILED DESCRIPTION

[0051] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0052] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0054] It should be understood that the term "and / or" used in this article is only a description of the same field of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0055] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0056] Figure 1 FIG. 2 is a flow chart showing a method for estimating a posture according to an exemplary embodiment of the present disclosure. Figure 1 As shown,

[0057] The posture estimation method of the exemplary embodiment of the present disclosure includes:

[0058] S100. Obtain an original image containing a target object and perform annotation processing on the original image; this step specifically includes:

[0059] S110. The original image or original video of human behavior posture is captured by a camera in an electronic device, and an image containing a target object is obtained in the original image or original video; the target object may be a human or animal object with key point position information for which posture estimation is required.

[0060] S120. If it is an original video, split the original video into individual images by frame, and screen out images that meet the resolution requirements as valid images; if it is an original image, directly screen out clear images as valid images; clear images can be selected according to specific pixel requirements or according to the top 50% of clarity ranking. Specifically, for example, the resolution requirement here is at least 1024*1024, or 2048*2048, which varies according to different devices.

[0061] The joint point positions and categories of the target objects in the valid image are annotated, and the initial heat map of the target joint points is generated based on the key points and the Gaussian kernel function with a fixed standard deviation.

[0062] In practical applications, the location annotation of key points needs to be completed by professionals, and the fixed value of the standard deviation needs to be confirmed based on the image data and imaging pixel information actually containing the target object. Figure 2 FIG. 4 shows a schematic diagram of image annotation in this specific embodiment. Figure 2 As shown in the figure, the coordinate (3, 3) is taken as the key point, and the coordinate distance scores of all neighboring pixels and the key point are calculated. The higher the coordinate distance score, the closer the distance, and the lower the coordinate distance score, the farther the distance. When the coordinate distance score is less than 0.5, it is changed to 0. According to the key point position of the image, a Gaussian kernel function with a standard deviation of 2 is selected to generate the initial heat map of the key points in the image, and its category is labeled according to its key point information.

[0063] The coordinate distance score is calculated as follows:

[0064]

[0065] Among them, yi and zi are the coordinate positions that need to be calculated. are the coordinate positions of the key points in the image respectively, σ is the standard deviation, σ=2 in this specific implementation, and in actual needs, it can be customized according to the actual situation.

[0066] Step 200. Preprocess the original image, which includes denoising and enhancing the original image, decomposing the original image by wavelet transform, then denoising the details at different scales in the wavelet domain, and enhancing the image by adaptive histogram equalization. Specifically, the image is divided into small blocks (the number of image blocks is preferably 4 or 9), and histogram equalization is performed in each small block, so that the processed image has a better local contrast enhancement effect.

[0067] Step 300. Use the target detection model to perform target detection on the preprocessed image, determine the position information of the target object in the image, and crop the image according to the position information to obtain a cropped image of a single target object; the target detection model adopts one of YOLOV7, YOLOV8, EfficientDet, RetinaNet, and Faster R-CNN, but is not limited to the above, and common target detection models can be used. The target detection model is an existing technology, and the annotation of its key points and the training and prediction process of the target detection model are not described in detail here. In addition, in actual applications, the selection of the pre-trained model should be determined according to the actual equipment and response speed requirements, and should be considered according to the actual situation, so no specific description is given here.

[0068] Step 400. Use the heat map feature extraction module to extract high-dimensional posture key point regions from the cropped image of a single target object to obtain a high-dimensional heat map; the heat map feature extraction module is as follows: Figure 3 As shown, the heat map feature extraction module consists of a multi-layer downsampling unit and a high-dimensional heat map prediction head, the input is the enhanced image X_img, and the output is the predicted high-dimensional heat map H_h;

[0069] Among them, the multi-layer downsampling unit includes three layers of downsampling convolution structures connected in sequence; the downsampling convolution structure includes a first convolution layer, a first maximum pooling layer and a second convolution layer, the convolution kernel size of the first convolution layer is k1*k1, the step size is 1, the number of input channels is c1, and the number of output channels is c2; the convolution kernel size of the first maximum pooling layer is 2, the step size is 2; the convolution kernel size of the second convolution layer is k2*k2, the step size is 1, the number of input channels is c2, and the number of output channels is c2;

[0070] The convolution kernel k1 of each downsampling convolution structure remains the same, and c1 and c2 change with the number of layers. The c1 and c2 of the first downsampling convolution module are 3 (when the input data is a 3-channel image) or 1 (when the input image is a grayscale image) and 64 respectively. The c1 and c2 of the second downsampling convolution module are 64 and 128 respectively. The c1 and c2 of the third downsampling convolution module are 128 and 256 respectively.

[0071] The high-dimensional heat map prediction head includes a first prediction head, a second prediction head and a third prediction head. The output of the high-dimensional heat map prediction head is the result of merging the outputs of the first prediction head, the second prediction head and the third prediction head. The merging here is the addition and averaging operation. The first prediction head receives the output of the first layer of down-sampling convolution structure, the second prediction head receives the output of the second layer of down-sampling convolution structure, and the third prediction head receives the output of the third layer of down-sampling convolution structure. The first prediction head includes a fourth convolution layer and a first up-sampling layer. The convolution kernel size of the fourth convolution layer is 1. *1, the step size is 1, the number of input channels is c2, the number of output channels is 1, and the first upsampling layer is 2 times upsampling; the second prediction head includes the fifth convolution layer and the second upsampling layer, the convolution kernel size of the fifth convolution layer is 1*1, the step size is 1, the number of input channels is c3, the number of output channels is 1, and the second upsampling layer is 4 times upsampling; the third prediction head includes the sixth convolution layer and the third upsampling layer, the convolution kernel size of the sixth convolution layer is 1*1, the step size is 1, the number of input channels is c4, the number of output channels is 1, and the third upsampling layer is 8 times upsampling.

[0072] Specifically, the heat map feature extraction module inputs an image X_img with a dimension size of C*H*W. After passing through the first layer of downsampling convolution structure, a feature image X_1 with a dimension size of c2*H / 2*W / 2 is obtained. After passing through the second layer of downsampling convolution structure, a feature image X_2 with a dimension size of c3*H / 4*W / 4 is obtained. After passing through the second layer of downsampling convolution structure, a feature image X_3 with a dimension size of c4*H / 8*W / 8 is obtained.

[0073] The inputs of the first prediction head, the second prediction head and the third prediction head of the high-dimensional heat map prediction head are feature images X_1, feature images X_2 and feature images X_3 respectively; the feature channels are compressed by convolution operations with a convolution kernel of 1*1, and the number of channels is compressed from c2, c3, and c4 to 1, and then the features are scaled by upsampling by 2 times, 4 times and 8 times respectively to the size of the image X_img. At this time, the dimension is 1*H*W, and the three sampling results are numerically merged to obtain the high-dimensional heat map estimation H_h. The merging formula is:

[0074] H_h=(H1+H2+H3) / 3

[0075] Among them, H1, H2 and H3 are the three corresponding sampling results of the first prediction head, the second prediction head and the third prediction head respectively.

[0076] Step 500: stitching the high-dimensional heat map and the cropped image of a single target object to obtain a first stitched image;

[0077] Step 600: Input the first stitched image into a multi-level multi-scale feature extraction module, such as Figure 4 As shown in FIG. 1 , the multi-level multi-scale feature extraction module includes multiple layers of multi-scale feature extraction modules connected in sequence, and each layer of multi-scale feature extraction modules includes connected multi-scale convolution units and spatial attention units; in each layer of multi-scale feature extraction modules, the multi-scale convolution unit extracts multi-scale features from the feature image (the input of the first layer of multi-scale feature extraction module is the first spliced ​​image), and the spatial attention unit adaptively allocates weights to each region in the feature image extracted by the multi-scale convolution unit of this layer. An example of a multi-level multi-scale feature extraction module is shown in FIG. Figure 4 As shown, only some layers are shown in the figure, and the rest of the repeated layers are not drawn, but you can still combine them from the text. Figure 4 The complete structure of the multi-layer multi-scale feature extraction module is obtained. In specific operations, for those skilled in the art, the padding size can be customized, and when the calculated output scale is inconsistent, an upsampling module can be added to correct the dimension size.

[0078] The multi-level multi-scale feature extraction module includes 4 layers of multi-scale feature extraction modules connected in sequence. Specifically, the multi-level multi-scale feature extraction module includes 4 layers of multi-scale convolution units and 4 layers of spatial attention units, which are arranged alternately in a structure of first multi-scale convolution units and then spatial attention units. One multi-scale convolution unit and one spatial attention unit constitute a layer of multi-scale feature extraction module, as shown in the figure. The spatial attention unit of the same layer calculates the attention weight of the spatial information of the image features extracted by the multi-scale convolution unit of the same layer, and then uses the attention weight to optimize the extracted image features; through the residual connection, the image features before optimization and the image features after optimization are added, and the result is input to the next layer;

[0079] In the 4-layer multi-scale feature extraction module, the multi-scale convolution unit of each layer includes three parallel dilated convolution structures, which are used to extract features of different scales from the input feature image;

[0080] The spatial attention unit is used to obtain a weight matrix. The spatial attention unit includes three parallel weight matrix acquisition units. Each weight matrix acquisition unit has the same structure, such as Figure 5 As shown, the specific operation of each weight matrix acquisition unit is:

[0081] Assuming that the dimension of the input feature image is C1*H1*W1, perform global pooling on the input feature image on the horizontal axis to obtain the pooling matrix 1; then use a window of size 3 to perform sliding average processing on the pooling matrix 1 in the vertical direction with a step size of 3 to obtain an attention matrix with a dimension of C1*1*W1;

[0082] Perform global pooling on the input feature image on the vertical axis to obtain pooling matrix 2; then use a window of size 3 to perform sliding average processing on pooling matrix 2 in the horizontal direction with a step size of 3 to obtain an attention matrix with a dimension of C1*H1*1;

[0083] Use matrix multiplication to multiply the attention matrix with dimension C1*H1*1 and the attention matrix with dimension C*1*W to obtain the total attention matrix with dimension C1*H1*W1;

[0084] The Sigmoid function is used to scale the value of the total attention matrix to obtain a weight matrix with a weight constraint between 0 and 1;

[0085] The weight matrices acquired by the three parallel weight matrix acquisition units are respectively the first weight matrix, the second weight matrix and the third weight matrix; the outputs of the three parallel dilated convolution structures of the same layer are respectively multiplied by the three parallel weight matrices (connected by the residual structure, as shown in the figure), the three multiplication results are added, and the added result is input into the multi-scale feature extraction module of the next layer. Here, if the sizes of the three multiplication results are different, the other two are scaled according to the largest size or scaled according to a fixed uniform size.

[0086] The dilated convolution structure includes a dilated convolution layer, a second maximum pooling layer and a BN layer. The dilated convolution layers in the three parallel dilated convolution structures are the first dilated convolution layer, the second dilated convolution layer and the third dilated convolution layer respectively; the parameters of the first dilated convolution layer are: Kernel_size is 5*5, padding is 2, stride is 1, and dilation is 2; the parameters of the second dilated convolution layer are: Kernel_size is 4*4, padding is 1, stride is 1, and dilation is 2; the parameters of the second dilated convolution layer are: Kernel_size is 3*3, padding is 1, stride is 2, and dilation is 1. The parameters of the maximum pooling layers in the three dilated convolution structures are: Kernel_size is 3*3, and stride is 2. In the 4-layer multi-scale convolution unit, the number of input channels and output channels of the dilated convolution structure in each layer of the multi-scale convolution unit are different. The input image of the multi-level multi-scale feature extraction module is the first spliced ​​image. Therefore, in the dilated convolution structure in the multi-scale convolution unit of the first layer, the input channel is 4 (when the input image is a 3-channel image) or 2 (when the input image is a grayscale image), and the output channel is 64; in the dilated convolution structure in the multi-scale convolution unit of the second layer, the input channel is 64, and the output channel is 128; in the dilated convolution structure in the multi-scale convolution unit of the third layer, the input channel is 128, and the output channel is 256; in the dilated convolution structure in the multi-scale convolution unit of the fourth layer, the input channel is 256, and the output channel is 256.

[0087] Dilated convolution is an existing technology. The formula for calculating the output feature size after the dilated convolution structure is:

[0088]

[0089] Among them, out_size is the height or width of the output feature, size is the height or width of the input feature, p is the padding size, k is the convolution kernel Kernel_size, d is the dilation size, and s is the stride size. When the input feature passes through three parallel dilated convolution structures in the same layer, if the size of the output feature is inconsistent, the output features of the three parallel dilated convolution structures in the same layer are sampled and supplemented according to the maximum output size of the three output features, and the nearest neighbor interpolation method is used.

[0090] For the features after fusion of the i-th layer, that is, the input features of the i+1-th layer are:

[0091]

[0092] Among them, x i+1is the output of the i-th layer multi-scale feature extraction module and the input of the i+1-th layer multi-scale feature extraction module. Xi_1, Xi_2 and Xi_3 are the outputs of three parallel dilated convolution structures respectively. Wi_1, Wi_2 and Wi_3 are the first weight matrix, the second weight matrix and the third weight matrix respectively.

[0093] Take the input features of the second layer as an example:

[0094]

[0095] S700. Input the output of the multi-level and multi-scale feature extraction module to the key point classification head to obtain the key point classification result, and obtain the behavior posture recognition result according to the key point classification result.

[0096] The weighted fusion of multi-scale features obtains the low-dimensional key point heat map and key point classification results. The features after fusion of multiple layers of weights are updated separately, and the prediction results of different layers are weighted according to the overall prediction performance of the prediction results. The weighted fusion of multi-scale features obtains the low-dimensional key point heat map, and the classification of key points is predicted by merging multiple layers of prediction features.

[0097] During the training process of the heat map feature extraction module and the multi-level multi-scale feature extraction module, the heat map representations in each layer of the multi-scale feature extraction module are respectively input into the multi-dimensional heat map prediction head, and then the results output by each multi-dimensional heat map prediction head are fused to obtain a multi-dimensional heat map; the heat map feature extraction module and the multi-level multi-scale feature extraction module are trained using high-dimensional heat map loss, multi-dimensional heat map loss, and key point classification loss.

[0098] For prediction results of different dimensions at different layers, upsampling is required to change the dimensions to obtain the same dimensions as the original input image.

[0099] The high-dimensional heat map loss, the multi-dimensional heat map loss and the key point classification loss are specifically: loss1, loss2 and loss3, respectively as follows:

[0100] loss1 = MSE(H_h, Heat_1)

[0101] Among them, H_h represents the predicted high-dimensional heat map, Heat_1 represents the real high-dimensional heat map, and MSE represents the mean square error loss;

[0102] loss2 = MSE(Heatmap, Heat_2)

[0103] Heat_2 represents the real multidimensional heat map; Heatmap represents the predicted multidimensional heat map, which is the result of fusing the output of each multidimensional heat map prediction head. Specifically, the fusion formula of the multidimensional heat map is:

[0104]

[0105]

[0106] sum = 1 / loss min1 +1 / loss min2 +1 / loss min3 +1 / loss min4

[0107] Among them, loss mini represents the heat map loss of the i-th layer, that is, the predicted heat map of the i-th layer multi-scale convolution unit and the spatial attention unit in the multi-level multi-scale feature extraction module and the previous round prediction loss of the corresponding label, xi represents the predicted heat map of the i-th layer multi-scale feature extraction module in the multi-level multi-scale feature extraction module, ai represents the weight parameter corresponding to xi, i is 1, 2, 3 or 4, and the default initial values ​​a1, a2, a3 and a4 are 0.25, 0.25, 0.25, 0.25;

[0108] loss3 = CES(pred, label)

[0109] Among them, pred is the predicted category of each heat map location, label is the category of the actual heat map location, and CES is the cross entropy loss.

[0110] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A behavior posture recognition and analysis method, characterized in that: The steps include: Obtain the original image containing the target object and perform annotation processing on the original image; Preprocess the original image; The target detection model is used to perform target detection on the preprocessed image, determine the position information of the target object in the image, and crop the image according to the position information to obtain a cropped image of a single target object; The heat map feature extraction module is used to extract the high-dimensional posture key point area of ​​the cropped image of a single target object to obtain a high-dimensional heat map; Splicing the high-dimensional heat map and the cropped image of a single target object to obtain a first spliced ​​image; Inputting the first spliced ​​image into a multi-level multi-scale feature extraction module, the multi-level multi-scale feature extraction module includes multiple layers of multi-scale feature extraction modules connected in sequence, each layer of multi-scale feature extraction modules includes connected multi-scale convolution units and spatial attention units; in each layer of multi-scale feature extraction modules, the multi-scale convolution unit extracts multi-scale features from the feature image, and the spatial attention unit adaptively allocates weights to each region in the feature map extracted by the multi-scale convolution unit of this layer; The output of the multi-level and multi-scale feature extraction module is input into the key point classification head to obtain the key point classification result, and the behavior posture recognition result is obtained according to the key point classification result; During the training process of the heat map feature extraction module and the multi-level multi-scale feature extraction module, the heat map representations in each layer of the multi-scale feature extraction module are respectively input into the multi-dimensional heat map prediction head, and then the results output by each multi-dimensional heat map prediction head are fused to obtain a multi-dimensional heat map; the heat map feature extraction module and the multi-level multi-scale feature extraction module are trained using high-dimensional heat map loss, multi-dimensional heat map loss, and key point classification loss.

2. A behavior posture recognition and analysis method according to claim 1, characterized in that: The obtaining of the original image containing the target object and the labeling of the original image include: Capturing an original image or original video of a human body's behavior posture through a camera in an electronic device, wherein the original image or original video contains an image of a target object; If it is an original video, split the original video into individual images by frame, and select the images that meet the resolution requirements as valid images; if it is an original image, directly select the clear image as the valid image; The joint point positions and categories of the target objects in the valid image are annotated, and the initial heat map of the target joint points is generated based on the key points and the Gaussian kernel function with a fixed standard deviation.

3. The behavior posture recognition and analysis method according to claim 1, characterized in that: The preprocessing of the original image comprises: The original image is decomposed by wavelet transform, and then the details at different scales are denoised in the wavelet domain; Image enhancement via adaptive histogram equalization.

4. The method for behavior posture recognition and analysis according to claim 1, characterized in that: The target detection model adopts one of YOLOV7, YOLOV8, EfficientDet, RetinaNet, and Faster R-CNN.

5. The method for behavior posture recognition and analysis according to claim 1, characterized in that: The heat map feature extraction module consists of a multi-layer downsampling unit and a high-dimensional heat map prediction head. The input is the enhanced image X_img, and the output is the predicted high-dimensional heat map H_h; Among them, the multi-layer downsampling unit includes three layers of downsampling convolution structures connected in sequence; the downsampling convolution structure includes a first convolution layer, a first maximum pooling layer and a second convolution layer, the convolution kernel size of the first convolution layer is k1*k1, and the step size is 1; the convolution kernel size of the first maximum pooling layer is 2, and the step size is 2; the convolution kernel size of the second convolution layer is k2*k2, and the step size is 1; The high-dimensional heat map prediction head includes a first prediction head, a second prediction head and a third prediction head. The output of the high-dimensional heat map prediction head is the result of merging the outputs of the first prediction head, the second prediction head and the third prediction head. The first prediction head receives the output of the first layer of downsampling convolution structure, the second prediction head receives the output of the second layer of downsampling convolution structure, and the third prediction head receives the output of the third layer of downsampling convolution structure. The first prediction head includes a fourth convolution layer and a first upsampling layer. The convolution kernel size of the fourth convolution layer is 1*1 and the step size is 1. The first upsampling layer is 2 times upsampling. The second prediction head includes a fifth convolution layer and a second upsampling layer. The convolution kernel size of the fifth convolution layer is 1*1 and the step size is 1. The second upsampling layer is 4 times upsampling. The third prediction head includes a sixth convolution layer and a third upsampling layer. The convolution kernel size of the sixth convolution layer is 1*1 and the step size is 1. The third upsampling layer is 8 times upsampling.

6. The behavior posture recognition and analysis method according to claim 1, characterized in that: The multi-level multi-scale feature extraction module includes 4 layers of multi-scale feature extraction modules connected in sequence; In the 4-layer multi-scale feature extraction module, the multi-scale convolution unit of each layer includes three parallel dilated convolution structures, which are used to extract features of different scales from the input feature image; The spatial attention unit is used to obtain a weight matrix. The spatial attention unit includes three parallel weight matrix acquisition units. The structure of each weight matrix acquisition unit is the same. The specific operation of each weight matrix acquisition unit is as follows: Assuming that the dimension of the input feature image is C1*H1*W1, perform global pooling on the input feature image on the horizontal axis to obtain the pooling matrix 1; then use a window of size 3 to perform sliding average processing on the pooling matrix 1 in the vertical direction with a step size of 3 to obtain an attention matrix with a dimension of C1*1*W1; Perform global pooling on the input feature image on the vertical axis to obtain pooling matrix 2; then use a window of size 3 to perform sliding average processing on pooling matrix 2 in the horizontal direction with a step size of 3 to obtain an attention matrix with a dimension of C1*H1*1; Use matrix multiplication to multiply the attention matrix with dimension C1*H1*1 and the attention matrix with dimension C*1*W to obtain the total attention matrix with dimension C1*H1*W1; The Sigmoid function is used to scale the value of the total attention matrix to obtain a weight matrix with a weight constraint between 0 and 1; The weight matrices acquired by the three parallel weight matrix acquisition units are respectively the first weight matrix, the second weight matrix and the third weight matrix; the outputs of the three parallel dilated convolution structures of the same layer are respectively multiplied by the three parallel weight matrices, the three multiplication results are added, and the added results are input into the multi-scale feature extraction module of the next layer.

7. A behavior posture recognition and analysis method according to claim 6, characterized in that: The atrous convolution structure includes a atrous convolution layer, a second maximum pooling layer and a BN layer. The atrous convolution layers in the three parallel atrous convolution structures are respectively the first atrous convolution layer, the second atrous convolution layer and the third atrous convolution layer; the parameters of the first atrous convolution layer are: Kernel_size is 5*5, padding is 2, stride is 1, and dilation is 2; the parameters of the second atrous convolution layer are: Kernel_size is 4*4, padding is 1, stride is 1, and dilation is 2; the parameters of the second atrous convolution layer are: Kernel_size is 3*3, padding is 1, stride is 2, and dilation is 1.

8. The method for behavior posture recognition and analysis according to claim 1, characterized in that: The high-dimensional heat map loss, the multi-dimensional heat map loss and the key point classification loss are specifically: loss1, loss2 and loss3, respectively as follows: loss1 = MSE(H_h, Heat_1) Among them, H_h represents the predicted high-dimensional heat map, Heat_1 represents the real high-dimensional heat map, and MSE represents the mean square error loss; loss2 = MSE(Heatmap, Heat_2) sum=1 / loss min1 +1 / loss min2 +1 / loss min3 +1 / loss min4 Among them, loss mini represents the heat map loss of the i-th layer, that is, the predicted heat map of the i-th layer multi-scale convolution unit and spatial attention unit in the multi-level multi-scale feature extraction module and the previous round prediction loss of the corresponding label, xi represents the predicted heat map of the i-th layer multi-scale feature extraction module in the multi-level multi-scale feature extraction module, ai represents the weight parameter corresponding to xi, and i is 1, 2, 3 or 4; Heatmap represents the predicted multi-dimensional heat map; Heat_2 represents the real multi-dimensional heat map; loss3 = CES(pred, label) Among them, pred is the predicted category of each heat map location, label is the category of the actual heat map location, and CES is the cross entropy loss.

9. A behavior posture recognition system, characterized in that: The system is used to perform behavior posture recognition on images or videos, and the method adopted is a behavior posture recognition method described in any one of claims 1-8.

Images