Human pose estimation method based on dynamic attention and cross-branch dynamic fusion

Abstract

The present invention discloses a human posture estimation method based on dynamic attention and cross-branch dynamic fusion, which realizes human posture estimation by constructing a key point prediction model combining HRNet (multi-resolution feature extraction module), CSAM (cross-branch dynamic fusion module), DAM (dynamic enhancement module) and SimCC (key point coordinate classification module). By introducing CSAM and DAM, combined with HRNet and SimCC technologies, the accuracy and efficiency of human key point positioning are significantly improved. The dynamic attention mechanism and multi-resolution dynamic fusion effectively reduce redundant calculations and significantly reduce the model inference time; the key point coordinates are directly regressed through SimCC to avoid the ambiguity introduced by the heat map and achieve pixel-level precise positioning; it still performs well in complex scenes (such as severe occlusion and large changes in perspective) and has strong robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human pose estimation, in particular to a human pose estimation method based on dynamic attention and cross-branch dynamic fusion. BACKGROUND

[0002] Human pose estimation is a key task in the field of computer vision, and is widely used in motion capture, virtual reality, human behavior analysis, human-computer interaction and other scenarios. With the development of deep learning technology, human pose estimation based on deep learning has become the mainstream. For example, the Chinese invention patent application with the application publication number CN 115830630 A "A small-scale perception enhanced human pose estimation method", first acquires a large number of pictures containing people, and labels the key point positions of the people in the pictures; then the model framework based on SSA-NET is trained using the acquired pictures and their labeled information, wherein the model framework based on SSA-NET includes a BackBone module for feature extraction of the input picture, a TAA module for feature enhancement of the extracted feature map, and a SimCC module for predicting the key point positions according to the enhanced feature map; then input the picture containing the person to be estimated into the trained model, and the position coordinates of each key point of the person in the picture can be predicted. However, the model framework based on SSA-NET used in this method still faces the following challenges in practical application: 1) Insufficient multi-resolution feature fusion: the BackBone module uses simple weighting or splicing operation to fuse the feature map branches, which fails to fully exploit the complementarity between multi-resolution features, and the modeling ability of complex correlations between global and local is insufficient, especially in complex human pose or severe occlusion scenes. 2) Insufficient dynamic feature enhancement: the TAA module uses fixed weights, which cannot dynamically adjust the feature importance of each channel and spatial region according to the context of the input image, and background noise often interferes with key point prediction, resulting in decreased model robustness and accuracy. SUMMARY

[0003] The present application solves the problem of insufficient multi-resolution feature fusion and dynamic feature enhancement in existing human pose estimation methods based on deep learning, which leads to poor human pose estimation results, and provides a human pose estimation method based on dynamic attention and cross-branch dynamic fusion.

[0004] To solve the above problems, the present application is realized by the following technical scheme:

[0005] The human pose estimation method based on dynamic attention and cross-branch dynamic fusion includes the following steps:

[0006] Step 1, pre-processing the input image to obtain a pre-processed input image;

[0007] Step 2: The preprocessed input image is fed into the key point prediction model to obtain the predicted coordinates of the key points of the human body; the key point prediction model consists of a multi-resolution feature extraction module, a cross-branch dynamic fusion module, a dynamic enhancement module and a key point coordinate classification module. The preprocessed input image is fed into the input of the multi-resolution feature extraction module, the output of the multi-resolution feature extraction module is connected to the input of the cross-branch dynamic fusion module, the output of the cross-branch dynamic fusion module is connected to the input of the dynamic enhancement module, the output of the dynamic enhancement module is connected to the input of the key point coordinate classification module, and the output of the key point coordinate classification module obtains the predicted coordinates of the key points of the human body;

[0008] Step 3: Map the predicted coordinates of the human body key points to the original image coordinate system, and connect the key points according to the human skeleton topology to generate a human body posture graph.

[0009] In the above scheme, the preprocessing of the input image includes normalization and image size adjustment.

[0010] In the above scheme, the specific processing process of the multi-resolution feature extraction module includes the following:

[0011] Step 2.1.1: Preprocessed input image I∈R W×H×3 After 3 convolution operations, the feature map F 1 ;

[0012] Step 2.1.2, feature map F 1 On the one hand, after two convolution operations, the feature map F is obtained 2 On the other hand, after downsampling and convolution operations, the feature map F is obtained 3 ;

[0013] Step 2.1.3, feature map F 2 On the one hand, after the convolution operation, the feature map F is obtained 4 On the other hand, after downsampling, the feature map F is obtained 5 ;

[0014] Step 2.1.4, feature map F 3 On the one hand, after upsampling operation, the feature map F is obtained 6 On the other hand, after the convolution operation, the feature map F is obtained 7 ;

[0015] Step 2.1.5, feature map F 4 and feature map F 6 After the fusion operation, the feature map F is obtained 8 , feature map F 5 and feature map F 7 After the fusion operation, the feature map F is obtained 9;

[0016] Step 2.1.6, feature map F 8 After 4 convolution operations, the feature map F is obtained 10 ;

[0017] Step 2.1.7, feature map F 9 After 4 convolution operations, the feature map F is obtained 11 , and after down-sampling and 3 convolution operations, the feature map F is obtained 12 ;

[0018] Step 2.1.8, feature map F 10 After 4 convolution operations, the feature map F is obtained 13 , and after down-sampling, the feature map F is obtained 14 ;

[0019] Step 2.1.9, feature map F 11 After 4 convolution operations, the feature map F is obtained 15 , and after down-sampling, the feature map F is obtained 16 , and after down-sampling, the feature map F is obtained 17 ;

[0020] Step 2.1.10, feature map F 12 After 4 convolution operations, the feature map F is obtained 18 , and after down-sampling, the feature map F is obtained 19 ;

[0021] Step 2.1.11, feature map F 13 , feature map F 15 and feature map F 18 After fusion operation, the feature map F is obtained 20 , feature map F 14 , feature map F 16 and feature map F 18 After fusion operation, the feature map F is obtained 21 , feature map F 14 , feature map F 17 and feature map F 19 After fusion operation, the feature map F is obtained 22 ;

[0022] Step 2.1.12, feature map F 20 After 3 convolution operations, the feature map F is obtained 23 ;

[0023] Step 2.1.13, feature map F 21 After 3 convolution operations, the feature map F is obtained 24;

[0024] Step 2.1.14, feature map F 22 is obtained by a convolution operation, and feature map F 25 is obtained by a down-sampling operation and two convolution operations; 26 ;

[0025] Step 2.1.15, feature map F 23 is obtained by a convolution operation, and feature map F 27 is obtained by a down-sampling operation; 28 ;

[0026] Step 2.1.16, feature map F 24 is obtained by an up-sampling operation, feature map F 29 is obtained by a convolution operation, and feature map F 30 is obtained by a down-sampling operation; 31 ;

[0027] Step 2.1.17, feature map F 25 is obtained by an up-sampling operation, feature map F 32 is obtained by a convolution operation, and feature map F 33 is obtained by a down-sampling operation; 34 ;

[0028] Step 2.1.18, feature map F 26 is obtained by an up-sampling operation, and feature map F 35 is obtained by a convolution operation; 36 ;

[0029] Step 2.1.19, feature map F 27 is obtained by fusing feature map F 29 , feature map F 32 , and feature map F 35 , feature map F high is obtained by fusing feature map F 28 , feature map F 30 , feature map F 32 , and feature map F 35 , and feature map F mid is obtained by fusing feature map F 28 , feature map F 31 , feature map F 33 , and feature map F 35 ; and feature map F low is obtained by fusing feature map F 28 , feature map F 31, feature map F 34 and feature map F 36 After the fusion operation, a feature map F lowest .

[0030] In the above scheme, the specific processing process of the cross-branch dynamic fusion module includes the following:

[0031] Step 2.2.1, for the four different resolution feature maps output by the resolution feature extraction module, the three lower resolution feature maps are respectively adjusted to the same resolution as the highest resolution feature map by upsampling, and the highest resolution feature map remains unchanged, obtaining four same resolution feature maps;

[0032] Step 2.2.2, adjust the four same resolution feature maps to a unified channel number C using 1x1 convolution, obtaining four same channel response maps; wherein C is a set channel number;

[0033] Step 2.2.3, extract the global semantics of the four same channel response maps using global average pooling, obtaining four global semantic vectors;

[0034] Step 2.2.4, input the four global semantic vectors into the weight-shared double-layer MLP respectively, and generate four dynamic weights through the Sigmoid activation function;

[0035] Step 2.2.5, according to the four dynamic weights, weighted sum of the four same channel response maps is performed to generate a fusion feature map.

[0036] In the above scheme, the specific processing process of the dynamic enhancement module includes the following:

[0037] Step 2.3.1, perform global average pooling on the fusion feature map output by the cross-branch dynamic fusion module through global average pooling, obtaining a channel response vector;

[0038] Step 2.3.2, input the channel response vector into the weight-shared double-layer MLP, and generate a channel weight through the Sigmoid activation function;

[0039] Step 2.3.3, apply the channel weight to each channel of the fusion feature map, that is, weighted sum of each channel of the fusion feature map according to the channel weight is performed to obtain a channel enhanced feature map;

[0040] Step 2.3.4, perform global average pooling and global maximum pooling on the channel enhanced feature map respectively, generating two single-channel feature maps;

[0041] Step 2.3.5, concatenate the two single-channel feature maps in the channel dimension to obtain a concatenated feature map;

[0042] Step 2.3.6, the spliced feature map is passed through a 7x7 convolution, and a spatial weight is generated through a Sigmoid activation function;

[0043] Step 2.3.7, the spatial weight is applied to each channel of the channel enhancement feature map, that is, the spatial weight is used to weight and sum each pixel point of the channel enhancement feature map, to obtain a final dynamic enhancement feature map.

[0044] In the above scheme, the specific processing process of the key point coordinate classification module includes the following:

[0045] Step 2.4.1, the channel number of the dynamic enhancement feature map is converted from C to N by using a 1x1 convolution to obtain an output feature map; wherein N is the set number of key points;

[0046] Step 2.4.2, the output feature map F is divided into N groups along the channel dimension, each group containing a single-channel feature map corresponding to a feature vector of a key point; and the spatial dimension of the single-channel feature map corresponding to each key point is flattened into a one-dimensional vector as the feature vector of the key point;

[0047] Step 2.4.3, for the feature vector of each key point:

[0048] the feature vector of the key point is mapped to W1 classification intervals by using the horizontal coordinate classifier to obtain the horizontal coordinate original output probability of the key point, and the horizontal coordinate with the maximum horizontal coordinate original output probability in the horizontal coordinate original output probability of the key point is selected; at the same time, the feature vector of the key point is mapped to H1 classification intervals by using the vertical coordinate classifier to obtain the vertical coordinate original output probability of the key point, and the vertical coordinate with the maximum vertical coordinate original output probability in the vertical coordinate original output probability of the key point is selected; then, the selected horizontal coordinate and vertical coordinate are taken as the predicted coordinate output of the key point;

[0049] The above W1 represents the width of the dynamic enhancement feature map, and H1 represents the height of the dynamic enhancement feature map.

[0050] Compared with the prior art, the present application has the following advantages:

[0051] 1. A key point prediction model combining HRNet (multi-resolution feature extraction module), CSAM (cross-branch dynamic fusion module), DAM (dynamic enhancement module) and SimCC (key point coordinate classification module) is constructed, which significantly improves the human pose estimation accuracy while greatly reducing the computational complexity, and is suitable for real-time application scenarios, providing a new scheme for efficient and accurate human pose estimation;

[0052] 2、Cross-branch dynamic fusion module realizes efficient fusion of different resolution feature maps through dynamic weighting mechanism, and uses channel alignment and global attention generation strategy to fully exploit the complementarity of different resolution features.

[0053] 3、By combining channel attention and spatial attention, the dynamic enhancement module dynamically adjusts the importance of each channel and pixel region in the feature map, effectively suppresses the interference of background noise on key point prediction, and enhances the feature expression ability of the key point region. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The principle diagram of the human pose estimation method based on dynamic attention and cross-branch dynamic fusion.

[0055] Figure 2 The principle diagram of the multi-resolution feature extraction module.

[0056] Figure 3 The principle diagram of the cross-branch dynamic fusion module.

[0057] Figure 4 The principle diagram of the dynamic enhancement module.

[0058] Figure 5 The principle diagram of the key point coordinate classification module. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the present application will be further described in detail below with reference to specific examples and drawings.

[0060] The human pose estimation method based on dynamic attention and cross-branch dynamic fusion, as shown in Figure 1 , includes the following steps:

[0061] Step 1, input image preprocessing: pre-processing the input image to obtain the pre-processed input image.

[0062] The input image is an RGB color image HxWx3, W is the width of the input image, H is the height of the input image, and 3 is the RGB three color channels of the input image.

[0063] Normalization processing: scale the pixel value of the input image to the interval [0, 1] to adapt to the input standard of the neural network. Assuming that the original image is I, the normalization formula is:

[0064]

[0065] Where, I min and I max are the minimum and maximum values of the image pixels, respectively.

[0066] Image resizing: The normalized input image is resized to a fixed size, such as 256x192 or 384x288, to accommodate the input requirements of the subsequent model.

[0067] Step 2: Prediction of human key point coordinates: The preprocessed input image is input into the key point prediction model to obtain the predicted coordinates of the human key points.

[0068] The key point prediction model of the present application is composed of a multi-resolution feature extraction module, a cross-branch dynamic fusion module, a dynamic enhancement module, and a key point coordinate classification module connected in sequence. The preprocessed input image is input into the input of the multi-resolution feature extraction module, the output of the multi-resolution feature extraction module is connected to the input of the cross-branch dynamic fusion module, the output of the cross-branch dynamic fusion module is connected to the input of the dynamic enhancement module, the output of the dynamic enhancement module is connected to the input of the key point coordinate classification module, and the output of the key point coordinate classification module obtains the predicted coordinates of the human key points.

[0069] Step 2.1, multi-resolution feature extraction module (HRNet), as shown in Figure 2 .

[0070] HRNet (High-Resolution Network) is a network framework widely used in human pose estimation. The HRNet backbone network is composed of multiple parallel connected branches of different resolutions, each branch containing multiple convolution modules. Through repeated feature fusion (such as upsampling low-resolution features and adding them to high-resolution features, and downsampling high-resolution features and adding them to low-resolution features), the high-resolution branch can continuously receive global context information from the low-resolution branch, so that it has rich global context information while maintaining high resolution and preserving local details. The multi-resolution feature extraction module of the present application directly uses the backbone network of HRNet, which uses its parallel multi-branch structure and internal feature fusion mechanism to extract multi-resolution features.

[0071] The preprocessed input image I∈R WΔHΔ3 is input into the multi-resolution feature extraction module. The multi-resolution feature extraction module extracts four different resolution feature maps through its unique multi-branch parallel architecture and internal feature fusion mechanism: and where the high-resolution feature retains detailed information, and the low-resolution feature contains more rich global semantic vectors. W is the width of the input image, H is the height of the input image, C m is the number of channels of the mth resolution feature map, m∈{high, mid, low, lowest}.

[0072] Step 2.1.1, pre-processed input image I e R W×H×3 Feature map F after 3 convolution operations 1 ;

[0073] Step 2.1.2, feature map F 1 Feature map F after 2 convolution operations on one hand 2 and feature map F after down-sampling and convolution operations on the other hand 3 ;

[0074] Step 2.1.3, feature map F 2 Feature map F after convolution operations on one hand 4 and feature map F after down-sampling operations on the other hand 5 ;

[0075] Step 2.1.4, feature map F 3 Feature map F after up-sampling operations on one hand 6 and feature map F after convolution operations on the other hand 7 ;

[0076] Step 2.1.5, feature map F 4 and feature map F 6 Feature map F after fusion operations 8 , feature map F 5 and feature map F 7 Feature map F after fusion operations 9 ;

[0077] Step 2.1.6, feature map F 8 Feature map F after 4 convolution operations 10 ;

[0078] Step 2.1.7, feature map F 9 Feature map F after 4 convolution operations on one hand 11 and feature map F after down-sampling and 3 convolution operations on the other hand 12 ;

[0079] Step 2.1.8, feature map F 10 Feature map F after convolution operations on one hand 13 and feature map F after down-sampling on the other hand 14 ;

[0080] Step 2.1.9, feature map F 11 Feature map F after up-sampling operations on one hand 15 and feature map F after convolution operations on the other hand 16, and the feature map F 17 ;

[0081] Step 2.1.10, the feature map F 12 , and the feature map F 18 , and the feature map F 19 ;

[0082] Step 2.1.11, the feature map F 13 , the feature map F 15 , and the feature map F 18 , the feature map F 20 , the feature map F 14 , the feature map F 16 , and the feature map F 18 , the feature map F 21 , the feature map F 14 , the feature map F 17 , and the feature map F 19 , the feature map F 22 ;

[0083] Step 2.1.12, the feature map F 20 , the feature map F 23 ;

[0084] Step 2.1.13, the feature map F 21 , the feature map F 24 ;

[0085] Step 2.1.14, the feature map F 22 , and the feature map F 25 , and the feature map F 26 ;

[0086] Step 2.1.15, the feature map F 23 , and the feature map F 27 , and the feature map F 28 ;

[0087] Step 2.1.16, the feature map F 24 , and the feature map F 29 , and the feature map F 30 , and the feature map F 31 ;

[0088] Step 2.1.17, the feature map F25 On the one hand, the feature map F 32 is obtained after an up-sampling operation 33 , and on the other hand, the feature map F 34 is obtained after a convolution operation 26 ;

[0089] Step 2.1.18, the feature map F 35 is obtained after an up-sampling operation 36 ;

[0090] Step 2.1.19, the feature map F 27 , the feature map F 29 , the feature map F 32 and the feature map F 35 are fused to obtain the feature map F high , the feature map F 28 , the feature map F 30 , the feature map F 32 and the feature map F 35 are fused to obtain the feature map F mid , the feature map F 28 , the feature map F 31 , the feature map F 33 and the feature map F 35 are fused to obtain the feature map F low ; the feature map F 28 , the feature map F 31 , the feature map F 34 and the feature map F 36 are fused to obtain the feature map F lowest .

[0091] Step 2.2, a cross-branch dynamic fusion module (CSAM), as shown in Figure 3 .

[0092] The CSAM module is integrated into the last stage of the HRNet backbone network, and dynamically weighted fusion is performed on the four feature maps of different resolutions output by the HRNet backbone network. Unlike the original fusion mechanism of HRNet, the CSAM uses channel alignment and global attention generation strategy to realize dynamic weighting, so as to more fully tap the complementarity between features of different resolutions. In this way, the CSAM generates a feature map F CSAM that fuses multi-scale information and has the same resolution as the highest resolution feature map F high , significantly improving the feature representation capability.

[0093] Step 2.2.1, resolution alignment: the feature map F highthree feature maps F mid , F mid , and F lowest are up-sampled to the same resolution, i.e., the highest resolution W / 4 x H / 4, where the feature map F high remains unchanged, resulting in four feature maps F high with the same resolution

[0094]

[0095] Step 2.2.2, Channel Alignment: Adjust the number of channels of each resolution feature map F m,aligned to a uniform value C (C is a set value) by 1x1 convolution, resulting in four feature response maps F m ' ,aligned ∈ R W / 4×H / 4×C :

[0096] F′ m,aligned = Conv 1×1 (F m,aligned ), m∈{high,mid,low,lowest}

[0097] Step 2.2.3, Extracting Global Semantic Vector: Extract the global semantics of each feature map F m ' ,aligned using global average pooling (GAP), resulting in four one-dimensional global semantic vectors w m :

[0098] w m = GAP(F′ m,aligned ), m∈{high,mid,low,lowest}

[0099] Step 2.2.4, Dynamic Weight Generation: Input each global semantic vector w m into a weight-shared two-layer MLP (Multilayer Perceptron) and pass it through a Sigmoid activation function to generate four dynamic weights

[0100]

[0101] where W and W are the weight matrices of the two layers of the weight-shared two-layer MLP, r is the channel compression ratio, C is the number of channels, ReLU is the activation function, and σ is the Sigmoid activation function.

[0102] Step 2.2.5, Feature Weighted Fusion: According to the four dynamic weights F m ,aligned F CSAM ∈R W / 4×H / 4×C

[0103]

[0104] F CSAM which preserves global semantic information and detailed expression ability, and can be directly used for subsequent modules to accurately locate key points.

[0105] Step 2.3, Dynamic Enhancement Module (DAM), as shown in Figure 4

[0106] The dynamic enhancement module further dynamically enhances the fusion feature map F CSAM output by the CSAM. First, it enhances the channel features related to key points using a channel attention mechanism, and then further enhances the features in the region where the key points are located using a spatial attention mechanism. By applying channel attention and spatial attention, the importance of each channel and pixel region in the feature map is dynamically adjusted, the feature map is dynamically enhanced, the interference of background noise on key point prediction is effectively suppressed, and the feature expression ability of the key point region is enhanced, improving the accuracy and robustness of key point positioning.

[0107] Step 2.3.1, Channel Attention Mechanism: This mechanism models the interdependence between channels in the feature map, learns the importance weight of each channel, and thus enhances the channel features related to key point positioning and suppresses the response of irrelevant channels.

[0108] Step 2.3.1.1, Average Pooling to Generate Channel Response Vector: The fusion feature map F CSAM is globally averaged using global average pooling to obtain a one-dimensional channel response vector z of length C, where z(c) in z corresponds to the response value of the c-th channel, and the calculation formula is:

[0109]

[0110] Step 2.3.1.2, Multi-layer Perceptron to Generate Channel Weights: The channel response vector z is input to a weight-shared two-layer MLP, and the channel weights w1 are generated after Sigmoid activation function, where w1(c) in the channel weights w1 corresponds to the weight of the c-th channel, and the calculation formula is:

[0111] w1(c) = σW2·ReLU(W1·z(c))), c = 1, 2,... C

[0112] where,​​​ and is the weight matrix of the two-layer MLP in the weight-sharing two-layer MLP, r is the channel compression ratio, C is the number of channels, ReLU is the activation function, and σ is the Sigmoid activation function.

[0113] Step 2.3.1.3, weights act on channels: apply channel weight w1 to the fusion feature map F CSAM On each channel of the fusion feature map, the weighted sum of each channel of the fusion feature map is performed according to the channel weight to obtain the channel enhanced feature map F CA ∈R W / 4×H / 4×C , where the channel enhanced feature map F CA The eigenvalue F of the pixel (i, j) on the feature map of the cth channel CA (c,i,j) is:

[0114] F CA (c,i,j)=w1(c)·F CSAM (c,i,j),i=1,2,...W / 4,j=1,2,...H / 4,c=1,2,...C

[0115] Step 2.3.2, spatial attention mechanism: This mechanism focuses on the importance of different spatial positions of the feature map, enhances the features of the area where the key points are located, and suppresses the interference of the background area.

[0116] Step 2.3.2.1. Pooling to generate single channel feature map: Enhance the channel feature map F CA Perform global average pooling and global maximum pooling respectively to generate two single-channel feature maps F avg ∈R W / 4×H / 4 and F max ∈R W / 4×H / 4 :in

[0117] Single channel feature map F avg The eigenvalue F at pixel (i, j) avg (i,j) is:

[0118]

[0119] Single channel feature map F max The eigenvalue F at pixel (i, j) max (i,j) is:

[0120]

[0121] Step 2.3.2.2, Generate spatial weights: transform the two single-channel response vectors F avg and F maxConcatenate in the channel dimension to obtain the concatenated feature map F cat ∈R W / 4×H / 4×2 where the concatenated feature map F cat is the feature value at the pixel point (i, j) of the concatenated feature map F cat (i, j) is:

[0122] F cat (i, j) = Concat (F avg (i, j), F max (i, j)), i = 1, 2,... W / 4, j = 1, 2,... H / 4

[0123] Step 2.3.2.3, concatenate the concatenated feature map F cat to generate a spatial weight w2 through a 7x7 convolution and pass through a Sigmoid activation function, where the spatial weight value w2(i, j) of the spatial weight w2 at the pixel point (i, j) is:

[0124] w2(i, j) = σ(Conv 7×7 (F cat (i, j))), i = 1, 2,... W / 4, j = 1, 2,... H / 4

[0125] Step 2.3.2.4, weight acting on space: apply the spatial weight w2 to each channel of the channel enhancement feature map F CA , that is, according to the spatial weight, the weighted sum of each space (pixel point) of the channel enhancement feature map is obtained. Dynamic enhancement feature map F DAM ∈R W / 4×H / 4×C where the dynamic enhancement feature map F DAM is the feature value at the pixel point (i, j) of the dynamic enhancement feature map F DAM (c, i, j) is:

[0126] F DAM (c, i, j) = w2(i, j) · F CA (c, i, j), i = 1, 2,... W / 4, j = 1, 2,... H / 4, c = 1, 2,... C

[0127] Step 2.4, key point coordinate classification module (SimCC), as shown in Figure 5 .

[0128] Considering that the heat map method relies on the probability distribution to locate the joint position, it is difficult to achieve pixel-level accuracy, especially in scenes where the key point spacing is small or there is occlusion, which is prone to ambiguity and uncertainty. In addition, generating a high-resolution heat map requires a large amount of computing resources, especially when the number of key points is large. Subsequent analysis, such as maximum value search or coordinate regression, further increases the computational burden, making it unsuitable for resource-constrained devices or real-time tasks. The present application uses a key point coordinate classification module to replace the traditional heat map generation and analysis process. The SimCC method directly regresses joint coordinates, achieving efficient and accurate joint positioning. Compared with the traditional heat map method, the key point coordinate classification module avoids the complex process of high-resolution heat map generation and analysis, achieves pixel-level accurate positioning, and significantly improves computational efficiency.

[0129] Step 2.4.1, Channel Adjustment: Convert the channel number of the dynamic enhancement feature map F DAM from C to N through 1x1 convolution to obtain the output feature map F W / 4×H / 4×N :

[0130] F = Conv 1×1 (F DAM )

[0131] The above N is the set number of key points, which is set according to the data set. For example, the COCO data set defines 17 key points, and the MPII data set defines 16 key points, which are located at the top of the head, neck, shoulder, elbow, wrist, hip, knee, and ankle.

[0132] Step 2.4.2, Flatten Feature: Divide the output feature map F along the channel dimension into N groups, each containing a single-channel feature map corresponding to a key point feature vector. Then, flatten the spatial dimension (WxH) of the single-channel feature map corresponding to each key point into a one-dimensional vector f n ∈R W / 4×H / 4 , as the feature vector of the key point.

[0133] f n = Flatten(F), n = 1, 2,... N

[0134] Step 2.4.3, Coordinate Prediction:

[0135] To predict the horizontal and vertical coordinates of each key point, two classifiers need to be designed: a horizontal coordinate classifier (horizontal coordinate classifier) and a vertical coordinate classifier (vertical coordinate classifier). Both classifiers consist of a linear layer (fully connected layer) followed by a Softmax function activation layer. Assuming the feature map F DAMThe image is divided into W1 horizontal intervals and H1 vertical intervals, each interval corresponding to a pixel coordinate, which is used to represent the horizontal and vertical coordinates of the key point, respectively.

[0136] For each key point n (n = 1, 2,... N), the feature vector f n The following operations are performed, wherein:

[0137] 1) Map to W1 classification intervals using the horizontal coordinate classifier to obtain the horizontal coordinate raw output probability of the key point n

[0138]

[0139] Map to H1 classification intervals using the vertical coordinate classifier to obtain the vertical coordinate raw output probability of the key point n

[0140]

[0141] wherein W x and W y are the weight matrices of the linear layer, b x and b y are the bias terms, and Softmax(·) is an activation function for normalizing the output into a probability distribution.

[0142] 2) Take the horizontal coordinate raw output probability of the key point with the maximum horizontal coordinate raw output probability in the horizontal coordinate raw output probability Take the vertical coordinate raw output probability of the key point with the maximum vertical coordinate raw output probability in the vertical coordinate raw output probability and take as the predicted coordinate output of the key point.

[0143]

[0144] wherein, represents the horizontal coordinate raw output probability of the key point n at the horizontal coordinate i, the vertical coordinate raw output probability of the key point n at the vertical coordinate j,

[0145] Step 3, pose generation: map each predicted key point coordinate to the original image coordinate system, and connect the key points according to the human body skeleton topology to generate a human body pose graph (skeleton graph), which can be directly used for subsequent analysis or display.

[0146] The application significantly improves the accuracy and efficiency of human key point positioning by introducing a cross-branch dynamic fusion module (CSAM) and a dynamic enhancement module (DAM), combining HRNet and SimCC technology; the dynamic attention mechanism and multi-resolution dynamic fusion effectively reduce redundant calculation, significantly reducing the model inference time; directly regressing key point coordinates through SimCC avoids the ambiguity introduced by the heat map, achieving pixel-level accurate positioning; it still performs well in complex scenes (such as severe occlusion and large view angle change), and has strong robustness.

[0147] It should be noted that, although the above embodiments of the application are illustrative, this is not a limitation of the application, therefore the application is not limited to the above specific embodiments. Any other embodiments obtained by those skilled in the art under the inspiration of the application without departing from the principles of the application are considered to be within the protection of the application.

Claims

1. A human pose estimation method based on dynamic attention and cross-branch dynamic fusion, characterized by: The steps are as follows: Step 1: preprocessing the input image to obtain a preprocessed input image; Step 2: The preprocessed input image is fed into the key point prediction model to obtain the predicted coordinates of the key points of the human body; the key point prediction model consists of a multi-resolution feature extraction module, a cross-branch dynamic fusion module, a dynamic enhancement module and a key point coordinate classification module. The preprocessed input image is fed into the input of the multi-resolution feature extraction module, the output of the multi-resolution feature extraction module is connected to the input of the cross-branch dynamic fusion module, the output of the cross-branch dynamic fusion module is connected to the input of the dynamic enhancement module, the output of the dynamic enhancement module is connected to the input of the key point coordinate classification module, and the output of the key point coordinate classification module obtains the predicted coordinates of the key points of the human body; The specific processing process of the cross-branch dynamic fusion module includes the following: Step 2.2.

1. For the four feature maps with different resolutions output by the resolution feature extraction module, adjust the three feature maps with lower resolutions to the same resolution as the feature map with the highest resolution by upsampling. The feature map with the highest resolution remains unchanged, thus obtaining four feature maps with the same resolution. Step 2.2.2: Use 1×1 convolution to adjust the four feature maps of the same resolution to a unified channel number C, obtaining four response maps of the same channel; where C is the set number of channels; Step 2.2.3: Use global average pooling to extract the global semantics of the four identical channel response maps and obtain four global semantic vectors. Step 2.2.4: Input the four global semantic vectors into the weight-sharing two-layer MLP and generate four dynamic weights through the Sigmoid activation function; Step 2.2.5: Perform weighted summation on the four identical channel response maps according to the four dynamic weights to generate a fused feature map; The dynamic enhancement module further integrates the fusion feature map F output by the cross-branch dynamic fusion module CSAM Perform dynamic enhancement, first using the channel attention mechanism to enhance the channel features related to the key points, and then using the spatial attention mechanism to further enhance the features of the area where the key points are located; By applying channel attention and spatial attention, the importance of each channel and pixel area in the feature map is dynamically adjusted to dynamically enhance the feature map; Step 3: Map the predicted coordinates of the human body key points to the original image coordinate system, and connect the key points according to the human skeleton topology to generate a human body posture graph.

2. The human posture estimation method based on dynamic attention and cross-branch dynamic fusion according to claim 1 is characterized in that: The preprocessing of the input image includes normalization and image size adjustment.

3. The human posture estimation method based on dynamic attention and cross-branch dynamic fusion according to claim 1 is characterized in that: The specific processing of the multi-resolution feature extraction module includes the following: Step 2.1.1: Preprocessed input image I∈R W×H×3 After 3 convolution operations, the feature map F 1 ; Step 2.1.2, feature map F 1 On the one hand, after two convolution operations, the feature map F is obtained 2 On the other hand, after downsampling and convolution operations, the feature map F is obtained 3 ; Step 2.1.3, feature map F 2 On the one hand, after the convolution operation, the feature map F is obtained 4 On the other hand, after downsampling, the feature map F is obtained 5 ; Step 2.1.4, feature map F 3 On the one hand, after upsampling operation, the feature map F is obtained 6 On the other hand, after the convolution operation, the feature map F is obtained 7 ; Step 2.1.5, feature map F 4 and feature map F 6 After the fusion operation, the feature map F is obtained 8 , feature map F 5 and feature map F 7 After the fusion operation, the feature map F is obtained 9 ; Step 2.1.6, feature map F 8 After 4 convolution operations, the feature map F is obtained 10 ; Step 2.1.7, feature map F 9 On the one hand, after 4 convolution operations, the feature map F is obtained 11 On the other hand, after downsampling and 3 convolution operations, the feature map F is obtained 12 ; Step 2.1.8, feature map F 10 On the one hand, after the convolution operation, the feature map F is obtained 13 On the other hand, after downsampling, the feature map F is obtained 14 ; Step 2.1.9, feature map F 11 On the one hand, after upsampling operation, the feature map F is obtained 15 On the other hand, after the convolution operation, the feature map F is obtained 16 On the other hand, after downsampling, the feature map F is obtained 17 ; Step 2.1.10, feature map F 12 On the one hand, after upsampling operation, the feature map F is obtained 18 On the other hand, after convolution, we get the feature map F 19 ; Step 2.1.

11. Feature map F 13 , feature map F 15 and feature map F 18 After the fusion operation, the feature map F is obtained 20 , feature map F 14 , feature map F 16 and feature map F 18 After the fusion operation, the feature map F is obtained 21 , feature map F 14 , feature map F 17 and feature map F 19 After the fusion operation, the feature map F is obtained 22 ; Step 2.1.12, feature map F 20 After 3 convolution operations, the feature map F is obtained 23 ; Step 2.1.13, feature map F 21 After 3 convolution operations, the feature map F is obtained 24 ; Step 2.1.14, feature map F 22 On the one hand, after three convolution operations, the feature map F is obtained 25 On the other hand, after downsampling and 2 convolution operations, the feature map F is obtained 26 ; Step 2.1.15, feature map F 23 On the one hand, after the convolution operation, the feature map F is obtained 27 On the other hand, after downsampling, the feature map F is obtained 28 ; Step 2.1.16, feature map F 24 On the one hand, after upsampling operation, the feature map F is obtained 29 On the other hand, after the convolution operation, the feature map F is obtained 30 On the other hand, after downsampling, the feature map F is obtained 31 ; Step 2.1.17, feature map F 25 On the one hand, after upsampling operation, the feature map F is obtained 32 On the other hand, after the convolution operation, the feature map F is obtained 33 On the other hand, after downsampling, the feature map F is obtained 34 ; Step 2.1.18, feature map F 26 On the one hand, after upsampling operation, the feature map F is obtained 35 On the other hand, after convolution, we get the feature map F 36 ; Step 2.1.19, feature map F 27 , feature map F 29 , feature map F 32 and feature map F 35 After the fusion operation, the feature map F is obtained high , feature map F 28 , feature map F 30 , feature map F 32 and feature map F 35 After the fusion operation, the feature map F is obtained mid , feature map F 28 , feature map F 31 , feature map F 33 and feature map F 35 After the fusion operation, the feature map F is obtained low ; Feature map F 28 , feature map F 31 , feature map F 34 and feature map F 36 After the fusion operation, the feature map F is obtained lowest .

4. The human posture estimation method based on dynamic attention and cross-branch dynamic fusion according to claim 1 is characterized in that: The specific processing process of the dynamic enhancement module includes the following: Step 2.3.1: Perform global average pooling on the fusion feature map output by the cross-branch dynamic fusion module to obtain a channel response vector. Step 2.3.2: Input the channel response vector into the weight-sharing two-layer MLP and generate channel weights through the Sigmoid activation function; Step 2.3.3, apply the channel weight to each channel of the fused feature map, that is, perform weighted summation on each channel of the fused feature map according to the channel weight to obtain the channel enhanced feature map; Step 2.3.4: Perform global average pooling and global maximum pooling on the channel enhancement feature map to generate two single-channel feature maps; Step 2.3.5: Concatenate the two single-channel feature maps in the channel dimension to obtain a concatenated feature map. Step 2.3.6: Pass the concatenated feature map through a 7×7 convolution and a Sigmoid activation function to generate spatial weights; Step 2.3.7: Apply the spatial weight to each channel of the channel enhancement feature map, that is, perform weighted summation on each pixel of the channel enhancement feature map according to the spatial weight to obtain the final dynamic enhancement feature map.

5. The human posture estimation method based on dynamic attention and cross-branch dynamic fusion according to claim 1 is characterized in that: The specific processing process of the key point coordinate classification module includes the following: Step 2.4.

1. Use 1×1 convolution to convert the number of channels of the dynamic enhancement feature map from C to N to obtain the output feature map; where N is the set number of key points; Step 2.4.2, divide the output feature map F into N groups along the channel dimension, each group contains a single-channel feature map corresponding to the feature vector of a key point; then flatten the spatial dimension of the single-channel feature map corresponding to each key point into a one-dimensional vector as the feature vector of the key point; Step 2.4.3, for each key point's feature vector: Use the horizontal coordinate classifier to map the feature vector of the key point to W1 classification intervals respectively, obtain the horizontal coordinate original output probability of the key point, and select the horizontal coordinate with the maximum horizontal coordinate original output probability among the horizontal coordinate original output probabilities of the key point; at the same time, use the vertical coordinate classifier to map the feature vector of the key point to H1 classification intervals respectively, obtain the vertical coordinate original output probability of the key point, and select the vertical coordinate with the maximum vertical coordinate original output probability among the vertical coordinate original output probabilities of the key point; Afterwards, the selected horizontal and vertical coordinates are output as the predicted coordinates of the key point; The above W1 represents the width of the dynamic enhancement feature map, and H1 represents the height of the dynamic enhancement feature map.

Images