A human pose estimation method and system

Abstract

The application discloses a human posture estimation method and system, and the human posture estimation method comprises the following steps: using a segmentation model composed of a Swin Transformer and a region proposal network RPN to perform segmentation extraction on an original image which needs to be subjected to human posture estimation, so as to obtain a human-containing image; processing the human-containing image to obtain a patch feature map; using channel attention to process the non-overlapping patch feature map with a random position and a fixed size to obtain a feature vector; using spatial attention to process the patch feature map with a non-fixed size and capable of covering the whole image to obtain a feature vector; mixing the feature vectors obtained by using the two kinds of attention to obtain a mixed attention feature vector based on patches; and fusing the feature vectors and classifying to obtain a posture estimation result. The application realizes more accurate human posture estimation.

Description

Technical Field

[0001] This invention relates to a method and system for estimating human posture. Background Technology

[0002] Human pose estimation is often used to solve the problem of detecting abnormal human behavior. For example, in a production setting, the pose of the human body is estimated to detect whether the worker's behavior will endanger production safety.

[0003] Existing human pose estimation methods primarily rely on keypoint detection. However, when faced with occlusion, insufficient keypoints can easily lead to pose estimation failures. Furthermore, when human poses are overly complex or concentrated, keypoint detection may fail, resulting in poor pose estimation performance. Additionally, keypoint detection is prone to anomalies when estimating poses that deviate from conventional patterns.

[0004] In summary, keypoint detection-based human pose estimation methods cannot accurately estimate human pose when faced with occlusion or complex human behavior in complex scenes. Summary of the Invention

[0005] Purpose of the invention: The first purpose of the invention is to provide a highly accurate human posture estimation method; the second purpose of the invention is to provide a corresponding human posture estimation system.

[0006] Technical solution: The human pose estimation method of the present invention includes:

[0007] (1) Using a segmentation model consisting of a Swing Transformer and a Region Proposal Network (RPN), the original image M1 for which human pose estimation is required is segmented and extracted to obtain images A1, A2, ..., A1 containing only humans. n ;

[0008] (2) For images A1, A2, ..., A n Each patch is processed separately to obtain non-overlapping patch blocks with random positions and fixed sizes, and patch blocks with fixed sizes that can cover the entire image. Feature extraction is performed on each patch block using a SwinTransformer with class label l to obtain the corresponding feature maps.

[0009] (3) For non-overlapping patch feature maps with random positions and fixed sizes, channel attention processing is used to obtain the feature vector F. CA Spatial attention is used to process the patch feature map, which is of non-fixed size and can cover the entire image, to obtain the feature vector F. SA The feature vectors obtained from the two attention methods are blended to obtain a patch-based hybrid attention feature vector F. PbA ;

[0010] (4) The feature vectors are fused, and after being processed by a combination of linear and nonlinear connection layers, Gaussian dropout and batch normalization are used for regularization and classification to obtain the pose estimation results.

[0011] Further, step (1) includes:

[0012] The original image M1 was feature-extracted using the Swing Transformer to obtain the feature-extracted map.

[0013] The Region Proposal Network (RPN) is used to obtain the bounding box coordinates of the regions in the feature extraction image with the human body as an instance. The bounding box coordinates are then used to obtain the segmentation mask, and the segmented images are resized to be consistent.

[0014] Furthermore, the formula used by the Swin Transformer is as follows:

[0015]

[0016] in, For querying the matrix, Let be the key matrix, M is a value matrix. 2 Here, d is the number of window patches, d is the dimension of the query matrix and the key matrix, and B is taken from a smaller bias matrix. SoftMax(.) is the normalization function.

[0017] Furthermore, step (2) includes:

[0018] Images A1, A2, ..., A n The data is divided into a set D of non-overlapping patch blocks with random positions and fixed sizes. Feature extraction is performed using a Swing Transformer with class label l to obtain the corresponding feature map F. h, w, and c represent the height, width, and channels of the feature map, respectively; D = {d1, d2, ..., dc} e |I l},e=(h×w) / a 2 a×a is the spatial size of the rectangular patch d, and d i The spatial dimension is defined as p i =[x i ,y i ,Δw,Δh],Δw=Δh=a,[x i ,y i [x] represents the pixel coordinates of the patch block on the image, with the first pixel at the bottom left as the origin. A random function is used to obtain the [x] coordinates of the patch block. i ,y i ]coordinate;

[0019] Images A1, A2, ..., A n The image is then divided into patches of varying sizes that cover the entire image. This time, the patch scale is randomly varied, but all patches do not overlap, filling the entire image. Feature extraction is then performed again using a Swing Transformer with class labels l, yielding the corresponding feature map F. i Each patch is represented as d i =[F i p i ], F i For patch p i Feature map, p i =[x i ,y i ,Δw i ,Δh i [Regarding p] i The spatial dimension of the block, Δw i ,Δh i Width and height of each patch block;

[0020] The images A1, A2, ..., A n The feature map F corresponding to the set P of n non-overlapping patches of the same scale and non-fixed-size patches that can cover the entire image is upsampled to obtain a higher-resolution feature map k. The features of each patch are then pooled using bilinear interpolation. p i For a single patch block, F i For p i The feature map, k, has a size of h×w. Upsampling can be viewed as a mapping operation. h, w, and c represent the height, width, and channels of the feature map, respectively.

[0021] Furthermore, the bilinear interpolation formula is as follows:

[0022]

[0023] Among them, Q 11 = (x1, y1), Q 12 = (x1, y2), Q 21 = (x2, y1), Q 22 = (x2, y2); (x1, y1), (x1, y2), (x2, y1), (x2, y2) are the coordinates of the points.

[0024] Furthermore, step (3) includes:

[0025] From the non-overlapping patch feature map F with random positions and fixed size output in step (2), we learn the query Q, keyword K, and value V. The attention weight matrix is ​​the dot product of Q and K, multiplied by V to generate the attention feature descriptor. The channel attention formula is as follows:

[0026]

[0027] Where F is the feature map obtained in step (2), and M c σ is the weight coefficient, MLP(.) is the activation function, AvgPool(.) is the multilayer perceptron, MaxPool(.) is the average pooling layer, and W1 and W0 are matrix weights. The average feature map of C neurons, The maximum feature map of C neurons;

[0028] The importance of each pi is calculated using a weighted sum of the attention scores of all patches, as shown in the following formula:

[0029] δ i,j =softmax(M c +b δ ),

[0030] Where, δ i,j Let π be the mathematical constant, softmax(.) be the normalization function, and b be the normalization function. δ For bias vectors, For the feature space of aggregation, F j For patch feature maps;

[0031] Global average pool aggregation Generate all blocks in P Finally, the weighted attention matrix φ is generated using Softmax. i The formula is as follows:

[0032]

[0033] in, The average feature map of all patch blocks. F is the global average pooling function. CA This is the weighted channel attention feature vector. W φ As the weight, b φ For bias.

[0034] Furthermore, step (3) also includes:

[0035] Global max pooling is performed on the patch feature map output from step (2), which has a non-fixed size and can cover the entire image, to generate an optimized feature vector F. gmp The feature vector F is generated by global average pooling with the patch feature map, which is of non-fixed size and can cover the entire image. gap The feature maps are fused to obtain the fused feature map H, as shown in the following formula:

[0036]

[0037] in, concat(.) is the fusion function. This is the global average pooling function. It is the global max pooling function.

[0038] Applying a multilayer perceptron to generate spatial attention feature vector F SA The specific formula is as follows:

[0039]

[0040] in, For a multilayer perceptron, σ(.) is an element-wise nonlinear activation function using GELU. X is the vector obtained by Flattening the feature map H, and XW represents a linear transformation on X. λ GD+BN For regularization processes with Gaussian discarding and batch normalization;

[0041] The spatial attention feature vector F SA The element-by-element and weighted channel attention feature vector F CA Multiply, and enhance distinguishability by capturing subtle details and focusing on global structural information, by combining residual paths and F CA The concatenation yields a patch-based hybrid attention feature vector F. PbA The specific formula is as follows:

[0042] Further, step (4) includes:

[0043] The upsampled feature map F from step (2) is subjected to a gap layer. Compression produces a vector with (1×c) channels, which is correlated with the attention feature map F. PbA Add them together, and the final feature vector F f The results are processed by linear and nonlinear connection layers, summed, regularized, and then passed to Softmax to calculate the output vector of latent class probabilities. The resulting class is the human pose estimation result, as shown in the following formula:

[0044] Y pred =softmax(λ(F) f ))

[0045] Where λ(.) represents the Gaussian loss and batch normalization regularization processing, with Gaussian loss using multiplicative noise and the loss rate φ mapped to the noise standard deviation. ρ represents the noise level.

[0046] The human pose estimation system of the present invention includes:

[0047] The human pose estimation unit includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the human pose estimation method is implemented.

[0048] Furthermore, the human pose estimation system further includes:

[0049] The image acquisition unit is used to acquire the original image M1 for which human pose estimation is required.

[0050] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0051] Before performing human pose estimation, an image containing only the human figure is first segmented from the original image. Then, a dual-patch method is used to extract the image, and feature vectors are extracted using channel attention and spatial attention respectively, to achieve more accurate human pose estimation. Using object detection to segment the human image effectively avoids interference from other irrelevant factors in the image. The dual-channel attention mechanism acquires human pose information, and when encountering occlusion, the position can be deduced through the attention mechanism—an advantage that cannot be achieved using only keypoint detection. Furthermore, complex pose detection can be achieved by using random patch images and global spatial feature maps for mutual mapping, and the positional information extracted by the attention mechanism is used to fit the pose. Attached Figure Description

[0052] Figure 1 This is a flowchart of a human pose estimation method provided in an embodiment of this application;

[0053] Figure 2 This is a schematic diagram of the network structure of the human pose estimation method in the embodiments of this application;

[0054] Figure 3 This is a schematic diagram of the structure of a human posture estimation system provided in an embodiment of this application. Detailed Implementation

[0055] The invention will now be further described with reference to the accompanying drawings.

[0056] like Figure 1 As shown in the figure, this application provides a human pose estimation method, including:

[0057] (1) Using a segmentation model consisting of a Swing Transformer and a Region Proposal Network (RPN), the original image M1 for which human pose estimation is required is segmented and extracted to obtain images A1, A2, ..., A1 containing only humans. n ;

[0058] Step (1) specifically includes:

[0059] The original image M1 was feature-extracted using the Swing Transformer to obtain the feature-extracted map.

[0060] The Region Proposal Network (RPN) is used to obtain the bounding box coordinates of the regions in the feature extraction image with the human body as an instance. The bounding box coordinates are then used to obtain the segmentation mask, and the segmented images are resized to be consistent.

[0061] The formulas used by Swin Transformer are as follows:

[0062]

[0063] in, For querying the matrix, Let be the key matrix, M is a value matrix. 2 Here, d is the number of window patches, d is the dimension of the query matrix and the key matrix, and B is taken from a smaller bias matrix. SoftMax(.) is the normalization function.

[0064] (2) For images A1, A2, ..., A n Each patch is processed separately to obtain non-overlapping patch blocks with random positions and fixed sizes, and patch blocks with fixed sizes that can cover the entire image. Feature extraction is performed on each patch block using a SwinTransformer with class label l to obtain the corresponding feature maps.

[0065] Step (2) specifically includes:

[0066] Images A1, A2, ..., A n The data is divided into a set D of non-overlapping patch blocks with random positions and fixed sizes. Feature extraction is performed using a Swing Transformer with class label l to obtain the corresponding feature map F. h, w, and c represent the height, width, and channels of the feature map, respectively; D = {d1, d2, ..., dc} e |I l},e=(h×w) / a 2 a×a is the spatial size of the rectangular patch d, and d i The spatial dimension is defined as p i =[x i ,y i ,Δw,Δh],Δw=Δh=a,[x i ,y i [x] represents the pixel coordinates of the patch block on the image, with the first pixel at the bottom left as the origin. A random function is used to obtain the [x] coordinates of the patch block. i ,y i ]coordinate;

[0067] Images A1, A2, ..., A n The image is then divided into patches of varying sizes that cover the entire image. This time, the patch scale is randomly varied, but all patches do not overlap, filling the entire image. Feature extraction is then performed again using a Swing Transformer with class labels l, yielding the corresponding feature map F. i Each patch is represented as d i =[F i p i ], F i For patch p i Feature map, p i =[x i ,y i ,Δw i ,Δh i [Regarding p] i The spatial dimension of the block, Δw i ,Δh i Width and height of each patch block;

[0068] The images A1, A2, ..., A n The feature map F corresponding to the set P of n non-overlapping patches of the same scale and non-fixed-size patches that can cover the entire image is upsampled to obtain a higher-resolution feature map k. The features of each patch are then pooled using bilinear interpolation. p i For a single patch block, F i For p i The feature map, k, has a size of h×w. Upsampling can be viewed as a mapping operation. h, w, and c represent the height, width, and channels of the feature map, respectively.

[0069] The formula for bilinear interpolation is as follows:

[0070]

[0071] Among them, Q 11 = (x1, y1), Q 12 = (x1, y2), Q 21 = (x2, y1), Q 22 = (x2, y2); (x1, y1), (x1, y2), (x2, y1), (x2, y2) are the coordinates of the points.

[0072] (3) For non-overlapping patch feature maps with random positions and fixed sizes, channel attention processing is used to obtain the feature vector F. CA Spatial attention is used to process the patch feature map, which is of non-fixed size and can cover the entire image, to obtain the feature vector F. SA The feature vectors obtained from the two attention methods are blended to obtain a patch-based hybrid attention feature vector F. PbA ;

[0073] Step (3) specifically includes:

[0074] From the non-overlapping patch feature map F with random positions and fixed size output in step (2), we learn the query Q, keyword K, and value V. The attention weight matrix is ​​the dot product of Q and K, multiplied by V to generate the attention feature descriptor. The channel attention formula is as follows:

[0075]

[0076] Where F is the feature map obtained in step (2), and M c σ is the weight coefficient, MLP(.) is the activation function, AvgPool(.) is the multilayer perceptron, MaxPool(.) is the average pooling layer, and W1 and W0 are matrix weights. The average feature map of C neurons, The maximum feature map of C neurons;

[0077] The importance of each pi is calculated using a weighted sum of the attention scores of all patches, as shown in the following formula:

[0078] δ i,j =softmax(M c +b δ ),

[0079] Where, δ i,j Let π be the mathematical constant, softmax(.) be the normalization function, and b be the normalization function. δ For bias vectors, For the feature space of aggregation, F j For patch feature maps;

[0080] Global average pool aggregation Generate all blocks in P Finally, the weighted attention matrix φ is generated using Softmax. i The formula is as follows:

[0081]

[0082] in, The average feature map of all patch blocks. F is the global average pooling function. CA This is the weighted channel attention feature vector. W φ As the weight, b φ For bias.

[0083] Global max pooling is performed on the patch feature map output from step (2), which has a non-fixed size and can cover the entire image, to generate an optimized feature vector F. gmp The feature vector F is generated by global average pooling with the patch feature map, which is of non-fixed size and can cover the entire image. gap The feature maps are fused to obtain the fused feature map H, as shown in the following formula:

[0084]

[0085] in, concat(.) is the fusion function. This is the global average pooling function. It is the global max pooling function.

[0086] Applying a multilayer perceptron to generate spatial attention feature vector F SA The specific formula is as follows:

[0087]

[0088] in, For a multilayer perceptron, σ(.) is an element-wise nonlinear activation function using GELU. X is the vector obtained by Flattening the feature map H, and XW represents a linear transformation on X. λ GD+BN For regularization processes with Gaussian discarding and batch normalization;

[0089] The spatial attention feature vector F SA The element-by-element and weighted channel attention feature vector F CA Multiply, and enhance distinguishability by capturing subtle details and focusing on global structural information, by combining residual paths and F CA The concatenation yields a patch-based hybrid attention feature vector F.PbA The specific formula is as follows:

[0090] (4) The feature vectors are fused, and after being processed by a combination of linear and nonlinear connection layers, Gaussian dropout and batch normalization are used for regularization and classification to obtain the pose estimation results.

[0091] Step (4) specifically includes:

[0092] The upsampled feature map F from step (2) is subjected to a gap layer. Compression produces a vector with (1×c) channels, which is correlated with the attention feature map F. PbA Add them together, and the final feature vector F f The results are processed by linear and nonlinear connection layers, summed, regularized, and then passed to Softmax to calculate the output vector of latent class probabilities. The resulting class is the human pose estimation result, as shown in the following formula:

[0093] Y pred =softmax(λ(F) f ))

[0094] Where λ(.) represents the Gaussian loss and batch normalization regularization processing, with Gaussian loss using multiplicative noise and the loss rate φ mapped to the noise standard deviation. ρ represents the noise level.

[0095] like Figure 2The diagram shown is a schematic of the network structure of the human pose estimation method provided in this embodiment. After feature extraction from the original image using the Swin Transformer, the image is processed through two convolutional layers and then segmented using the RPN segmentation model to obtain bounding boxes of the human figure. After segmenting the bounding boxes, an image containing only the human figure is obtained. The obtained image containing only people is divided into two patches to obtain non-overlapping patches with random locations and fixed sizes, and patches of fixed sizes that cover the entire image. Features are extracted using a Swing Transformer with class labels and input into the Upsample module. The features of the non-overlapping patches with random locations and fixed sizes are then input into the GMP module. The result is then combined with the result of inputting the features of the patches of fixed sizes that cover the entire image into the GAP module and input into the Concatenation module. The results are processed by linear and non-linear GELU layers respectively, summed, and then input into Gaussian Dropput and BatchNorm layers for regularization. After summing, a Softmat layer is used for normalization. Channel attention and spatial attention are then applied to the feature maps of the non-overlapping patches with random locations and fixed sizes, and the patches of fixed sizes that cover the entire image, respectively. The outputs are then processed again by linear and non-linear GELU layers. The sums of the results are then input into the Gaussian... The Dropput and BatchNorm layers perform regularization operations. The regularization results are then added together and fed into the Softmax layer for normalization. Finally, the results are classified to obtain the human pose estimation result.

[0096] like Figure 3 As shown in the illustration, this application also provides a human pose estimation system, including an image acquisition unit and a human pose estimation unit. The image acquisition unit is used to acquire an original image M1 for which human pose estimation is required. The human pose estimation unit includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the human pose estimation method described in this application embodiment.

Claims

1. A method for estimating human pose, characterized in that, include: (1) A segmentation model consisting of a Swing Transformer and a Region Proposal Network (RPN) is used to perform human pose estimation on the original images. Segmentation and extraction are performed to obtain images containing only people. , … ; (2) For the image , … Each patch is processed separately to obtain non-overlapping patch blocks with random positions and fixed sizes, and patch blocks of non-fixed sizes that can cover the entire image. All patches are labeled with class tags. The SwinTransformer is used to extract features and obtain the corresponding feature map. (3) For non-overlapping patch feature maps with random positions and fixed sizes, channel attention processing is used to obtain feature vectors. ; Spatial attention is used to process the feature maps of non-fixed-size patches that can cover the entire image to obtain feature vectors. The feature vectors obtained from the two attention methods are blended to obtain a patch-based hybrid attention feature vector. ; (4) The feature vectors are fused, and after being processed by a combination of linear and nonlinear connection layers, regularization and classification are performed using Gaussian dropout and batch normalization to obtain the pose estimation results. Step (2) includes: Image , … Divide into a set D of non-overlapping patch blocks with random positions and fixed sizes, using class labels. The Swing Transformer is used for feature extraction to obtain the corresponding feature map F, where... h, w, and c represent the height, width, and channels of the feature map, respectively; , , Let d be the space size of the rectangular patch. Spatial dimension is defined as , , Let [the location] be the pixel coordinates of the patch block on the image, with the first pixel at the bottom left as the origin. A random function is used to obtain the [[...]] coordinates of the patch block. ]coordinate; Image , … The image is then divided again into non-fixed-size patches that cover the entire image. This time, the patch scale is randomly varied, but all patches do not overlap, filling the entire image. Class labels are then used again. The Swing Transformer is used for feature extraction to obtain the corresponding feature map. Each patch is represented as , For patch Feature map, For about The spatial dimension of a block , Width and height of each patch block; Will the image , … The feature map F corresponding to the set P of n non-overlapping patches of the same scale and non-fixed-size patches that can cover the entire image is upsampled to obtain a higher-resolution feature map k. The features of each patch are then pooled using bilinear interpolation. , , For a single patch block, for The feature map, k has a size of Upsampling can be viewed as a mapping operation. h, w, and c represent the height, width, and channels of the feature map, respectively.

2. The human posture estimation method according to claim 1, characterized in that, Step (1) includes: Use the Swing Transformer on the original image Perform feature extraction to obtain a feature extraction map; The Region Proposal Network (RPN) is used to obtain the bounding box coordinates of the regions in the feature extraction image with the human body as an instance. The bounding box coordinates are then used to obtain the segmentation mask, and the segmented images are resized to be consistent.

3. The human posture estimation method according to claim 2, characterized in that, The formulas used by Swin Transformer are as follows: ； in, For querying the matrix, The key matrix, For value matrices, Here, d is the number of window patches, d is the dimension of the query matrix and the key matrix, and B is taken from a bias matrix. , SoftMax(.) is the normalization function.

4. The human posture estimation method according to claim 1, characterized in that, The formula for bilinear interpolation is as follows: ； in, , , , ; , , , represents the position coordinates of the point.

5. The human posture estimation method according to claim 1, characterized in that, Step (3) includes: From the non-overlapping patch feature map F with random positions and fixed size output in step (2), we learn the query Q, keyword K, and value V. The attention weight matrix is ​​the dot product of Q and K, multiplied by V to generate the attention feature descriptor. The channel attention formula is as follows: ； Where F is the feature map obtained in step (2), These are weighting coefficients. It is an activation function. It is a multilayer perceptron. For average pooling layer, For max pooling layer, , All are matrix weights; The average feature map of C neurons, The maximum feature map of C neurons; The importance of each pi is calculated using a weighted sum of the attention scores of all patches, as shown in the following formula: ； in, Pi For normalization function, For bias vectors, For the feature space of aggregation, For patch feature maps; Global average pool aggregation Generate all blocks in P Finally, a weighted attention matrix is ​​generated using Softmax. The formula is as follows: ； in, The average feature map of all patch blocks. , This is the weighted channel attention feature vector. , As weight, For bias.

6. The human posture estimation method according to claim 5, characterized in that, Step (3) also includes: Global max pooling is performed on the patch feature map output from step (2), which has a non-fixed size and can cover the entire image, to generate an optimized feature vector. The feature vector is generated by global average pooling with a patch feature map of non-fixed size that can cover the entire image. The feature maps are fused to obtain the fused feature map H, as shown in the following formula: ； in, , For fusion function, , It is the global max pooling function. ; Applying a multilayer perceptron to generate spatial attention feature vectors The specific formula is as follows: ； in, It is a multilayer perceptron. It is an element-wise nonlinear activation function, employing the GELU nonlinear activation function. X is the vector obtained by Flattening the feature map H, and XW represents a linear transformation on X. For regularization processes with Gaussian discarding and batch normalization; Spatial attention feature vector element-by-element and weighted channel attention feature vector Multiplication, and enhanced distinguishability by capturing subtle details and focusing on global structural information, combines residual paths and The concatenation yields a patch-based hybrid attention feature vector. The specific formula is as follows: .

7. The human posture estimation method according to claim 6, characterized in that, Step (4) includes: The upsampled feature map F from step (2) is subjected to a gap layer. Compression produces (1) c) Vectors of the channels, which are related to the attention feature map. Add them together to get the final feature vector. The results are processed by linear and nonlinear connection layers, summed, regularized, and then passed to Softmax to calculate the output vector of latent class probabilities. The resulting class is the human pose estimation result, as shown in the following formula: ； in, For Gaussian loss and batch normalization regularization processing, Gaussian loss uses multiplicative noise, and the loss rate is... Mapped to noise standard deviation , Noise level.

8. A human pose estimation system, characterized in that, include: A human pose estimation unit includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the human pose estimation method according to any one of claims 1 to 7.

9. The human posture estimation system according to claim 8, characterized in that, Also includes: The image acquisition unit is used to acquire raw images for human pose estimation. .

Images