Lightweight human body posture estimation method

Abstract

The invention belongs to the field of computer vision based on a deep neural network, and discloses a lightweight human body posture estimation method. According to the method, human body postures are recognized in real time through a designed lightweight method. A lightweight global modeling network LGM-Pose which is approximate to a single branch is provided. In the network, a global feature extraction module is designed, and lightweight extraction of global information is realized through a proposed parameter-free dimension transformation sub-module and MLP Block. And a convolution feature fusion module is designed to fuse multi-scale information through shuffling operation and grouping convolution, so that the problem of performance reduction caused by lack of multi-scale combination of a single-branch structure is relieved. Experimental results on a COCO data set show that compared with a current lightweight method in the field of human body posture estimation, the method provided by the invention has fewer parameters and obtains more excellent performance at the same time.

Description

Technical Field

[0001] The present invention relates to the field of computer vision based on deep neural networks, and in particular to a lightweight method for estimating human posture. Background Art

[0002] The purpose of human pose estimation is to simultaneously detect the key points of each human body from the input image. It is a prerequisite technology for many computer vision tasks, such as human action / activity recognition, virtual animation modeling, human pose tracking and motion analysis. In practical applications, these tasks often need to be deployed on resource-constrained devices. Therefore, the research on high-performance lightweight human pose estimation models has become a hot issue in this field. These models not only need to maintain high-precision pose estimation capabilities, but also reduce the consumption of computing resources to adapt to the limitations of resource-constrained terminal environments.

[0003] In recent years, with the development of deep convolutional neural networks, human pose estimation has made significant progress. Since the key points of the human body are small and extremely sensitive to position, some high-resolution convolutional networks have been proposed to improve the prediction accuracy of human key point detection. At present, most lightweight methods for top-down multi-person pose estimation are based on multi-branch parallel network architectures. The multi-branch structure represented by the high-resolution parallel network (HRNet) can combine multi-scale features to obtain local and global information, and has achieved remarkable performance in the top-down human pose estimation task. Therefore, inspired by HRNet, many current lightweight methods have adopted a multi-branch architecture and conducted lightweight research on this basis. However, the multi-branch frequent up-down and down-sampling fusion operations and a large number of 1x1 convolutions brought about by the multi-resolution parallel structure adopted by these models still make their networks have high latency. At the same time, these lightweight methods are all based on pure CNN construction. Due to the inherent defects of the network volume and convolution kernel, the model's ability to obtain global information is limited, resulting in the key points with complex semantics in the scene being difficult to identify or mispositioned.

[0004] In recent years, the Transformer architecture has been applied to human pose estimation methods due to its excellent global learning representation ability, and has significant key point detection capabilities. Some models combine Transformer with CNN, using the position sensitivity of CNN and the global information interaction ability of Transformer to achieve stronger expression of pose estimation. However, the high computational cost of Transformer makes it difficult to directly apply these methods to resource-constrained scenarios. MobileViT designs a lightweight attention model by dividing the feature map into patches and performing attention calculations on different patches. However, the Transformer part used to extract global information in MobileViT still occupies a considerable computational cost. Summary of the invention

[0005] The purpose of the present invention is to overcome the problems of computational redundancy, insufficient global information acquisition and expensive Transformer computation in pure CNN multi-branch architecture. Computational redundancy is reduced and latency is reduced by reasonably designing an approximate single-branch architecture and a convolutional fusion module. And by designing a global information interaction module to obtain long-distance information of features, a significant reduction in computational parameters is achieved on the basis of achieving better performance than using Transformer. Thereby achieving human posture estimation with higher accuracy and lower latency. In order to achieve the above-mentioned purpose of the invention, the present invention provides the following technical solutions:

[0006] A lightweight human body posture estimation method comprises the following steps:

[0007] S1: construct a human posture estimation model, the model includes an initial feature extraction module, a global feature extraction module, a high-resolution feature map generation module, and a key point heat map generation module connected in sequence;

[0008] The initial feature extraction module is used to perform initial feature extraction on the input image, extract shallow semantic information, and reduce the resolution so as to reduce the amount of calculation in subsequent operations;

[0009] The global feature extraction module is used to extract deep semantic information from the image after passing through the initial feature extraction module, and obtain information of different scales;

[0010] The high-resolution feature map generation module is used to generate high-quality high-resolution feature maps from low-resolution feature maps, and to combine multi-scale features to facilitate key point prediction

[0011] The key point heat map generation module is used to obtain the feature map of semantic information to generate a key point heat map and perform the final posture estimation task;

[0012] S2: Setting a training set and a loss function to train the human posture estimation model;

[0013] S3: Input the image to be estimated containing the human body into the trained human posture estimation model to obtain a heat map of the key points of the human body.

[0014] Furthermore, the initial feature extraction module uses conventional 3x3 convolution and mobile device-friendly MobileNetV2Block for downsampling and initial feature extraction; one 3x3 convolution and four MobileNetV2Blocks are connected in sequence.

[0015] Furthermore, the MobileNetV2Block first performs 1x1 convolution on the input feature map to increase the dimension, then performs depth-wise separable convolution on the increased-dimensional feature map, and finally reduces the dimension of the feature map of the extracted local features for subsequent feature extraction.

[0016] Furthermore, the global feature extraction module first uses a standard 3x3 convolution to encode the local information of the feature map, followed by a 1x1 convolution to learn the linear combination of input channels to project the tensor into a high-dimensional space; then, the feature map is input into the global interaction module for global modeling; finally, the input features are convolutionally fused with the features after global modeling and dimensionality reduction by 1x1 convolution.

[0017] Furthermore, the global interaction module transforms the feature map R H x W x d Will be expanded into N non-overlapping flat surfaces R P x N x d ;

[0018] Where P = wh, N = HW / P is the number of patches, and h≤n, w≤n are the height and width of the patch respectively; then R P x N x d Convert to R P x dx N , input into MLPBlock, the input dimension is N; MLPBlock can fuse pixels at the same position in different patches to obtain tensor R P x dx N , thereby obtaining global interaction information;

[0019] The MLPBlock is a two-layer perceptron, and the formula is as follows

[0020] X U →LayerNorm→linear→GELU→linear→X G

[0021] The first linear layer maps the input to a high-dimensional space to enhance the nonlinear expression, and the second linear layer maps the high-dimensional information to the original dimension.

[0022] Furthermore, the global interaction module involves a parameter-free dimension transformation part, wherein the first transformation is to divide the feature map into patches to obtain R P x N x d , flatten each patch to (1, N) and arrange them vertically. The dimension of the feature map is R P x dx N ; When performing feature fusion, N is used as input for nonlinear transformation, which means that pixels at the same position in different patches are used as input; the second dimensional transformation is to convert R P x dx N Flip up to get the feature map R N x dx P At this time, the different pixels in the same patch are input to the MLPBlock module for information exchange; the third time is to fold the feature map after information fusion back to the initial feature map for feature extraction in the next stage.

[0023] Furthermore, the high-resolution feature map generation module uses deconvolution upsampling on the low-resolution feature map to increase the resolution; and fuses the feature maps with the same resolution through the convolution fusion module to obtain the final high-quality feature map.

[0024] Furthermore, the convolution fusion module splices the deconvolved feature map with the feature map of the same resolution in the feature extraction stage, performs a shuffle operation, then performs a group convolution, and finally performs a 1x1 convolution adjustment channel.

[0025] Furthermore, the human body posture estimation model is trained using the MSE loss function, and the loss function expression is as follows:

[0026]

[0027] Among them, Y ij is the heat map of the real key points, X ij is the keypoint heatmap predicted by the model, N is the number of samples, K is the number of keypoints, and ||·|| represents the Euclidean norm.

[0028] The present invention provides a lightweight human body posture estimation algorithm. Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. This paper proposes a lightweight global modeling network named LGM-PoseNet for multi-person pose estimation. The network can obtain multi-scale and global information with a simple structure, obtain accurate key point positioning and have low latency, and achieve the best performance among the current representative models.

[0030] 2. The global interaction module designed by the present invention effectively establishes remote dependencies and captures global information, thereby improving the ability to accurately locate complex key points. It uses a simple and lightweight method to effectively cope with edge devices with limited resources and perform real-time human posture estimation tasks.

[0031] Instruction Manual

[0032] Figure 1 is a flow chart of the present invention;

[0033] Figure 2 It is a model structure diagram of the present invention;

[0034] Figure 3 It is a structural diagram of the global feature extraction module of the present invention;

[0035] Figure 4 This is a structural diagram of the convolution fusion module of the present invention;

[0036] Figure 5 It is a schematic diagram of the conclusion of the present invention. DETAILED DESCRIPTION

[0037] Here, exemplary embodiments will be described in detail, and unless otherwise indicated, the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are only examples consistent with some aspects of the present disclosure as detailed in the appended claims.

[0038] like Figure 1 As shown, this embodiment provides a lightweight human body posture estimation method, comprising the following steps:

[0039] S1: Build a human posture estimation model, which includes the initial feature extraction module, the global feature extraction module, the high-resolution feature map generation module, and the key point heat map generation module.

[0040] The initial feature extraction module uses conventional 3x3 convolution and downsampling of the mobile device's MobileNetV2Block and initial feature extraction. The input resized image resolution is HxW, where H is the height of the feature image and W is the width of the feature image. A 3x3 convolution with a step of 2 is sequentially connected to downsample to And 4 MobileNetV2Blocks, only the convolution step number in the second MobileNetV2Bloc is 2, and the output feature map dimension is

[0041] MobileNetV2Block first performs 1x1 convolution on the input feature map to increase the dimension, then performs depth-separable convolution on the increased-dimensional feature map, and finally extracts the local features of the feature map. Figure 1 x1 dimension reduction for subsequent feature extraction.

[0042] For the subsequent feature map feature extraction stage, a MobileNetV2 module and a global feature extraction module are used together three times, where the MobileNetV2 module has a convolution step of 2, which can downsample while extracting local information. The global feature extraction module first uses a standard 3x3 convolution to encode the local information of the feature map, followed by a 1x1 convolution to learn the linear combination of the input channels to project the tensor into a high-dimensional space. Then, the feature map is input into the global interaction module for global modeling. Finally, the input features are convolved and fused with the features after global modeling and dimensionality reduction by 1x1 convolution.

[0043] The first time, the input feature map resolution is After the MobileNetV2 module, the feature map resolution is Then, the global interaction module transforms the feature map R H x W x d Will be expanded into N non-overlapping flat surfaces R P x N x d Here, P = wh, N = HW / P is the number of patches, and h≤n, w≤n are the height and width of the patch, which can be set to 2 or 4. Then R P x N x d Convert to R P x dx N , input into MLPBlock, the input dimension is N. MLPBlock can fuse pixels at the same position in different patches to obtain tensor R P x dx N , thereby obtaining global interaction information.

[0044] The MLPBlock is a two-layer perceptron, and the formula is as follows:

[0045] X U →LayerNorm→linear→GELU→linear→X G

[0046] The first linear layer maps the input to a high-dimensional space to enhance the nonlinear expression, and the second linear layer maps the high-dimensional information back to the original dimension. This involves parameter-free dimensional transformation. The first transformation is to divide the feature map into patches to obtain R P x N x d , flatten each patch to (1, N) and arrange them vertically. The dimension of the feature map is R P x dx N When performing feature fusion, N is used as input for nonlinear transformation, which means that pixels at the same position in different patches are used as input. The second dimensional transformation is to convert R P x dx N Flip up to get the feature map R N x dx P , at this time, the different pixels in the same patch are input to the MLPBlock module for information exchange. The third time is to fold the feature map after information fusion back to the initial feature map for the next stage of feature extraction. The resolution of the feature map obtained at this time is

[0047] The second time, the input feature map resolution is After the MobileNetV2 module, the feature map resolution is After the global interaction module, the feature map resolution is

[0048] The third time, the input feature map resolution is After the MobileNetV2 module, the feature map resolution is After the global interaction module, the feature map resolution is

[0049] The high-resolution feature map generation module has a resolution of The low-resolution feature map is upsampled using deconvolution to increase the resolution to And the feature maps with the same resolution are fused through the convolution fusion module to obtain the final high-quality feature map.

[0050] The convolution fusion module concatenates the deconvolved feature map with the feature map of the same resolution in the feature extraction stage, performs a shuffle operation, and then performs a group convolution, in which the number of groups is set to half of the input dimension, and finally performs a 1x1 convolution adjustment channel.

[0051] The convolution fusion module is also performed three times. The feature map resolution output after the second convolution fusion is

[0052] The feature map resolution output after the third convolution fusion is

[0053] For the high-resolution feature map that has extracted local and global information, it is input into the key point heat map generation module. This module contains a MobileNetV2 module to enhance semantic information and a 1x1 convolution to generate the number of key points corresponding to the dataset.

[0054] S2: Setting a training set and a loss function to train the human body posture estimation model of the present invention;

[0055] The human posture estimation model of the present invention is trained based on the public data set COCO data set, and the number of human key points in the COCO data set is M=17. In the training stage, the images of the training set are rotated, cropped and flipped and then input into the human posture estimation model of the present invention. The M heat maps output by the model are Gaussian distribution predictions of the coordinates of the human key points. The heat maps predicted by the model are compared with the given real labels in the data set to measure the loss for optimization. The present invention uses the MSE loss function to train the human posture estimation model. The loss function expression is as follows

[0056]

[0057] Among them, Y ij is the heat map of the real key points, X ij is the key point heat map predicted by the model, N is the number of samples, K is the number of key points, and ||·|| represents the Euclidean norm

[0058] The test results of the proposed method and other existing methods on the COCO validation set are shown in Table 1. In Table 1, LGM-Pose represents the model proposed by us. Params represents the number of parameters of the operation, GFLOPs represents the number of floating-point operations, AP represents the average precision, AP 50 ,AP 75 ,APM and AP L They represent the average precision at an IoU threshold of 0.50, the average precision at an IoU threshold of 0.75, the average precision at all IoU thresholds, and the average precision at lower IoU thresholds. AR represents the average recall rate. FPS represents the FPS test value in CPU and GPU environments. It can be seen that our model maintains comparable performance and has low latency while having low computational complexity.

[0059] Table 1: Comparison results on the COCO dataset validation set

[0060]

[0061]

[0062] S3: Input the image to be estimated containing the human body into the trained human posture estimation model to obtain a heat map of the key points of the human body.

[0063] The pose estimation prediction pictures on the COCO dataset are as shown in the attached manual. Figure 5 As shown in the figure, each person predicts 17 key points. It can be seen that the model of the present invention can still accurately predict complex points and small-sized characters.

[0064] The above is only one of the specific implementations of the present invention, but the technical features of the present invention are not limited thereto. Any modification, equivalent replacement or improvement made within the technical concept of the present invention shall be included in the protection scope of the present invention.

Claims

1. A lightweight human body posture estimation method, characterized in that: The steps include: S1: constructing a human posture estimation model, the model comprising an initial feature extraction module, a global feature extraction module, a high-resolution feature map generation module, and a key point heat map generation module connected in sequence; The initial feature extraction module is used to perform initial feature extraction on the input image, extract shallow semantic information, and reduce the resolution so as to reduce the amount of calculation in subsequent operations; The global feature extraction module is used to extract deep semantic information from the image after passing through the initial feature extraction module, and obtain information of different scales; The high-resolution feature map generation module is used to generate a high-quality high-resolution feature map from a low-resolution feature map, and to combine multi-scale features to facilitate key point prediction; The key point heat map generation module is used to obtain the feature map of semantic information to generate a key point heat map and perform the final posture estimation task; S2: Setting a training set and a loss function to train the human posture estimation model; S3: Inputting the image to be estimated containing the human body into the trained human posture estimation model to obtain a heat map of key points of the human body.

2. The lightweight human body posture estimation method according to claim 1, characterized in that: The initial feature extraction module uses conventional 3x3 convolution and mobile device-friendly MobileNetV2 Block for downsampling and initial feature extraction; one 3x3 convolution and four MobileNetV2 Blocks are connected in sequence.

3. The lightweight human body posture estimation method according to claim 2, characterized in that: The MobileNetV2Block first performs 1x1 convolution on the input feature map to increase the dimension, then performs depth-wise separable convolution on the increased-dimensional feature map, and finally reduces the dimension of the feature map of the extracted local features for subsequent feature extraction.

4. The lightweight human body posture estimation method according to claim 1, characterized in that: The global feature extraction module first uses a standard 3x3 convolution to encode the local information of the feature map, followed by a 1x1 convolution to learn the linear combination of input channels to project the tensor into a high-dimensional space; then, the feature map is input into the global interaction module for global modeling; finally, the input features are convolved and fused with the features after global modeling and dimensionality reduction by 1x1 convolution.

5. The lightweight human body posture estimation method according to claim 4, characterized in that: The global interaction module transforms the feature map R HxWxd Will be expanded into N non-overlapping flat surfaces R PxNxd ; Where P = wh, N = HW / P is the number of patches, and h≤n, w≤n are the height and width of the patch respectively; then R PxNxd Convert to R PxdxN , input into the MLP Block, the input dimension is N; MLP Block can fuse pixels at the same position in different patches to obtain a tensor R PxdxN , thereby obtaining global interaction information; The MLP Block is a two-layer perceptron, and the formula is as follows X U →LayerNorm→linear→GELU→linear→X G The first linear layer maps the input to a high-dimensional space to enhance the nonlinear expression, and the second linear layer maps the high-dimensional information back to the original dimension.

6. The lightweight human body posture estimation method according to claim 4, characterized in that: The global interaction module involves a parameter-free dimension transformation part; the first transformation is to divide the feature map into patches to obtain R PxNxd , flatten each patch to (1, N) and arrange them vertically. The dimension of the feature map is R PxdxN ; When performing feature fusion, N is used as input for nonlinear transformation, which means that pixels at the same position of different patches are used as input; The second dimensional transformation is to transform R PxdxN Flip up to get the feature map R NxdxP At this time, the different pixels in the same patch are input to the MLPBlock module for information exchange; the third time is to fold the feature map after information fusion back to the initial feature map for feature extraction in the next stage.

7. The lightweight human body posture estimation method according to claim 1, characterized in that: The high-resolution feature map generation module uses deconvolution upsampling on the low-resolution feature map to increase the resolution; and fuses the feature maps with the same resolution through the convolution fusion module to obtain the final high-quality feature map.

8. The lightweight human body posture estimation method according to claim 7, characterized in that: The convolution fusion module concatenates the deconvolved feature map with the feature map of the same resolution in the feature extraction stage, performs a shuffle operation, then performs a group convolution, and finally performs a 1x1 convolution adjustment channel.

9. The lightweight human body posture estimation method according to claim 1, characterized in that: The human body posture estimation model is trained using the MSE loss function, and the loss function expression is as follows: Among them, Y ij is the heat map of the real key points, X ij is the keypoint heatmap predicted by the model, N is the number of samples, K is the number of keypoints, and ||·|| represents the Euclidean norm.

Images