A multi-level efficient human posture estimation system and method

Abstract

The present invention relates to the technical field of human posture estimation, and solves the technical problem that it is difficult for traditional methods to improve the efficiency of key points on single-scale features, and in particular to a multi-level efficient human posture estimation system and method, the system comprising: extracting features from an input image in sequence, and generating a local feature extraction module, a spatial feature extraction module, and a global feature extraction module for low-level features, intermediate features, and high-level features, respectively. The method comprises: obtaining low-level features, intermediate features, and high-level features of the input image in sequence, and converting the spatial scale of the high-level features into two-dimensional spatial features, and using deconvolution to restore the scale of the two-dimensional spatial features to obtain the key point positions corresponding to the high-level features in the input image. The present invention overcomes the defect that it is difficult for traditional methods to improve the efficiency of key points on single-scale features, can effectively improve the accuracy of key points, and maintain a good running speed.

Description

Technical Field

[0001] The present invention relates to the technical field of human body posture estimation, and in particular to a multi-level efficient human body posture estimation system and method. Background Art

[0002] Human pose estimation aims to locate key points of the human body on an image and is a fundamental task in academia and industry, such as human-computer interaction, activity recognition, and fitness. With the advent of deep learning, the performance of human pose estimation has been greatly improved.

[0003] Convolutional neural network (CNN) obtains local information through convolution kernels and identifies key points of the human body with higher accuracy than traditional manual filters. In order to improve the accuracy of the network, HRNet proposes a multi-scale fusion scheme to produce reliable high-resolution representations by iteratively fusing features generated from high to low subnetworks. Compared with single-scale feature extraction, multi-scale fusion ensures that key point heat maps retain detailed information in the network, and strategies have also been found in bottom-up methods for human posture estimation. Although multi-scale fusion is better in key point recognition, it seriously affects the recognition efficiency of key nodes. In real-world applications, the accuracy and efficiency of key point recognition will restrict the development of applications.

[0004] Currently, researchers have begun to apply Transformer to human posture estimation. Unlike CNN, which captures low-level features, Transformer can capture high-level global dependencies with self-attention. In many methods, CNN and simple Transformer are directly combined to obtain primary and high-level features from the image, so that the network can learn local and global features. This is more conducive to the accuracy of key point recognition than pure single-scale convolution or single-scale Transformer methods, and overcomes the disadvantages of multi-scale methods in network operation efficiency. Therefore, people began to think about how to use the combination of multi-scale Transformers to improve the accuracy of key points, but this approach undoubtedly increases the operating burden of the network, making it difficult to ensure that the network maintains high efficiency in terms of both operating speed and accuracy. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a multi-level efficient human posture estimation system and method, which solves the problem of human posture estimation in a new way, overcomes the defect that traditional methods are difficult to improve the efficiency of key points on single-scale features, can effectively improve the accuracy of key points, and maintain a good operating speed.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a multi-stage efficient human posture estimation system based on single-scale feature extraction, the human posture estimation system only obtains the estimation result of the human posture in the final output image of a single scale of the input image, and the system comprises:

[0007] Extract features from the input image in sequence, and generate a local feature extraction module, a spatial feature extraction module, and a global feature extraction module for low-level features, intermediate features, and high-level features, respectively;

[0008] The low-level features are reduced in spatial scale by the spatial reducer to obtain a feature size of C s ×H s ×W s The spatial reduction features are used as inputs of the spatial feature extraction module, the intermediate features are added to the spatial reduction features, and their spatial dimensions are flattened to obtain flattened features for operation in the global feature extraction module;

[0009] After the global feature extraction module outputs the high-level features, the spatial scale of the high-level features of the input image is converted into two-dimensional spatial features by using dimensional conversion, and the scale of the two-dimensional spatial features is restored by using deconvolution, and the estimated result of the human body posture in the image is output.

[0010] Furthermore, the local feature extraction module includes performing channel squeezing and space squeezing on the image input features respectively to obtain channel squeezing features and space squeezing features, and channel squeezing attention and space squeezing attention of the channel squeezing features and the space squeezing features.

[0011] Further, the spatial feature extraction module includes a channel separator that splits the second channel feature into 1 / 4, 1 / 4 and 1 / 2 of the original size after obtaining the feature before splitting the second channel, and obtains the first channel feature, the second channel feature and the third channel feature with the same spatial scale and adjacent channels;

[0012] And, fusing the channel information of the first channel feature and the second channel feature, acquiring the spatial relationship between the two, and obtaining a first channel fuser of the first channel fusion feature;

[0013] And, the channel information of the first channel fusion feature and the third channel feature are fused to obtain the spatial relationship between the two, and the second channel fuser of the second channel fusion feature is obtained.

[0014] Furthermore, the global feature extraction module includes N Transformer encoders, each of which includes a multi-head attention for generating a multi-head self-attention feature, and a position sensor for calculating the multi-head self-attention feature to obtain a degree of perception of the multi-head self-attention feature to a spatial position;

[0015] The position sensor includes a first linear layer that expands the channel of the multi-head self-attention feature by twice to obtain the channel-expanded feature, and a second linear layer that reduces the channel of the channel-expanded feature to half of its original value after being activated by the activation function ReLU.

[0016] The technical solution also provides a multi-level efficient human posture estimation method applied to the above multi-level efficient human posture estimation system, the method comprising:

[0017] Input any input image into the local feature extraction module to obtain the low-level features of the input image;

[0018] A spatial reducer is used to reduce the spatial scale of low-level features to generate spatial reduction features, which are then input into a spatial feature extraction module to obtain intermediate features of the input image.

[0019] Adding the intermediate features of the input image to the spatial reduction features and flattening their spatial dimensions to obtain the flattened features for operation in the global feature extraction module;

[0020] Add position encoding to the flattened features to obtain three operational signatures of the same size, and process the operational signatures through the Transformer encoder to obtain high-level features of the input image;

[0021] The spatial scale of the high-level features is converted into two-dimensional spatial features, and the scale of the two-dimensional spatial features is restored by deconvolution to obtain the key point positions corresponding to the high-level features in the input image.

[0022] Furthermore, the process of obtaining the low-level features of the input image is as follows:

[0023] The input image with feature size of H×W×3 is convolved with a convolution kernel size of 7×7 to obtain a feature size of C i ×H i ×W i Image input features;

[0024] Among them, H and W represent the height and width of the input image respectively. i =1 / 4H,W i =1 / 4W, C i =256;

[0025] The channel squeezing attention and the space squeezing attention perform channel squeezing and space squeezing on the image input features respectively, obtaining a channel squeezing feature and a space squeezing feature;

[0026] After the channel squeezing feature and the space squeezing feature are added, they are fused with the image input feature using a residual connection to obtain a low-level feature of the input image.

[0027] Furthermore, after the channel squeezing attention performs channel squeezing on the image input feature, the process of obtaining the channel squeezing feature is as follows:

[0028] The channel squeeze attention obtains the image input features, and uses two convolutions to compress the channels of the image input features respectively. i to 1 / 2 C i and 1, and keep the spatial size scale unchanged, and obtain the first channel compression feature and the second channel compression feature;

[0029] The first channel compression feature is subjected to dimension conversion and convolution processing to obtain a first channel calculation feature for calculating channel extrusion;

[0030] The second channel compression feature is subjected to dimension conversion and Softmax activation function to obtain the second channel calculation feature for calculating channel squeezing;

[0031] The first channel calculation feature and the second channel calculation feature are fused by matrix dot product operation, and a channel squeeze fusion feature is obtained by convolution and Sigmoid activation function, wherein the spatial size of the channel squeeze fusion feature is 1×1;

[0032] The channel squeeze fusion feature is fused with the image input feature by using an element-by-element multiplication operation to obtain a channel squeeze feature.

[0033] Furthermore, after the spatial squeezing attention performs spatial squeezing on the image input feature, the process of obtaining the spatial squeezing feature is as follows:

[0034] The spatial squeezing attention obtains the image input features, and uses two convolutions to compress the channels of the image input features respectively. i to 1 / 2 C i and 1 / 2 C i , maintaining the spatial size scale unchanged, obtaining the first spatial compression feature and the second spatial compression feature;

[0035] After the first spatial compression feature is pooled and dimensionally converted, a Softmax activation function is used to obtain a first spatial calculation feature for calculating spatial squeezing;

[0036] The second spatial compression feature is subjected to dimensional conversion and convolution to obtain a second spatial calculation feature for calculating spatial extrusion;

[0037] The first spatial calculation feature and the second spatial calculation feature are fused by using a matrix dot product operation, and a spatial squeeze fusion feature is obtained by dimension conversion and a Sigmoid activation function, wherein the number of channels of the spatial squeeze fusion feature is 1;

[0038] The spatial squeeze fusion feature is fused with the image input feature by using an element-by-element multiplication operation to obtain a spatial squeeze feature.

[0039] Furthermore, the process of obtaining the intermediate features of the input image is as follows:

[0040] The low-level features of the input image are reduced to their spatial scale by the spatial reducer, and the feature size is C s ×H s ×W s The space reduction feature of

[0041] Among them, H s =1 / 4H i , W s =1 / 4W i , C s =256;

[0042] The spatial reduction features are batch normalized BatchNorm, and then input into a convolution with a convolution kernel size of 1×1 and a deep convolution DWConv with a convolution kernel size of 5×5, respectively, to obtain feature sizes of C s ×H s ×W s , and used to perceive the first pre-channel splitting features of the image spatial information before channel splitting, and used for subsequent channel splitting to obtain the second pre-channel splitting features of the spatial information relationship of the spatial reduction features;

[0043] After the channel separator obtains the features before the second channel splitting, it splits the channels of the features before the second channel splitting into 1 / 4, 1 / 4 and 1 / 2 of the original sizes, and obtains the first channel features, the second channel features and the third channel features with the same spatial scale and adjacent channels;

[0044] After the first channel feature and the second channel feature are subjected to a deep convolution DWConv with a convolution kernel size of 3×3 and a deep convolution DWConv with a convolution kernel size of 5×5 respectively, the channel information of the first channel feature and the second channel feature are fused using a first channel fuser to obtain a spatial relationship between the two, thereby obtaining a first channel fused feature;

[0045] After the first channel fusion feature and the third channel feature are respectively subjected to the deep convolution DWConv with a convolution kernel size of 5×5 and the deep convolution DWConv with a convolution kernel size of 7×7, the second channel fuser is used to fuse the channel information of the first channel fusion feature and the third channel feature, and the spatial relationship between the two is obtained to obtain the second channel fusion feature;

[0046] The second channel fusion feature is convolved with a convolution kernel size of 1×1, and then multiplied and fused element-by-element with the first channel feature before splitting to obtain the intermediate features of the input image.

[0047] Furthermore, the specific process of obtaining high-level features of the input image by calculating three operational signatures of the same size through the Transformer encoder includes:

[0048] Use multi-head attention to calculate the self-attention value for three operational signatures of the same size. The formula is:

[0049] ;

[0050] Among them, MSA is the self-attention value of multi-head attention, represents the transposition of the first operation signature K; the size of H can be 8 or other specific number of heads;

[0051] Add the self-attention value MSA to the third operation feature symbol V to obtain the multi-head self-attention feature;

[0052] After the position sensor obtains the multi-head self-attention features, it is layer-normalized and the first linear layer is used to expand its channels to twice the original ones to obtain the channel-expanded features;

[0053] The channel-expanded features are subjected to layer normalization, and after activation by the activation function ReLU, layer normalization is performed again, and the second linear layer is used to reduce the channel to half of the original, thus obtaining the channel-reduced features;

[0054] The channel-reduced features are layer-normalized and then added to the multi-head self-attention features to obtain position-aware features.

[0055] After passing through N Transformer encoders, the global feature extraction module outputs high-level features.

[0056] By means of the above technical solution, the present invention provides a multi-level efficient human posture estimation system and method, which has at least the following beneficial effects:

[0057] 1. The present invention overcomes the defect that traditional methods are difficult to improve the efficiency of key points on single-scale features, can effectively improve the accuracy of key points, and maintain a good running speed, and only obtains a single scale of the input image, realizing single-scale feature extraction of a single module, overcoming the defect of multi-scale methods in running speed.

[0058] 2. The present invention adds two major attentions, channel squeezing attention and space squeezing attention, to the local feature extraction module, and completes the attention to its channels and spaces based on the image input features, further improving the present invention's ability to obtain spatial information and the relationship between channels in the image. The proposed channel squeezing attention and space squeezing attention do not occupy too many parameters, and only convolution can be used to complete the attention to channels and spaces, digging into the deep information in the image, and improving the overall accuracy of human posture estimation.

[0059] 3. In the spatial feature extraction module of the present invention, taking into account that local and global features are explorations of feature details and the global situation, in order to further explore the intermediate features between primary and advanced features, the spatial feature extraction module is used to obtain the interactive information of different spaces, and channel fusion is used to complete the extraction of intermediate features, so that the spatial scale of the input remains unchanged, and two channel fusions are performed in the structure, which not only effectively obtains the interactive information of the space, but also improves the overall operating speed.

[0060] 4. In the global feature extraction module, the present invention proposes a position sensor, which is a different structure from the original multi-layer sensor in the Transformer, to enhance the Transformer encoder's perception of position. By changing the number of channels, increasing the number of channels can effectively improve the sensitivity to spatial position, further improving the robustness of the overall method, and without changing the feature dimension, it can better balance the accuracy and efficiency of the network.

[0061] 5. The present invention adds a spatial feature extraction module between obtaining the primary features and high-level features of the input image, so that the method of the present invention sequentially obtains the primary, intermediate and high-level features of the input image, and systematically completes the extraction of human body posture estimation features in images of different degrees, thereby ensuring the accuracy of human body posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0063] Figure 1 It is a schematic diagram of the structure of the prior art multi-scale network in the present invention;

[0064] Figure 2 It is a schematic diagram of the multi-stage efficient human posture estimation system of the present invention;

[0065] Figure 3 A network model structure diagram of a multi-level efficient human posture estimation system of the present invention;

[0066] Figure 4This is a network model diagram of channel squeezing attention of the present invention;

[0067] Figure 5 A network model diagram of spatial squeezing attention of the present invention;

[0068] Figure 6 This is a graph showing the experimental results of the multi-level efficient human posture estimation method of the present invention on the COCO dataset. DETAILED DESCRIPTION

[0069] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0070] In the process of extracting features at different levels of features, this embodiment uses a single scale, which is completely different from the previous multi-scale method, such as the HRNet given in the prior art Deep high-resolution representation learning for human pose estimation. Figure 1 Under this limitation, this embodiment starts from a different perspective and aims to use single-scale features to improve the accuracy and running speed of human posture estimation, thereby completing multi-level feature capture of the image based on a single scale and realizing the feature extraction process of the input image.

[0071] Please refer to Figure 1 - Figure 3 This embodiment proposes a multi-stage efficient human posture estimation system based on single-scale feature extraction. The human posture estimation system only obtains the estimation result of the human posture in the final output image of a single scale of the input image, such as Figure 2 As shown, the system includes: extracting features from the input image in sequence, and generating a local feature extraction module, a spatial feature extraction module and a global feature extraction module for low-level features, intermediate features and high-level features respectively.

[0072] Specific as Figure 3As shown, the local feature extraction module includes channel squeezing attention and space squeezing attention for obtaining channel squeezing features and space squeezing features after performing channel squeezing and space squeezing on the image input features respectively. This embodiment adds two major attentions, channel squeezing attention and space squeezing attention, to the local feature extraction module, and completes the attention to its channels and space based on the image input features, further improving the ability of the present invention to obtain spatial information and the relationship between channels in the image. The proposed channel squeezing attention and space squeezing attention do not occupy too many parameters, and only convolution is used to complete the attention to channels and spaces, excavating deep information in the image, and improving the overall accuracy of human posture estimation.

[0073] The spatial feature extraction module includes a channel separator that splits the channel into 1 / 4, 1 / 4 and 1 / 2 of the original size after obtaining the features before the second channel splitting, and obtains the first channel features, second channel features and third channel features with the same spatial scale and adjacent channels; and a first channel fuser that fuses the channel information of the first channel features and the second channel features, obtains the spatial relationship between the two, and obtains the first channel fusion features; and a second channel fuser that fuses the channel information of the first channel fusion features and the third channel features, obtains the spatial relationship between the two, and obtains the second channel fusion features.

[0074] In this embodiment, taking into account that local and global features are explorations of feature details and the global situation, in order to further explore the intermediate features between primary and advanced features, the spatial feature extraction module is used to obtain the interactive information of different spaces, and channel fusion is used to complete the extraction of intermediate features, so that the spatial scale of the input remains unchanged, and two channel fusions are performed in the structure, which not only effectively obtains the interactive information of the space, but also improves the overall running speed.

[0075] The global feature extraction module includes N Transformer encoders, each of which includes a multi-head attention for generating multi-head self-attention features, and a position sensor for calculating the multi-head self-attention features to obtain the degree of perception of the multi-head self-attention features to the spatial position. Among them, the position sensor includes a first linear layer that expands the channel of the multi-head self-attention feature by twice to obtain the channel-expanded feature, and a second linear layer that reduces the channel of the channel-expanded feature to half of its original value after activation by the activation function ReLU.

[0076] In this embodiment, a position sensor is proposed, which is different from the multi-layer sensor in the original Transformer, in order to improve the Transformer encoder's perception of position. By changing the number of channels, increasing the number of channels can effectively improve the sensitivity to spatial position, further improving the robustness of the overall method, and without changing the feature dimension, it can better balance the accuracy and efficiency of the network.

[0077] The low-level features are reduced in spatial size by the spatial reducer to obtain a feature size of C s ×H s ×W s The spatial reduction features are used as inputs to the spatial feature extraction module, the intermediate features are added to the spatial reduction features, and their spatial dimensions are flattened to obtain flattened features for operation in the global feature extraction module;

[0078] After the global feature extraction module outputs high-level features, it uses dimensionality conversion to convert the spatial scale of the high-level features of the input image into two-dimensional spatial features, and uses deconvolution to restore the scale of the two-dimensional spatial features, and outputs the estimated results of the human body posture in the image.

[0079] In this embodiment, a spatial feature extraction module is added between obtaining the primary features and the high-level features of the input image, so that the method of the present invention sequentially obtains the primary, intermediate and high-level features of the input image, and systematically completes the extraction of human body posture estimation features in images of different degrees, thereby ensuring the accuracy of human body posture estimation.

[0080] Corresponding to the above multi-level efficient human posture estimation system, please refer to Figure 3 - Figure 6 ,This embodiment also proposes a multi-level efficient human posture estimation method based on single-scale feature extraction, such as Figure 3 As shown, the method comprises the following steps:

[0081] S1. Input any input image into the local feature extraction module to obtain the low-level features of the input image. The specific process includes:

[0082] The input image with feature size of H×W×3 is convolved with a convolution kernel size of 7×7 to obtain a feature size of C i ×H i ×W i The image input features; where H and W represent the height and width of the input image respectively, H i =1 / 4H,W i =1 / 4W, C i =256.

[0083] After the channel squeeze attention and the space squeeze attention perform channel squeeze and space squeeze on the image input features respectively, the channel squeeze feature and the space squeeze feature are obtained. As a preferred implementation of this embodiment, Figure 4 As shown in the figure, after the channel squeeze attention performs channel squeeze on the image input features, the process of obtaining the channel squeeze feature is as follows:

[0084] Channel squeeze attention obtains image input features, and uses two convolutions to compress the channels of image input features respectively. i to 1 / 2 C i and 1, and keep the spatial size scale unchanged, and obtain the first channel compression feature and the second channel compression feature;

[0085] The first channel compression feature is subjected to dimension conversion and convolution processing to obtain the first channel calculation feature for calculating channel extrusion;

[0086] The second channel compression feature is converted into a dimension and activated by the Softmax function to obtain the second channel calculation feature for calculating channel squeezing;

[0087] The matrix dot product operation is used to fuse the features calculated by the first channel and the features calculated by the second channel, and the channel squeeze fusion features are obtained through convolution and Sigmoid activation function. The spatial size of the channel squeeze fusion features is 1×1. The matrix dot product operation can be regarded as an operation that keeps the batch number input to the network unchanged in a multidimensional matrix and performs matrix multiplication in the subsequent matrix. The convolution in the channel squeeze attention and spatial squeeze attention mentioned above can be regarded as a 1×1 2D ordinary convolution implementation.

[0088] The channel squeeze fusion feature is fused with the image input feature using element-by-element multiplication operation to obtain the channel squeeze feature.

[0089] In this embodiment, if Figure 5 As shown in Figure 1, after the spatial squeeze attention performs spatial squeezing on the image input features, the process of obtaining the spatial squeeze features is as follows:

[0090] The spatial squeeze attention obtains the image input features and uses two convolutions to compress the channels of the image input features respectively. i to 1 / 2 C i and 1 / 2 C i , maintaining the spatial size scale unchanged, obtaining the first spatial compression feature and the second spatial compression feature;

[0091] After the first spatial compression feature is pooled and dimensionally converted, the Softmax activation function is used to obtain the first spatial calculation feature for calculating spatial squeezing;

[0092] The second spatial compression feature is subjected to dimension conversion and convolution to obtain a second spatial calculation feature for calculating spatial squeezing;

[0093] The first spatial calculation feature and the second spatial calculation feature are fused by matrix dot product operation, and the spatial squeeze fusion feature is obtained after dimension conversion and Sigmoid activation function. The number of channels of the spatial squeeze fusion feature is 1.

[0094] The spatial squeeze fusion feature is fused with the image input feature using element-wise multiplication operation to obtain the spatial squeeze feature.

[0095] In the process of acquiring channel squeezing features and space squeezing features, the essence is to use the image input features' perception of channels and space to complete the feature processing process. By element-by-element multiplication, the importance of the image input features and the obtained channel squeezing fusion features and space squeezing fusion features is calculated to calculate the influence of channels and space on the image input features.

[0096] After adding the channel squeeze features and the space squeeze features, they are fused with the image input features using residual connections to obtain the low-level features of the input image.

[0097] S2, using a spatial reducer to reduce the spatial scale of the low-level features to generate spatial reduction features, and inputting them into a spatial feature extraction module to obtain intermediate features of the input image; in a specific embodiment of the present invention, the spatial feature extraction module is used to obtain the intermediate features of the input image based on the low-level features of the input image, and the specific process includes:

[0098] The low-level features of the input image are reduced to their spatial scale by the spatial reducer, and the feature size is C s ×H s ×W s The spatial reduction characteristics of s =1 / 4H i , W s =1 / 4W i , C s =256.

[0099] The spatial reduction features are batch normalized BatchNorm, and then input into a convolution with a convolution kernel size of 1×1 and a deep convolution DWConv with a convolution kernel size of 5×5 to obtain the features before the first channel split and the features before the second channel split; the feature size of the features before the first channel split is C s ×H s ×W s , used to perceive the image space information before channel splitting; the feature size of the second channel before splitting is C s ×H s ×Ws , used for subsequent channel splitting to obtain the spatial information relationship of spatial reduction features.

[0100] In order to complete channel separation, the spatial scale of the input second channel feature before splitting is maintained unchanged, and its space is split into three equal parts to achieve sequential channel fusion.

[0101] Specifically, after the channel separator obtains the features before the second channel splitting, it splits the channels of the features before the second channel splitting into 1 / 4, 1 / 4 and 1 / 2 of the original size, and obtains the first channel features, second channel features and third channel features with the same spatial scale and adjacent channels; the feature sizes of the first channel features, the second channel features and the third channel features are H s ×W s ×1 / 4C s , H s ×W s ×1 / 4C s and H s ×W s ×1 / 2C s ;

[0102] After the first channel feature and the second channel feature are subjected to the deep convolution DWConv with a convolution kernel size of 3×3 and the deep convolution DWConv with a convolution kernel size of 5×5 respectively, the first channel fuser is used to fuse the channel information of the first channel feature and the second channel feature to obtain the spatial relationship between the two and obtain the first channel fused feature;

[0103] After the first channel fusion feature and the third channel feature are respectively subjected to the deep convolution DWConv with a convolution kernel size of 5×5 and the deep convolution DWConv with a convolution kernel size of 7×7, the second channel fuser is used to fuse the channel information of the first channel fusion feature and the third channel feature to obtain the spatial relationship between the two and obtain the second channel fusion feature;

[0104] Compared with the ordinary convolution mentioned above, deep convolution DWConv performs a separate convolution operation on each input channel. In the process of channel fusion, the use of deep convolution can improve the channel perception ability during network training, which is conducive to channel fusion and improves the friendliness of the spatial feature interaction process.

[0105] The second channel fusion feature is convolved with a convolution kernel size of 1×1, and then element-wise multiplied and fused with the first channel feature before splitting to obtain the intermediate features of the input image.

[0106] S3, adding the intermediate features of the input image to the spatial reduction features, and flattening the spatial dimensions thereof to obtain the flattened features for operation in the global feature extraction module;

[0107] As a preferred technical solution of the present invention, after obtaining the intermediate features of the input image, the intermediate features of the input image are added to the spatial reduction features, and the spatial dimensions are flattened to obtain the flattened features for operation in the global feature extraction module. The dimension size of the flattened features is (H s ·W s )×C o , where C o = C s .

[0108] S4. Add position encoding to the flattened features to obtain three operational features of the same size, and process the operational features through the Transformer encoder to obtain high-level features of the input image. In a specific embodiment of the present application, after the global feature extraction module obtains the flattened features, it adds position encoding to the flattened features to obtain three operational features of the same size, namely the first operational feature K, the second operational feature Q and the third operational feature V. The three operational features of the same size are processed through the Transformer encoder to obtain high-level features of the input image.

[0109] In this application, the specific process of obtaining high-level features of the input image by calculating three operational signatures of the same size through the Transformer encoder includes:

[0110] Use multi-head attention to calculate the self-attention value for three operational signatures of the same size. The formula is:

[0111] ;

[0112] Among them, MSA is the self-attention value of multi-head attention, Represents the transposition of the first operation signature K; the size of H can be 8 or other specific number of heads.

[0113] Add the self-attention value MSA to the third operation feature symbol V to obtain the multi-head self-attention feature;

[0114] The position sensor is used to calculate the multi-head self-attention features to obtain the degree of perception of the multi-head self-attention features to the spatial position.

[0115] In fact, the above multi-head self-attention acquisition process is exactly the same as that of the naive Transformer. The original TransPose (TransPose: Towards Explainable Human Pose Estimation by Transformer) paper describes the self-attention acquisition process in detail. This article is exactly the same as it, so the multi-head acquisition process will not be described in detail.

[0116] The difference between the original naive Transformer and the position sensor is that the position sensor includes the first linear layer, the second linear layer and the activation function ReLU. After the position sensor obtains the multi-head self-attention features, it undergoes layer normalization and uses the first linear layer to expand its channels to twice the original ones to obtain the channel-expanded features.

[0117] The channel-expanded features are subjected to layer normalization, and after activation by the activation function ReLU, layer normalization is performed again, and the second linear layer is used to reduce the channel to half of the original, thus obtaining the channel-reduced features;

[0118] The channel-reduced features are layer-normalized and then added to the multi-head self-attention features to obtain position-aware features.

[0119] In this embodiment, the position sensor improves the position perception of spatial features of the present invention by directly transforming the linear layer and only changing the channel layer. At the same time, it does not need to perform nonlinear transformation, retains the invariance of spatial scale, and always maintains consistency with the multi-head self-attention features in dimension, which helps to reduce the difficulty of training.

[0120] After passing through N Transformer encoders, the global feature extraction module obtains the high-level features of the input image.

[0121] S5. Convert the spatial scale of the high-level features into two-dimensional spatial features, and use deconvolution to restore the scale of the two-dimensional spatial features to obtain the key point positions corresponding to the high-level features in the input image. After obtaining the high-level features of the image, use dimensionality conversion to convert them into two-dimensional spatial features, and then use deconvolution to expand the scale of the two-dimensional spatial features. Finally, the key point positions of the image input features are obtained, and the loss value is calculated with the true value to complete the training of the entire method. Figure 6 As shown, the key point positions in the image constitute the posture of the human body, and finally the estimated result of the human body posture in the image is output.

[0122] The following is an experimental comparison of the present invention and related methods on the COCO dataset, as shown in Table 1:

[0123] Table 1 Experimental results on COCO val

[0124]

[0125] Among them, HRNet-W32 is the HRNet paper, which has been introduced above;

[0126] The source of PPT-S is the paper "PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation", published in ECCV in 2022;

[0127] The source of SRPose is the paper "Lightweight Super-Resolution Head for Human PoseEstimation", which was preprinted on arXiv in 2023.

[0128] In the experiment, #Params represents the number of network parameters, Backbone represents the backbone network used, GFLOPs represents the amount of calculation, AP represents the average accuracy, and FPS represents the number of frames per second at which the network runs.

[0129] According to the experimental results, the method of the present invention has an accuracy of 74.3% and a speed of 175 on the COCO val dataset, which exceeds the relevant methods. This further confirms the effectiveness of the present invention and achieves better results in balancing accuracy and speed.

[0130] After acquiring the input image, this embodiment uses the proposed local feature extraction module, spatial feature extraction module, and global feature extraction module to process the input image in sequence. In each module, only a single scale of the input image is acquired, thereby realizing single-scale feature extraction of a single module, overcoming the defect of the multi-scale method in running speed, and sequentially acquiring primary, intermediate, and advanced features of the input image.

[0131] At the same time, in the local feature extraction module, the two major attentions of channel squeezing attention and space squeezing attention are used, and only convolution is used to complete the attention to channels and spaces, and deep information in the image is mined, which improves the overall accuracy of human posture estimation. After obtaining the primary features of the input image, the spatial feature extraction module obtains the interactive information of different spaces, and uses channel fusion to complete the extraction of intermediate features. The spatial scale of the input remains unchanged, and two channel fusions are performed in the structure, which effectively obtains the interactive information of the space and improves the overall running speed. After obtaining the intermediate features of the input image, the global feature extraction module uses the position sensor to improve the Transformer encoder's perception effect on the position, so that the robustness of the overall method is further improved, and there is no need to change the feature dimension. The high-level features of the input image are obtained, and the key point position information of the characters in the image is aligned using deconvolution to complete the network training.

[0132] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0133] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A multi-level efficient human posture estimation system, characterized in that: The human body posture estimation system only obtains the estimation result of the human body posture in the final output image of a single scale of the input image, and the system includes: Extract features from the input image in sequence, and generate a local feature extraction module, a spatial feature extraction module, and a global feature extraction module for low-level features, intermediate features, and high-level features, respectively; The low-level features are reduced in spatial scale by the spatial reducer to obtain a feature size of C s ×H s ×W s The spatial reduction features are used as inputs of the spatial feature extraction module, the intermediate features are added to the spatial reduction features, and their spatial dimensions are flattened to obtain flattened features for operation in the global feature extraction module; After the global feature extraction module outputs the high-level features, the spatial scale of the high-level features of the input image is converted into a two-dimensional spatial feature by using dimensional conversion, and the scale of the two-dimensional spatial feature is restored by using deconvolution, and the estimation result of the human body posture in the image is output; The local feature extraction module comprises performing channel squeezing and space squeezing on the image input features respectively, and obtaining channel squeezing attention and space squeezing attention of the channel squeezing features and the space squeezing features; The spatial feature extraction module includes a channel separator that splits the second channel into 1 / 4, 1 / 4 and 1 / 2 of the original size after obtaining the features before splitting the second channel, and obtains the first channel features, the second channel features and the third channel features with the same spatial scale and adjacent channels; And, fusing the channel information of the first channel feature and the second channel feature, acquiring the spatial relationship between the two, and obtaining a first channel fuser of the first channel fusion feature; And, fusing the channel information of the first channel fusion feature and the third channel feature, acquiring the spatial relationship between the two, and obtaining a second channel fuser of the second channel fusion feature; The global feature extraction module includes N Transformer encoders, each of which includes a multi-head attention for generating multi-head self-attention features, and a position sensor for calculating the multi-head self-attention features to obtain the degree of perception of the multi-head self-attention features to the spatial position; The position sensor includes a first linear layer that expands the channel of the multi-head self-attention feature by twice to obtain the channel-expanded feature, and a second linear layer that reduces the channel of the channel-expanded feature to half of its original value after being activated by the activation function ReLU.

2. A multi-stage efficient human posture estimation method applied to the multi-stage efficient human posture estimation system according to claim 1, characterized in that: The method includes: Input any input image into the local feature extraction module to obtain the low-level features of the input image; A spatial reducer is used to reduce the spatial scale of low-level features to generate spatial reduction features, which are then input into a spatial feature extraction module to obtain intermediate features of the input image. Adding the intermediate features of the input image to the spatial reduction features and flattening their spatial dimensions to obtain the flattened features for operation in the global feature extraction module; Add position encoding to the flattened features to obtain three operational signatures of the same size, and process the operational signatures through the Transformer encoder to obtain high-level features of the input image; The spatial scale of the high-level features is converted into two-dimensional spatial features, and the scale of the two-dimensional spatial features is restored by deconvolution to obtain the key point positions corresponding to the high-level features in the input image.

3. The multi-stage efficient human posture estimation method according to claim 2, characterized in that: The process of obtaining the low-level features of the input image is as follows: The input image with feature size of H×W×3 is convolved with a convolution kernel size of 7×7 to obtain a feature size of C i ×H i ×W i Image input features; Among them, H and W represent the height and width of the input image respectively. i =1 / 4H,W i =1 / 4W, C i =256; Channel squeezing attention and space squeezing attention perform channel squeezing and space squeezing on the image input features respectively to obtain channel squeezing features and space squeezing features; After the channel squeezing feature and the space squeezing feature are added, they are fused with the image input feature using a residual connection to obtain a low-level feature of the input image.

4. The multi-stage efficient human posture estimation method according to claim 3, characterized in that: After the channel squeezing attention performs channel squeezing on the image input feature, the process of obtaining the channel squeezing feature is as follows: The channel squeeze attention obtains the image input features, and uses two convolutions to compress the channels of the image input features respectively. i to 1 / 2 C i and 1, and keep the spatial size scale unchanged, and obtain the first channel compression feature and the second channel compression feature; The first channel compression feature is subjected to dimension conversion and convolution processing to obtain a first channel calculation feature for calculating channel extrusion; The second channel compression feature is subjected to dimension conversion and Softmax activation function to obtain the second channel calculation feature for calculating channel squeezing; The first channel calculation feature and the second channel calculation feature are fused by matrix dot product operation, and a channel squeeze fusion feature is obtained by convolution and Sigmoid activation function, wherein the spatial size of the channel squeeze fusion feature is 1×1; The channel squeeze fusion feature is fused with the image input feature by using an element-by-element multiplication operation to obtain a channel squeeze feature.

5. The multi-stage efficient human posture estimation method according to claim 3, characterized in that: After the spatial squeezing attention performs spatial squeezing on the image input feature, the process of obtaining the spatial squeezing feature is as follows: The spatial squeezing attention obtains the image input features, and uses two convolutions to compress the channels of the image input features respectively. i to 1 / 2 C i and 1 / 2 C i , maintaining the spatial size scale unchanged, obtaining the first spatial compression feature and the second spatial compression feature; After the first spatial compression feature is pooled and dimensionally converted, a Softmax activation function is used to obtain a first spatial calculation feature for calculating spatial squeezing; The second spatial compression feature is subjected to dimensional conversion and convolution to obtain a second spatial calculation feature for calculating spatial extrusion; The first spatial calculation feature and the second spatial calculation feature are fused by using a matrix dot product operation, and a spatial squeeze fusion feature is obtained by dimension conversion and a Sigmoid activation function, wherein the number of channels of the spatial squeeze fusion feature is 1; The spatial squeeze fusion feature is fused with the image input feature by using an element-by-element multiplication operation to obtain a spatial squeeze feature.

6. The multi-stage efficient human posture estimation method according to claim 2, characterized in that: The process of obtaining the intermediate features of the input image is as follows: The low-level features of the input image are reduced to their spatial scale by the spatial reducer, and the feature size is C s ×H s ×W s The space reduction feature of Among them, H s =1 / 4H i , W s =1 / 4W i , C s =256; The spatial reduction features are batch normalized BatchNorm, and then input into a convolution with a convolution kernel size of 1×1 and a deep convolution DWConv with a convolution kernel size of 5×5, respectively, to obtain feature sizes of C s ×H s ×W s , and used to perceive the first pre-channel splitting features of the image spatial information before channel splitting, and used for subsequent channel splitting to obtain the second pre-channel splitting features of the spatial information relationship of the spatial reduction features; After the channel separator obtains the features before the second channel splitting, it splits the channels of the features before the second channel splitting into 1 / 4, 1 / 4 and 1 / 2 of the original size, and obtains the first channel features, the second channel features and the third channel features with the same spatial scale and adjacent channels; After the first channel feature and the second channel feature are subjected to a deep convolution DWConv with a convolution kernel size of 3×3 and a deep convolution DWConv with a convolution kernel size of 5×5 respectively, the channel information of the first channel feature and the second channel feature are fused using a first channel fuser to obtain a spatial relationship between the two, thereby obtaining a first channel fused feature; After the first channel fusion feature and the third channel feature are respectively subjected to the deep convolution DWConv with a convolution kernel size of 5×5 and the deep convolution DWConv with a convolution kernel size of 7×7, the second channel fuser is used to fuse the channel information of the first channel fusion feature and the third channel feature, and the spatial relationship between the two is obtained to obtain the second channel fusion feature; The second channel fusion feature is convolved with a convolution kernel size of 1×1, and then element-wise multiplied and fused with the first channel feature before splitting to obtain the intermediate features of the input image.

7. The multi-stage efficient human posture estimation method according to claim 2, characterized in that: The specific process of obtaining the high-level features of the input image by calculating three operational signatures of the same size through the Transformer encoder includes: Use multi-head attention to calculate the self-attention value for three operational signatures of the same size. The formula is: ； Among them, MSA is the self-attention value of multi-head attention, represents the transposition of the first operation signature K; the value of H is 8; Add the self-attention value MSA to the third operation feature symbol V to obtain the multi-head self-attention feature; After the position sensor obtains the multi-head self-attention features, it is layer-normalized and the first linear layer is used to expand its channels to twice the original ones to obtain the channel-expanded features; The channel-expanded features are subjected to layer normalization, and after activation by the activation function ReLU, layer normalization is performed again, and the second linear layer is used to reduce the channel to half of the original, thus obtaining the channel-reduced features; The channel-reduced features are layer-normalized and then added to the multi-head self-attention features to obtain position-aware features. After passing through N Transformer encoders, the global feature extraction module outputs high-level features.

Images