2D human pose estimation method based on expanded mixed attention and related device

Abstract

The application discloses a 2D human body posture estimation method based on expanded mixed attention and related devices, and the method comprises the following steps: obtaining image data to be estimated, and inputting the image data to be estimated into a high-resolution backbone network to perform different scale feature extraction processing, and obtaining four feature images of different scales; inputting the four feature images of different scales into a cascaded expansion convolution module respectively to perform feature expression processing, and obtaining four high-resolution feature images; inputting the four high-resolution feature images into a mixed self-attention module to perform feature context information extraction processing, and obtaining context information corresponding to the four high-resolution feature images; inputting the context information corresponding to the four high-resolution feature images into a key point regression network to perform human body posture estimation processing on the image data to be estimated, and obtaining a human body posture estimation result. In the embodiment of the application, the accuracy of multi-person posture estimation is improved, and the calculation amount is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a 2D human pose estimation method based on dilated mixed attention and a related device. BACKGROUND

[0002] 2D multi-person human pose estimation (MHPE) needs to correctly estimate the joint key points of all human targets; from the data space division of human-related features, 2D multi-person human pose estimation is mainly divided into two ideas: Top-Down method and Bottom-Up method, both of which use key point heat maps for positioning; among them, the core of the Top-Down method is to regard multi-person pose estimation as a human detection and single-person pose estimation problem, and the calculation amount will quickly rise with the increase of the number of people in the picture, the advantage is that the cropped human region image is usually scaled to a uniform scale, which helps the network model to implicitly learn the position distribution of each joint, such as MSPN, Fast HPE, HRNet and other methods; the Top-Down method is limited by human detection frame cropping, ROI align and other operations, which will lose the context information outside the detection frame, and after the network detects the heat map, the argmax algorithm is also needed to locate the key points, and the processing speed is relatively slow.

[0003] And the core of the multi-person pose estimation method of the Bottom-Up method is the key point matching optimization and classification problem in the feature space, and the processing speed is very fast, such as DEKR, KAPAO, Open Pose and other methods, in 2021, KAPAO proposed to model a single key point and a set of spatially related key points (i.e. pose) as objects in a dense, single-stage and anchor-free detection framework, by simultaneously detecting human poses and key point objects, and fusing the detection results, faster and more accurate results are obtained; in 2022, Xiao et al. proposed the Adaptive Pose method, which uses a region perception module to regress seven adaptive human part-related points from the center of the human instance, and then predicts the center heat map by aggregating the features of the adaptive points, this new representation method can better capture various pose deformations, and adaptively decompose the long-distance center to joint displacement, making multi-person pose estimation more compact and efficient.

[0004] Overall, the Bottom-Up multi-person pose estimation method is faster in speed, but the accuracy is lower than the Top-Down method, because the Bottom-Up method is limited to directly understanding the original image, optimizing and recombining classification according to template matching, and when the picture contains complex scenes such as human body overlap, group similarity motion, etc., it faces the challenges of overlapping joint key point classification and depth blur.

[0005] HRNet obtains more feature information of different levels by parallel connection of high-resolution to low-resolution convolution, improves the field of view range of the whole network model, and through multi-scale repeated fusion in the network, the learned result is closer to the correct representation, and the predicted heat map is more accurate in space; based on the above research, the technical solution adopts HRNet (high-resolution backbone network) as the backbone network to extract human multi-scale features, and then combines the cascaded dilated convolution to dynamically expand the receptive field of the multi-scale features to obtain more small target feature information. SUMMARY

[0006] The present application aims to overcome the shortcomings of the prior art, and provides a 2D human pose estimation method based on dilated mixed attention and related devices, which improves the accuracy of multi-person pose estimation and reduces the computational complexity.

[0007] To solve the above technical problems, the present application embodiment provides a 2D human pose estimation method based on dilated mixed attention, which comprises:

[0008] Obtain the image data to be estimated, and input the image data to be estimated into a high-resolution backbone network for different scale feature extraction processing to obtain four different scale feature images corresponding to the image data to be estimated;

[0009] Respectively input the four different scale feature images corresponding to the image data to be estimated into a cascaded dilated convolution module for feature expression processing to obtain four high-resolution feature images, the cascaded dilated convolution module contains four branches, and each branch processes a scale feature image;

[0010] Input the four high-resolution feature images into a mixed self-attention module for feature context information extraction processing to obtain the context information corresponding to the four high-resolution feature images;

[0011] Input the context information corresponding to the four high-resolution feature images into a key point regression network to estimate the human pose in the image data to be estimated, and obtain the human pose estimation result.

[0012] Optionally, the different scale feature extraction processing of the high-resolution backbone network is performed on the to-be-estimated image data, and four different scale feature images corresponding to the to-be-estimated image data are obtained.

[0013] After the to-be-estimated image data is input into the high-resolution backbone network, the different scale feature extraction processing of the high-resolution backbone network is performed on the to-be-estimated image data in turn through three high-resolution steps, and four different scale feature images corresponding to the to-be-estimated image data are obtained.

[0014] The first high-resolution step of the three high-resolution steps processes the input to-be-estimated image data and generates two different scale feature images input into the second high-resolution step, and the second high-resolution step processes the generated three different scale feature images and inputs them into the third high-resolution step.

[0015] Optionally, the four different scale feature images corresponding to the to-be-estimated image data are input into the cascaded dilated convolution module respectively for feature expression processing, and four high-resolution feature images are obtained.

[0016] After the four different scale feature images corresponding to the to-be-estimated image data are input into the cascaded dilated convolution module respectively, the four different scale feature images input are processed by four branches of the dilated convolution module in the cascaded dilated convolution module, and four high-resolution feature images are obtained, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0017] Optionally, the structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel.

[0018] The second layer structure is that the second branch of the four branches except the first branch is connected with a 1*3 convolution kernel after a 1*1 expansion kernel, the third branch is connected with a 3*1 convolution kernel after a 1*1 expansion kernel, and the fourth branch is connected with a 3*3 convolution kernel after a 1*1 expansion kernel.

[0019] The third layer structure is that a 3*3 dilated convolution kernel is connected to the four branches, and the relationship between the actual size of the dilated convolution kernel and the size of the original convolution kernel is:

[0020] K=k+(k-1)(r-1);

[0021] Wherein, K represents the equivalent kernel size; k represents the original convolution kernel size; and r represents the expansion rate of the dilated convolution.

[0022] Optionally, the input of the expansion convolution of the other three branches except the first branch in the four branches of the expansion convolution module is based on the receptive field of the previous layer and is stacked layer by layer, and the fourth branch equivalently contains four receptive fields.

[0023] The output of each branch is associated based on Concat to perform multi-level expansion branch feature fusion to obtain four high-resolution feature images.

[0024] Optionally, the four high-resolution feature images are input into a hybrid self-attention module for feature context information extraction processing to obtain context information corresponding to the four high-resolution feature images, including:

[0025] After the four high-resolution feature images are input into the hybrid self-attention module, the four high-resolution feature images are normalized in the hybrid self-attention module to form normalized high-resolution feature images.

[0026] The normalized high-resolution feature images are input into convolution self-attention CSA and coordinate attention CA respectively for processing, and the processed output is fused with the four high-resolution feature images to form fused high-resolution feature images.

[0027] The fused high-resolution feature images are input into a multi-layer perception MLP for processing to output context information corresponding to the four high-resolution feature images.

[0028] Optionally, the normalized high-resolution feature images are input into the convolution self-attention CSA for processing, including:

[0029] Q, K and V components are arranged at the input end of the convolution self-attention CSA, and the Q, K and V components are generated by 1*1 convolution.

[0030] After the normalized high-resolution feature images are processed by the Q and K components, they are input into a SiLU activation function through a fully connected layer FC to obtain a first processed image feature.

[0031] The first processed image feature is input into a Tanh activation function through a fully connected layer FC for processing to obtain a second processed image feature.

[0032] The second processed image feature is multiplied with the output of the V component, and the multiplication result is connected in residual with the input normalized high-resolution feature image based on a fully connected layer FC to obtain the output of the convolution self-attention CSA.

[0033] In addition, the embodiment of the present application also provides a 2D human pose estimation device based on dilated mixed attention, which comprises:

[0034] A scale feature extraction module is configured to obtain image data to be estimated and input the image data to be estimated into a high-resolution backbone network for different scale feature extraction processing to obtain four feature images of different scales corresponding to the image data to be estimated.

[0035] A feature expression module is configured to input the four feature images of different scales corresponding to the image data to be estimated into a cascaded dilated convolution module for feature expression processing to obtain four high-resolution feature images, wherein the cascaded dilated convolution module comprises four branches, and each branch processes a feature image of one scale.

[0036] An information extraction module is configured to input the four high-resolution feature images into a mixed self-attention module for feature context information extraction processing to obtain context information corresponding to the four high-resolution feature images.

[0037] A pose estimation module is configured to input the context information corresponding to the four high-resolution feature images into a key point regression network to perform human pose estimation processing on the image data to be estimated to obtain a human pose estimation result.

[0038] In addition, the embodiment of the present application also provides an electronic device comprising a processor and a memory, wherein the processor runs a computer program or code stored in the memory to realize the 2D human pose estimation method according to any one of the above.

[0039] In addition, the embodiment of the present application also provides a computer readable storage medium for storing a computer program or code, wherein when the computer program or code is executed by a processor, the 2D human pose estimation method according to any one of the above is realized.

[0040] In the embodiment of the present application, by sequentially arranging the HRNet backbone network, the cascaded dilated convolution module, the mixed attention module and the key point regression four parts; after obtaining the image data to be estimated, first, the HRNet backbone network is used to obtain four feature maps of different scales, then the cascaded dilated convolution module is used to expand the receptive field of the small human body posture, and then the mixed attention module is used to fuse the context information into the network, so that the network learns the correlation of local features and increases the representation ability of the network to global features; finally, the key point regression network is used to obtain the key point matching result of the multi-person posture; the cascaded dilated convolution module is used to obtain the local features of different key points of the small human body posture, and then the mixed self-attention is used to strengthen the correlation of context information, so as to improve the detection performance of the network for multi-person posture estimation and the accuracy of human body posture estimation, and reduce the calculation amount; at the same time, the detection accuracy of small targets is improved. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 is a flowchart of the 2D human body posture estimation method based on dilated mixed attention in the embodiment of the present application;

[0043] Figure 2 is a flowchart of the 2D human body posture estimation method based on dilated mixed attention in another embodiment of the present application;

[0044] Figure 3 is a structural composition schematic diagram of the 2D human body posture estimation device based on dilated mixed attention in the embodiment of the present application;

[0045] Figure 4 is a structural composition schematic diagram of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] Embodiment one, please refer to Figure 1 , Figure 1is a flowchart of a 2D human pose estimation method based on an expanded mixed attention in an embodiment of the present application.

[0048] As shown in Figure 1 A 2D human pose estimation method based on an expanded mixed attention, the method comprises:

[0049] S101: Obtain image data to be estimated, and input the image data to be estimated into a high-resolution backbone network for different scale feature extraction processing to obtain four different scale feature images corresponding to the image data to be estimated.

[0050] In the specific implementation process of the present application, the input of the image data to be estimated into the high-resolution backbone network for different scale feature extraction processing to obtain four different scale feature images corresponding to the image data to be estimated comprises: after inputting the image data to be estimated into the high-resolution backbone network, based on the high-resolution backbone network, sequentially passing through three high-resolution steps for different scale feature extraction processing, four different scale feature images corresponding to the image data to be estimated are obtained; the first high-resolution step in the three high-resolution steps processes the input image data to be estimated and generates two different scale feature images input into the second high-resolution step, and the second high-resolution step inputs the three different scale feature images generated by processing into the third high-resolution step.

[0051] Specifically, after obtaining the image data to be estimated, the image data to be estimated needs to be input into the high-resolution backbone network HRNet, three high-resolution steps (HRstage) are sequentially arranged in the HRNet, and different scale feature extraction processing is realized by sequentially passing through the three HRstage, thereby obtaining four different scale feature images corresponding to the image data to be estimated; wherein the first HRstage in the three HRstage processes the input image data to be estimated, outputs two different scale feature images, then inputs the two different scale feature images into the second HRstage, and outputs three different scale feature images, then inputs the three different scale feature images into the third HRstage, and finally outputs four different scale feature images corresponding to the image data to be estimated.

[0052] S102: Input the four different scale feature images corresponding to the image data to be estimated into a cascaded expanded convolution module for feature expression processing to obtain four high-resolution feature images, the cascaded expanded convolution module contains four branches, and each branch processes a scale feature image;

[0053] In the implementation of the present application, the four different scale feature images corresponding to the image data to be estimated are respectively input into the cascaded dilated convolution module for feature expression processing to obtain four high-resolution feature images, including: after the four different scale feature images corresponding to the image data to be estimated are respectively input into the cascaded dilated convolution module, the four different scale feature images input are processed based on the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0054] Further, the structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel; the second layer structure is that the second branch except the first branch in the four branches is connected with a 1*3 convolution kernel after a 1*1 expansion kernel, the third branch is connected with a 3*1 convolution kernel after a 1*1 expansion kernel, and the fourth branch is connected with a 3*3 convolution kernel after a 1*1 expansion kernel; the third layer structure is that a 3*3 dilated convolution kernel is connected to the four branches, and the relationship between the actual size of the dilated convolution kernel and the size of the original convolution kernel is:

[0055] K=k+(k-1)(r-1);

[0056] Wherein, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0057] Further, the input of the dilated convolution of the three branches except the first branch in the four branches of the dilated convolution module is based on the receptive field of the previous layer and is stacked layer by layer, and the fourth branch equivalently contains four receptive fields; the output of each branch is associated based on Concat to perform multi-level dilated branch feature fusion to obtain four high-resolution feature images.

[0058] Specifically, in the application, the more network convolution kernels, the more the channel number and the richer the feature expression, but the network calculation time and cost will also increase; in order to balance the calculation time and cost and the balance between feature rich expression, the cascaded dilated convolution module is adopted in the embodiment; that is, after the four different scale feature images corresponding to the image data to be estimated are respectively input into the cascaded dilated convolution module, the four different scale feature images input are processed by the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0059] In this embodiment, the cascaded dilated convolution module DCM is used to improve the perception of small targets and reduce the amount of calculation. For each scale feature map output by the HRnet, four branches in the DCM are used for processing. This is because the dilated convolution has inherent grid effect which easily leads to local information inconsistency. Therefore, four dilated convolution layers with different expansion rates (R = 1, 3, 3, 5) are used to alleviate this grid effect.

[0060] The structure of the four branches of the dilated convolution module is as follows: the first layer structure of each branch is a 1*1 convolution kernel; the second layer structure is that the second branch of the four branches except the first branch is connected with a 1*3 convolution kernel after a 1*1 expansion kernel, the third branch is connected with a 3*1 convolution kernel after a 1*1 expansion kernel, and the fourth branch is connected with a 3*3 convolution kernel after a 1*1 expansion kernel; the third layer structure is that a 3*3 dilated convolution kernel is connected to the four branches; the relationship between the actual size of the dilated convolution kernel and the size of the original convolution kernel is as follows:

[0061] K = k + (k-1)(r-1);

[0062] wherein, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0063] The input of the dilated convolution of the other three branches except the first branch is based on the previous layer, and the receptive field is gradually stacked, so the fourth branch contains four receptive fields. Finally, each branch is concatenated to realize the fusion of multi-level dilated branch features. Through this expansion from small to large area, information can be captured from a larger context area, different local rich representations can be extracted, the receptive field of small targets can be improved, and higher resolution feature maps can be generated.

[0064] In order to improve the efficiency, the number of channels in the block is gradually reduced to maintain a low calculation cost.

[0065] S103: input the four high-resolution feature maps into the mixed self-attention module for feature context information extraction processing to obtain the context information corresponding to the four high-resolution feature maps;

[0066] In the implementation of the present application, the input of the four high-resolution feature images into the mixed self-attention module for feature context information extraction processing to obtain the context information corresponding to the four high-resolution feature images includes: after inputting the four high-resolution feature images into the mixed self-attention module, normalizing the four high-resolution feature images in the mixed self-attention module to form normalized high-resolution feature images; inputting the normalized high-resolution feature images into convolution self-attention CSA and coordinate attention CA respectively for processing, and fusing the output results after processing with the four high-resolution feature images to form fused high-resolution feature images; inputting the fused high-resolution feature images into a multi-layer perception MLP for processing to output the context information corresponding to the four high-resolution feature images.

[0067] Further, the input of the normalized high-resolution feature images into the convolution self-attention CSA for processing includes: setting Q, K and V three components at the input end of the convolution self-attention CSA, and generating Q, K and V three components through 1*1 convolution; after the normalized high-resolution feature images are processed through Q and K components, inputting them into a SiLU activation function through a fully connected layer FC to obtain the first image feature after processing; inputting the first image feature after processing into a Tanh activation function through a fully connected layer FC for processing to obtain the second image feature after processing; multiplying the second image feature after processing with the output result of the V component, and performing residual connection processing on the multiplication result based on a fully connected layer FC and the input normalized high-resolution feature images to obtain the output result after processing of the convolution self-attention CSA.

[0068] Specifically, in order to perceive the context information of the feature map extracted in the last step, enhance the representation of small-scale human local features, and solve the problem of serious loss of pose position information, the feature map X is normalized in the layer (LN) as X1, fused through convolution self-attention (CSA) and coordinate attention (CA) two blocks as X2, and Y is the processing result output through the MLP layer, which can be represented by the following formula:

[0069] X1 = LN(X);

[0070] X2 = CA(X1) + CSA(X1) + X;

[0071] Y = MLP(LN(X2)) + X2;

[0072] And in the convolution self-attention CSA, in order to be able to obtain local and global features at the same time and improve the fusion ability of the context, Q, K and V three components are arranged at the input end of the convolution self-attention CSA, and Q, K and V three components are generated through 1*1 convolution; X input The input feature is represented as follows:

[0073] Q, K, V = Conv(X input );

[0074] After the multiplication of Q and K matrices, a series of nonlinear processing is carried out, and then the full connection layer (FC) is input to the SiLU activation function to increase the linear expression ability, and X A is obtained.

[0075]

[0076] X A is input to the full connection layer FC and the Tanh activation function again to obtain the weight information, and X B is obtained, as follows:

[0077]

[0078] X B is multiplied by V again, and then the full connection layer FC is connected with Xinput to obtain the final output result Xoutput; as follows:

[0079]

[0080] After the full connection layer, the Tanh activation function is used for processing, and the output result is a matrix multiplied by the linearly transformed V component; compared with the original MSA, the CSA can make the attention network have stronger context perception ability after the convolution to obtain more local features, and further enhance the use of the network to the feature utilization range; in addition, more nonlinear transformations make the network have stronger nonlinear expression ability.

[0081] The coordinate attention (CA) module is adopted to enhance the position information fusion ability of the network. The CA module can effectively enhance the association between different channels and improve the perception ability of the network to remote position information; the ablation experiment results show that the added coordinate attention can improve the detection accuracy of small targets.

[0082] S104: input the context information corresponding to the four high-resolution feature images into the key point regression network to estimate the human pose in the image data to be estimated, and obtain the human pose estimation result.

[0083] In the implementation of the present application, after the context information corresponding to the four high-resolution feature images is obtained, the context information corresponding to the four high-resolution feature images needs to be input into the key point regression network to estimate the human pose in the image data to be estimated; that is, the key point matching result of the multi-person pose is obtained through the key point regression network, and the human pose estimation result is output according to the key point matching result.

[0084] In the embodiment of the present application, the HRNet backbone network, the cascaded dilated convolution module, the hybrid attention module and the key point regression are sequentially arranged; after the image data to be estimated is obtained, the HRNet backbone network is used to obtain four feature maps of different scales, then the cascaded dilated convolution module is used to expand the receptive field of the human pose of different scales, and then the hybrid attention module is used to fuse the context information into the network, so that the network learns the correlation of local features and increases the representation ability of the network to global features; finally, the key point matching result of the multi-person pose is obtained through the key point regression network; the cascaded dilated convolution module is used to obtain the local features of different key points of the small human pose, and then the hybrid self-attention is used to strengthen the correlation of the context information, so as to improve the detection performance of the network for multi-person pose estimation and the accuracy of human pose estimation, and reduce the amount of calculation; at the same time, the detection accuracy of small targets is improved.

[0085] Embodiment two, please refer to Figure 2 , Figure 2 is a flowchart of a 2D human pose estimation method based on dilated hybrid attention in another embodiment of the present application.

[0086] As Figure 2 shown, a 2D human pose estimation method based on dilated hybrid attention, the method comprises:

[0087] S201: obtaining image data to be estimated, and inputting the image data to be estimated into a high-resolution backbone network for different scale feature extraction processing to obtain four feature images of different scales corresponding to the image data to be estimated;

[0088] S202: after the four feature images of different scales corresponding to the image data to be estimated are input into the cascaded dilated convolution module, the four feature images of different scales input are processed based on the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module;

[0089] S203: after the four high-resolution feature images are input into the hybrid self-attention module, the four high-resolution feature images are normalized in the hybrid self-attention module to form normalized high-resolution feature images;

[0090] S204: input the normalized high-resolution feature images into convolution self-attention CSA and coordinate attention CA respectively for processing, and fuse the output results after processing with the four high-resolution feature images to form fused high-resolution feature images;

[0091] S205: input the fused high-resolution feature images into a multi-layer perception MLP for processing, and output context information corresponding to the four high-resolution feature images;

[0092] S206: input the context information corresponding to the four high-resolution feature images into a key point regression network to estimate the human pose in the image data to be estimated, and obtain a human pose estimation result.

[0093] The specific implementation of the second embodiment can refer to the above-mentioned embodiments, which will not be repeated here.

[0094] Embodiment three, please refer to Figure 3 , Figure 3 is a structural composition schematic diagram of a 2D human pose estimation device based on expanded mixed attention in the embodiment of the application.

[0095] As shown in Figure 3 , a 2D human pose estimation device based on expanded mixed attention, the device comprises:

[0096] The scale feature extraction module 301 is used to obtain image data to be estimated, and input the image data to be estimated into a high-resolution backbone network for different scale feature extraction processing to obtain four feature images of different scales corresponding to the image data to be estimated.

[0097] In the specific implementation process of the application, the input of the image data to be estimated into the high-resolution backbone network for different scale feature extraction processing to obtain four feature images of different scales corresponding to the image data to be estimated comprises: after inputting the image data to be estimated into the high-resolution backbone network, based on the high-resolution backbone network, sequentially passing through three high-resolution steps for different scale feature extraction processing to obtain four feature images of different scales corresponding to the image data to be estimated; the first high-resolution step in the three high-resolution steps processes the input image data to be estimated and generates two feature images of different scales input into the second high-resolution step, and the second high-resolution step inputs three feature images of different scales generated by processing into the third high-resolution step.

[0098] Specifically, after obtaining the to-be-estimated image data, the to-be-estimated image data needs to be input into a high-resolution backbone network HRNet, three high-resolution stages (HRstage) are sequentially arranged in the HRNet, and different scale feature extraction processing is sequentially implemented through the three HRstage, so that four feature images of different scales corresponding to the to-be-estimated image data are obtained; wherein the first HRstage of the three HRstage processes the input to-be-estimated image data, outputs two feature images of different scales, then inputs the two feature images of different scales into the second HRstage, and outputs three feature images of different scales, then inputs the three feature images of different scales into the third HRstage, and finally outputs four feature images of different scales corresponding to the to-be-estimated image data.

[0099] The feature expression module 302 is configured to input the four feature images of different scales corresponding to the to-be-estimated image data into a cascaded dilated convolution module respectively for feature expression processing, so as to obtain four high-resolution feature images, wherein the cascaded dilated convolution module comprises four branches, and each branch processes a feature image of one scale.

[0100] In the specific implementation of the present application, the input of the four feature images of different scales corresponding to the to-be-estimated image data into the cascaded dilated convolution module for feature expression processing to obtain four high-resolution feature images comprises: after the input of the four feature images of different scales corresponding to the to-be-estimated image data into the cascaded dilated convolution module, the four feature images of different scales are processed by four branches of a dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0101] Further, the structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel; the second layer structure is that the second branch of the four branches except the first branch is connected with a 1*3 convolution kernel after a 1*1 dilated kernel, the third branch is connected with a 3*1 convolution kernel after a 1*1 dilated kernel, and the fourth branch is connected with a 3*3 convolution kernel after a 1*1 dilated kernel; the third layer structure is that a 3*3 dilated kernel is connected to the four branches, and the relationship between the actual size of the dilated kernel and the size of the original convolution kernel is:

[0102] K=k+(k-1)(r-1);

[0103] Wherein, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0104] Further, the input of the dilated convolution of the three branches other than the first branch in the four branches of the dilated convolution module is based on the receptive field of the previous layer and is stacked layer by layer, and the fourth branch equivalently contains four receptive fields; the output of each branch is associated based on Concat to perform multi-level dilated branch feature fusion to obtain four high-resolution feature images.

[0105] Specifically, in the implemented application, the more network convolution kernels, the more the channel number and the richer the feature expression, but the network calculation time and cost will also increase; in order to balance the calculation time and cost and the balance between feature rich expression, the embodiment adopts a cascaded dilated convolution module; that is, after the four different scale feature images corresponding to the to-be-estimated image data are input into the cascaded dilated convolution module, the four different scale feature images input are processed by the four branches of the dilated convolution module in the cascaded dilated convolution module, thereby obtaining four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0106] In the embodiment, the cascaded dilated convolution module DCM is used to improve the perception of small targets and reduce the amount of calculation; for each scale feature map output by the HRnet, the four branches in the DCM are used for processing; this is because the dilated convolution has inherent grid effect which is easy to cause local information inconsistency, and four dilated convolution layers with different expansion rates (R=1, 3, 3, 5) are used to alleviate this grid effect.

[0107] The structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel; the second layer structure is that the second branches other than the first branch in the four branches are connected with a 1*3 convolution kernel after a 1*1 dilated kernel, the third branch is connected with a 3*1 convolution kernel after a 1*1 dilated kernel, and the fourth branch is connected with a 3*3 convolution kernel after a 1*1 dilated kernel; the third layer structure is that a 3*3 dilated kernel is connected to the four branches; the relationship between the actual size of the dilated kernel and the size of the original convolution kernel is:

[0108] K=k+(k-1)(r-1);

[0109] Wherein, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0110] In addition to the first branch, the input of the expansion convolution of the other three branches is based on the previous layer, and the domain is gradually stacked, so the fourth branch equivalently contains four receptive fields, and finally, each branch is associated by Concat to realize multi-level expansion branch feature fusion; Through this region from small to large expansion, information can be captured from a larger context area, different local rich representations can be extracted, the receptive field of small targets can be improved, and higher resolution feature maps can be generated.

[0111] In order to improve efficiency, the number of channels in this block gradually decreases to maintain a low calculation cost.

[0112] The information extraction module 303 is configured to input the four high-resolution feature images into a hybrid self-attention module for feature context information extraction processing, and obtain context information corresponding to the four high-resolution feature images.

[0113] In the specific implementation process of the present application, the input of the four high-resolution feature images into the hybrid self-attention module for feature context information extraction processing, and the obtaining of the context information corresponding to the four high-resolution feature images, comprises: after inputting the four high-resolution feature images into the hybrid self-attention module, normalizing the four high-resolution feature images in the hybrid self-attention module to form normalized high-resolution feature images; inputting the normalized high-resolution feature images into convolution self-attention CSA and coordinate attention CA respectively for processing, and fusing the output results after processing with the four high-resolution feature images to form fused high-resolution feature images; inputting the fused high-resolution feature images into a multi-layer perception MLP for processing, and outputting the context information corresponding to the four high-resolution feature images.

[0114] Further, the input of the normalized high-resolution feature image into the convolution self-attention CSA for processing comprises: setting Q, K and V three components at the input end of the convolution self-attention CSA, and generating Q, K and V three components through 1*1 convolution; after the normalized high-resolution feature image is processed by the Q and K components, it is input into the SiLU activation function through the full connection layer FC to obtain the first image feature after processing; the first image feature after processing is input into the Tanh activation function through the full connection layer FC for processing to obtain the second image feature after processing; the second image feature after processing is multiplied with the output result of the V component, and the multiplication result is residual connected with the input normalized high-resolution feature image based on the full connection layer FC to obtain the output result after processing of the convolution self-attention CSA.

[0115] Specifically, in order to perceive the context information of the feature map extracted in the last step, enhance the representation of small-scale human local features, and solve the problem of serious loss of pose position information, the feature map X is normalized as X1 after layer normalization (LN), and then fused through two blocks of convolution self-attention (CSA) and coordinate attention (CA) to obtain X2, Y is the processing result output through the MLP layer, and can be expressed as the following formula:

[0116] X1 = LN(X);

[0117] X2 = CA(X1) + CSA(X1) + X;

[0118] Y = MLP(LN(X2)) + X2;

[0119] In the convolution self-attention CSA, in order to be able to obtain local and global features at the same time and improve the context fusion capability, three components Q, K and V are set at the input end of the convolution self-attention CSA, and Q, K and V are generated through 1*1 convolution; X input represents the input feature, as follows:

[0120] Q, K, V = Conv(X input );

[0121] After Q and K matrices are multiplied, a series of nonlinear processing is performed, and then input to the SiLU activation function through the fully connected layer (FC), the linear expression ability is increased, and X A is obtained.

[0122]

[0123] X A is input to the fully connected layer FC and the Tanh activation function again to obtain the weight information, and X B is obtained, as follows:

[0124]

[0125] X B is multiplied by V again, and then connected in residual connection with Xinput through the fully connected layer FC to obtain the final output result Xoutput; as follows:

[0126]

[0127] After the full connection layer, a Tanh activation function is used for processing, and the output result is a matrix multiplied by a linear transformation V component; compared with the original MSA, after the CSA obtains more local features through convolution, global calculation is performed again, so that the attention network has stronger context awareness ability, and the use range of the network for feature utilization is further enhanced; in addition, more nonlinear transformations are used, which makes the network have stronger nonlinear expression ability.

[0128] A coordinate attention (CA) module is adopted to enhance the position information fusion capability of the network. The CA module can effectively enhance the association between different channels and improve the perception ability of the network for remote position information; the ablation experiment results show that the added coordinate attention can improve the detection accuracy of small targets.

[0129] The pose estimation module 304 is configured to input the context information corresponding to the four high-resolution feature images into a key point regression network to estimate the human pose in the image data to be estimated, and obtain a human pose estimation result.

[0130] In the implementation of the present application, after obtaining the context information corresponding to the four high-resolution feature images, the context information corresponding to the four high-resolution feature images needs to be input into a key point regression network to estimate the human pose in the image data to be estimated; that is, the key point matching result of the multi-person pose is obtained through the key point regression network, and the human pose estimation result is output according to the key point matching result.

[0131] In the embodiment of the present application, the HRNet backbone network, the cascaded dilated convolution module, the hybrid attention module and the key point regression are sequentially arranged; after obtaining the image data to be estimated, the HRNet backbone network is used to obtain four feature maps of different scales, then the cascaded dilated convolution module is used to expand the receptive field of the human pose of different scales, and then the hybrid attention module is used to fuse the context information into the network, so that the network learns the correlation of local features and increases the representation ability of the network for global features; finally, the key point regression network is used to obtain the key point matching result of the multi-person pose; the cascaded dilated convolution module is used to obtain the local features of different key points of the small human pose, and then the hybrid self-attention is used to strengthen the correlation of the context information, so as to improve the detection performance of the network for multi-person pose estimation and the accuracy of human pose estimation, and reduce the calculation amount; at the same time, the detection accuracy of small targets is improved.

[0132] This invention provides a computer-readable storage medium storing a computer program. When executed by a processor, this program implements the 2D human pose estimation method of any of the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium that can store or transmit information in a readable form by a device (e.g., a computer, a mobile phone), and can be a read-only memory, a disk, or an optical disk, etc.

[0133] This invention also provides a computer application that runs on a computer and is used to execute the 2D human pose estimation method of any of the above embodiments.

[0134] also, Figure 4 This is a schematic diagram of the structural composition of the electronic device in an embodiment of the present invention.

[0135] This invention also provides an electronic device, such as... Figure 4 As shown. The electronic device includes components such as a processor 402, a memory 403, an input unit 404, and a display unit 405. Those skilled in the art will understand that... Figure 4 The structural components of the illustrated electronic device do not constitute a limitation on all devices and may include more or fewer components than illustrated, or combine certain components. Memory 403 can be used to store application program 401 and various functional modules. Processor 402 runs application program 401 stored in memory 403, thereby performing various functional applications and data processing of the device. Memory can be internal memory or external memory, or both. Internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. External memory may include hard disks, floppy disks, ZIP disks, USB flash drives, magnetic tapes, etc. The memory disclosed in this invention includes, but is not limited to, these types of memory. The memory disclosed in this invention is only an example and not a limitation.

[0136] The input unit 404 is configured to receive input of signals and receive a keyword input by a user. The input unit 404 can include a touch panel and other input devices. The touch panel can collect a touch operation (such as an operation of a user on or near the touch panel using a finger, a stylus, or any suitable object or accessory) of the user and drive a corresponding connection device according to a preset program; the other input devices can include, but are not limited to, one or more of a physical keyboard, function keys (such as play control buttons, switch buttons, etc.), a trackball, a mouse, a joystick, etc. The display unit 405 can be configured to display information input by the user or information provided to the user and various menus of the terminal device. The display unit 405 can take the form of a liquid crystal display, an organic light-emitting diode, etc. The processor 402 is a control center of the terminal device, connects all parts of the entire device through various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 403 and calling data stored in the memory.

[0137] As an embodiment, the electronic device includes: one or more processors 402, a memory 403, and one or more application programs 401, wherein the one or more application programs 401 are stored in the memory 403 and configured to be executed by the one or more processors 402, and the one or more application programs 401 are configured to perform the corresponding 2D human pose estimation method in any one of the above embodiments.

[0138] In the embodiment of the present application, by sequentially arranging the HRNet backbone network, the cascaded dilated convolution module, the hybrid attention module and the key point regression four parts; after obtaining the image data to be estimated, first, the HRNet backbone network is used to obtain four feature maps of different scales, then the cascaded dilated convolution module is used to expand the receptive field of the human pose of different scales, and then the hybrid attention module is used to fuse the context information into the network, so that the network learns the correlation of local features and increases the representation ability of the network to global features; finally, the key point regression network is used to obtain the key point matching result of the multi-person pose; the cascaded dilated convolution module is used to obtain the local features of different key points of small human poses, and then the hybrid self-attention is used to strengthen the correlation of context information, thereby improving the detection performance of the network for multi-person pose estimation and the accuracy of human pose estimation, reducing the amount of calculation; and improving the detection accuracy of small targets.

[0139] In addition, the above describes in detail the 2D human pose estimation method and related device based on the expanded mixed attention provided by the embodiment of the present application. The principle and implementation manner of the present application are described by using specific examples in this paper. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A 2D human pose estimation method based on extended hybrid attention, characterized in that, The method includes: The image data to be estimated is obtained and input into a high-resolution backbone network for feature extraction at different scales to obtain four feature images at different scales corresponding to the image data to be estimated. The four feature images of different scales corresponding to the image data to be estimated are respectively input into the cascaded dilated convolution module for feature representation processing to obtain four high-resolution feature images. The cascaded dilated convolution module contains four branches, and each branch corresponds to processing a feature image of one scale. The four high-resolution feature images are input into the hybrid self-attention module for feature context information extraction processing to obtain the context information corresponding to the four high-resolution feature images; The context information corresponding to the four high-resolution feature images is input into the keypoint regression network to estimate the human pose in the image data to be estimated, and the human pose estimation result is obtained. The step involves inputting the image data to be estimated into a high-resolution backbone network for feature extraction at different scales to obtain four feature images at different scales corresponding to the image data to be estimated, including: After the image data to be estimated is input into the high-resolution backbone network, feature extraction processing at different scales is performed sequentially through three high-resolution steps in the high-resolution backbone network to obtain four feature images at different scales corresponding to the image data to be estimated. The first high-resolution step in the three high-resolution steps processes the input image data to be estimated and generates two feature images at different scales, which are then input into the second high-resolution step. The second high-resolution step inputs the three feature images at different scales generated in the second high-resolution step into the third high-resolution step. The four feature images at different scales corresponding to the image data to be estimated are then input into a cascaded dilated convolution module for feature representation processing to obtain four high-resolution feature images, including: After inputting the four feature images of different scales corresponding to the image data to be estimated into the cascaded dilated convolution module, the four feature images of different scales are processed based on the four branches of the dilated convolution module to obtain four high-resolution feature images. The dilated convolution module uses four dilated convolution layers with different dilation rates. The structure of the four branches of the dilated convolution module is that the first layer of each branch is a 1*1 convolution kernel. The second layer structure consists of four branches, with the second branch (excluding the first branch) connected to a 1*3 convolution kernel after a 1*1 expanded kernel, the third branch connected to a 3*1 convolution kernel after a 1*1 expanded kernel, and the fourth branch connected to a 3*3 convolution kernel after a 1*1 expanded kernel. The third layer structure consists of 3x3 dilated convolutional kernels connected to all four branches, and the relationship between the actual size of the dilated convolutional kernel and the size of the original convolutional kernel is as follows: ； in, Indicates the equivalent kernel size; Indicates the original kernel size; This represents the dilation rate of the dilated convolution.

2. The 2D human pose estimation method according to claim 1, characterized in that, Except for the first branch, the inputs of the dilated convolutions in the other three branches of the dilated convolution module are based on the receptive fields of the previous layer, and the fourth branch effectively contains four receptive fields. The output of each branch is fused into multi-level expanded branch features based on Concat association to obtain four high-resolution feature images.

3. The 2D human pose estimation method according to claim 1, characterized in that, The step of inputting the four high-resolution feature images into the hybrid self-attention module for feature context information extraction processing to obtain the context information corresponding to the four high-resolution feature images includes: After the four high-resolution feature images are input into the hybrid self-attention module, the four high-resolution feature images are normalized in the hybrid self-attention module to form normalized high-resolution feature images. The normalized high-resolution feature images are input into convolutional self-attention (CSA) and coordinate attention (CA) for processing, and the output of the processing is fused with the four high-resolution feature images to form a fused high-resolution feature image. The fused high-resolution feature images are input into a multilayer perceptron (MLP) for processing, and the context information corresponding to the four high-resolution feature images is output.

4. The 2D human pose estimation method according to claim 3, characterized in that, The step of inputting the normalized high-resolution feature image into a convolutional self-attention (CSA) system for processing includes: Three components, Q, K, and V, are set at the input of the convolutional self-attention (CSA) and are generated by a 1x1 convolution. After normalization, the high-resolution feature image is processed by Q and K components, and then input into the SiLU activation function through a fully connected layer (FC) to obtain the processed first image features. The processed first image features are passed through a fully connected layer (FC) and then input into the Tanh activation function for further processing to obtain the processed second image features. The processed second image features are multiplied with the output of the V component, and the multiplication result is then processed by residual connection between the fully connected layer FC and the input normalized high-resolution feature image to obtain the output of the convolutional self-attention CSA.

5. A 2D human pose estimation device based on extended hybrid attention, characterized in that, The device includes: Scale feature extraction module: used to obtain the image data to be estimated, and input the image data to be estimated into the high-resolution backbone network for feature extraction processing at different scales, to obtain four feature images at different scales corresponding to the image data to be estimated; Feature representation module: used to input the four feature images of different scales corresponding to the image data to be estimated into the cascaded dilated convolution module for feature representation processing to obtain four high-resolution feature images. The cascaded dilated convolution module contains four branches, and each branch processes a feature image of one scale. Information extraction module: used to input the four high-resolution feature images into the hybrid self-attention module for feature context information extraction processing, and obtain the context information corresponding to the four high-resolution feature images; Pose estimation module: used to input the context information corresponding to the four high-resolution feature images into the keypoint regression network to estimate the human pose in the image data to be estimated, and obtain the human pose estimation result; The step involves inputting the image data to be estimated into a high-resolution backbone network for feature extraction at different scales to obtain four feature images at different scales corresponding to the image data to be estimated, including: After the image data to be estimated is input into the high-resolution backbone network, feature extraction processing at different scales is performed sequentially through three high-resolution steps in the high-resolution backbone network to obtain four feature images at different scales corresponding to the image data to be estimated. The first high-resolution step of the three high-resolution steps processes the input image data to be estimated and generates two feature images at different scales, which are then input into the second high-resolution step. The second high-resolution step then inputs the three feature images at different scales generated in the process into the third high-resolution step. The process involves inputting four feature images of different scales corresponding to the image data to be estimated into a cascaded dilated convolution module for feature representation processing to obtain four high-resolution feature images, including: After inputting the four feature images of different scales corresponding to the image data to be estimated into the cascaded dilated convolution module, the four feature images of different scales are processed based on the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images. The dilated convolution module uses four dilated convolution layers with different dilation rates. The structure of the four branches of the dilated convolution module is that the first layer of each branch is a 1*1 convolution kernel; The second layer structure consists of four branches, with the second branch (excluding the first branch) connected to a 1*3 convolution kernel after a 1*1 expanded kernel, the third branch connected to a 3*1 convolution kernel after a 1*1 expanded kernel, and the fourth branch connected to a 3*3 convolution kernel after a 1*1 expanded kernel. The third layer structure consists of 3x3 dilated convolutional kernels connected to all four branches, and the relationship between the actual size of the dilated convolutional kernel and the size of the original convolutional kernel is as follows: ； in, Indicates the equivalent kernel size; Indicates the original kernel size; This represents the dilation rate of the dilated convolution.

6. An electronic device comprising a processor and a memory, characterized in that, The processor runs a computer program or code stored in the memory to implement the 2D human pose estimation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium for storing computer programs or code, characterized in that, When the computer program or code is executed by a processor, the 2D human pose estimation method as described in any one of claims 1 to 4 is implemented.

Images