2D human body posture estimation method based on dilatation mixed attention and related device

Abstract

The invention discloses a 2D human body posture estimation method and related device based on expanded mixed attention, and the method comprises the steps: obtaining to-be-estimated image data, inputting the to-be-estimated image data into a high-resolution backbone network, and carrying out the extraction of features of different scales, and obtaining four feature images of different scales; respectively inputting the four feature images with different scales into a cascade expansion convolution module for feature expression processing to obtain four high-resolution feature images; the four high-resolution feature images are input into a hybrid self-attention module for feature context information extraction processing, and context information corresponding to the four high-resolution feature images is obtained; and inputting context information corresponding to the four high-resolution feature images into the key point regression network to perform estimation processing on a human body posture in the to-be-estimated image data to obtain a human body posture estimation result. According to the embodiment of the invention, the precision of multi-person attitude estimation is improved, and the calculation amount is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a 2D human body posture estimation method based on expanded mixed attention and a related device. Background Art

[0002] 2D multi-person pose estimation (MHPE) requires that the joint key points of all human targets can be correctly estimated. From the data space division of human-related features, 2D multi-person pose estimation is mainly divided into two approaches: top-down method and bottom-up method, both of which use key point heat map for positioning. Among them, the core of the top-down method is to regard multi-person pose estimation as human detection and single-person pose estimation problems. The amount of calculation will increase rapidly with the increase of the number of people contained in the picture. The advantage is that the cropped human area image is usually scaled to a uniform scale, which helps the network model implicitly learn the position distribution of each joint point, such as MSPN, Fast HPE, HRNet and other methods. The Top-Down method is limited by operations such as human detection box cropping and ROI align. These operations will lose contextual information outside the detection box, and the argmax algorithm needs to be used to locate the key points after the network detects the heat map, and the processing speed is relatively slow.

[0003] The core of the bottom-up multi-person pose estimation method is the key point matching optimization and classification problem in the feature space, and the processing speed is very fast, such as DEKR, KAPAO, Open Pose and other methods. In 2021, KAPAO proposed to model single key points and sets of spatially related key points (ie, postures) as objects in a dense, single-stage and anchor-free detection framework, and achieve faster and more accurate results by simultaneously detecting human postures and key point objects and fusing the detection results; in 2022, Xiao et al. proposed the Adaptive Pose method, which uses a part perception module to regress seven adaptive human part-related points from the center of the human instance, and then predicts the center heat map by aggregating the features of the adaptive points. This new representation method can better capture various posture deformations and adaptively decompose long-distance center-to-joint displacements, making multi-person pose estimation more compact and efficient.

[0004] In general, the Bottom-Up multi-person pose estimation method is faster, but its accuracy is lower than that of the Top-Down method. This is because the Bottom-Up method is limited to the process of directly understanding the original image, optimizing and recombining the image according to template matching. When the image contains complex scenes such as overlapping occlusion of human bodies and similar group movements, it faces the challenges of classifying overlapping joint key points and depth blur.

[0005] HRNet obtains more feature information at different levels by connecting convolutions from high resolution to low resolution in parallel, improving the field of view of the entire network model, and through repeated fusion of multiple scales in the network, the learning results are closer to the correct representation, and the predicted heat map is more accurate in space. Based on the above research, this technical solution uses HRNet (high-resolution backbone network) as the backbone network to extract multi-scale features of the human body, and then combines cascaded dilated convolutions to dynamically expand the receptive field of multi-scale features to obtain more small target feature information. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a 2D human posture estimation method and related devices based on expanded mixed attention, which improves the accuracy of multi-person posture estimation and reduces the amount of calculation.

[0007] In order to solve the above technical problems, an embodiment of the present invention provides a 2D human posture estimation method based on extended mixed attention, the method comprising:

[0008] Obtaining image data to be estimated, and inputting the image data to be estimated into a high-resolution backbone network to perform feature extraction processing at different scales, so as to obtain feature images of four different scales corresponding to the image data to be estimated;

[0009] Inputting the feature images of four different scales corresponding to the image data to be estimated into the cascade dilated convolution module for feature expression processing respectively to obtain four high-resolution feature images, wherein the cascade dilated convolution module includes four branches, and each branch processes a feature image of one scale;

[0010] Inputting the four high-resolution feature images into a hybrid self-attention module to perform feature context information extraction processing to obtain context information corresponding to the four high-resolution feature images;

[0011] The context information corresponding to the four high-resolution feature images is input into a key point regression network to estimate the human body posture in the image data to be estimated, and obtain a human body posture estimation result.

[0012] Optionally, the step of inputting the image data to be estimated into a high-resolution backbone network to perform feature extraction processing at different scales to obtain feature images of four different scales corresponding to the image data to be estimated includes:

[0013] After the image data to be estimated is input into the high-resolution backbone network, three high-resolution steps in sequence in the high-resolution backbone network are used to perform feature extraction processing of different scales to obtain feature images of four different scales corresponding to the image data to be estimated;

[0014] The first high-resolution step of the three high-resolution steps processes the input image data to be estimated and generates two feature images of different scales which are input into the second high-resolution step. The second high-resolution step inputs the processed three feature images of different scales into the third high-resolution step.

[0015] Optionally, the four feature images of different scales corresponding to the image data to be estimated are respectively input into a cascade dilated convolution module for feature expression processing to obtain four high-resolution feature images, including:

[0016] After the feature images of four different scales corresponding to the image data to be estimated are respectively input into the cascade dilated convolution module, the input feature images of four different scales are processed based on the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with four different expansion rates are used in the dilated convolution module.

[0017] Optionally, the structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel;

[0018] The second layer structure is that the second branch except the first branch of the four branches is connected to a 1*3 convolution kernel after a 1*1 expansion kernel, the third branch is connected to a 3*1 convolution kernel after a 1*1 expansion kernel, and the fourth branch is connected to a 3*3 convolution kernel after a 1*1 expansion kernel;

[0019] The third layer structure is to connect 3*3 dilated convolution kernels to all four branches, and the relationship between the actual size of the dilated convolution kernel and the size of the original convolution kernel is:

[0020] K = k + (k-1) (r-1);

[0021] Among them, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0022] Optionally, among the four branches of the dilated convolution module, except for the first branch, the inputs of the dilated convolutions of the other three branches are stacked layer by layer based on the receptive field of the previous layer, and the fourth branch equivalently includes four receptive fields;

[0023] The output of each branch is subjected to multi-level expansion branch feature fusion based on Concat association to obtain 4 high-resolution feature images.

[0024] Optionally, the step of inputting the four high-resolution feature images into a hybrid self-attention module to perform feature context information extraction processing to obtain context information corresponding to the four high-resolution feature images includes:

[0025] After inputting the four high-resolution feature images into the hybrid self-attention module, the four high-resolution feature images are normalized in the hybrid self-attention module to form normalized high-resolution feature images;

[0026] The normalized high-resolution feature images are input into the convolution self-attention CSA and coordinate attention CA for processing respectively, and the processed output results are fused with the four high-resolution feature images to form a fused high-resolution feature image;

[0027] The fused high-resolution feature images are input into a multi-layer perceptron MLP for processing, and the context information corresponding to the four high-resolution feature images is output.

[0028] Optionally, the normalized high-resolution feature images are input into convolutional self-attention CSA for processing, including:

[0029] Set the three components Q, K and V at the input end of the convolutional self-attention CSA, and the three components Q, K and V are generated by 1*1 convolution;

[0030] The normalized high-resolution feature image is processed by the Q and K components and then input into the SiLU activation function through the fully connected layer FC to obtain the processed first image feature;

[0031] The processed first image features are input into the Tanh activation function through the fully connected layer FC for processing to obtain the processed second image features;

[0032] The processed second image feature is multiplied with the result of the output of the V component, and the multiplication result is residually connected with the normalized high-resolution feature image of the input based on the fully connected layer FC to obtain the processed output result of the convolutional self-attention CSA.

[0033] In addition, an embodiment of the present invention further provides a 2D human posture estimation device based on expanded mixed attention, the device comprising:

[0034] Scale feature extraction module: used to obtain image data to be estimated, and input the image data to be estimated into a high-resolution backbone network to perform feature extraction processing at different scales, so as to obtain feature images of four different scales corresponding to the image data to be estimated;

[0035] Feature expression module: used to input the feature images of four different scales corresponding to the image data to be estimated into the cascade dilated convolution module for feature expression processing, so as to obtain four high-resolution feature images. The cascade dilated convolution module contains four branches, and each branch processes a feature image of one scale;

[0036] Information extraction module: used for inputting the four high-resolution feature images into the hybrid self-attention module to perform feature context information extraction processing, so as to obtain context information corresponding to the four high-resolution feature images;

[0037] Posture estimation module: used to input the context information corresponding to the four high-resolution feature images into the key point regression network to estimate the human body posture in the image data to be estimated, and obtain the human body posture estimation result.

[0038] In addition, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the processor runs a computer program or code stored in the memory to implement the 2D human body posture estimation method as described in any one of the above.

[0039] In addition, an embodiment of the present invention further provides a computer-readable storage medium for storing a computer program or code. When the computer program or code is executed by a processor, the 2D human body posture estimation method as described in any one of the above is implemented.

[0040] In an embodiment of the present invention, four parts are sequentially arranged, namely, a HRNet backbone network, a cascade dilated convolution module, a hybrid attention module and a key point regression module; after obtaining the image data to be estimated, firstly, feature maps of four different scales are obtained through the HRNet backbone network, and then the receptive field of the small-scale human body posture is expanded through the cascade dilated convolution module, and then the context information is integrated into the network through the hybrid attention module, so that the network can learn the correlation of local features and increase the network's representation ability of global features; finally, the key point matching results of multi-person postures are obtained through the key point regression network; the local features of different key points of the small human body posture are obtained by using the cascade dilated convolution module, and then the correlation of the context information is strengthened by the hybrid self-attention, thereby improving the network's detection performance for multi-person posture estimation and the accuracy of human body posture estimation, reducing the amount of calculation; and at the same time, improving the detection accuracy of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of a 2D human body posture estimation method based on expanded mixed attention in an embodiment of the present invention;

[0043] Figure 2 is a flow chart of a 2D human body posture estimation method based on expanded mixed attention in another embodiment of the present invention;

[0044] Figure 3 Schematic diagram of the structure of a 2D human body posture estimation device based on expanded mixed attention in an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] For example, see Figure 1 , Figure 1It is a flowchart of a 2D human body posture estimation method based on expanded mixed attention in an embodiment of the present invention.

[0048] like Figure 1 As shown, a 2D human posture estimation method based on expanded mixed attention, the method comprising:

[0049] S101: obtaining image data to be estimated, and inputting the image data to be estimated into a high-resolution backbone network to perform feature extraction processing at different scales, so as to obtain feature images of four different scales corresponding to the image data to be estimated;

[0050] In the specific implementation process of the present invention, the image data to be estimated is input into a high-resolution backbone network for feature extraction processing of different scales to obtain four feature images of different scales corresponding to the image data to be estimated, including: after the image data to be estimated is input into the high-resolution backbone network, three high-resolution steps in sequence are performed based on the high-resolution backbone network to extract features of different scales to obtain four feature images of different scales corresponding to the image data to be estimated; wherein the first high-resolution step of the three high-resolution steps processes the input image data to be estimated, and generates two feature images of different scales which are input into the second high-resolution step, and the second high-resolution step inputs the three feature images of different scales generated by the processing into the third high-resolution step.

[0051] Specifically, after obtaining the image data to be estimated, the image data to be estimated needs to be input into the high-resolution backbone network HRNet, in which three high-resolution steps (HRstage) are sequentially arranged, and the three HRstages are sequentially used to realize the extraction and processing of features of different scales, so as to obtain four feature images of different scales corresponding to the image data to be estimated; wherein, the first HRstage of the three HRstages processes the input image data to be estimated accordingly, outputs two feature images of different scales, and then inputs the two feature images of different scales into the second HRstage, and outputs three feature images of different scales, and then inputs the three feature images of different scales into the third HRstage, and finally outputs four feature images of different scales corresponding to the image data to be estimated.

[0052] S102: inputting the feature images of four different scales corresponding to the image data to be estimated into a cascade dilated convolution module for feature expression processing to obtain four high-resolution feature images, wherein the cascade dilated convolution module includes four branches, and each branch processes a feature image of one scale;

[0053] In the specific implementation process of the present invention, the four feature images of different scales corresponding to the image data to be estimated are respectively input into the cascade dilated convolution module for feature expression processing to obtain four high-resolution feature images, including: after the four feature images of different scales corresponding to the image data to be estimated are respectively input into the cascaded dilated convolution module, the four feature images of different scales input are processed based on the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0054] Furthermore, the structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel; the second layer structure is that the second branches except the first branch of the four branches are connected to a 1*3 convolution kernel after the 1*1 dilation kernel, the third branch is connected to a 3*1 convolution kernel after the 1*1 dilation kernel, and the fourth branch is connected to a 3*3 convolution kernel after the 1*1 dilation kernel; the third layer structure is that the 3*3 dilation kernel is connected to the four branches, and the relationship between the actual size of the dilation kernel and the size of the original convolution kernel is:

[0055] K = k + (k-1) (r-1);

[0056] Among them, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0057] Furthermore, among the four branches of the dilated convolution module, except for the first branch, the inputs of the dilated convolutions of the other three branches are stacked layer by layer based on the receptive field of the previous layer, and the fourth branch equivalently contains four receptive fields; the output of each branch is subjected to multi-level dilated branch feature fusion based on Concat association to obtain 4 high-resolution feature images.

[0058] Specifically, in the implemented application, the more network convolution kernels there are, the higher the number of channels and the richer the feature expression, but the network calculation time and cost will also increase; in order to balance the calculation time and cost with the rich feature expression, this embodiment is implemented using a cascaded dilated convolution module; that is, after the four feature images of different scales corresponding to the image data to be estimated are respectively input into the cascaded dilated convolution module, the four feature images of different scales are processed by the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0059] In this embodiment, a cascaded dilated convolution module DCM is used to improve the perception of small targets and reduce the amount of calculation. For each scale feature map output by HRnet, it is processed by four branches in DCM. This is because the inherent grid effect of dilated convolution can easily lead to the problem of inconsistent local information. Four dilated convolution layers with different expansion rates (R=1, 3, 3, 5) are used to alleviate this grid effect.

[0060] The structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel; the second layer structure is that the second branch except the first branch of the four branches is connected to a 1*3 convolution kernel after the 1*1 dilation kernel, the third branch is connected to a 3*1 convolution kernel after the 1*1 dilation kernel, and the fourth branch is connected to a 3*3 convolution kernel after the 1*1 dilation kernel; the third layer structure is that the 3*3 dilation kernel is connected to all four branches; the relationship between the actual size of the dilated convolution kernel and the original convolution kernel size is:

[0061] K = k + (k-1) (r-1);

[0062] Among them, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0063] Except for the first branch, the inputs of the dilated convolutions of the other three branches are based on the previous layer, and the receptive fields are gradually superimposed. Therefore, the fourth branch equivalently contains four receptive fields. Finally, each branch is concat-associated to achieve multi-level dilated branch feature fusion; through this area expansion from small to large, information can be captured from larger context areas, different local rich representations can be extracted, the receptive field of small targets can be improved, and higher resolution feature maps can be generated.

[0064] To improve efficiency, the number of channels in a block is gradually reduced to keep the computational cost low.

[0065] S103: Inputting the four high-resolution feature images into a hybrid self-attention module to perform feature context information extraction processing to obtain context information corresponding to the four high-resolution feature images;

[0066] In the specific implementation process of the present invention, the four high-resolution feature images are input into the hybrid self-attention module for feature context information extraction processing to obtain the context information corresponding to the four high-resolution feature images, including: after the four high-resolution feature images are input into the hybrid self-attention module, the four high-resolution feature images are normalized in the hybrid self-attention module to form normalized high-resolution feature images; the normalized high-resolution feature images are respectively input into the convolution self-attention CSA and the coordinate attention CA for processing, and the processed output results are fused with the four high-resolution feature images to form a fused high-resolution feature image; the fused high-resolution feature image is input into the multi-layer perceptron MLP for processing, and the context information corresponding to the four high-resolution feature images is output.

[0067] Furthermore, the normalized high-resolution feature image is input into the convolutional self-attention CSA for processing, including: setting three components Q, K and V at the input end of the convolutional self-attention CSA, and the three components Q, K and V are generated by 1*1 convolution; the normalized high-resolution feature image is processed by the Q and K components, and then input into the SiLU activation function through the fully connected layer FC to obtain the processed first image feature; the processed first image feature is input into the Tanh activation function through the fully connected layer FC for processing to obtain the processed second image feature; the processed second image feature is multiplied by the result of the output of the V component, and the multiplication result is residually connected based on the fully connected layer FC and the input normalized high-resolution feature image to obtain the processed output result of the convolutional self-attention CSA.

[0068] Specifically, in order to perceive the contextual information of the feature map extracted in the previous step and enhance the representation of small-scale local features of the human body, a hybrid attention mechanism is used to solve the problem of severe loss of posture and position information. The feature map X is X1 after layer normalization (LN), and is fused by the convolutional self-attention (CSA) and coordinate attention (CA) blocks to become X2. Y is the final output processing result of the MLP layer, which can be expressed as the following formula:

[0069] X1=LN(X);

[0070] X2=CA(X1)+CSA(X1)+X;

[0071] Y = MLP (LN (X2)) + X2;

[0072] In the convolutional self-attention CSA, in order to simultaneously obtain local and global features and improve the context fusion capability; three components Q, K and V are set at the input end of the convolutional self-attention CSA, and the three components Q, K and V are generated by 1*1 convolution; X input Represents the input features as follows:

[0073] Q, K, V = Conv(X input );

[0074] After the Q and K matrices are multiplied, a series of nonlinear processing is performed, and then the fully connected layer (FC) is input into the SiLU activation function to increase the linear expression ability, and X is obtained. A ;

[0075]

[0076] In the above formula, X A Then input the fully connected layer FC and Tanh activation function to obtain weight information and get X B ,as follows:

[0077]

[0078] X B After multiplication with V, it passes through the fully connected layer FC and then performs a residual connection with Xinput to obtain the final output result Xoutput; as follows:

[0079]

[0080] After passing through the fully connected layer, it is processed using the Tanh activation function, and the output result is a matrix multiplied by the V component of the linear transformation. Compared with the original MSA, CSA obtains more local features through convolution and then performs global calculations, which can make the attention network have stronger context perception and further enhance the network's use of the feature utilization range. In addition, it has more nonlinear transformations, which makes the network have stronger nonlinear expression capabilities.

[0081] The coordinate attention (CA) module is used to enhance the network's location information fusion capability. The CA module can effectively enhance the association between different channels and improve the network's perception of remote location information; the ablation experiment results show that the added coordinate attention can improve the model's detection accuracy for small targets.

[0082] S104: Inputting the context information corresponding to the four high-resolution feature images into a key point regression network to perform estimation processing on the human body posture in the image data to be estimated, and obtaining a human body posture estimation result.

[0083] In the specific implementation process of the present invention, after obtaining the context information corresponding to the four high-resolution feature images, it is necessary to input the context information corresponding to the four high-resolution feature images into the key point regression network to estimate the human body posture in the image data to be estimated; that is, the key point matching results of multi-person postures are obtained through the key point regression network, and the human body posture estimation results are output based on the key point matching results.

[0084] In an embodiment of the present invention, four parts are sequentially arranged, namely, a HRNet backbone network, a cascade dilated convolution module, a hybrid attention module and a key point regression module; after obtaining the image data to be estimated, firstly, feature maps of four different scales are obtained through the HRNet backbone network, and then the receptive field of the small-scale human body posture is expanded through the cascade dilated convolution module, and then the context information is integrated into the network through the hybrid attention module, so that the network can learn the correlation of local features and increase the network's representation ability of global features; finally, the key point matching results of multi-person postures are obtained through the key point regression network; the local features of different key points of the small human body posture are obtained by using the cascade dilated convolution module, and then the correlation of the context information is strengthened by the hybrid self-attention, thereby improving the network's detection performance for multi-person posture estimation and the accuracy of human body posture estimation, reducing the amount of calculation; and at the same time, improving the detection accuracy of small targets.

[0085] For example 2, please refer to Figure 2 , Figure 2 It is a flowchart of a 2D human posture estimation method based on expanded mixed attention in another embodiment of the present invention.

[0086] like Figure 2 As shown, a 2D human posture estimation method based on expanded mixed attention, the method comprising:

[0087] S201: Obtain image data to be estimated, and input the image data to be estimated into a high-resolution backbone network to perform feature extraction processing at different scales, so as to obtain feature images of four different scales corresponding to the image data to be estimated;

[0088] S202: After inputting the feature images of four different scales corresponding to the image data to be estimated into the cascade dilated convolution module respectively, the input feature images of four different scales are processed based on the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different dilation rates are used in the dilated convolution module;

[0089] S203: after inputting the four high-resolution feature images into the hybrid self-attention module, normalizing the four high-resolution feature images in the hybrid self-attention module to form normalized high-resolution feature images;

[0090] S204: inputting the normalized high-resolution feature image into the convolution self-attention CSA and the coordinate attention CA for processing respectively, and fusing the processed output results with the four high-resolution feature images to form a fused high-resolution feature image;

[0091] S205: inputting the fused high-resolution feature image into a multi-layer perceptron MLP for processing, and outputting context information corresponding to the four high-resolution feature images;

[0092] S206: Input the context information corresponding to the four high-resolution feature images into a key point regression network to perform estimation processing on the human body posture in the image data to be estimated, and obtain a human body posture estimation result.

[0093] The specific implementation of the second embodiment can be found in the above embodiments, which will not be described in detail here.

[0094] For example 3, please refer to Figure 3 , Figure 3 It is a schematic diagram of the structural composition of a 2D human posture estimation device based on expanded mixed attention in an embodiment of the present invention.

[0095] like Figure 3 As shown, a 2D human posture estimation device based on expanded mixed attention, the device comprising:

[0096] Scale feature extraction module 301: used to obtain image data to be estimated, and input the image data to be estimated into a high-resolution backbone network to perform different scale feature extraction processing, so as to obtain four feature images of different scales corresponding to the image data to be estimated;

[0097] In the specific implementation process of the present invention, the image data to be estimated is input into a high-resolution backbone network for feature extraction processing of different scales to obtain four feature images of different scales corresponding to the image data to be estimated, including: after the image data to be estimated is input into the high-resolution backbone network, three high-resolution steps in sequence are performed based on the high-resolution backbone network to extract features of different scales to obtain four feature images of different scales corresponding to the image data to be estimated; wherein the first high-resolution step of the three high-resolution steps processes the input image data to be estimated, and generates two feature images of different scales which are input into the second high-resolution step, and the second high-resolution step inputs the three feature images of different scales generated by the processing into the third high-resolution step.

[0098] Specifically, after obtaining the image data to be estimated, the image data to be estimated needs to be input into the high-resolution backbone network HRNet, in which three high-resolution steps (HRstage) are sequentially arranged, and the three HRstages are sequentially used to realize the extraction and processing of features of different scales, so as to obtain four feature images of different scales corresponding to the image data to be estimated; wherein, the first HRstage of the three HRstages processes the input image data to be estimated accordingly, outputs two feature images of different scales, and then inputs the two feature images of different scales into the second HRstage, and outputs three feature images of different scales, and then inputs the three feature images of different scales into the third HRstage, and finally outputs four feature images of different scales corresponding to the image data to be estimated.

[0099] Feature expression module 302: used to input the feature images of four different scales corresponding to the image data to be estimated into the cascade dilated convolution module for feature expression processing, so as to obtain four high-resolution feature images. The cascade dilated convolution module includes four branches, and each branch processes a feature image of one scale.

[0100] In the specific implementation process of the present invention, the four feature images of different scales corresponding to the image data to be estimated are respectively input into the cascade dilated convolution module for feature expression processing to obtain four high-resolution feature images, including: after the four feature images of different scales corresponding to the image data to be estimated are respectively input into the cascaded dilated convolution module, the four feature images of different scales input are processed based on the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0101] Furthermore, the structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel; the second layer structure is that the second branches except the first branch of the four branches are connected to a 1*3 convolution kernel after the 1*1 dilation kernel, the third branch is connected to a 3*1 convolution kernel after the 1*1 dilation kernel, and the fourth branch is connected to a 3*3 convolution kernel after the 1*1 dilation kernel; the third layer structure is that the 3*3 dilation kernel is connected to the four branches, and the relationship between the actual size of the dilation kernel and the size of the original convolution kernel is:

[0102] K = k + (k-1) (r-1);

[0103] Among them, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0104] Furthermore, among the four branches of the dilated convolution module, except for the first branch, the inputs of the dilated convolutions of the other three branches are stacked layer by layer based on the receptive field of the previous layer, and the fourth branch equivalently contains four receptive fields; the output of each branch is subjected to multi-level dilated branch feature fusion based on Concat association to obtain 4 high-resolution feature images.

[0105] Specifically, in the implemented application, the more network convolution kernels there are, the higher the number of channels and the richer the feature expression, but the network calculation time and cost will also increase; in order to balance the calculation time and cost with the rich feature expression, this embodiment is implemented using a cascaded dilated convolution module; that is, after the four feature images of different scales corresponding to the image data to be estimated are respectively input into the cascaded dilated convolution module, the four feature images of different scales are processed by the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with different expansion rates are used in the dilated convolution module.

[0106] In this embodiment, a cascaded dilated convolution module DCM is used to improve the perception of small targets and reduce the amount of calculation. For each scale feature map output by HRnet, it is processed by four branches in DCM. This is because the inherent grid effect of dilated convolution can easily lead to the problem of inconsistent local information. Four dilated convolution layers with different expansion rates (R=1, 3, 3, 5) are used to alleviate this grid effect.

[0107] The structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel; the second layer structure is that the second branch except the first branch of the four branches is connected to a 1*3 convolution kernel after the 1*1 dilation kernel, the third branch is connected to a 3*1 convolution kernel after the 1*1 dilation kernel, and the fourth branch is connected to a 3*3 convolution kernel after the 1*1 dilation kernel; the third layer structure is that the 3*3 dilation kernel is connected to all four branches; the relationship between the actual size of the dilated convolution kernel and the original convolution kernel size is:

[0108] K = k + (k-1) (r-1);

[0109] Among them, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

[0110] Except for the first branch, the inputs of the dilated convolutions of the other three branches are based on the previous layer, and the receptive fields are gradually superimposed. Therefore, the fourth branch equivalently contains four receptive fields. Finally, each branch is concat-associated to achieve multi-level dilated branch feature fusion; through this area expansion from small to large, information can be captured from larger context areas, different local rich representations can be extracted, the receptive field of small targets can be improved, and higher resolution feature maps can be generated.

[0111] To improve efficiency, the number of channels in a block is gradually reduced to keep the computational cost low.

[0112] Information extraction module 303: used to input the four high-resolution feature images into the hybrid self-attention module to perform feature context information extraction processing, so as to obtain context information corresponding to the four high-resolution feature images;

[0113] In the specific implementation process of the present invention, the four high-resolution feature images are input into the hybrid self-attention module for feature context information extraction processing to obtain the context information corresponding to the four high-resolution feature images, including: after the four high-resolution feature images are input into the hybrid self-attention module, the four high-resolution feature images are normalized in the hybrid self-attention module to form normalized high-resolution feature images; the normalized high-resolution feature images are respectively input into the convolution self-attention CSA and the coordinate attention CA for processing, and the processed output results are fused with the four high-resolution feature images to form a fused high-resolution feature image; the fused high-resolution feature image is input into the multi-layer perceptron MLP for processing, and the context information corresponding to the four high-resolution feature images is output.

[0114] Furthermore, the normalized high-resolution feature image is input into the convolutional self-attention CSA for processing, including: setting three components Q, K and V at the input end of the convolutional self-attention CSA, and the three components Q, K and V are generated by 1*1 convolution; the normalized high-resolution feature image is processed by the Q and K components, and then input into the SiLU activation function through the fully connected layer FC to obtain the processed first image feature; the processed first image feature is input into the Tanh activation function through the fully connected layer FC for processing to obtain the processed second image feature; the processed second image feature is multiplied by the result of the output of the V component, and the multiplication result is residually connected based on the fully connected layer FC and the input normalized high-resolution feature image to obtain the processed output result of the convolutional self-attention CSA.

[0115] Specifically, in order to perceive the contextual information of the feature map extracted in the previous step and enhance the representation of small-scale local features of the human body, a hybrid attention mechanism is used to solve the problem of severe loss of posture and position information. The feature map X is X1 after layer normalization (LN), and is fused by the convolutional self-attention (CSA) and coordinate attention (CA) blocks to become X2. Y is the final output processing result of the MLP layer, which can be expressed as the following formula:

[0116] X1=LN(X);

[0117] X2=CA(X1)+CSA(X1)+X;

[0118] Y = MLP (LN (X2)) + X2;

[0119] In the convolutional self-attention CSA, in order to simultaneously obtain local and global features and improve the context fusion capability; three components Q, K and V are set at the input end of the convolutional self-attention CSA, and the three components Q, K and V are generated by 1*1 convolution; X input Represents the input features as follows:

[0120] Q,K,V=Conv(X input );

[0121] After the Q and K matrices are multiplied, a series of nonlinear processing is performed, and then the fully connected layer (FC) is input into the SiLU activation function to increase the linear expression ability, and X is obtained. A ;

[0122]

[0123] In the above formula, X A Then input the fully connected layer FC and Tanh activation function to obtain weight information and get X B ,as follows:

[0124]

[0125] X B After multiplication with V, it passes through the fully connected layer FC and then performs a residual connection with Xinput to obtain the final output result Xoutput; as follows:

[0126]

[0127] After passing through the fully connected layer, it is processed using the Tanh activation function, and the output result is a matrix multiplied by the V component of the linear transformation. Compared with the original MSA, CSA obtains more local features through convolution and then performs global calculations, which can make the attention network have stronger context perception and further enhance the network's use of the feature utilization range. In addition, it has more nonlinear transformations, which makes the network have stronger nonlinear expression capabilities.

[0128] The coordinate attention (CA) module is used to enhance the network's location information fusion capability. The CA module can effectively enhance the association between different channels and improve the network's perception of remote location information; the ablation experiment results show that the added coordinate attention can improve the model's detection accuracy for small targets.

[0129] The posture estimation module 304 is used to input the context information corresponding to the four high-resolution feature images into the key point regression network to estimate the human posture in the image data to be estimated, and obtain the human posture estimation result.

[0130] In the specific implementation process of the present invention, after obtaining the context information corresponding to the four high-resolution feature images, it is necessary to input the context information corresponding to the four high-resolution feature images into the key point regression network to estimate the human body posture in the image data to be estimated; that is, the key point matching results of multi-person postures are obtained through the key point regression network, and the human body posture estimation results are output based on the key point matching results.

[0131] In an embodiment of the present invention, four parts are sequentially arranged, namely, a HRNet backbone network, a cascade dilated convolution module, a hybrid attention module and a key point regression module; after obtaining the image data to be estimated, firstly, feature maps of four different scales are obtained through the HRNet backbone network, and then the receptive field of the small-scale human body posture is expanded through the cascade dilated convolution module, and then the context information is integrated into the network through the hybrid attention module, so that the network can learn the correlation of local features and increase the network's representation ability of global features; finally, the key point matching results of multi-person postures are obtained through the key point regression network; the local features of different key points of the small human body posture are obtained by using the cascade dilated convolution module, and then the correlation of the context information is strengthened by the hybrid self-attention, thereby improving the network's detection performance for multi-person posture estimation and the accuracy of human body posture estimation, reducing the amount of calculation; and at the same time, improving the detection accuracy of small targets.

[0132] A computer-readable storage medium provided in an embodiment of the present invention stores a computer program on the computer-readable storage medium, and when the program is executed by a processor, a 2D human posture estimation method of any one of the above embodiments is implemented. Wherein, the computer-readable storage medium includes but is not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (EraSable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the storage device includes any medium that can store or transmit information in a readable form by a device (for example, a computer, a mobile phone), which can be a read-only memory, a disk or an optical disk, etc.

[0133] An embodiment of the present invention further provides a computer application program, which runs on a computer and is used to execute the 2D human body posture estimation method of any one of the above embodiments.

[0134] also, Figure 4 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention.

[0135] The embodiment of the present invention further provides an electronic device, such as Figure 4 The electronic device includes a processor 402, a memory 403, an input unit 404, a display unit 405 and other devices. Those skilled in the art will understand that Figure 4 The structural components of the electronic device shown do not constitute a limitation on all devices, and may include more or fewer components than shown, or combine certain components. The memory 403 can be used to store the application 401 and various functional modules, and the processor 402 runs the application 401 stored in the memory 403, thereby executing various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both internal and external memories. The internal memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory. The external memory may include a hard disk, a floppy disk, a ZIP disk, a U disk, a magnetic tape, etc. The memory disclosed in the present invention includes but is not limited to these types of memories. The memory disclosed in the present invention is only used as an example and not as a limitation.

[0136] The input unit 404 is used to receive the input of the signal and the keyword input by the user. The input unit 404 may include a touch panel and other input devices. The touch panel can collect the user's touch operation on or near it (such as the user's operation on or near the touch panel using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program; other input devices may include but are not limited to one or more of a physical keyboard, a function key (such as a playback control key, a switch key, etc.), a trackball, a mouse, a joystick, etc. The display unit 405 can be used to display the information input by the user or the information provided to the user and various menus of the terminal device. The display unit 405 can be in the form of a liquid crystal display, an organic light emitting diode, etc. The processor 402 is the control center of the terminal device, which uses various interfaces and lines to connect the various parts of the entire device, and executes various functions and processes data by running or executing software programs and / or modules stored in the memory 403, and calling the data stored in the memory.

[0137] As an embodiment, the electronic device includes: one or more processors 402, a memory 403, and one or more applications 401, wherein the one or more applications 401 are stored in the memory 403 and are configured to be executed by the one or more processors 402, and the one or more applications 401 are configured to execute the corresponding 2D human body posture estimation method in any one of the above-mentioned embodiments.

[0138] In an embodiment of the present invention, four parts are sequentially arranged, namely, a HRNet backbone network, a cascade dilated convolution module, a hybrid attention module and a key point regression module; after obtaining the image data to be estimated, firstly, feature maps of four different scales are obtained through the HRNet backbone network, and then the receptive field of the small-scale human body posture is expanded through the cascade dilated convolution module, and then the context information is integrated into the network through the hybrid attention module, so that the network can learn the correlation of local features and increase the network's representation ability of global features; finally, the key point matching results of multi-person postures are obtained through the key point regression network; the local features of different key points of the small human body posture are obtained by using the cascade dilated convolution module, and then the correlation of the context information is strengthened by the hybrid self-attention, thereby improving the network's detection performance for multi-person posture estimation and the accuracy of human body posture estimation, reducing the amount of calculation; and at the same time, improving the detection accuracy of small targets.

[0139] In addition, the above is a detailed introduction to a 2D human posture estimation method based on expanded mixed attention and related devices provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a general technician in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A 2D human pose estimation method based on extended mixed attention, characterized in that: The method comprises: Obtaining image data to be estimated, and inputting the image data to be estimated into a high-resolution backbone network to perform feature extraction processing at different scales, so as to obtain feature images of four different scales corresponding to the image data to be estimated; Inputting the feature images of four different scales corresponding to the image data to be estimated into the cascade dilated convolution module for feature expression processing respectively to obtain four high-resolution feature images, wherein the cascade dilated convolution module includes four branches, and each branch processes a feature image of one scale; Inputting the four high-resolution feature images into a hybrid self-attention module to perform feature context information extraction processing to obtain context information corresponding to the four high-resolution feature images; The context information corresponding to the four high-resolution feature images is input into a key point regression network to estimate the human body posture in the image data to be estimated, and obtain a human body posture estimation result.

2. The 2D human body posture estimation method according to claim 1, characterized in that: The step of inputting the image data to be estimated into a high-resolution backbone network to extract features of different scales to obtain feature images of four different scales corresponding to the image data to be estimated includes: After the image data to be estimated is input into the high-resolution backbone network, three high-resolution steps in sequence in the high-resolution backbone network are used to perform feature extraction processing of different scales to obtain feature images of four different scales corresponding to the image data to be estimated; The first high-resolution step of the three high-resolution steps processes the input image data to be estimated and generates two feature images of different scales which are input into the second high-resolution step. The second high-resolution step inputs the processed three feature images of different scales into the third high-resolution step.

3. The 2D human body posture estimation method according to claim 1, characterized in that: The four feature images of different scales corresponding to the image data to be estimated are respectively input into the cascade dilated convolution module for feature expression processing to obtain four high-resolution feature images, including: After the feature images of four different scales corresponding to the image data to be estimated are respectively input into the cascade dilated convolution module, the input feature images of four different scales are processed based on the four branches of the dilated convolution module in the cascaded dilated convolution module to obtain four high-resolution feature images, wherein four dilated convolution layers with four different expansion rates are used in the dilated convolution module.

4. The 2D human body posture estimation method according to claim 3, characterized in that: The structure of the four branches of the dilated convolution module is that the first layer structure of each branch is a 1*1 convolution kernel; The second layer structure is that the second branch except the first branch of the four branches is connected to a 1*3 convolution kernel after a 1*1 expansion kernel, the third branch is connected to a 3*1 convolution kernel after a 1*1 expansion kernel, and the fourth branch is connected to a 3*3 convolution kernel after a 1*1 expansion kernel; The third layer structure is to connect 3*3 dilated convolution kernels to all four branches, and the relationship between the actual size of the dilated convolution kernel and the size of the original convolution kernel is: K = k + (k-1) (r-1); Among them, K represents the equivalent kernel size; k represents the original convolution kernel size; r represents the expansion rate of the dilated convolution.

5. The 2D human body posture estimation method according to claim 4, characterized in that: Except for the first branch, the inputs of the dilated convolutions of the other three branches of the dilated convolution module are stacked layer by layer based on the receptive fields of the previous layer, and the fourth branch equivalently contains four receptive fields; The output of each branch is subjected to multi-level expansion branch feature fusion based on Concat association to obtain 4 high-resolution feature images.

6. The 2D human body posture estimation method according to claim 1, characterized in that: The step of inputting the four high-resolution feature images into a hybrid self-attention module to perform feature context information extraction processing to obtain context information corresponding to the four high-resolution feature images includes: After inputting the four high-resolution feature images into the hybrid self-attention module, the four high-resolution feature images are normalized in the hybrid self-attention module to form normalized high-resolution feature images; The normalized high-resolution feature images are input into the convolution self-attention CSA and coordinate attention CA for processing respectively, and the processed output results are fused with the four high-resolution feature images to form a fused high-resolution feature image; The fused high-resolution feature images are input into a multi-layer perceptron MLP for processing, and the context information corresponding to the four high-resolution feature images is output.

7. The 2D human body posture estimation method according to claim 6, characterized in that: The normalized high-resolution feature image is input into the convolutional self-attention CSA for processing, including: Set the three components Q, K and V at the input of the convolutional self-attention CSA, and the three components Q, K and V are generated by 1*1 convolution; The normalized high-resolution feature image is processed by the Q and K components and then input into the SiLU activation function through the fully connected layer FC to obtain the processed first image feature; The processed first image features are input into the Tanh activation function through the fully connected layer FC for processing to obtain the processed second image features; The processed second image feature is multiplied with the result of the output of the V component, and the multiplication result is residually connected with the normalized high-resolution feature image of the input based on the fully connected layer FC to obtain the processed output result of the convolutional self-attention CSA.

8. A 2D human posture estimation device based on extended mixed attention, characterized in that: The device comprises: Scale feature extraction module: used to obtain image data to be estimated, and input the image data to be estimated into a high-resolution backbone network to perform feature extraction processing at different scales, so as to obtain feature images of four different scales corresponding to the image data to be estimated; Feature expression module: used to input the feature images of four different scales corresponding to the image data to be estimated into the cascade dilated convolution module for feature expression processing, so as to obtain four high-resolution feature images. The cascade dilated convolution module contains four branches, and each branch processes a feature image of one scale; Information extraction module: used for inputting the four high-resolution feature images into the hybrid self-attention module to perform feature context information extraction processing, so as to obtain context information corresponding to the four high-resolution feature images; Posture estimation module: used to input the context information corresponding to the four high-resolution feature images into the key point regression network to estimate the human body posture in the image data to be estimated, and obtain the human body posture estimation result.

9. An electronic device comprising a processor and a memory, characterized in that: The processor runs a computer program or code stored in the memory to implement the 2D human posture estimation method according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing a computer program or code, characterized in that: When the computer program or code is executed by a processor, the 2D human posture estimation method according to any one of claims 1 to 7 is implemented.

Images