Posture estimation method based on global information perception and related device

Abstract

The invention belongs to the technical field of computer vision, and discloses an attitude estimation method based on global information perception and a related device, and the method comprises the steps: obtaining a to-be-estimated original image; inputting the to-be-estimated original image into a pre-constructed global information perception attitude network to carry out human body attitude estimation, and outputting to obtain a human body attitude estimation result; wherein the pre-constructed global information sensing attitude network is an improved HRNet network; specifically, according to the improved HRNet network, a parallel replacement convolution block attention module is introduced into a branch expansion conversion layer of the HRNet network, fast Fourier convolution up-sampling operation is introduced in the fourth stage of the HRNet network, and down-sampling operation based on 3 * 3 convolution with the step length being 2 in the HRNet network is replaced with recombined pixel down-sampling operation; according to the method, perception and understanding of the model on global information are enhanced, so that the accuracy of an attitude estimation result is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and relates to a method for estimating human body posture, and in particular to a method for estimating human body posture based on global information perception and a related device. Background Art

[0002] In the field of computer vision, human posture estimation refers to locating key points of the human body in images or videos through computer algorithms, thereby realizing the recognition and analysis of human posture. Its core is to extract the position and motion trajectory of key points by capturing human motion information in images or videos.

[0003] Human posture is usually closely related to the surrounding environment and context. When dealing with full-body posture estimation, it is necessary to be able to capture the long-distance dependencies between various parts of the body. However, existing human posture estimation models often rely on local features such as pixel information around key points to predict the positions of key points. When dealing with long-distance dependencies, they often ignore global context information such as interactions between people and interactions between the human body and objects, which limits the model's ability to capture global information. However, when dealing with human posture estimation tasks in complex scenes such as occlusion, overlap or complex background, local features may not be able to accurately capture occluded or overlapping key parts. If the model only relies on local features, it may mistakenly connect key points of different people together, or connect unrelated parts of the body. Secondly, in complex scenes, the semantic information module caused by occlusion or truncation of key points and degradation of image quality increases the model's limitations in global information processing, resulting in missed detection or false detection of the model, which in turn makes the posture estimation results prone to errors or mistakes. Summary of the invention

[0004] In response to the technical problems existing in the prior art, the present invention provides a posture estimation method and related devices based on global information perception to solve the technical problem that the existing human posture estimation model often relies on local features, resulting in errors or mistakes in the posture estimation results.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention provides a posture estimation method based on global information perception, comprising:

[0007] Obtaining the original image to be estimated;

[0008] Inputting the original image to be estimated into a pre-built global information perception posture network to perform human posture estimation, and outputting a human posture estimation result;

[0009] Among them, the pre-constructed global information perception posture network is an improved HRNet network; specifically, the improved HRNet network is: introducing a parallel permutation convolution block attention module in the branch expansion conversion layer of the HRNet network, introducing a fast Fourier convolution upsampling operation in the fourth stage of the HRNet network, and replacing the downsampling operation based on 3×3 convolution with a step size of 2 in the HRNet network with a reorganized pixel downsampling operation.

[0010] Furthermore, the spatial dimension attention mechanism and the channel dimension attention mechanism are introduced into the attention module of the permutation convolution block to capture the global correlation in the image; the fast Fourier convolution upsampling operation is used to utilize fast Fourier convolution and pixel reorganization to achieve efficient conversion from a smaller-sized feature map to a larger-sized feature map; the reorganized pixel downsampling operation is used to retain spatial information while reducing the size of the feature map by reorganizing pixels.

[0011] Furthermore, the working process of the permuted convolution fast attention module is as follows:

[0012] The input feature map F with the dimension C×H×W x Based on the average segmentation of the channel, several sub-feature groups are obtained Wherein, each sub-feature group includes several sub-feature graphs;

[0013] For each sub-feature group Divide the sub-feature group into two sub-parts with the same number of channels along the channel direction

[0014] Using the channel attention module, the channel dimension attention mechanism is applied to the first sub-part, and the feature map with the channel dimension attention mechanism is multiplied by the first sub-part to obtain the sub-feature map F s3 ;

[0015] Using the spatial attention module, the spatial dimension attention mechanism is applied to the second sub-part, and the feature map with the spatial dimension attention mechanism is multiplied by the second sub-part to obtain the sub-feature map F s4 ;

[0016] For the sub-feature map F s3 and the sub-feature map F s4 Perform feature concatenation to obtain the sub-feature graph F s5 ;

[0017] Perform sub-feature graph F on all sub-feature groups s5 After aggregation, the Channel Shuffle operation is used to mix and exchange feature information across groups, and the output is a feature map processed by the permuted convolution block attention module.

[0018] Furthermore, the channel attention module is used to apply the channel dimension attention mechanism to the first sub-part, as follows:

[0019] Performing a maximum pooling operation and an average pooling operation on the first sub-part respectively to obtain a maximum pooled sub-part and an average pooled sub-part; using a multi-layer perceptron MLP to process the maximum pooled sub-part and the average pooled sub-part respectively, adding them together, and processing them through an activation function to obtain a feature map applying a channel dimension attention mechanism;

[0020] Using the spatial attention module, the process of applying the attention mechanism of the spatial dimension to the second sub-part is as follows:

[0021] Perform a maximum pooling operation or an average pooling operation on the second sub-part to obtain a pooled sub-part; perform convolution and activation function processing on the pooled sub-part to obtain a feature map applying a spatial dimension attention mechanism.

[0022] Furthermore, the process of the fast Fourier convolution upsampling operation is as follows:

[0023] For the input feature map F with dimension c×h×w f ∈R c×h×w Perform fast Fourier convolution operation to obtain the feature map F f1 ∈R (d*d*c)×h×w ;

[0024] For the feature map F f1 ∈R( d*d*c)×h×w Perform pixel reorganization PixelShuffle operation to obtain feature map F f1 ∈R (d*d*c)×(d*h)×w ; Where d is the upsampling factor in the pixel reorganization PixelShuffle operation.

[0025] Furthermore, the process of the reorganized pixel downsampling operation is as follows:

[0026] For the input feature map F with dimension a×a×b spd0 ∈R a×a×b Perform the Shuffle Pixel operation to obtain the reorganized feature map F spd1 ;

[0027] The reorganized feature map F spd1 Apply 4 1×1 convolution kernels to the reorganized feature map F spd1 Perform size transformation to obtain feature map F spd2 .

[0028] The present invention also provides a posture estimation system based on global information perception, comprising:

[0029] An image acquisition module, used for acquiring an original image to be estimated;

[0030] A posture estimation module is used to input the original image to be estimated into a pre-built global information perception posture network to perform human posture estimation, and output a human posture estimation result;

[0031] Among them, the pre-constructed global information perception posture network is an improved HRNet network; specifically, the improved HRNet network is: introducing a parallel permutation convolution block attention module in the branch expansion conversion layer of the HRNet network, introducing a fast Fourier convolution upsampling operation in the fourth stage of the HRNet network, and replacing the downsampling operation based on 3×3 convolution with a step size of 2 in the HRNet network with a reorganized pixel downsampling operation.

[0032] The present invention also provides a posture estimation device based on global information perception, comprising:

[0033] a processor suitable for executing a computer program;

[0034] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the posture estimation method based on global information perception is executed.

[0035] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the posture estimation method based on global information perception is implemented.

[0036] The present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the posture estimation method based on global information perception is implemented.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The posture estimation method based on global information perception provided by the present invention introduces a parallel permutation convolution block attention module, a fast Fourier convolution upsampling operation and a recombined pixel downsampling operation into the HRNet network to construct a global information perception posture network, and uses the global information perception posture network to perform human posture estimation processing on the original image to be estimated, thereby expanding the receptive field of the model and minimizing the loss of spatial details in the feature extraction process, thereby significantly improving the model's understanding and perception of global contextual information; secondly, by effectively capturing the human body key point information in the image, higher key point positioning accuracy is achieved when performing the human body posture estimation task, thereby enhancing the model's perception and understanding of global information, thereby improving the accuracy of the posture estimation result.

[0039] The posture estimation system based on global information perception, the posture estimation device based on global information perception and the computer-readable storage medium provided by the present invention have all the advantages of the above-mentioned posture estimation method based on global information perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A flowchart of a posture estimation method based on global information perception provided in Example 1;

[0042] Figure 2 Schematic diagram of the structure of the global information perception posture network in Example 1;

[0043] Figure 3 Schematic diagram of the structure of the permuted convolution block attention module in Example 1;

[0044] Figure 4 This is a flowchart of the reorganized pixel downsampling operation in Example 1;

[0045] Figure 5 This is a visualization result diagram of the feature graph output by HRNet and GIPNet at the same network level in Example 1;

[0046] Figure 6 A structural block diagram of a posture estimation system based on global information perception provided in Example 2;

[0047] Figure 7 This is a structural block diagram of a posture estimation device based on global information perception provided in Example 3. DETAILED DESCRIPTION

[0048] In order to make the technical problems, technical solutions and beneficial effects solved by this application clearer, the technical solutions in the embodiments of this application will be described clearly and completely in combination with the drawings in the embodiments of this application; obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] As attached Figure 1 As shown, this embodiment 1 provides a posture estimation method based on global information perception, comprising the following steps:

[0050] Step 1: Get the original image to be estimated.

[0051] Step 2: Input the original image to be estimated into a pre-built global information perception posture network (GIPNet) for human body posture estimation, and output the human body posture estimation result.

[0052] As attached Figure 2 As shown in the figure, the pre-built global information perception posture network (GIPNet) is an improved HRNet network; wherein, the pre-built global information perception posture network is an improved HRNet network; specifically, the improved HRNet network is: a parallel permutation convolution block attention module (Shuffle Convolutional Block Attention Module, SCBAM) is introduced in the branch expansion conversion layer of the HRNet network, a fast Fourier convolution up-sampling operation (Fast Fourier Convolution Up-sampling, FFCU) is introduced in the fourth truncation of the HRNet network, and the down-sampling operation based on 3×3 convolution with a step size of 2 in the HRNet network is replaced by a reorganized pixel down-sampling operation (Shuffle Pixel Down-sampling, SPD); wherein, the spatial dimension attention mechanism and the channel dimension attention mechanism are introduced into the permutation convolution block attention module to capture the global correlation in the image; the fast Fourier convolution up-sampling operation is used to utilize fast Fourier convolution and pixel reorganization to achieve efficient conversion from a smaller-sized feature map to a larger-sized feature map; the reorganized pixel down-sampling operation is used to retain spatial information while reducing the size of the feature map by reorganizing pixels.

[0053] Specifically, the original image to be estimated is input into a pre-built global information perception posture network (GIPNet) for human posture estimation, and the process of outputting the human posture estimation result is as follows:

[0054] Step 11: perform a reconstructed pixel downsampling operation (SDP) on the original image to be estimated to obtain a feature map Stem_Output, and output the feature map Stem_Output to the first stage (Stage 1) of the global information perception posture network (GIPNet).

[0055] Step 12, in the first stage (Stage 1) of the global information perception posture network (GIPNet), the feature map Stem_Output is feature extracted through two parallel resolution sub-network branches to obtain feature maps SPD_1_S1 and feature maps SPD_2_S1, and the feature maps SPD_1_S1 and feature maps SPD_2_S1 are output to the first branch extension conversion layer.

[0056] Specifically, in the first stage (Stage 1) of the global information perception posture network (GIPNet), the feature map Stem_Output is feature extracted through two parallel resolution sub-network branches to obtain the feature map SPD_1_S1 and the feature map SPD_2_S1, as follows:

[0057] In the first resolution sub-network branch of the first stage (Stage1), the feature map Stem_Output is upsampled to obtain the feature map SPD_1_S1.

[0058] In the second resolution sub-network branch of the first stage (Stage1), the feature map Stem_Output is subjected to a reorganized pixel downsampling operation to obtain a feature map SPD_2_S1.

[0059] Step 13, the first branch expansion conversion layer includes two parallel permutation convolution block attention modules, which respectively convert the feature map SPD_1_S1 and the feature map SPD_2_S1 through two parallel permutation convolution block attention modules, and output the feature map SCBAM_1_S2 and the feature map SCBAM_2_S2 to the second stage (Stage 2) of the global information perception posture network (GIPNet).

[0060] Specifically, the feature map SPD_1_S1 is converted by using the first permutation convolution block attention module in the first branch expansion conversion layer, and the feature map SCBAM_1_S2 is output to the second stage (Stage2) of the global information perception posture network (GIPNet); the feature map SPD_2_S1 is converted by using the second permutation convolution block attention module in the first branch expansion conversion layer, and the feature map SCBAM_2_S2 is output to the second stage (Stage2) of the global information perception posture network (GIPNet).

[0061] Step 14, in the second stage (Stage 2) of the global information perception posture network (GIPNet), the feature map SCBAM_1_S2 and the feature map SCBAM_2_S2 are feature extracted through three parallel resolution sub-network branches to obtain feature maps Fuse_1_S2, feature maps Fuse_2_S2 and feature maps Fuse_3_S2, and the feature maps Fuse_1_S2, the feature maps Fuse_2_S2 and the feature maps Fuse_3_S2 are output to the second branch expansion conversion layer.

[0062] Specifically, in the second stage (Stage 2) of the global information perception posture network (GIPNet), the feature map SCBAM_1_S2 and the feature map SCBAM_2_S2 are feature extracted through three parallel resolution sub-network branches to obtain the feature map Fuse_1_S2, the feature map Fuse_2_S2 and the feature map Fuse_3_S2, as follows:

[0063] In the first resolution sub-network branch of the second stage (Stage2), upsampling and reorganized pixel downsampling operations are performed on the feature map SCBAM_1_S2, and the upsampling result of the feature map SCBAM_1_S2 is fused with the upsampling result of the feature map SCBAM_2_S2 to obtain the feature map Fuse_1_S2.

[0064] In the second resolution subnetwork branch of the second stage (Stage2), upsampling and reorganized pixel downsampling operations are performed on the feature map SCBAM_2_S2, and the reorganized pixel downsampling result of the feature map SCBAM_1_S2 is fused with the upsampling result of the feature map SCBAM_2_S2 to obtain the feature map Fuse_2_S2.

[0065] In the third resolution sub-network branch of the second stage (Stage2), the recombined pixel downsampling result of the feature map SCBAM_1_S2 and the recombined pixel downsampling result of the feature map SCBAM_2_S2 are fused to obtain the feature map Fuse_3_S2.

[0066] Step 15, the second branch expansion conversion layer includes three parallel permutation convolution block attention modules, which respectively convert the feature map Fuse_1_S2, the feature map Fuse_2_S2 and the feature map Fuse_3_S2 through the three parallel permutation convolution block attention modules, and output the feature map SCBAM_1_S3, the feature map SCBAM_2_S3 and the feature map SCBAM_3_S3 to the third stage (Stage 3) of the global information perception posture network (GIPNet).

[0067] Step 16. In the third stage (Stage 3) of the global information perception posture network (GIPNet), feature extraction is performed on the feature map SCBAM_1_S3, the feature map SCBAM_2_S3 and the feature map SCBAM_3_S3 through four parallel resolution sub-network branches to obtain feature maps Fuse_1_S3, feature maps Fuse_2_S3, feature maps Fuse_3_S3 and feature maps Fuse_4_S3, and the feature maps Fuse_1_S3, the feature maps Fuse_2_S3, the feature maps Fuse_3_S3 and the feature maps Fuse_4_S3 are output to the third branch expansion conversion layer.

[0068] Specifically, in the third stage (Stage 3) of the global information perception posture network (GIPNet), the feature map SCBAM_1_S3, the feature map SCBAM_2_S3 and the feature map SCBAM_3_S3 are feature extracted through four parallel resolution sub-network branches to obtain the feature map Fuse_1_S3, the feature map Fuse_2_S3, the feature map Fuse_3_S3 and the feature map Fuse_4_S3, as follows:

[0069] In the first resolution subnetwork branch of the third stage (Stage3), the feature map SCBAM_1_S3 is upsampled and the reorganized pixel downsampled, and the upsampling result of the feature map SCBAM_1_S3, the upsampling result of the feature map SCBAM_2_S3 and the upsampling result of the feature map SCBAM_3_S3 are fused to obtain the feature map Fuse_1_S3.

[0070] In the second resolution subnetwork branch of the third stage (Stage3), the feature map SCBAM_2_S3 is upsampled and the reorganized pixel downsampling operations are performed, and the reorganized pixel downsampling result of the feature map SCBAM_1_S3, the upsampling result of the feature map SCBAM_2_S3 and the upsampling result of the feature map SCBAM_3_S3 are fused to obtain the feature map Fuse_2_S3.

[0071] In the third resolution subnetwork branch of the third stage (Stage3), upsampling and reorganized pixel downsampling operations are performed on the feature map SCBAM_3_S3, and the reorganized pixel downsampling result of the feature map SCBAM_1_S3, the reorganized pixel downsampling result of the feature map SCBAM_2_S3 and the upsampling result of the feature map SCBAM_3_S3 are fused to obtain the feature map Fuse_3_S3.

[0072] In the fourth resolution subnetwork branch of the third stage (Stage3), upsampling and reorganized pixel downsampling operations are performed on the feature map SCBAM_3_S3, and the reorganized pixel downsampling results of the feature map SCBAM_1_S3, the reorganized pixel downsampling results of the feature map SCBAM_2_S3, and the reorganized pixel downsampling results of the feature map SCBAM_3_S3 are fused to obtain the feature map Fuse_4_S3.

[0073] Step 17, the fourth branch expansion conversion layer includes four parallel permutation convolution block attention modules, through which the feature map Fuse_1_S3, the feature map Fuse_2_S3, the feature map Fuse_3_S3 and the feature map Fuse_4_S3 are converted and processed respectively, and the feature map SCBAM_1_S4, the feature map SCBAM_2_S4, the feature map SCBAM_3_S4 and the feature map SCBAM_4_S4 are output to the fourth stage (Stage 4) of the global information perception posture network (GIPNet).

[0074] Step 18. In the fourth stage (Stage 4) of the global information perception posture network (GIPNet), feature extraction is performed on the feature map SCBAM_1_S4, the feature map SCBAM_2_S4, the feature map SCBAM_3_S4 and the feature map SCBAM_4_S4 through four parallel resolution sub-network branches, and fast Fourier convolution upsampling operations are performed on the feature map SCBAM_2_S4, the feature map SCBAM_3_S4 and the feature map SCBAM_4_S4 respectively, and feature fusion is performed to obtain feature maps Fuse_1_S4, feature maps Fuse_2_S4, feature maps Fuse_3_S4 and feature maps Fuse_4_S4.

[0075] Specifically, in the fourth stage (Stage 4) of the global information perception posture network (GIPNet), through four parallel resolution sub-network branches, the feature map SCBAM_1_S4, the feature map SCBAM_2_S4, the feature map SCBAM_3_S4 and the feature map SCBAM_4_S4 are feature extracted, and the feature map SCBAM_2_S4, the feature map SCBAM_3_S4 and the feature map SCBAM_4_S4 are respectively subjected to fast Fourier convolution upsampling operations, and through feature fusion, the feature map Fuse_1_S4, the feature map Fuse_2_S4, the feature map Fuse_3_S4 and the feature map Fuse_4_S4 are obtained. The process is as follows:

[0076] In the first resolution subnetwork branch of the fourth stage (Stage4), the feature map SCBAM_1_S4 is upsampled and pixel reorganized downsampled, and the upsampling result of the feature map SCBAM_1_S4, the fast Fourier convolution upsampling result of the feature map SCBAM_2_S4, the fast Fourier convolution upsampling result of the feature map SCBAM_3_S4 and the fast Fourier convolution upsampling result of the feature map SCBAM_4_S4 are fused to obtain the feature map Fuse_1_S4.

[0077] In the second resolution subnetwork branch of the fourth stage (Stage4), the feature map SCBAM_2_S4 is upsampled, reorganized pixel downsampling operations and fast Fourier convolution upsampling operations are performed, and the reorganized pixel downsampling results of the feature map SCBAM_1_S4, the upsampling results of the feature map SCBAM_2_S4, the upsampling results of the feature map SCBAM_3_S4 and the upsampling results of the feature map SCBAM_4_S4 are fused to obtain the feature map Fuse_2_S4.

[0078] In the third resolution subnetwork branch of the fourth stage (Stage4), the feature map SCBAM_3_S4 is upsampled, reorganized pixel downsampling operations and fast Fourier convolution upsampling operations are performed, and the reorganized pixel downsampling results of the feature map SCBAM_1_S4, the reorganized pixel downsampling results of the feature map SCBAM_2_S4, the upsampling results of the feature map SCBAM_3_S4 and the upsampling results of the feature map SCBAM_4_S4 are fused to obtain the feature map Fuse_3_S4.

[0079] In the fourth resolution subnetwork branch of the fourth stage (Stage4), the feature map SCBAM_4_S4 is upsampled and fast Fourier convolution upsampling operations are performed, and the recombined pixel downsampling results of the feature map SCBAM_1_S4, the recombined pixel downsampling results of the feature map SCBAM_2_S4, the recombined pixel downsampling results of the feature map SCBAM_3_S4 and the upsampling results of the feature map SCBAM_4_S4 are fused to obtain the feature map Fuse_4_S4.

[0080] Step 19: convert the feature map Fuse_1_S4, the feature map Fuse_2_S4, the feature map Fuse_3_S4 and the feature map Fuse_4_S4 to obtain a thermal map of a desired output size; and post-process the thermal map of the desired output size to output a human body posture estimation result.

[0081] It should be noted that in the HRNet network, information exchange between different stages is achieved through the branch expansion conversion layer; in order to enhance the feature extraction capability of the model and the utilization of global information, this embodiment 1 introduces a permutation convolution block attention module (SCBAM) in the branch expansion conversion layer between different stages; integrating SCBAM into the network aims to enhance the feature representation capability of the model at all levels and enhance the model's ability to control global information, which can not only enhance the network's attention to details, but also enhance the network's sensitivity to a wide range of contextual information, so that the model can more effectively adapt to complex and changeable scenes and posture challenges, thereby enabling GIPNet to achieve more accurate and robust human posture estimation performance.

[0082] In the permuted convolutional block attention module (SCBAM), the input feature map is evenly divided into several sub-feature groups based on the channel, and each sub-feature group includes several sub-feature maps; for each sub-feature group, it is equally divided into two parts with the same number of channels along the channel direction, and the attention mechanism of the channel dimension and the spatial dimension are applied to the two parts of the score in parallel, allowing the network to simultaneously consider the channel importance and spatial details of the feature map; then, the processed sub-feature maps are re-aggregated, and the mixing and exchange of feature information across groups is performed through the Channel Shuffle operation; SCBAM enables the network to pay more attention to spatial and channel features without significantly increasing the computational burden, significantly improving the model's ability to estimate occluded postures.

[0083] Specifically, as attached Figure 3 As shown, the working process of the permuted convolution fast attention module is as follows:

[0084] The input feature map F with the dimension C×H×W x Based on the average segmentation of the channel, several sub-feature groups are obtained Each sub-feature group includes several sub-feature graphs.

[0085] For each sub-feature group Divide the sub-feature group into two sub-parts with the same number of channels along the channel direction

[0086] Using the channel attention module, the channel dimension attention mechanism is applied to the first sub-part, and the feature map with the channel dimension attention mechanism is multiplied by the first sub-part to obtain the sub-feature map F s3 ; Among them, the process of applying the channel dimension attention mechanism to the first sub-part by using the channel attention module is specifically as follows: performing maximum pooling operation and average pooling operation on the first sub-part respectively to obtain the maximum pooled sub-part and the average pooled sub-part; using a multi-layer perceptron MLP to process the maximum pooled sub-part and the average pooled sub-part respectively, and then adding them, and processing them through an activation function to obtain a feature map applying the channel dimension attention mechanism.

[0087] Using the spatial attention module, the spatial dimension attention mechanism is applied to the second sub-part, and the feature map with the spatial dimension attention mechanism is multiplied by the second sub-part to obtain the sub-feature map F s4 ; The process of applying the spatial dimension attention mechanism to the second sub-part by using the spatial attention module is specifically as follows: performing a maximum pooling operation or an average pooling operation on the second sub-part to obtain a pooled sub-part; performing convolution and activation function processing on the pooled sub-part to obtain a feature map applying the spatial dimension attention mechanism.

[0088] For the sub-feature map F s3 and the sub-feature map F s4 Perform feature concatenation to obtain the sub-feature graph F s5 ; Perform sub-feature graph F on all sub-feature groups s5 After aggregation, the Channel Shuffle operation is used to mix and exchange feature information across groups, and the output is a feature map processed by the permuted convolution block attention module.

[0089] It should also be noted that in the task of human posture estimation, the amount of information obtained by the model in the feature extraction stage is crucial, which directly affects the accuracy of key point positioning; by introducing the attention mechanism in the permuted convolutional block attention module (SCBAM), the model's ability to capture long-distance dependencies can be significantly improved, which is beneficial for the model to understand global contextual information; secondly, by learning and strengthening the key parts of the input data and applying the attention mechanism to assign higher weights to key information, the model can focus on processing the information that is most important for the current task, so as to enhance the model's feature representation capabilities at all levels and its control over global information.

[0090] Secondly, in order to improve the accuracy of the model in the deep feature sampling process, in the final stage of GIPNet, i.e., the fourth stage (stage 4), when the last three layers of feature maps with a resolution of 1 / 8, 1 / 16, and 1 / 32 of the original image are upsampled by 2 times, 4 times, and 8 times, fast Fourier convolution upsampling (FFCU) is used in this embodiment 1 to optimize the traditional interpolation-based upsampling method used by the HRNet network; wherein, fast Fourier convolution upsampling (FFCU) is based on fast Fourier convolution (FFC) to restore the spatial resolution of the image, while retaining the rich high-frequency information in the image, which can give full play to the potential of fast Fourier convolution and show significant advantages in image processing; fast Fourier convolution upsampling (FFCU) not only effectively expands the receptive field of the model, but also retains the detail information in the image; by introducing fast Fourier convolution upsampling (FFCU), it not only improves the accuracy of the upsampling process, but also enhances the model's recognition ability of occluded key points and cross key points, which is particularly critical for enhancing the model's ability to process human posture estimation in complex scenes.

[0091] In this embodiment 1, the core idea of ​​fast Fourier convolution upsampling (FFCU) is to use fast Fourier convolution FFC and pixel reorganization Pixel Shuffle to achieve efficient conversion from a smaller-sized feature map to a larger-sized feature map; specifically, FFC is first used to increase the number of channels of the feature map, and then the newly added channels are used to compensate for the reduction in length and width of the image, thereby maintaining the dimensionality and semantic richness of the feature map without losing too much spatial information.

[0092] Specifically, the process of the fast Fourier convolution upsampling operation is implemented in the following steps:

[0093] For the input feature map F with dimension c×h×w f ∈R c×h×w Perform fast Fourier convolution operation to obtain the feature map F f1 ∈R (d*d*c)×h×w ; For the feature map F f1 ∈R (d*d*c)×h×w Perform pixel reorganization PixelShuffle operation to obtain feature map F f1 ∈R( d*d*c)×(d*h)×w ; Where d is the upsampling factor in the pixel reorganization PixelShuffle operation.

[0094] It should be noted that in computer vision tasks, especially tasks such as human pose estimation that are highly sensitive to position information, high-resolution feature maps are very important because they can provide more accurate spatial information. Usually, the model will obtain feature representations with strong semantic information through downsampling, but this will lead to a decrease in the clarity of the feature map and the loss of some accurate spatial details. For this reason, high-resolution features are reconstructed from lower-resolution features through upsampling operations to restore information in the spatial dimension, allowing the model to perform more detailed spatial analysis.

[0095] In the HRNet network, an interpolation upsampling method is used to achieve the fusion of low-resolution feature maps to high-resolution feature maps; however, the interpolation upsampling method does not involve a learning process, and some important detail information may be lost during the upsampling process; in this embodiment 1, the fast Fourier convolution upsampling (FFCU) combines the advantages of the fast Fourier convolution FFC and the PixelShuffle upsampling technology, by fusing the Pixel Shuffle has the advantage of maintaining the richness of feature map details and using the ability of fast Fourier convolution FFC to expand the receptive field, which not only improves the upsampling efficiency of feature maps, but also effectively maintains and enhances the detail information of the image; therefore, fast Fourier convolution upsampling (FFCU) provides a more accurate and efficient way to restore high-resolution features; different from traditional zero-padding-based transposed convolution upsampling or non-learnable interpolation upsampling methods, fast Fourier convolution upsampling (FFCU) adjusts the number of channels by performing fast Fourier convolution operations on low-resolution feature maps, and then rearranges the pixels of the channel dimension to the spatial dimension to achieve high-quality and high-resolution feature recovery, which not only generates high-resolution feature maps, but also can predict details pixel by pixel, significantly expands the range of the receptive field, and effectively enriches the detail information of the feature maps.

[0096] In addition, in order to reduce the number of model parameters while maintaining the integrity of information and network performance, GIPNet uses reorganized pixel downsampling (SPD) to replace the original downsampling operation based on 3×3 convolution with a step size of 2 in the HRNet network. The introduction of reorganized pixel downsampling (SPD) not only reduces the number of model parameters, but also effectively retains the important spatial information of the image through the reorganized pixel operation, thereby improving the computational efficiency of the model, and can achieve more efficient feature extraction while maintaining the network representation capability.

[0097] Based on the PixelShuffle operation, SPD aims to retain a larger receptive field and reduce the number of model parameters while downsampling. SPD can retain a larger range of spatial information while reducing the size of the feature map by reorganizing pixels, thereby maintaining a larger effective receptive field during the downsampling process.

[0098] Specifically, as attached Figure 4 As shown, the process of recombined pixel down sampling (SPD) is implemented in the following steps:

[0099] For the input feature map F with dimension a×a×b spd0 ∈R a×a×b Perform the Shuffle Pixel operation to obtain the reorganized feature map F spd1; The reorganized feature map F spd1 Apply 4 1×1 convolution kernels to the reorganized feature map F spd1 Perform size transformation to obtain feature map F spd2 .

[0100] The following is an example of the processing of an input feature map with a dimension of 4×4×3.

[0101] First, when the input feature map of dimension 4×4×3 is downsampled by 2 times, the downsampling factor df is set to 2; then, the input feature map of dimension 4×4×3 is subjected to the Shuffle Pixel operation to obtain a feature map of dimension 2×2×12; wherein, the Shuffle Pixel operation reduces the spatial size of the feature map while increasing the channel depth by dividing the image in the spatial dimension, thereby retaining more spatial information; then, four 1×1 convolution kernels are applied to the feature map of dimension 2×2×12, and the reorganized feature map is further transformed into a size of 2×2×4; wherein, the 1×1 convolution kernel has the same receptive field as the 3×3 convolution kernel, which not only helps in feature extraction, but also significantly reduces the number of parameters in the network, achieving the dual purpose of optimizing performance and computational efficiency.

[0102] Experimental results and analysis:

[0103] (1) Experimental Dataset

[0104] In this embodiment 1, the OCHuman dataset is selected to verify the performance of the global information perception posture network (GIPNet).

[0105] (2) Quantitative comparison experimental results:

[0106] The experimental results of OCHumanval set are shown in Table 1. Table 1 shows the performance of GIPNet and several other advanced models in posture estimation on the OCHuman dataset. As can be seen from Table 1 below, GIPNet achieves a higher AP value, indicating that GIPNet can improve the detection accuracy under occlusion. Compared with HRNet, GIPNet has achieved a significant improvement of 2.7 percentage points in AP value, which highlights the effectiveness of GIPNet in enhancing the model's global information perception ability, especially when dealing with occlusion problems. Compared with DAG, a comprehensive framework that combines data, attention and graphs, GIPNet has improved its AP value by 1.3 percentage points. Although DAG is designed to solve the performance degradation caused by occlusion, its ability in global information perception is limited, so it is not as efficient as GIPNet when dealing with occlusion problems.

[0107] Table 1 Experimental results of different human pose estimation models on OCHumanval set

[0108]

[0109] (3) Qualitative comparison experimental results:

[0110] In order to intuitively verify the ability of GIPNet in perceiving global information, the GIPNet model and the HRNet model were used to infer the same image, and the feature maps output by the two models at the same network level were visually compared; as shown in the attached figure. Figure 5 As shown, attached Figure 5 The feature map visualization result diagram of HRNet and GIPNet at the same network level is given in Figure 1, which specifically shows the intermediate feature maps generated by GIPNet and HRNet during the reasoning process of the same image. Figure 5 It is divided into two parts, the first row of each part shows the feature map generated by the HRNet model, and the second row shows the corresponding feature map generated by the GIPNet model; Figure 5 It can be observed that in the shallow layers of the network (stage1, stage2), since the image is only processed by a small number of convolutional layers, the resolution of the feature map obtained is high, and it mainly captures some fine-grained information such as edges, corners, and textures, retaining rich location information and details, but relatively lacking sufficient semantic information. As the network depth increases, the receptive field gradually expands, and the network can capture global semantic information closer to the output; at the network levels of stage3 and stage4, the visualization results show that the feature map generated by GIPNet is more significant for the characteristics of the target human body and can better highlight the semantic information of key parts. Therefore, GIPNet shows better performance in deep feature extraction, which helps to improve the positioning and recognition of key points of the human body.

[0111] The posture estimation method described in this embodiment 1 uses a global information-aware posture network (GIPNet) to enhance the model's perception and understanding of global information, thereby improving the accuracy of posture estimation; specifically, GIPNet integrates three core components: 1) a permuted convolutional block attention module (SCBAM), which combines channel and spatial attention mechanisms to help the network efficiently capture long-distance global dependencies; 2) a fast Fourier convolution upsampling (FFCU), which finely restores high-resolution feature maps by refining the model's ability to recognize details, thereby further improving the accuracy of key point prediction; 3) a reorganized pixel downsampling (SPD), which reduces the amount of model parameters by reorganizing pixels while maintaining the model's efficient feature extraction capabilities; experimental verification results show that GIPNet achieves state-of-the-art performance on the OCHuman dataset, fully demonstrating its robustness to crowded and occluded scenes.

[0112] Example 2

[0113] As attached Figure 6 As shown, this embodiment 2 provides a posture estimation system based on global information perception, including an image acquisition module and a posture estimation module. The image acquisition module is used to acquire the original image to be estimated; the posture estimation module is used to input the original image to be estimated into a pre-constructed global information perception posture network for human body posture estimation, and output the human body posture estimation result; wherein the pre-constructed global information perception posture network is an improved HRNet network; specifically, the improved HRNet network is: introducing a parallel permutation convolution block attention module in the branch expansion conversion layer of the HRNet network, introducing a fast Fourier convolution upsampling operation in the fourth stage of the HRNet network, and replacing the downsampling operation based on 3×3 convolution with a step size of 2 in the HRNet network with a reorganized pixel downsampling operation.

[0114] Example 3

[0115] As attached Figure 7 As shown, this embodiment 3 provides a posture estimation device based on global information perception, including: a memory for storing a computer program; a processor for implementing the steps of a posture estimation method based on global information perception when executing the computer program or for implementing the functions of each module in the above-mentioned posture estimation system based on global information perception when executing the computer program.

[0116] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing preset functions, and the instruction segments are used to describe the execution process of the computer program in the posture estimation device based on global information perception.

[0117] The posture estimation device based on global information perception can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The posture estimation device based on global information perception may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above is an example of a posture estimation device based on global information perception, and does not constitute a limitation on the posture estimation device based on global information perception, and may include more components than the above, or a combination of certain components, or different components. For example, the posture estimation device based on global information perception may also include input and output devices, network access devices, buses, etc.

[0118] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the posture estimation device based on global information perception, and various interfaces and lines are used to connect the various parts of the posture estimation device based on global information perception.

[0119] The memory can be used to store the computer program and / or module, and the processor implements various functions of the posture estimation device based on global information perception by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory.

[0120] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0121] Example 4

[0122] This embodiment 4 also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the posture estimation method based on global information perception are implemented.

[0123] If the module / unit integrated in the posture estimation system based on global information perception is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0124] Based on such understanding, the present invention implements all or part of the processes in the above-mentioned posture estimation method based on global information perception, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned posture estimation method based on global information perception when executed by a processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or preset intermediate form, etc.

[0125] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0126] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of patent practice. For example, according to patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0127] Example 5

[0128] This embodiment 5 provides a computer product, which computer program product includes a computer program, which is stored in a computer-readable storage medium; the processor of the posture estimation device based on global information perception reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the posture estimation device based on global information perception can execute the posture estimation method based on global information perception described in Embodiment 1, which will not be repeated here.

[0129] It should be noted that a person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0130] In the present invention, the accuracy of the model in key point positioning is significantly improved by expanding the receptive field of the model and reducing the loss of spatial details in the feature map; specifically, by introducing a parallel permuted convolution block attention module, the spatial attention and channel attention are effectively combined, thereby helping the network model to capture the global correlation in the image more carefully; secondly, through fast Fourier convolution upsampling (FFCU), the high-resolution feature map can be restored in a more refined way to more accurately predict the key point position; in addition, a reorganized pixel downsampling (SPD) method is proposed, which can maintain efficient feature extraction capabilities while reducing the amount of model parameters.

[0131] The posture estimation method based on global information perception described in the present invention improves the accuracy of key point detection of the human body by enhancing the model's ability to perceive and utilize global information, and can more accurately locate and identify key points of the human body when dealing with occlusion and crowded environments; in intensive detection tasks such as human posture estimation, the results of key point detection are highly dependent on high-level semantic information. High-level features can capture the global information of the human body, and these features usually contain richer semantic information, which helps to distinguish different human body parts and postures.

[0132] It should be noted that in multi-person scenes, human body parts may be occluded or confused with the background. Due to their strong abstractness, high-level features help to distinguish and associate the postures of different people and improve the accuracy of posture estimation. Expanding the receptive field is crucial to obtaining high-level features because these features need to cover a wider range of global contextual information. A larger receptive field enables the model to learn long-distance dependencies, thereby enhancing its adaptability to different scenes and changes and improving generalization capabilities; in view of this, the present invention proposes a global information-aware posture network (GIPNet). The core of GIPNet is to expand the receptive field of the model while minimizing the loss of spatial details during feature extraction, thereby significantly improving the model's understanding and perception of global contextual information; GIPNet can more effectively capture the human key point information in the image, thereby achieving higher key point positioning accuracy when performing human posture estimation tasks; wherein, the key points are represented by heat map vectors.

[0133] Specifically, the permuted convolutional block attention module (SCBAM) is introduced into the global information perception posture network (GIPNet). Through parallel processing and combining the attention mechanism of spatial dimension and channel dimension, the network model can capture the global correlation in the image more carefully, thereby significantly improving the accuracy of key point recognition. Among them, by introducing the attention mechanism, the model can automatically adjust the size and structure of the receptive field, focusing more on the key information in the input data, thereby improving performance and generalization ability. Especially when dealing with complex scenes, the spatial perception advantage brought by the introduction of the attention mechanism is particularly obvious.

[0134] Secondly, in the task of human pose estimation, upsampling technology is crucial. It not only improves the resolution of feature maps, but also enhances the model's ability to accurately locate key points in the image. Most pose estimation models tend to use interpolation upsampling methods to achieve fast processing, but this upsampling method does not have learning capabilities and may lose detail information during the upsampling process. In order to overcome this limitation, a fast Fourier convolution upsampling (FFCU) is proposed to provide a more accurate method to restore high-resolution feature maps, thereby better predicting the exact location of key points.

[0135] In addition, in order to maintain efficient feature extraction while reducing the number of model parameters, a restructured pixel downsampling (SPD) is proposed as an alternative to the traditional 3×3 convolution downsampling method with a step size of 2. It not only reduces the complexity of the model, but more importantly, it effectively maintains the key spatial information of the image while improving computational efficiency.

[0136] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention.

Claims

1. A posture estimation method based on global information perception, characterized in that: include: Obtaining the original image to be estimated; Inputting the original image to be estimated into a pre-built global information perception posture network to perform human posture estimation, and outputting a human posture estimation result; Among them, the pre-constructed global information perception posture network is an improved HRNet network; specifically, the improved HRNet network is: introducing a parallel permutation convolution block attention module in the branch expansion conversion layer of the HRNet network, introducing a fast Fourier convolution upsampling operation in the fourth stage of the HRNet network, and replacing the downsampling operation based on 3×3 convolution with a step size of 2 in the HRNet network with a reorganized pixel downsampling operation.

2. A posture estimation method based on global information perception according to claim 1, characterized in that: The spatial dimension attention mechanism and the channel dimension attention mechanism are introduced into the permutation convolution block attention module to capture the global correlation in the image; the fast Fourier convolution upsampling operation is used to utilize fast Fourier convolution and pixel reorganization to achieve efficient conversion from a smaller-sized feature map to a larger-sized feature map; the reorganized pixel downsampling operation is used to retain spatial information while reducing the size of the feature map by reorganizing pixels.

3. A posture estimation method based on global information perception according to claim 2, characterized in that: The working process of the permuted convolution fast attention module is as follows: The input feature map F with the dimension C×H×W x Based on the average segmentation of the channel, several sub-feature groups are obtained Wherein, each sub-feature group includes several sub-feature graphs; For each sub-feature group Divide the sub-feature group into two sub-parts with the same number of channels along the channel direction Using the channel attention module, the channel dimension attention mechanism is applied to the first sub-part, and the feature map with the channel dimension attention mechanism is multiplied by the first sub-part to obtain the sub-feature map F s3 ; Using the spatial attention module, the spatial dimension attention mechanism is applied to the second sub-part, and the feature map with the spatial dimension attention mechanism is multiplied by the second sub-part to obtain the sub-feature map F s4 ; For the sub-feature map F s3 and the sub-feature map F s4 Perform feature concatenation to obtain the sub-feature graph F s5 ; Perform sub-feature graph F on all sub-feature groups s5 After aggregation, the Channel Shuffle operation is used to mix and exchange feature information across groups, and the output is a feature map processed by the permuted convolution block attention module.

4. A posture estimation method based on global information perception according to claim 3, characterized in that: Using the channel attention module, the process of applying the channel dimension attention mechanism to the first sub-part is as follows: Performing a maximum pooling operation and an average pooling operation on the first sub-part respectively to obtain a maximum pooled sub-part and an average pooled sub-part; using a multi-layer perceptron MLP to process the maximum pooled sub-part and the average pooled sub-part respectively, adding them together, and processing them through an activation function to obtain a feature map applying a channel dimension attention mechanism; Using the spatial attention module, the process of applying the attention mechanism of the spatial dimension to the second sub-part is as follows: Perform a maximum pooling operation or an average pooling operation on the second sub-part to obtain a pooled sub-part; perform convolution and activation function processing on the pooled sub-part to obtain a feature map applying a spatial dimension attention mechanism.

5. The method for posture estimation based on global information perception according to claim 2, characterized in that: The process of the fast Fourier convolution upsampling operation is as follows: For the input feature map F with dimension c×h×w f ∈R c×h×w Perform fast Fourier convolution operation to obtain the feature map F f1 ∈R (d*d*c)×h×w ; For the feature map F f1 ∈R (d*d*c)×h×w Perform pixel reorganization PixelShuffle operation to obtain feature map F f1 ∈R (d *d*c)×(d*h)×w ; Where d is the upsampling factor in the PixelShuffle operation.

6. A posture estimation method based on global information perception according to claim 2, characterized in that: The process of the reorganized pixel downsampling operation is as follows: For the input feature map F with dimension a×a×b spd0 ∈R a×a×b Perform the Shuffle Pixel operation to obtain the reorganized feature map F spd1 ; The reorganized feature map F spd1 Apply 4 1×1 convolution kernels to the reorganized feature map F spd1 Perform size transformation to obtain feature map F spd2 .

7. A posture estimation system based on global information perception, characterized in that: include: An image acquisition module, used for acquiring an original image to be estimated; A posture estimation module is used to input the original image to be estimated into a pre-built global information perception posture network to perform human posture estimation, and output a human posture estimation result; Among them, the pre-constructed global information perception posture network is an improved HRNet network; specifically, the improved HRNet network is: introducing a parallel permutation convolution block attention module in the branch expansion conversion layer of the HRNet network, introducing a fast Fourier convolution upsampling operation in the fourth stage of the HRNet network, and replacing the downsampling operation based on 3×3 convolution with a step size of 2 in the HRNet network with a reorganized pixel downsampling operation.

8. A posture estimation device based on global information perception, characterized in that: include: a processor suitable for executing a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the posture estimation method based on global information perception as described in any one of claims 1 to 6 is executed.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the posture estimation method based on global information perception is implemented as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method for posture estimation based on global information perception is implemented as described in any one of claims 1 to 6.

Images