Multi-scale fusion human body posture estimation method based on improved HRNet

Abstract

The invention relates to the technical field of human body posture estimation, in particular to a multi-scale fusion human body posture estimation method based on an improved HRNet, and the method is specifically as follows: the method is based on an HRNet fusion encoder and decoder architecture, the encoder is designed into three stages, feature branches with different resolutions are generated in parallel in each stage, and the feature branches with different resolutions are generated in parallel; an improved lightweight module MBConv is introduced into each characteristic branch; the corresponding decoder and encoder are of a symmetrical structure and are also designed into three stages, low-resolution branches are gradually reduced until the third stage only retains a high-resolution feature map for accurately predicting the position of a joint point; constructing jump connection between the encoder and the decoder; at the end of each stage of an encoder and a decoder, a repeated multi-scale feature fusion strategy is implemented. According to the method, by improving the network structure of the HRNet, the spatial relation between the human body joint points can be better captured, meanwhile, it is guaranteed that the fine-grained features are not lost, and therefore the performance of human body posture estimation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human body posture estimation, and in particular to a multi-scale fusion human body posture estimation method based on improved HRNet. Background Art

[0002] The goal of human posture estimation is to accurately locate the joints of the human body. It has many applications, such as human behavior recognition, human-computer interaction, and animation production. It is a very challenging problem in the field of computer vision.

[0003] Some representative works to solve the problem of human posture estimation are to perform step-by-step downsampling through strided convolution or pooling layers during the encoding process to generate low-resolution representations to extract high-level semantic information; during the decoding process, the low-resolution feature maps are gradually restored to high-resolution feature maps through transposed convolution or bilinear interpolation upsampling to obtain high-resolution representations. There are obvious drawbacks in this process. In the process of downsampling and then restoring to high resolution, more spatial information in the length and width directions is often lost. This is very unfavorable for dense prediction tasks such as posture estimation that are pixel-dense and position-sensitive.

[0004] In addition, in terms of feature fusion, most of them connect the feature maps of the same scale of the encoder and decoder through the jump connection mechanism, helping the model to fuse shallow information with deep information to achieve effective fusion of coarse-grained features and fine-grained features. However, this process is limited in improving the representation ability of high-resolution feature maps, which is crucial for dense prediction tasks such as human posture estimation.

[0005] Therefore, it is necessary to further explore and study new human pose estimation methods to enhance the spatial representation capability of high-resolution feature maps and enrich the semantic information of high-resolution feature maps. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention develops a multi-scale fusion human posture estimation method based on improved HRNet, which can significantly improve the positioning accuracy of human joints, thereby improving the performance of human posture estimation.

[0007] The technical solution of the present invention to solve the technical problem is: A multi-scale fusion human posture estimation method based on improved HRNet includes the following steps: S1. Obtaining human pose estimation dataset , Represents the human pose estimation dataset The pictures, , for the dataset Perform data preprocessing to obtain the preprocessed human posture estimation data set ; S2. Construct an improved HRNet network structure, and transform the first Pictures Input into the initial stage of the improved HRNet network structure to obtain a high-resolution feature map ; S3. In the first stage of the encoder of the improved HRNet network structure, the high-resolution feature map After the Transition1 module, a high-resolution feature map is obtained and medium resolution feature maps , the feature map and After the StageModule1 module, a high-resolution feature map is obtained and medium resolution feature maps ; S4. In the second stage of the encoder of the improved HRNet network structure, the high-resolution feature map and medium resolution feature maps After the Transition2 module, a high-resolution feature map is obtained , medium resolution feature map and low-resolution feature maps , the feature map , and After two StageModule2 modules, a high-resolution feature map is obtained , medium resolution feature map and low-resolution feature maps ; S5. In the third stage of the encoder of the improved HRNet network structure, the high-resolution feature map , medium resolution feature map and low-resolution feature maps After the Transition3 module, a high-resolution feature map is obtained , medium resolution feature map , low-resolution feature map and small resolution feature maps , the feature map , , and After the StageModule3 module, a high-resolution feature map is obtained and medium resolution feature maps ; S6. In the first stage of the decoder of the improved HRNet network structure, the high-resolution feature map and medium resolution feature maps The high-resolution feature map is obtained through the multi-scale fusion module Fusion4 , medium resolution feature map and low-resolution feature maps , the feature map , and After two StageModule2 modules, a high-resolution feature map is obtained , medium resolution feature map and low-resolution feature maps , the high-resolution feature map and A high-resolution feature map is obtained by adding the skip connection , the medium resolution feature map and The medium-resolution feature map is obtained by adding them through skip connections. , the low-resolution feature map and The low-resolution feature map is obtained by adding the skip connection ; S7. In the second stage of the decoder of the improved HRNet network structure, the high-resolution feature map , medium resolution feature map and low-resolution feature maps The high-resolution feature map is obtained through the multi-scale fusion module Fusion5 and medium resolution feature maps , the feature map , After the StageModule1 module, a high-resolution feature map is obtained and medium resolution feature maps , the high-resolution feature map and A high-resolution feature map is obtained by adding the skip connection , the medium resolution feature map and The medium-resolution feature map is obtained by adding the skip connection ; S8. In the third stage of the decoder of the improved HRNet network structure, the high-resolution feature map and medium resolution feature maps After the multi-scale fusion module Fusion6, a high-resolution feature map is obtained ; S9. The high-resolution feature map Input into the detection head and get the output feature map , output feature map Each channel of is used to represent the position of each joint point, and finally the human body posture estimation result is obtained.

[0008] Furthermore, the training set obtained after preprocessing in step S1 And the preprocessed validation set The specific ones include: Human pose estimation dataset All sample images included in the dataset are uniformly scaled, and then data enhancement strategies are performed, including random horizontal flipping, random scaling and rotation of the human body bounding box, and affine transformation, to obtain the preprocessed human posture estimation dataset. ; Human pose estimation dataset after using data enhancement strategy Generate labels for training. The specific operation is as follows: centering on the position of each human joint in the image, use a 2D Gaussian kernel function with a standard deviation of 1 pixel to generate a true value heat map as the label.

[0009] Furthermore, the step S3 specifically includes: The Transition1 module of the first stage of the encoder contains two branches: and , branch It includes a convolution layer with a kernel size of 3x3, a stride of 1, and a padding of 1, a batch normalization layer, a ReLU activation function, and a branch It also includes a convolutional layer with a kernel size of 3x3, a stride of 1, and a padding of 1, a batch normalization layer, and a ReLU activation function; high-resolution feature maps Through the branch After mapping, a high-resolution feature map is obtained , through the branch After mapping, we get a medium-resolution feature map ; The StageModule1 module of the first stage of the encoder includes a feature extraction module Extract1 and a multi-scale fusion module Fusion1. The feature extraction module Extract1 includes two branches: and , feature map Through the branch After mapping, a high-resolution feature map is obtained , feature map Through the branch After mapping, a medium-resolution feature map is obtained ; In the multi-scale fusion module Fusion1, a high-resolution feature map is obtained. and medium resolution feature maps The mapping formula is: , , in, represents the identity mapping, It means that the 2x upsampling module of the high-resolution feature map is obtained. It represents a 2x downsampling module for obtaining a medium-resolution feature map.

[0010] Furthermore, the step S4 specifically includes: The Transition2 module of the second stage of the encoder contains three branches: , and , branch are all identity mappings, branches Includes convolutional layers with kernel size 3x3, stride 2, padding 1, batch normalization layers, and ReLU activation functions; high-resolution feature maps Through the branch After mapping, a high-resolution feature map is obtained , medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , medium resolution feature map Through the branch After mapping, we get a low-resolution feature map ; The StageModule2 module of the second stage of the encoder includes a feature extraction module Extract2 and a multi-scale feature fusion module Fusion2. The feature extraction module Extract2 includes three branches: , and , high-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , medium resolution feature map Through the branch After mapping, we get the medium resolution feature map , low-resolution feature map Through the branch After mapping, we get a low-resolution feature map ; In the multi-scale fusion module Fusion2, a high-resolution feature map is obtained. , medium resolution feature map and low-resolution feature maps The mapping formula is: , , , in, represents the identity mapping, It means that the 4-fold upsampling module of the high-resolution feature map is obtained. Indicates a 2x upsampling module for medium-resolution feature maps. It means that the 4-fold downsampling module of the low-resolution feature map is obtained. Represents a 2x downsampling module for obtaining a low-resolution feature map.

[0011] Furthermore, the step S5 specifically includes: The Transition3 module of the third stage of the encoder contains four branches: , , and , branch are all identity mappings, branches Includes convolutional layers with kernel size 3x3, stride 2, padding 1, batch normalization layers, and ReLU activation functions; high-resolution feature maps Through the branch After mapping, a high-resolution feature map is obtained , medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , low-resolution feature map Through the branch After mapping, we get a low-resolution feature map , the low-resolution feature map is branched After mapping, a small resolution feature map is obtained ; The StageModule3 module of the third stage of the encoder includes a feature extraction module Extract3 and a multi-scale feature fusion module Fusion3. The feature extraction module Extract3 includes four branches: , , and , resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , low-resolution feature map Through the branch After mapping, we get a low-resolution feature map , small resolution feature map Through the branch After mapping, a small resolution feature map is obtained ; In the multi-scale fusion module Fusion3, a high-resolution feature map is obtained. and medium resolution feature maps The mapping formula is: , , in, Indicates an 8-fold upsampling module for obtaining a high-resolution feature map. Indicates a 4x upsampling module for obtaining a medium-resolution feature map.

[0012] Furthermore, in the multi-scale fusion module Fusion4 in step S6, a high-resolution feature map is obtained. , medium resolution feature map and low-resolution feature maps The mapping formula is: , , .

[0013] Furthermore, in the multi-scale fusion module Fusion5 in step S7, a high-resolution feature map is obtained. and medium resolution feature maps The mapping formula is: , .

[0014] Furthermore, in the multi-scale fusion module Fusion6 of step S8, a high-resolution feature map is obtained. The mapping formula is: .

[0015] Furthermore, the branch in the feature extraction module Extract1 and , branches in the feature extraction module Extract2 , and And the branch in the feature extraction module Extract3 , , and , each comprising 4 improved MBConv modules, wherein the improved MBConv module comprises 5 consecutive parts: Part 1: Convolutional layer with kernel size of 1x1, stride of 1, padding of 0, batch normalization layer, SiLU activation function; Part 2: Depthwise separable convolution with kernel size 3x3, stride 1, padding 1, batch normalization layer, SiLU activation function; Part 3: SE module: It first undergoes an average pooling operation, and then performs two fully connected layer operations. The first fully connected layer uses the SiLU activation function, and the second fully connected layer uses the Sigmoid activation function. Part 4: Convolutional layer with kernel size of 1x1, stride of 1, padding of 0, and batch normalization layer; Part 5: Perform the DropPath operation, then make a residual connection between the input feature map and the output feature map in Part 4, and finally obtain the output feature map of the improved MBConv module.

[0016] The present invention also proposes a computing device, comprising a memory configured to store computer-executable instructions; and a processor configured to execute a multi-scale fusion human posture estimation method based on an improved HRNet when the computer-executable instructions are executed by the processor.

[0017] In general, the above technical solutions conceived by the present invention have the following beneficial effects: A multi-scale fusion human posture estimation method based on improved HRNet of the present invention mainly consists of two parts: an encoder and a decoder. The encoder is divided into three stages, each stage generates a low-resolution branch to realize parallel connection of branches with different resolutions, and the use of an improved lightweight module MBConv in each branch can further improve the feature extraction capability of the encoder, and a multi-scale fusion module is used at the tail of each stage to complete repeated multi-scale feature fusion, which can enrich the semantic information of feature maps with different resolutions; the decoder is divided into three stages, which is symmetrical with the encoder, and each stage eliminates a low-resolution branch, so that the generated high-resolution feature map can retain more detail information; between the encoder and the decoder, a jump connection is introduced to connect the features of the corresponding layers of the encoder and the decoder, thereby increasing the feature map detail information and improving the accuracy of joint point prediction. The method of the present invention can better capture the spatial relationship between human joints, while ensuring that fine-grained features are not lost, thereby improving the performance of human posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0019] Figure 1 It is a network flow chart of the present invention.

[0020] Figure 2 This is the improved MBConv module of the present invention.

[0021] Figure 3 It is the multi-scale fusion-upsampling module of the present invention.

[0022] Figure 4 It is the multi-scale fusion-downsampling module of the present invention.

[0023] Figure 5 It is the human body posture estimation result of the present invention. DETAILED DESCRIPTION

[0024] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below.

[0025] Example 1 In this embodiment, Figure 1 As shown in FIG. 1 , a multi-scale fusion human posture estimation method based on an improved HRNet includes the following steps: a) Obtaining a human pose estimation dataset , Indicates pictures, , for the dataset Perform data preprocessing to obtain the preprocessed data set .

[0026] a-1) Human pose estimation dataset All sample images contained in are uniformly scaled, and then data augmentation strategies are performed, including: random horizontal flipping, random scaling and rotation of human bounding boxes, and affine transformation.

[0027] a-2) For the human pose estimation dataset after using the data enhancement strategy , generate labels for training. The specific operation is: taking the position of each human joint in the picture as the center, use a 2D Gaussian kernel function with a standard deviation of 1 pixel to generate a true value heat map as the label.

[0028] b) Construct an improved HRNet network structure and convert the preprocessed human posture estimation dataset The pictures, Input into the initial stage of the network structure to obtain a high-resolution feature map .

[0029] b-1) Input sample image The size is , after mapping Then we get the feature map , whose size is , feature map After mapping , get the feature map , the size is , mapping It includes convolution layer, batch normalization layer, and ReLU activation function. The convolution kernel size of the convolution layer is 3x3, the step size is 2, and the padding is 1.

[0030] b-2) Feature map After passing through the BottleNeck module with a downsampling layer, the feature map is obtained , whose size is . BottleNeck module with downsampling, for the input feature map , after mapping , the size of the feature map is ; After mapping , the size of the feature map is ; After mapping , get the feature map , the size is . ; Perform a downsampling operation to obtain the feature map , the size is . Then pass the function: Get feature map , the size is . The mapping It includes a convolutional layer with a kernel size of 1x1, a stride of 1, and a padding of 0, a batch normalization layer, and a ReLU activation function. It includes a convolutional layer with a kernel size of 3x3, a stride of 1, and a padding of 1, a batch normalization layer, and a ReLU activation function. It includes convolutional layers with kernel size of 1x1, stride of 1, padding of 0, and batch normalization layers. The downsampling operation includes convolutional layers with kernel size of 1x1, stride of 1, padding of 0, and batch normalization layers.

[0031] b-3) Feature map After 3 BottleNeck modules without downsampling layers, a high-resolution feature map is obtained , whose size is BottleNeck module without downsampling, for the input feature map , after mapping , the size of the feature map is ; After mapping , the size of the feature map is ; After mapping , get the feature map , the size is . Then through the function: Get high-resolution feature map , the size is . The mapping It includes a convolutional layer with a kernel size of 1x1, a stride of 1, and a padding of 0, a batch normalization layer, and a ReLU activation function. It includes a convolutional layer with a kernel size of 3x3, a stride of 1, and a padding of 1, a batch normalization layer, and a ReLU activation function. It includes convolutional layers with a kernel size of 1x1, a stride of 1, and padding of 0, and a batch normalization layer.

[0032] c) In the first stage of the encoder, the high-resolution feature map After the Transition1 module, a high-resolution feature map is obtained , medium resolution feature map ; The feature map , After the StageModule1 module, a high-resolution feature map is obtained , medium resolution feature map .

[0033] c-1) Transition1 module contains two branches: , high-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , through the branch After mapping, we get a medium-resolution feature map . Feature map Size , feature map The size is . Branch , including a convolutional layer with a kernel size of 3x3, a stride of 1, and a padding of 1, a batch normalization layer, and a ReLU activation function. It also contains a convolutional layer with a kernel size of 3x3, a stride of 1, and a padding of 1, a batch normalization layer, and a ReLU activation function.

[0034] c-2) StageModule1 consists of two parts: feature extraction module Extract1 and multi-scale fusion module Fusion1. Feature extraction module Extract1 consists of two branches: , feature map Through the branch After mapping, a high-resolution feature map is obtained , the size is ; Feature map Through the branch After mapping, a medium-resolution feature map is obtained , the size is . Branch and Both contain 4 improved MBConv modules. Figure 2 As shown in the figure, the improved MBConv module contains 5 consecutive parts: (1) convolution layer with a kernel size of 1x1, a stride of 1, and a padding of 0, a batch normalization layer, and a SiLU activation function; the width and height remain unchanged, and the number of channels is adjusted to 6 times the number of input channels. (2) Depthwise separable convolution with a kernel size of 3x3, a stride of 1, and a padding of 1, a batch normalization layer, and a SiLU activation function; the width and height remain unchanged, and the number of channels is still 6 times the number of input channels. (3) SE module: first, after an average pooling operation, the feature map is adjusted to a one-dimensional vector, and the dimension is still 6 times the number of input channels; then two fully connected layer operations are performed. The first fully connected layer uses the SiLU activation function, and the output dimension is 6 times the number of input channels. The second fully connected layer uses the Sigmoid activation function, and the output dimension is 6 times the number of input channels. The input feature map is multiplied by it channel by channel to obtain the final output feature map of the SE module. The width and height remain unchanged, and the number of channels is 6 times the number of input channels. (4) The convolution layer with a convolution kernel size of 1x1, a stride of 1, and a padding of 0, and a batch normalization layer, the number of channels is reduced by 6 times, and the size of the output feature map is the same as the input feature map. Figure 1 (5) Perform the DropPath operation with a probability of 0.2, and then make a residual connection between the input feature map and the output feature map in (4), and obtain the size of the output feature map of the improved MBConv module and the input feature map. Figure 1 To.

[0035] c-3) In the multi-scale fusion module Fusion1, a high-resolution feature map is obtained The mapping relationship is: ;in, represents the identity mapping, Get feature map , the size is , Represents an upsampling module. Get feature map , the size is So the high-resolution feature map The size is .like Figure 3 As shown, the upsampling module The input feature map is upsampled by a factor of 2, which includes two parts: (1) a convolution layer with a kernel size of 1x1, a stride of 1, and padding of 0, and a batch normalization layer with the width and height unchanged and the number of output channels being times the number of input channels. (2) Use the nearest neighbor interpolation method to upsample by 2 times to obtain the upsampling module The output feature map of is adjusted to twice the width and height of the input feature map, and the number of channels is adjusted to .

[0036] c-4) The mapping relationship of the medium-resolution feature map obtained in the multi-scale fusion module Fusion1 is: .in, represents the identity mapping, Represents the downsampling module. Medium resolution feature map The size is .like Figure 4 As shown, the downsampling module The input feature map is downsampled by 2 times, including: a convolution layer with a kernel size of 3x3, a stride of 2, and a padding of 0, and a batch normalization layer. The width and height of the output feature map are the width and height of the input feature map. , the number of channels is twice that of the input feature map.

[0037] d) In the second stage of the encoder, the high-resolution feature map , medium resolution feature map After the Transition2 module, a high-resolution feature map is obtained ,Medium resolution feature map , low-resolution feature map . The feature map , , After two StageModule2 modules, a high-resolution feature map is obtained. , medium resolution feature map , low-resolution feature map .

[0038] d-1) Transition2 module contains 3 branches: . High-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , the size is ; Medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , the size is The medium-resolution feature map is branched After mapping, we get a low-resolution feature map , the size is . Branch All are identity mappings and do not contain any operations. Contains convolutional layers with kernel size 3x3, stride 2, padding 1, batch normalization layers, and ReLU activation functions.

[0039] d-2) StageModule2 consists of two modules: feature extraction module Extract2 and multi-scale feature fusion module Fusion2. Feature extraction module Extract2 consists of three branches: . The high-resolution feature map is branched After mapping, a high-resolution feature map is obtained , the size is ; The medium-resolution feature map is branched After mapping, we get the medium resolution feature map , the size is ; Low-resolution feature maps are branched After mapping, we get a low-resolution feature map , the size is . Branch , both contain 4 improved MBConv modules.

[0040] d-3) In the multi-scale fusion module Fusion2, a high-resolution feature map is obtained The mapping relationship is: ; is the identity mapping, Get feature map , the size is . It is an upsampling module. Get feature map , the size is , Get feature map , the size is So the high-resolution feature map The size is .like Figure 3 As shown, the upsampling module The input feature map is upsampled by 4 times, which includes two parts: (1) a convolution layer with a kernel size of 1x1, a stride of 1, and padding of 0, and a batch normalization layer with unchanged width and height, and the number of output channels is times the number of input channels. (2) Use the nearest neighbor interpolation method to upsample by 4 times to obtain the upsampling module The output feature map has a width and height that is 4 times that of the input feature map, and the number of channels is the number of input channels .

[0041] d-4) Get the medium resolution feature map The mapping relationship is: ;in is the identity mapping, Get feature map , the size is ; is the upsampling module, Get feature map , the size is ; is the downsampling module, Get feature map , the size is So the medium resolution feature map The size is .like Figure 3 As shown, the upsampling module The input feature map is upsampled by a factor of 2, which includes two parts: (1) a convolutional layer with a kernel size of 1x1, a stride of 1, and padding of 0, and a batch normalization layer with the width and height unchanged and the number of channels being the same as the number of input channels. (2) Use the nearest neighbor interpolation method to upsample by 2 times to obtain the upsampling module The output feature map has a width and height that is twice that of the input feature map, and the number of channels is that of the input feature map .

[0042] d-5) Get the low-resolution feature map The mapping relationship is: .in is the identity mapping, Get feature map , the size is ; is the downsampling module, Get feature map , the size is ; Get feature map , the size is So the low-resolution feature map The size is .like Figure 4 As shown, the downsampling module The input feature map is downsampled by 4 times, which includes two parts: (1) a convolution layer with a kernel size of 3x3, a stride of 2, and a padding of 1, a batch normalization layer, a ReLU activation function, and a width and height adjusted to the input feature map. , the number of channels remains unchanged. (2) A convolutional layer with a kernel size of 3x3, a stride of 2, and a padding of 1, and a batch normalization layer to obtain a downsampling module Output feature map, width and height are the input feature map , the number of channels is 4 times the number of input channels. Figure 4 As shown, the downsampling module The input feature map is downsampled by 2 times, including: a convolution layer with a convolution kernel size of 3x3, a step size of 2, and a padding of 1, and a batch normalization layer to obtain a downsampling module Output feature map, width and height are the input feature map , the number of channels is twice the number of input channels.

[0043] e) In the third stage of the encoder, the high-resolution feature map , medium resolution feature map , low-resolution feature map After the Transition3 module, a high-resolution feature map is obtained ,Medium resolution feature map , low-resolution feature map , small resolution feature map . The feature map , , , After the StageModule3 module, a high-resolution feature map is obtained , medium resolution feature map .

[0044] e-1) Transition3 module contains 4 branches: , high-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , the size is ; Medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , the size is ; Low-resolution feature map Through the branch After mapping, we get a low-resolution feature map , the size is ; Low-resolution feature maps are branched After mapping, a small resolution feature map is obtained , the size is . Branch All are identity mappings and do not contain any operations. Contains convolutional layers with kernel size 3x3, stride 2, padding 1, batch normalization layers, and ReLU activation functions.

[0045] e-2) StageModule3 consists of two parts: feature extraction module Extract3 and multi-scale fusion module Fusion3. Feature extraction module Extract3 consists of four branches: . High-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , the size is ; Medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , the size is ; Low-resolution feature map Through the branch After mapping, we get a low-resolution feature map , the size is ; Small resolution feature map Through the branch After mapping, a small resolution feature map is obtained , the size is . Branch , both contain 4 improved MBConv modules.

[0046] e-3) In the multi-scale fusion module Fusion3, a high-resolution feature map is obtained The mapping relationship is: ;in, is the identity mapping, Get feature map , the size is , It is an upsampling module. Get feature map , the size is ; Get feature map , the size is ; Get feature map , the size is So the high-resolution feature map The size is .like Figure 3 As shown, the upsampling module The input feature map is upsampled by 8 times, which includes two parts: (1) a convolution layer with a kernel size of 1x1, a stride of 1, and padding of 0, and a batch normalization layer with the width and height unchanged and the number of channels being the number of input channels. (2) Use the nearest neighbor interpolation method to upsample by 8 times to obtain the upsampling module The output feature map has a width and height that is 8 times that of the input feature map, and the number of channels is the number of input channels .

[0047] e-4) Get the medium resolution feature map The mapping relationship is: ;in, is the identity mapping, Get feature map , the size is , is the upsampling module, Get feature map , the size is ; Get feature map , the size is . It is a downsampling module. Get feature map , the size is So the medium resolution feature map The size is ,like Figure 3 As shown, the upsampling module The input feature map is upsampled by 4 times, which includes two parts: (1) a convolution layer with a kernel size of 1x1, a stride of 1, and padding of 0, and a batch normalization layer. The width and height of the output feature map remain unchanged, and the number of channels is times the number of input channels. (2) Use the nearest neighbor interpolation method to upsample by 4 times to obtain the upsampling module The output feature map has a width and height that is 4 times that of the input feature map, and the number of channels is the number of input channels .

[0048] f) In the first stage of the decoder, called the fourth stage, the high-resolution feature map , medium resolution feature map After the multi-scale fusion module Fusion4, a high-resolution feature map is obtained , medium resolution feature map , low-resolution feature map . The feature map , , After two StageModule2 modules, a high-resolution feature map is obtained. , medium resolution feature map , low-resolution feature map .Will and Add together to get a high-resolution feature map ;Will and Add together to get the medium resolution feature map ;Will and Add together to get a low-resolution feature map .

[0049] f-1) In the multi-scale fusion module Fusion4, a high-resolution feature map is obtained The mapping relationship is: ; is the identity mapping, Get feature map , the size is ; is the upsampling module, Get feature map , the size is So we get a high-resolution feature map Size . Get the medium resolution feature map The mapping relationship is: ; is the identity mapping, Get feature map , the size is ; is the downsampling module, Get feature map , the size is So we get a medium-resolution feature map The size is . Get a low-resolution feature map The mapping relationship is: . It is a downsampling module. Get feature map , the size is , Get feature map , the size is So we get a low-resolution feature map The size is .

[0050] f-2) Feature map , , After two StageModule2 modules, a high-resolution feature map is obtained. , the size is ; Medium resolution feature map , the size is ; Low-resolution feature map , the size is . , the size is ; ; Size is ; , the size is .

[0051] g) In the second stage of the decoder, called the fifth stage, the high-resolution feature map , medium resolution feature map , low-resolution feature map After the multi-scale fusion module Fusion5, a high-resolution feature map is obtained , medium resolution feature map , the feature map , After the StageModule1 module, a high-resolution feature map is obtained , medium resolution feature map .Will and Add together to get a high-resolution feature map ;Will and Add together to get the medium resolution feature map .

[0052] g-1) In the multi-scale fusion module Fusion5, a high-resolution feature map is obtained The mapping relationship is: ; is the identity mapping, Get feature map , the size is . is the upsampling module, Get feature map , the size is ; Get feature map , the size is So we get a high-resolution feature map The size is . Get the medium resolution feature map The mapping relationship is: .in, is the identity mapping, is the upsampling module, Get feature map , the size is . is the downsampling module, Get feature map , the size is So we get a medium-resolution feature map The size is .

[0053] g-2) Feature map , After the StageModule1 module, a high-resolution feature map is obtained , the size is ; Medium resolution feature map , the size is . , the size is ; , the size is .

[0054] h) In the third stage of the decoder, called the sixth stage, the high-resolution feature map , medium resolution feature map After the multi-scale fusion module Fusion6, a high-resolution feature map is obtained .

[0055] h-1) In the multi-scale fusion module Fusion6, a high-resolution feature map is obtained The mapping relationship is: ;in is the identity mapping, Get feature map , the size is . is the upsampling module, Get feature map So we get a high-resolution feature map The size is .

[0056] i) The obtained high-resolution feature map Input into the detection head and get the output feature map , output feature map Each channel of is used to represent the position of each joint point.

[0057] i-1) The detection head is a convolutional layer with a kernel size of 1x1, a stride of 1, and a padding of 0. The size of the output feature map is ,in Represents the number of joints in the human body, and each channel predicts the position of a joint.

[0058] Example 2 In this embodiment, Figure 5 The human pose estimation results of the method of the present invention on the COCO validation set are shown in Figure 1. We selected 17 pictures in total, including different scenes, different perspectives, different human poses, different crowd densities, different occlusion levels, different light intensities, and different human sizes. From these pictures, we can see that the method of the present invention has achieved good human pose estimation results under various conditions. The method of the present invention can be applied to many aspects, such as: human tracking, disease rehabilitation, human-computer interaction, video monitoring, motion analysis, etc.

[0059] Example 3 Table 1: In this embodiment, Table 1 shows the experimental comparison results of the method of the present invention and other mainstream algorithms on the COCO validation set. We used six mainstream algorithms for experimental comparison, and the evaluation indicators were AP (average precision) index, AP index with a threshold of 0.5, AP index with a threshold of 0.75, AP index of medium-sized objects, AP index of large objects, and AR (average recall rate) index. It can be seen that the AP index of the method of the present invention reached 76.7%, which is higher than the AP index of the mainstream algorithm frameworks SimpleBaseline, Transpose, AggPose, MIPNet, HRNet-32 and HRNet-48. Compared with AggPose with the highest AP in the mainstream algorithm framework, the AP index of the present invention has increased by 0.3%. The method of the present invention is an improvement based on HRNet-32 (the number of channels of the high-resolution branch is 32), and the AP index has increased by 0.9% compared with the original HRNet-32 network. In addition, compared with the more complex and more accurate HRNet-48 (the number of channels of the high-resolution branch is 48), the AP index has increased by 0.4%. The above data further illustrate that the method of the present invention has better human posture estimation performance.

[0060] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A multi-scale fusion human posture estimation method based on improved HRNet, characterized in that: The following steps are involved: S1. Obtaining human pose estimation dataset , Represents the human pose estimation dataset The pictures, , for the dataset Perform data preprocessing to obtain the preprocessed human posture estimation data set ; S2. Construct an improved HRNet network structure, the improved HRNet network structure includes: an initial stage, an encoder, a decoder and a detection head, and convert the preprocessed human posture estimation dataset The Pictures Input to the initial stage to obtain a high-resolution feature map ; S3. The encoder includes three stages: the first stage, the second stage and the third stage, each stage includes a Transition module and a StageModule module, wherein the StageModule module includes a feature extraction module Extract module and a multi-scale fusion module Fusion module, the Extract module adopts an improved MBConv module, and the improved MBConv module includes 5 consecutive parts: the first part, the convolution layer, the batch normalization layer and the SiLU activation function, the second part, the depth-separable convolution, the batch normalization layer and the SiLU activation function, the third part, the SE module, the fourth part, the convolution layer and the batch normalization layer, the fifth part, the DropPath operation and the residual connection; the high-resolution feature map Get a high-resolution feature map after the encoder and medium resolution feature maps ; S4. The decoder also includes three stages, which are symmetrical with the encoder. The first and second stages both include a StageModule module and an independent Fusion module. The third stage includes an independent Fusion module. and medium resolution feature maps After the decoder, a high-resolution feature map is obtained ; S5. The high-resolution feature map The output feature map is obtained after the detection head , output feature map Each channel of is used to represent the position of each joint point, and finally the human body posture estimation result is obtained.

2. According to the multi-scale fusion human posture estimation method based on improved HRNet according to claim 1, it is characterized in that: Step S1 obtains the preprocessed human posture estimation data set The specific ones include: Human pose estimation dataset All sample images included in the dataset are uniformly scaled, and then data enhancement strategies are performed, including random horizontal flipping, random scaling and rotation of the human body bounding box, and affine transformation, to obtain the preprocessed human posture estimation dataset. ; The human posture estimation dataset after using the data enhancement strategy Generate labels for training. The specific operation is as follows: centering on the position of each human joint in the image, use a 2D Gaussian kernel function with a standard deviation of 1 pixel to generate a true value heat map as the label.

3. The multi-scale fusion human posture estimation method based on improved HRNet according to claim 1, characterized in that: The first stage of the encoder in step S3 is specifically: The first stage of the encoder includes: Transition1 module and StageModule1 module; The Transition1 module of the first stage of the encoder contains two branches: and , high-resolution feature map Through the branch and After mapping, high-resolution feature maps are obtained and medium resolution feature maps ; The StageModule1 module of the first stage of the encoder includes a feature extraction module Extract1 module and a multi-scale fusion module Fusion1 module. The feature extraction module Extract1 module includes two branches: and , the high-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , the medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , the high-resolution feature map and low-resolution feature maps After being mapped by the multi-scale fusion module Fusion1 module, a high-resolution feature map is obtained and medium resolution feature maps , the mapping relationship formula is: ， ， in, represents the identity mapping, It means that the 2x upsampling module of the high-resolution feature map is obtained. It represents a 2x downsampling module for obtaining a medium-resolution feature map.

4. The multi-scale fusion human posture estimation method based on improved HRNet according to claim 1, characterized in that: The second stage of the encoder in step S3 is specifically: The second stage of the encoder includes: Transition2 module and StageModule2 module; The Transition2 module of the second stage of the encoder contains three branches: , and , the high-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , the medium resolution feature map Through the branch and After mapping, we get the medium-resolution feature maps and low-resolution feature maps ; The StageModule2 module of the second stage of the encoder includes a feature extraction module Extract2 module and a multi-scale feature fusion module Fusion2 module. The feature extraction module Extract2 module includes three branches: , and , the high-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , the medium resolution feature map Through the branch After mapping, we get the medium resolution feature map , the low-resolution feature map Through the branch After mapping, we get a low-resolution feature map , the feature map , and After being mapped by the multi-scale fusion module Fusion2 module, a high-resolution feature map is obtained , medium resolution feature map and low-resolution feature maps , the mapping relationship formula is: ， ， ， in, represents the identity mapping, It means that the 4-fold upsampling module of the high-resolution feature map is obtained. Indicates a 2x upsampling module for medium-resolution feature maps. It means that the 4-fold downsampling module of the low-resolution feature map is obtained. Represents a 2x downsampling module for obtaining a low-resolution feature map.

5. The multi-scale fusion human posture estimation method based on improved HRNet according to claim 1, characterized in that: The third stage of the encoder in step S3 is specifically: The third stage of the encoder includes: Transition3 module and StageModule3 module; The Transition3 module of the third stage of the encoder contains four branches: , , and , the high-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , the medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , the low-resolution feature map Through the branch and After mapping, we get low-resolution feature maps and small resolution feature maps ; The StageModule3 module of the third stage of the encoder includes a feature extraction module Extract3 module and a multi-scale feature fusion module Fusion3 module. The feature extraction module Extract3 includes four branches: , , and , the high-resolution feature map Through the branch After mapping, a high-resolution feature map is obtained , the medium resolution feature map Through the branch After mapping, we get a medium-resolution feature map , the low-resolution feature map Through the branch After mapping, we get a low-resolution feature map , the small resolution feature map Through the branch After mapping, a small resolution feature map is obtained , the feature map , , and After being mapped by the multi-scale fusion module Fusion3 module, a high-resolution feature map is obtained and medium resolution feature maps , the mapping relationship formula is: ， ， in, Indicates an 8-fold upsampling module for obtaining a high-resolution feature map. Indicates a 4x upsampling module for obtaining a medium-resolution feature map.

6. The multi-scale fusion human posture estimation method based on improved HRNet according to claim 1, characterized in that: The first stage of the decoder in step S4 is specifically as follows: The first stage of the decoder includes: a multi-scale fusion module Fusion4 module and two StageModule2 modules; The feature map and After being mapped by the multi-scale fusion module Fusion4 module, a high-resolution feature map is obtained , medium resolution feature map and low-resolution feature maps , the mapping relationship formula is: ， ， ； The feature map , and After two StageModule2 modules, a high-resolution feature map is obtained , medium resolution feature map and low-resolution feature maps , the high-resolution feature map and A high-resolution feature map is obtained by adding the skip connection , the medium resolution feature map and The medium-resolution feature map is obtained by adding them through skip connections. , the low-resolution feature map and The low-resolution feature map is obtained by adding the skip connection .

7. The multi-scale fusion human posture estimation method based on improved HRNet according to claim 1, characterized in that: The second stage of the decoder in step S4 is specifically as follows: The second stage of the decoder includes: a multi-scale fusion module Fusion5 module and a StageModule1 module; The high-resolution feature map , and After being mapped by the multi-scale fusion module Fusion5, a high-resolution feature map is obtained and medium resolution feature maps , the mapping relationship formula is: ， ； The feature map , After the StageModule1 module, a high-resolution feature map is obtained and medium resolution feature maps , the high-resolution feature map and A high-resolution feature map is obtained by adding the skip connection , the medium resolution feature map and The medium-resolution feature map is obtained by adding the skip connection .

8. The multi-scale fusion human posture estimation method based on improved HRNet according to claim 1, characterized in that: The third stage of the decoder in step S4 is specifically as follows: The third stage of the decoder includes a multi-scale fusion module Fusion6 module; The feature map and After being mapped by the multi-scale fusion module Fusion6 module, a high-resolution feature map is obtained , the mapping relationship formula is: .

9. The multi-scale fusion human posture estimation method based on improved HRNet according to claims 3-5, characterized in that: The branch in the feature extraction module Extract1 and , branches in the feature extraction module Extract2 , and And the branch in the feature extraction module Extract3 , , and , each branch contains 4 improved MBConv modules.

10. A computing device, characterized in that: It includes a memory configured to store computer executable instructions; a processor configured to execute a multi-scale fusion human posture estimation method based on an improved HRNet as described in any one of claims 1-10 when the computer executable instructions are executed by the processor.

Images