A 2D human pose estimation method integrating integrated attention

Abstract

The present invention discloses a 2D human posture estimation method integrating integrated attention. An integrated attention module is added to the existing HRNet to construct an HRNet network integrating the integrated attention module. The HRNet network integrating the integrated attention module includes HRNet and the integrated attention module. The integrated attention module has a first branch, a second branch and a merging branch. The first branch is implemented by an average pooling layer, the second branch is implemented by an average pooling layer, the merging branch includes a concat layer, a first fully connected layer, a ReLu activation layer, a second fully connected layer, and a Sigmoid activation layer. The integrated attention module extracts layer domain feature sets with different receptive field sizes and aggregates information in the channel direction to achieve the purpose of local feature relearning. The advantage is that the human posture estimation speed is fast and the accuracy is high.

Description

Technical Field

[0001] The present invention relates to a 2D human body posture estimation method, and in particular to a 2D human body posture estimation method integrating integrated attention. Background Art

[0002] 2D human pose estimation (HPE) is a fundamental task in computer vision. Its purpose is to detect the position of various joints, such as the head, shoulders, wrists, and knees, in a two-dimensional image. It serves as an auxiliary and foundational task for many other vision tasks and has found widespread and important applications in intelligent video surveillance, human-computer interaction, autonomous driving, and smart healthcare. However, accurately estimating the position of human joints is a challenging task because human joints are small, poses vary, and are often affected by complex backgrounds and differences in surface features, such as self-occlusion and occlusion caused by clothing, body shape, and human movement.

[0003] In recent years, deep learning methods have achieved great success in the field of image recognition, and deep network-based methods have been widely used in 2D human pose estimation. Deep convolutional neural network models are the most commonly used deep network in current 2D human pose estimation methods. Currently, there are two main 2D human pose estimation methods based on deep convolutional neural network models. The first method is the SimpleBaseline method based on the ResNet backbone network, disclosed in Reference 1 (B. Xiao, H. Wu, Y. Wei. Simple baselines for human pose estimation and tracking [C] / / ECCV. 2018: 472-487.). This method uses multi-resolution images for supervision, allowing the deep convolutional neural network model to see information from images with different resolutions. However, low-resolution images may have errors in the localization of joints and are not completely reliable. Therefore, most current methods rely primarily on the highest-resolution image in the deep convolutional neural network model for supervision. The second method is the method disclosed in literature 2 (K.Sun, B.Xiao, D.Liu, J.Wang.Deep high-resolution representation learning for human pose estimation[C] / / CVPR.2019:5693-5703.) using a high-resolution multi-resolution parallel deep convolutional network HRNet. This method maintains the learning of high-resolution image information through a multi-resolution parallel subnet architecture, and repeatedly fusions multi-scale features to further refine the positioning of joint points.

[0004] However, both of the above two 2D human pose estimation methods based on deep convolutional neural network models have the following problems: First, the two deep convolutional neural network models have the problems of large number of parameters, slow calculation speed and slow inference speed, which leads to slow human pose estimation speed; Second, the human body joints account for a small proportion in the image and are small targets, which requires learning local image information, but both deep convolutional neural network models do not have the ability to learn advanced local information, which leads to low accuracy of human pose estimation. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a 2D human posture estimation method that integrates integrated attention and has fast human posture estimation speed and high accuracy.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a 2D human posture estimation method integrating integrated attention, comprising the following steps:

[0007] Step 1: Get a public dataset MS COCO for 2D human pose estimation from the official website https: / / cocodataset.org / #keypoints-2019. The public dataset MS COCO contains N images of people in natural scenes and the coordinates of 17 joints of the human body in each image of a person; each image of a person is a three-channel color image; among them, N = 175000, the 17 joints include left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, the coordinates of each joint are determined based on the image coordinate system, which consists of abscissa and ordinate. The image coordinate system takes the upper left corner vertex of the image as the coordinate origin, the horizontal direction from the coordinate origin to the right (that is, the direction of the line connecting the upper left vertex and the upper right vertex of the image) is the positive abscissa, and the vertical downward direction from the coordinate origin (that is, the direction of the line connecting the upper left vertex and the lower left vertex of the image) is the positive ordinate; get the public dataset MS We obtained 175,000 human images from COCO and the coordinates of 17 human joints in each of these 175,000 human images. We randomly divided these 175,000 human images from the public dataset MS COCO into three categories: training set, validation set, and test set. The training set contains 150,000 human images, the validation set contains 5,000 human images, and the test set contains 20,000 human images.

[0008] Step 2: Get all the images of people from the training set and the validation set, and then make 17 joint point heat maps corresponding to each person's image. The method of making the K-th joint point heat map corresponding to each person's image is as follows: design a blank image with the same size as the person's image, and each pixel in the person's image corresponds to the pixel at the same position in the blank image. Then define the coordinates of the K-th joint point in the person's image as the center coordinate, and mark the horizontal coordinate of the center coordinate as Vertical seat mark The pixel point in the blank image corresponding to the pixel point at the center coordinate is marked with a value of 1, where 1 indicates the correct position of the K-th joint point. Then, a circular area is determined on the human image with the center coordinate as the center and σ = 3 cm as the radius. The two-dimensional Gaussian function is used to calculate the Calculate the value of each pixel in the circular area except the pixel at the center coordinate, and mark the value of each pixel at the corresponding pixel in the blank image, where e represents the base of the natural logarithm, m K Indicates the horizontal coordinate of a pixel point other than the pixel point at the center coordinate in the circular area, n K Represents the vertical coordinate of a pixel point other than the pixel point at the center coordinate in the circular area. Except for the pixel point value at the center coordinate in the circular area, the values ​​of the remaining pixels are greater than 0 and less than 1 respectively. Their values ​​indicate that they are close to the correct position of the K-th joint point, but are not the correct position of the K-th joint point. Finally, all the remaining unlabeled pixels in the blank image are marked with a value of 0, which indicates that they are not the correct position of the K-th joint point. After all the corresponding pixels in the blank image are marked, the K-th joint point heat map corresponding to the human image is obtained;

[0009] All images of people in the training and validation sets and the 17 joint point heat maps corresponding to each person's image are stretched to 256 cm in length and 192 cm in width. That is, the size of all images of people and all joint point heat maps is 256×192. The 17 joint point heat maps corresponding to each person's image are the corresponding 17 labels.

[0010] Step 3: Add an integrated attention module to the existing HRNet to construct an HRNet network with an integrated attention module. The HRNet network with an integrated attention module includes HRNet and an integrated attention module. The HRNet has a first stage, a second stage, a third stage, a fourth stage and a decoding stage. The first stage is used to receive an image of size 256×192×3, and perform feature extraction on the image to generate a feature map output of size 64×48×256. The second stage is used to receive the feature map of size 64×48×256 output by the first stage, and perform feature extraction on the feature map to generate a feature map output of size 64×48×32. The third stage is used to access the output of the second stage of size 64×192×3. 4×48×32 feature map, and generate a feature map output of size 64×48×32; the integrated attention module is used to access the feature map of size 64×48×32 output by the second stage and the feature map of size 64×48×32 output by the third stage, and perform channel feature extraction on the two accessed feature maps respectively to generate a feature map output of size 64×48×32, the fourth stage is used to access the feature map of size 64×48×32 output by the integrated attention module, and perform feature extraction on the feature map to generate a feature map output of size 64×48×32; the decoding stage is used to access the feature map of size 64×48×32 output by the fourth stage, and generate a predicted joint point heat map output of size 64×48×17;

[0011] The integrated attention module has a first branch, a second branch and a merging branch. The first branch is implemented by an average pooling layer. The first branch is used to access the feature map of size 64×48×32 output by the second stage, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The second branch is implemented by an average pooling layer. The second branch is used to access the feature map of size 64×48×32 output by the third stage, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The merging branch includes a concat layer, a first fully connected layer, a ReLu activation layer, a second fully connected layer, and a Sigmoid activation layer. The concat layer of the merging branch is used to access the feature map of size 1×1×32 output by the first branch and the feature map of size 1×1×32 output by the second branch, and perform feature extraction to generate a feature map output of size 1×1×64. The first fully connected layer of the merging branch is used to access the The concat layer of the merging branch outputs a feature map of size 1×1×64, and performs feature extraction on the feature map to generate a feature map output of size 1×1×4. The ReLu activation layer of the merging branch is used to access the feature map of size 1×1×4 output by the first fully connected layer of the merging branch, and perform feature extraction on the feature map to generate a feature map output of size 1×1×4. The second fully connected layer of the merging branch is used to access the feature map of size 1×1×4 output by the ReLu activation layer, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The Sigmoid activation layer of the merging branch is used to access the feature map of size 1×1×32 output by the second fully connected layer of the merging branch and the feature map of size 64×48×32 output by the third stage, and after feature extraction, the feature map of size 1×1×32 is multiplied by the feature map of size 64×48×32 output by the third stage to generate a feature map output of size 64×48×32.

[0012] Step 4: Train the HRNet network that integrates the attention module. The specific process is as follows:

[0013] (1) Initializing the HRNet network of the integrated attention module using the he_normal parameter initialization method;

[0014] (2) Randomly divide the images of people in the training set into multiple batches, so that each batch contains batchsize images of people. If the total number of images of people in the training set is divisible by batchsize, the batches are divided into the total number of images of people / batchsize batches. If the total number of images of people in the training set is not divisible by batchsize, the remainder is discarded to obtain |total number of images of people / batchsize| batches, where batchsize = 32 and || is the rounding symbol.

[0015] (3) Take one of the batches and perform data augmentation on all images of the person in the batch using random rotation, where the rotation range is greater than 0° and less than or equal to 45°;

[0016] (4) The images of all people in the selected batch are used as inputs of the HRNet network of the fusion integrated attention module, and are input into the HRNet network of the fusion integrated attention module for recognition processing. The HRNet network of the fusion integrated attention module obtains 17 predicted joint point heat maps corresponding to the image of each person in the batch;

[0017] (5) For each person image in the selected batch, the human pose estimation loss value of each person image is calculated based on the corresponding 17 predicted joint point heat maps and the corresponding 17 labels, and the average human pose estimation loss value of all people's images in the selected batch is calculated as the final loss value. The human pose estimation loss value of each person image is calculated as follows:

[0018]

[0019] Among them, M = 256 × 192 × 17, GHeat j Heat represents the j-th joint point heat map corresponding to a person image in the batch. j represents the j-th predicted joint point heat map corresponding to the image of the person, where j = 1, 2, 3, ..., 17;

[0020] (6) According to the human posture estimation loss value of all images in the batch calculated in step (5), the parameters of the HRNet network of the fusion integrated attention module are trained using the Adam optimizer with a learning rate of 1e-3 to complete the training of the HRNet network of the fusion integrated attention module for the selected batch;

[0021] (7) Repeat steps (3)-(6) until all batches have trained the HRNet network of the fusion integrated attention module once, then input the images of all people in the verification set into the HRNet network of the fusion integrated attention module after training, and use the same method as step (5) to obtain the human posture estimation loss value of each person's image in the verification set, and calculate and obtain the average human posture estimation loss value of all people's images in the verification set;

[0022] (8) Repeat steps (2)-(7) for a total of Num times until the loss of the HRNet network of the fusion integrated attention module on the validation set converges, and finally obtain a trained HRNet network of the fusion integrated attention module, where Num ≥ 210;

[0023] Step 5: Stretch the image of the person whose posture needs to be estimated to 256 cm in length and 192 cm in width. Then input the stretched image of the person into the trained HRNet network with integrated attention module. The trained HRNet network with integrated attention module generates 17 predicted joint point heat maps as output. These 17 predicted joint point heat maps are the human posture estimation results.

[0024] Compared with the prior art, the advantage of the present invention is that an integrated attention module is added to the existing HRNet to construct an HRNet network integrating the integrated attention module. The HRNet network integrating the integrated attention module includes HRNet and the integrated attention module. HRNet has a first stage, a second stage, a third stage, a fourth stage and a decoding stage. The first stage is used to receive an image of size 256×192×3, and perform feature extraction on the image to generate a feature map output of size 64×48×256. The second stage is used to receive a feature map of size 64×48×256 output by the first stage, and perform feature extraction on the feature map to generate a feature map output of size 64×48×32. The third stage is used to access the second stage. The output size is 64×48×32 feature map, and a feature map output of size 64×48×32 is generated; the integrated attention module is used to access the feature map of size 64×48×32 output by the second stage and the feature map of size 64×48×32 output by the third stage, and perform channel feature extraction on the two accessed feature maps respectively to generate a feature map output of size 64×48×32. The fourth stage is used to access the feature map of size 64×48×32 output by the integrated attention module, perform feature extraction on the feature map, and generate a feature map output of size 64×48×32. The decoding stage is used to access the feature map of size 64×48×32 output by the fourth stage and generate a predicted joint point heat map output of size 64×48×17;The integrated attention module has a first branch, a second branch and a merge branch. The first branch is implemented by an average pooling layer. The first branch is used to access the feature map of size 64×48×32 output by the second stage, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The second branch is implemented by an average pooling layer. The second branch is used to access the feature map of size 64×48×32 output by the third stage, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The merge branch includes a concat layer, a first fully connected layer, a ReLu activation layer, a second fully connected layer, and a Sigmoid activation layer. The concat layer of the merge branch is used to access the first The branch outputs a feature map of size 1×1×32 and the second branch outputs a feature map of size 1×1×32, and performs feature extraction to generate a feature map output of size 1×1×64. The first fully connected layer of the merged branch is used to access the concat layer output of the merged branch with a size of 1×1×64, and perform feature extraction on the feature map to generate a feature map output of size 1×1×4. The ReLu activation layer of the merged branch is used to access the first fully connected layer output of the merged branch with a size of 1×1×4, and perform feature extraction on the feature map to generate a feature map output of size 1×1×4. The second fully connected layer of the merged branch is used to access the ReLu activation layer output with a size of 1×1 ×4 feature map, and extract the feature map to generate a feature map output of size 1×1×32. The Sigmoid activation layer of the merged branch is used to access the feature map of size 1×1×32 output by the second fully connected layer of the merged branch and the feature map of size 64×48×32 output by the third stage. The feature map of size 1×1×32 is extracted and multiplied with the feature map of size 64×48×32 output by the third stage to generate a feature map output of size 64×48×32. The integrated attention module extracts layer domain feature sets of different receptive field sizes, aggregates information in the channel direction, and synchronously learns local and global image features under two receptive field sizes to achieve layer domain-channel mixing. An attention mechanism combines feature maps learned from both high-level semantic features and local information with a deeper set of high-level features in the input, thereby relearning high-level local features. This improves the current progressive learning model of deep convolutional neural networks and relearns local information from high-level features, helping the existing HRNet learn contextual information about regions of interest from a new perspective. When the integrated attention module is embedded in the current human pose estimation backbone network (the existing HRNet), the feature expression capability of convolutional neurons is enhanced. With fewer parameters and computational complexity, a more efficient new backbone network model, namely the HRNet network with the integrated attention module, is obtained, which achieves fast and highly accurate human pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a partial visualization comparison experimental result of the 2D human pose estimation method integrating integrated attention of the present invention and the 2D human pose estimation method using the existing HRNet network on the test set. DETAILED DESCRIPTION

[0026] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0027] Embodiment: A 2D human pose estimation method integrating attention includes the following steps:

[0028] Step 1: Get a public dataset MS COCO for 2D human pose estimation from the official website https: / / cocodataset.org / #keypoints-2019. The public dataset MS COCO contains N images of people in natural scenes and the coordinates of 17 joints of the human body in each image of a person; each image of a person is a three-channel color image; among them, N = 175000, the 17 joints include left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, the coordinates of each joint are determined based on the image coordinate system, which consists of abscissa and ordinate. The image coordinate system takes the upper left corner vertex of the image as the coordinate origin, the horizontal direction from the coordinate origin to the right (that is, the direction of the line connecting the upper left vertex and the upper right vertex of the image) is the positive abscissa, and the vertical downward direction from the coordinate origin (that is, the direction of the line connecting the upper left vertex and the lower left vertex of the image) is the positive ordinate; get the public dataset MS We obtained 175,000 human images from COCO and the coordinates of 17 human joints in each of these 175,000 human images. We randomly divided these 175,000 human images from the public dataset MS COCO into three categories: training set, validation set, and test set. The training set contains 150,000 human images, the validation set contains 5,000 human images, and the test set contains 20,000 human images.

[0029] Step 2: Get all the images of people from the training set and the validation set, and then make 17 joint point heat maps corresponding to each person's image. The method of making the K-th joint point heat map corresponding to each person's image is as follows: design a blank image with the same size as the person's image, and each pixel in the person's image corresponds to the pixel at the same position in the blank image. Then define the coordinates of the K-th joint point in the person's image as the center coordinate, and mark the horizontal coordinate of the center coordinate as Vertical seat mark The pixel point in the blank image corresponding to the pixel point at the center coordinate is marked with a value of 1, where 1 indicates the correct position of the K-th joint point. Then, a circular area is determined on the human image with the center coordinate as the center and σ = 3 cm as the radius. The two-dimensional Gaussian function is used to calculate the Calculate the value of each pixel in the circular area except the pixel at the center coordinate, and mark the value of each pixel at the corresponding pixel in the blank image, where e represents the base of the natural logarithm, m K Indicates the horizontal coordinate of a pixel point other than the pixel point at the center coordinate in the circular area, n K Represents the vertical coordinate of a pixel point other than the pixel point at the center coordinate in the circular area. Except for the pixel point value at the center coordinate in the circular area, the values ​​of the remaining pixels are greater than 0 and less than 1 respectively. Their values ​​indicate that they are close to the correct position of the K-th joint point, but are not the correct position of the K-th joint point. Finally, all the remaining unlabeled pixels in the blank image are marked with a value of 0, which indicates that they are not the correct position of the K-th joint point. After all the corresponding pixels in the blank image are marked, the K-th joint point heat map corresponding to the human image is obtained;

[0030] All images of people in the training and validation sets and the 17 joint point heat maps corresponding to each person's image are stretched to 256 cm in length and 192 cm in width. That is, the size of all images of people and all joint point heat maps is 256×192. The 17 joint point heat maps corresponding to each person's image are the corresponding 17 labels.

[0031] Step 3: Add an integrated attention module to the existing HRNet to construct an HRNet network with integrated attention module. The HRNet network with integrated attention module includes HRNet and integrated attention module. HRNet has the first stage, the second stage, the third stage, the fourth stage and the decoding stage. The first stage is used to receive an image of size 256×192×3, and perform feature extraction on the image to generate a feature map output of size 64×48×256. The second stage is used to receive the feature map of size 64×48×256 output by the first stage, and perform feature extraction on the feature map to generate a feature map output of size 64×48×32. The third stage is used to access the feature map of size 64×48×32 output by the second stage. The integrated attention module is used to access the feature map of size 64×48×32 output by the second stage and the feature map of size 64×48×32 output by the third stage, and perform channel feature extraction on the two accessed feature maps to generate a feature map output of size 64×48×32. The fourth stage is used to access the feature map of size 64×48×32 output by the integrated attention module, perform feature extraction on the feature map, and generate a feature map output of size 64×48×32. The decoding stage is used to access the feature map of size 64×48×32 output by the fourth stage and generate a predicted joint point heat map output of size 64×48×17.

[0032] The integrated attention module has a first branch, a second branch and a merging branch. The first branch is implemented by an average pooling layer. The first branch is used to access the feature map of size 64×48×32 output by the second stage, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The second branch is implemented by an average pooling layer. The second branch is used to access the feature map of size 64×48×32 output by the third stage, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The merging branch includes a concat layer, a first fully connected layer, a ReLu activation layer, a second fully connected layer, and a Sigmoid activation layer. The concat layer of the merging branch is used to access the feature map of size 1×1×32 output by the first branch and the feature map of size 1×1×32 output by the second branch, and perform feature extraction to generate a feature map output of size 1×1×64. The first fully connected layer of the merging branch is used to access the c of the merging branch. The oncat layer outputs a feature map of size 1×1×64, and performs feature extraction on the feature map to generate a feature map output of size 1×1×4. The ReLu activation layer of the merging branch is used to access the feature map of size 1×1×4 output by the first fully connected layer of the merging branch, and performs feature extraction on the feature map to generate a feature map output of size 1×1×4. The second fully connected layer of the merging branch is used to access the feature map of size 1×1×4 output by the ReLu activation layer, and performs feature extraction on the feature map to generate a feature map output of size 1×1×32. The Sigmoid activation layer of the merging branch is used to access the feature map of size 1×1×32 output by the second fully connected layer of the merging branch and the feature map of size 64×48×32 output by the third stage, and after feature extraction, the feature map of size 1×1×32 is multiplied by the feature map of size 64×48×32 output by the third stage to generate a feature map output of size 64×48×32.

[0033] Step 4: Train the HRNet network that integrates the attention module. The specific process is as follows:

[0034] (1) Initialize the HRNet network that integrates the attention module using the he_normal parameter initialization method;

[0035] (2) Randomly divide the images of people in the training set into multiple batches, so that each batch contains batchsize images of people. If the total number of images of people in the training set is divisible by batchsize, the batches are divided into the total number of images of people / batchsize batches. If the total number of images of people in the training set is not divisible by batchsize, the remainder is discarded to obtain |total number of images of people / batchsize| batches, where batchsize = 32 and || is the rounding symbol.

[0036] (3) Take one of the batches and perform data augmentation on all images of the person in the batch using random rotation, where the rotation range is greater than 0° and less than or equal to 45°;

[0037] (4) The images of all people in the selected batch are used as the input of the HRNet network fused with the integrated attention module, and are input into the HRNet network fused with the integrated attention module for recognition processing. The HRNet network fused with the integrated attention module obtains 17 predicted joint point heat maps corresponding to the image of each person in the batch;

[0038] (5) For each person image in the selected batch, the human pose estimation loss value of each person image is calculated based on the corresponding 17 predicted joint point heat maps and the corresponding 17 labels, and the average human pose estimation loss value of all people's images in the selected batch is calculated as the final loss value. The human pose estimation loss value of each person image is calculated as follows:

[0039]

[0040] Among them, M = 256 × 192 × 17, GHeat j Heat represents the j-th joint point heat map corresponding to a person image in the batch. j represents the j-th predicted joint point heat map corresponding to the image of the person, where j = 1, 2, 3, ..., 17;

[0041] (6) Based on the human posture estimation loss values ​​of all images in the selected batch calculated in step (5), the parameters of the HRNet network fused with the integrated attention module are trained using the Adam optimizer with a learning rate of 1e-3 to complete the training of the HRNet network fused with the integrated attention module for the selected batch;

[0042] (7) Repeat steps (3)-(6) until all batches have trained the HRNet network of the integrated attention module, then input the images of all people in the verification set into the trained HRNet network of the integrated attention module, and use the same method as step (5) to obtain the human pose estimation loss value of each person's image in the verification set, and calculate and obtain the average human pose estimation loss value of all people's images in the verification set;

[0043] (8) Repeat steps (2)-(7) for a total of Num times until the loss of the HRNet network integrated with the attention module on the validation set converges, and finally obtain the trained HRNet network integrated with the attention module, where Num ≥ 210;

[0044] Step 5: Stretch the image of the person whose posture needs to be estimated to 256 cm in length and 192 cm in width. Then input the stretched image of the person into the trained HRNet network with integrated attention module. The trained HRNet network with integrated attention module generates 17 predicted joint point heat maps as output. These 17 predicted joint point heat maps are the human posture estimation results.

[0045] In order to verify the superiority of this method, the 2D human pose estimation method of the present invention is compared with the human pose estimation methods using several existing mainstream networks. Based on the test set of the present invention, the average accuracy AP, AP 50 、AP 75 , and average recall rate AR are used as evaluation indicators. The experimental comparison results are shown in Table 1:

[0046] Table 1 Comparative experimental results

[0047]

[0048] Note: The bold numbers in Table 1 represent the optimal values ​​of that column.

[0049] Analysis of the data in Table 1 shows that the human body posture estimation results of the method of the present invention are superior to those of other existing network-based human body posture estimation methods in terms of various indicators, which proves the superiority of the method.

[0050] The human body posture estimation method of the present invention and the human body posture estimation method using the existing HRNet are compared in a visualization experiment on the test set. Some visualization results are selected as follows: Figure 1 shown. Figure 1In each comparison chart, the left image is the result obtained by the existing HRNet human posture estimation method, and the right image is the result obtained by the method of the present invention. The circles represent the human posture positions predicted incorrectly by the existing HRNet human posture estimation method and the correct human posture positions predicted by the human posture estimation method of the present invention. Figure 1 It can be intuitively found that the human body posture estimation method of the present invention can obtain more accurate human body posture estimation results.

Claims

1. A 2D human pose estimation method integrating integrated attention, characterized by The following steps are involved: Step 1: Get a public dataset MS COCO for 2D human pose estimation from the website. The public dataset MSCOCO contains N images of people in natural scenes and the coordinates of 17 joints of the human body in each image of a person; each image of a person is a three-channel color image; where N = 175,000, and the 17 joints include left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle. The coordinates of each joint are determined based on the image coordinate system and consist of abscissa and ordinate. The image coordinate system takes the upper left corner vertex of the image as the coordinate origin, and the horizontal direction to the right from the coordinate origin is the positive abscissa, and the vertical direction downward from the coordinate origin is the positive ordinate; from the 175,000 images of people in the public dataset MS COCO and the coordinates of the 17 joints of the human body in each of these 175,000 images, convert MS The 175,000 images of people in the COCO dataset are randomly divided into three categories: training set, validation set, and test set. The training set contains 150,000 images of people, the validation set contains 5,000 images of people, and the test set contains 20,000 images of people. Step 2: Get all the images of people from the training set and the validation set, and then make 17 joint point heat maps corresponding to each person's image. The method of making the K-th joint point heat map corresponding to each person's image is as follows: design a blank image with the same size as the person's image, and each pixel in the person's image corresponds to the pixel at the same position in the blank image. Then define the coordinates of the K-th joint point in the person's image as the center coordinate, and mark the horizontal coordinate of the center coordinate as Vertical seat mark The pixel point in the blank image corresponding to the pixel point at the center coordinate is marked with a value of 1, where 1 indicates the correct position of the K-th joint point. Then, a circular area is determined on the human image with the center coordinate as the center and σ = 3 cm as the radius. The two-dimensional Gaussian function is used to calculate the Calculate the value of each pixel in the circular area except the pixel at the center coordinate, and mark the value of each pixel at the corresponding pixel in the blank image, where e represents the base of the natural logarithm, m K Indicates the horizontal coordinate of a pixel point other than the pixel point at the center coordinate in the circular area, n K Represents the vertical coordinate of a pixel point other than the pixel point at the center coordinate in the circular area. Except for the pixel point value at the center coordinate in the circular area, the values ​​of the remaining pixels are greater than 0 and less than 1 respectively. Their values ​​indicate that they are close to the correct position of the K-th joint point, but are not the correct position of the K-th joint point. Finally, all the remaining unlabeled pixels in the blank image are marked with a value of 0, which indicates that they are not the correct position of the K-th joint point. After all the corresponding pixels in the blank image are marked, the K-th joint point heat map corresponding to the human image is obtained; All images of people in the training and validation sets and the 17 joint point heat maps corresponding to each person's image are stretched to 256 cm in length and 192 cm in width. That is, the size of all images of people and all joint point heat maps is 256×192. The 17 joint point heat maps corresponding to each person's image are the corresponding 17 labels. Step 3: Add an integrated attention module to the existing HRNet to construct an HRNet network with an integrated attention module. The HRNet network with an integrated attention module includes HRNet and an integrated attention module. The HRNet has a first stage, a second stage, a third stage, a fourth stage and a decoding stage. The first stage is used to receive an image of size 256×192×3, and perform feature extraction on the image to generate a feature map output of size 64×48×256. The second stage is used to receive the feature map of size 64×48×256 output by the first stage, and perform feature extraction on the feature map to generate a feature map output of size 64×48×32. The third stage is used to access the output of the second stage of size 64×192×3. 4×48×32 feature map, and generate a feature map output of size 64×48×32; the integrated attention module is used to access the feature map of size 64×48×32 output by the second stage and the feature map of size 64×48×32 output by the third stage, and perform channel feature extraction on the two accessed feature maps respectively to generate a feature map output of size 64×48×32, the fourth stage is used to access the feature map of size 64×48×32 output by the integrated attention module, and perform feature extraction on the feature map to generate a feature map output of size 64×48×32; the decoding stage is used to access the feature map of size 64×48×32 output by the fourth stage, and generate a predicted joint point heat map output of size 64×48×17; The integrated attention module has a first branch, a second branch and a merging branch. The first branch is implemented by an average pooling layer. The first branch is used to access the feature map of size 64×48×32 output by the second stage, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The second branch is implemented by an average pooling layer. The second branch is used to access the feature map of size 64×48×32 output by the third stage, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The merging branch includes a concat layer, a first fully connected layer, a ReLu activation layer, a second fully connected layer and a Sigmoid activation layer. The concat layer of the merging branch is used to access the feature map of size 1×1×32 output by the first branch and the feature map of size 1×1×32 output by the second branch, and perform feature extraction to generate a feature map output of size 1×1×64. The first fully connected layer of the merging branch is used to access the The concat layer of the merging branch outputs a feature map of size 1×1×64, and performs feature extraction on the feature map to generate a feature map output of size 1×1×4. The ReLu activation layer of the merging branch is used to access the feature map of size 1×1×4 output by the first fully connected layer of the merging branch, and perform feature extraction on the feature map to generate a feature map output of size 1×1×4. The second fully connected layer of the merging branch is used to access the feature map of size 1×1×4 output by the ReLu activation layer, and perform feature extraction on the feature map to generate a feature map output of size 1×1×32. The Sigmoid activation layer of the merging branch is used to access the feature map of size 1×1×32 output by the second fully connected layer of the merging branch and the feature map of size 64×48×32 output by the third stage, and after feature extraction, the feature map of size 1×1×32 is multiplied by the feature map of size 64×48×32 output by the third stage to generate a feature map output of size 64×48×32. Step 4: Train the HRNet network that integrates the attention module. The specific process is as follows: (1) Initializing the HRNet network of the integrated attention module using the he_normal parameter initialization method; (2) Randomly divide the images of people in the training set into multiple batches, so that each batch contains batchsize images of people. If the total number of images of people in the training set is divisible by batchsize, the batches are divided into the total number of images of people / batchsize batches. If the total number of images of people in the training set is not divisible by batchsize, the remainder is discarded to obtain |total number of images of people / batchsize| batches, where batchsize = 32 and || is the rounding symbol. (3) Take one of the batches and perform data augmentation on all images of the person in the batch using random rotation, where the rotation range is greater than 0° and less than or equal to 45°; (4) The images of all people in the selected batch are used as inputs of the HRNet network of the fusion integrated attention module, and are input into the HRNet network of the fusion integrated attention module for recognition processing. The HRNet network of the fusion integrated attention module obtains 17 predicted joint point heat maps corresponding to the image of each person in the batch; (5) For each person image in the selected batch, the human pose estimation loss value of each person image is calculated based on the corresponding 17 predicted joint point heat maps and the corresponding 17 labels, and the average human pose estimation loss value of all people's images in the selected batch is calculated as the final loss value. The human pose estimation loss value of each person image is calculated as follows: Among them, M = 256 × 192 × 17, GHeat j Heat represents the j-th joint point heat map corresponding to a person image in the batch. j represents the j-th predicted joint point heat map corresponding to the image of the person, where j = 1, 2, 3, ..., 17; (6) According to the human posture estimation loss value of all images in the batch calculated in step (5), the parameters of the HRNet network of the fusion integrated attention module are trained using the Adam optimizer with a learning rate of 1e-3 to complete the training of the HRNet network of the fusion integrated attention module for the selected batch; (7) Repeat steps (3)-(6) until all batches have trained the HRNet network of the fusion integrated attention module once, then input the images of all people in the verification set into the HRNet network of the fusion integrated attention module after training, and use the same method as step (5) to obtain the human posture estimation loss value of each person's image in the verification set, and calculate and obtain the average human posture estimation loss value of all people's images in the verification set; (8) Repeat steps (2)-(7) for a total of Num times until the loss of the HRNet network of the fusion integrated attention module on the validation set converges, and finally obtain a trained HRNet network of the fusion integrated attention module, where Num ≥ 210; Step 5: Stretch the image of the person whose posture needs to be estimated to 256 cm in length and 192 cm in width. Then input the stretched image of the person into the trained HRNet network with integrated attention module. The trained HRNet network with integrated attention module generates 17 predicted joint point heat maps as output. These 17 predicted joint point heat maps are the human posture estimation results.

Images