A 2D human pose estimation method based on dual-branch human structure modeling

Abstract

The present invention discloses a 2D human body posture estimation method for dual-branch human body structure modeling. The method selects a data set and divides it into a training set, a validation set and a test set. For images in the training set and the validation set, heat maps of 17 key points and 9 components are extracted. A convolutional neural network including a backbone network, a dual-branch reasoning module and a feature fusion decoding module is constructed. The dual-branch reasoning module consists of a high-confidence key point reasoning branch and a component reasoning branch, which process key point and component information respectively. The feature fusion decoding module includes a multi-layer perceptron and a decoding stage, so that the network can fuse key point and component information and generate a final predicted key point heat map. The method trains the network through a training set and a validation set, and finally realizes high-precision 2D human body posture estimation on a test set, performs well in processing occlusion and complex background, and significantly improves the accuracy and robustness of 2D human body posture estimation.

Description

Technical Field

[0001] The invention belongs to the field of image or video recognition, and relates to a human body posture estimation technology, and in particular to a 2D human body posture estimation method for occluded scenes based on dual-branch human body structure modeling. Background Art

[0002] Human pose estimation (HPE) is one of the important research topics in the field of computer vision. The main goal of human pose estimation is to accurately locate the positions of key points of the human body from a given two-dimensional image or monocular video sequence. In many fields such as human action recognition, human-computer interaction, pedestrian re-identification, sports assessment, medical rehabilitation monitoring, etc., accurate human pose estimation can characterize the key structural information of the human body, thereby providing important clues for downstream tasks to accurately capture and understand human actions, and building a bridge between visual perception and high-level semantic understanding. In recent years, with the rapid development of artificial intelligence technology, human pose estimation is not only a frontier hot issue in theoretical research, but also one of the core key technologies in many intelligent application scenarios, attracting more and more researchers. However, in the face of various occlusions, rich posture changes and perspective changes, and complex environmental interference that are common in actual application scenarios, existing human pose estimation methods face great challenges in performance and generalization ability.

[0003] In practical application scenarios, occlusion is one of the most important challenges in human pose estimation. In the case of occlusion, the main difficulties in human pose estimation come from the information loss caused by occlusion and the destruction of dependencies between parts. Existing HPE models based on statistical learning, such as deep learning models, use the labeled data in the training data set to learn the complex spatial relationship between human key points during the training phase; in the test phase, when encountering partial occlusion, the approximate position of invisible human key points can be inferred from the information of visible human key points based on the knowledge accumulated during the training process. This HPE model will make inaccurate or even irrational wrong predictions in the case of severe occlusion. For example, due to self-occlusion caused by perspective and complex human posture and occlusion by objects or people in the foreground, some human key points cannot be directly observed. Large-area occlusion will lead to the loss of context and structural information. This HPE model cannot obtain enough context information to infer the accurate position of invisible human key points, resulting in large prediction errors or even illogical predictions.

[0004] At present, deep learning methods based on convolutional neural networks have become the mainstream technology for 2D human pose estimation. Convolutional neural networks, such as SimpleBaseline, Hourglass, HRNet, etc., extract rich visual features from input images through multi-layer convolution, pooling and nonlinear activation operations. These visual features cover information such as color, texture, edge, shape, etc., and can effectively capture local features near human key points and contextual information. However, although the existing 2D human pose estimation methods based on convolutional neural networks have shown strong capabilities in image feature extraction and can learn the discriminative information of each human key point based on the extracted features, they lack a deep understanding of human body structure and reasoning ability based on human body structure. For example, in occluded scenes, the position of occluded human key points may not be accurately predicted. In this way, the model needs to have the ability to model the global posture of the human body so as to be able to infer the position information of invisible human key points from visible human parts, which is exactly the ability that the existing 2D human pose estimation methods based on convolutional neural networks do not have. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a 2D human body posture estimation method for dual-branch human body structure modeling, which can adaptively reconstruct human body key points in occluded scenes with high reconstruction accuracy.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a 2D human body posture estimation method based on dual-branch human body structure modeling, characterized by comprising the following steps:

[0007] Step 1: Select a data set that contains several human images in natural scenes, and the coordinates of the key points of the human body visible in the human images are known; then divide the data set into a training set, a validation set, and a test set;

[0008] Step 2: For each human image in the training set and the validation set, obtain 17 key point heat maps and 9 component heat maps corresponding to each human body in each human image. When there is no occlusion, each human body has 17 human key points and 9 components. The 17 human key points are left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle. The 9 components are left upper arm composed of left shoulder and left elbow key point pairs, left upper arm composed of left elbow and left wrist key point pairs, right .... The left lower arm is composed of a right shoulder and a right elbow key point pair, the right lower arm is composed of a right elbow and a right wrist key point pair, the left thigh is composed of a left hip and a left knee key point pair, the right thigh is composed of a right hip and a right knee key point pair, the left calf is composed of a left knee and a left ankle key point pair, the right calf is composed of a right knee and a right ankle key point pair, and the torso is determined by two shoulders and two hips, the key point pair is two connected human key points, the two shoulders refer to the left shoulder and the right shoulder, and the two hips refer to the left hip and the right hip; then the key point heat map is used as the key point label, and the component heat map is used as the component label;

[0009] Step 3: Construct a convolutional neural network as a 2D human posture estimation network for dual-branch human structure modeling, which includes a backbone network, a dual-branch reasoning module and a feature fusion decoding module. The dual-branch reasoning module is composed of a parallel high-confidence key point reasoning branch and a component reasoning branch; first, the human image is input into the backbone network to obtain a visual feature map corresponding to the human image; then the visual feature map is input into the high-confidence key point reasoning branch, the visual feature map is passed through the first decoding stage to generate an initial predicted key point heat map, the initial predicted key point heat map is passed through the gating module to obtain a gated output matrix, the visual feature map and the gated output matrix are passed through the first linear mapping layer together to generate a first mapping matrix, the encoding matrix obtained by position encoding the randomly initialized key point matrix and the first mapping matrix is ​​passed through the first attention module together to generate a first key point token; at the same time, the visual feature map is input into the component reasoning branch, the visual feature map is passed through the second decoding stage to generate a predicted component heat map, and the visual feature map and the predicted component heat map are combined. Figure 1 A second mapping matrix is ​​obtained through a second linear mapping layer, and a coding matrix obtained by position encoding the randomly initialized key point matrix and the second mapping matrix is ​​passed through a second attention module to generate a second key point token; the first key point token and the second key point token are then input into a feature fusion decoding module to generate a final predicted key point heat map;

[0010] Step 4: Use the training set and the validation set to train the 2D human pose estimation network to obtain a trained 2D human pose estimation network;

[0011] Step 5: Input each human image in the test set into the trained 2D human pose estimation network, and output 17 final predicted key point heat maps corresponding to each human body in each human image in the test set, which are the human pose estimation results.

[0012] In step 2, the first k The process of obtaining a key point heat map is as follows:

[0013] Step 2.1a: Design a blank image of the same size as the human image, where each pixel in the human image corresponds to the pixel at the same position in the blank image;

[0014] Step 2.2a: If the first k If the key points of the individual body have been marked, then proceed to step 2.3a, where k =1, 2, ..., 17; if the first k If the key points of the human body are not marked, the value 0 is marked at all pixels in the blank image, where the value 0 indicates that it is not the first k The correct position of the key points of the individual body is determined, and then step 2.5a is performed;

[0015] Step 2.3a: The first k The coordinates of the key points of the individual body are defined as the central coordinates, and the horizontal and vertical coordinates of the central coordinates are recorded as and ; Then take the center coordinate as the center of the circle, Determine a circular region in this human body image for radius;

[0016] Step 2.4a: In the blank image, mark the pixel at the same position as the center coordinate with the value 1, where the value 1 indicates that it is the first pixel. k The correct position of the key points of the human body is obtained; then the two-dimensional Gaussian function is used to calculate the value of each pixel in the circular area except the pixel at the center coordinate, and the calculated value of each pixel is marked on the pixel at the same position in the blank image, where the calculated value of each pixel is greater than 0 and less than 1, and these values ​​represent the same as the first k The correct positions of the key points of the individual bodies are close, but not the k The correct position of the key points of the human body is obtained; and the value 0 is marked at all the unmarked pixels in the blank image, where the value 0 indicates that it is not the first pixel. k Correct location of key points of the individual body;

[0017] Step 2.5a: After marking the values ​​at all the pixels in the blank image, the first pixel corresponding to the processed human body in this human body image is obtained. k Key point heat map.

[0018] In step 2.4a, the two-dimensional Gaussian function is ,in, e represents the base of natural logarithms, and The corresponding horizontal and vertical coordinates represent the remaining pixel points in the circular area except the pixel point at the center coordinate.

[0019] In step 2, the first j The process of obtaining a component heat map is as follows:

[0020] Step 2.1b: Design a blank image of the same size as the human image, where each pixel in the human image corresponds to the pixel at the same position in the blank image;

[0021] Step 2.2b: If the first j If the components are visible, then execute step 2.3b, where j =1, 2, ..., 9; if the first j If all or part of the components are invisible, mark all pixels in the blank image with a value of 0 and then execute step 2.4b;

[0022] Step 2.3b: If the first j If the first part is composed of two connected human key points, then the first part of the human body processed in this human image j The coordinates of one of the key points of the human body in the components are defined as the first center coordinates, and the coordinates of the other key point of the human body are defined as the second center coordinates; then in this human body image, a rectangular area with half the width of 2 pixels is determined with the line connecting the first center coordinate and the second center coordinate as the center line in the length direction; then a two-dimensional Gaussian function is used to calculate the value of each pixel in the rectangular area, and the calculated value of each pixel is marked at the pixel at the same position in the blank image, wherein the calculated value of each pixel is greater than 0 and less than 1; then the value 0 is marked at all the pixels that are not marked in the blank image; finally, step 2.4b is executed;

[0023] If the first j The first part of the human body in this human body image is the torso defined by the shoulders and hips. jThe coordinates of the four key points of the human body in the parts are used as the vertex coordinates; then in this human body image, a quadrilateral area is determined by the four vertex coordinates; then the pixel points at the same position as each pixel point in the quadrilateral area in the blank image are marked with a value of 1; then all the unmarked pixel points in the blank image are marked with a value of 0; finally, step 2.4b is executed;

[0024] Step 2.4b: After marking the values ​​at all the pixels in the blank image, the first pixel corresponding to the processed human body in this human body image is obtained. j A heat map of components.

[0025] In step 2.3b, the two-dimensional Gaussian function is ,in, is a hyperparameter, e represents the base of natural logarithms, d Indicates the vertical distance from the pixel point in the rectangular area to the line connecting the first center coordinate and the second center coordinate.

[0026] In step 1, the coordinates of the key points of the human body are determined based on the image coordinate system of the human body image in which they are located. The image coordinate system takes the upper left corner vertex of the human body image as the coordinate origin, the direction horizontally to the right from the coordinate origin is defined as the positive horizontal coordinate, and the direction vertically downward from the coordinate origin is defined as the positive vertical coordinate.

[0027] In the step 3, the backbone network is a three-stage HRNet network, and the three-stage HRNet network consists of a first stage, a second stage, and a third stage connected in sequence; the first stage receives a human body image of a size of 256×192×3, and performs feature extraction on the received human body image to generate a feature map output of a size of 64×48×256; the second stage receives a feature map of a size of 64×48×256 output by the first stage, and performs feature extraction on the received feature map to generate a feature map output of a size of 64×48×32; the third stage receives a feature map of a size of 64×48×32 output by the second stage, and generates a visual feature map output of a size of 64×48×32.

[0028] In step 3, the processing process of the gate control module is:

[0029] The global maximum pooling operation is performed on the initial predicted key point heat map of size 64×48×17 received by the gating module to obtain a 17×1-dimensional confidence vector, in which each element is the confidence of a human key point; then the 17×1-dimensional confidence vector is input into In the function, The function outputs a 17×1 dimensional vector, where Function , Indicatesi The confidence of individual key points, i =1, 2, …, 17, e represents the base of natural logarithms, a and b are all constants; then the 17×1-dimensional vector is multiplied by the predicted key point heat map of size 64×48×17 received by the gating module by channel, and the multiplication result is flattened by channel to obtain a 17×3072-dimensional gating output matrix;

[0030] Before the visual feature map passes through the first linear mapping layer, it is first subjected to a global maximum pooling operation to obtain a feature map of size 64×48×1, and then the feature map of size 64×48×1 and the 17×3072-dimensional gated output matrix output by the gating module are passed through the first linear mapping layer together, and the size of the obtained first mapping matrix is ​​18×192 dimensions; the size of the key point matrix is ​​17×192 dimensions, and the first attention module generates 17 192-dimensional first key point tokens;

[0031] The size of the predicted component heat map is 64×48×9. Before the visual feature map passes through the second linear mapping layer, the visual feature map is first subjected to a global maximum pooling operation to obtain a feature map of size 64×48×1. The feature map of size 64×48×1 and the predicted component heat map are then pooled together. Figure 1 After passing through the second linear mapping layer, the size of the second mapping matrix obtained is 10×192 dimensions; the second attention module generates 17 192-dimensional second key point tokens;

[0032] The structures of the first attention module and the second attention module are the same, both of which are composed of four layers of Transformer layers connected in sequence.

[0033] In the step 3, the feature fusion decoding module is composed of an MLP and a third decoding stage, and 17 192-dimensional first key point tokens output by the high-confidence key point reasoning branch and 17 192-dimensional second key point tokens output by the component reasoning branch are added; the first layer of perceptrons in the MLP receives the result of the addition and generates a 17×384-dimensional matrix output; the second layer of perceptrons in the MLP receives the result of the 17×384-dimensional matrix output by the first layer of perceptrons after layer normalization, and generates a 17×3072-dimensional matrix output; the third decoding stage receives the result of the 17×3072-dimensional matrix output by the second layer of perceptrons in the MLP after layer normalization, and generates 17 final predicted key point heat map outputs with a size of 64×48.

[0034] The training process of step 4 is as follows:

[0035] Step 4.1: Import pre-trained parameters for the backbone network;

[0036] Step 4.2: Randomly divide all human images of size 256×192 in the training set into multiple batches, so that each batch contains batchsize human images;

[0037] Step 4.3: Take one batch of the training set and perform data augmentation on all human images in this batch by random rotation, where the rotation degree range is greater than 0° and less than or equal to 45°;

[0038] Step 4.4: All human images in this batch are used as inputs of the 2D human pose estimation network, and are input into the 2D human pose estimation network for recognition processing. In the middle, 17 initial predicted key point heat maps and 9 corresponding predicted component heat maps corresponding to each human body in each human image in this batch are obtained, and finally 17 final predicted key point heat maps corresponding to each human body in each human image in this batch are obtained;

[0039] Step 4.5: For each person in each human image in this batch, calculate the initial predicted key point loss , Parts loss and the final predicted key point loss , and the total loss of human pose estimation To supervise the network training, MSE loss is used. , , , , is the two-norm operator symbol, Indicates the first i Zhang initial prediction key point heat map, Indicates the first i Zhang key point labels, Indicates the first j A heat map of predicted components, Indicates the first j Component labels, Indicates the first i Zhang's final predicted key point heat map, It is a device for adjusting and Hyperparameters that influence training;

[0040] Step 4.6: After the total loss of human pose estimation corresponding to all human bodies in all human images in this batch is calculated, the parameters of the 2D human pose estimation network are trained using the Adam optimizer with a learning rate of 1e-3 to complete the training of the 2D human pose estimation network for this batch;

[0041] Step 4.7: Repeat the process from step 4.3 to step 4.6 until all batches of the training set have trained the 2D human pose estimation network once;

[0042] Step 4.8: Randomly divide all human images of size 256×192 in the validation set into multiple batches, so that each batch contains batchsize human images;

[0043] Step 4.9: Take one of the batches of the validation set; then input all the human images in this batch into the 2D human pose estimation network trained with the training set, and obtain 17 initial predicted key point heat maps and 9 corresponding predicted component heat maps for each human body in each human image in this batch in the middle, and finally obtain 17 final predicted key point heat maps for each human body in each human image in this batch; then follow the process of step 4.5, in the same way, calculate the total loss of human pose estimation corresponding to each human body in each human image in this batch;

[0044] Step 4.10: Repeat the process of step 4.9 until all batches of the validation set have been processed once; then calculate the average loss of the total loss of human posture estimation corresponding to all human bodies in all human images in the validation set;

[0045] Step 4.11: Repeat the process from step 4.2 to step 4.10 until the average loss of the 2D human pose estimation network trained with the training set on the validation set converges, and finally obtains a trained 2D human pose estimation network.

[0046] Compared with the prior art, the advantages of the present invention are:

[0047] First, a dual-branch design based on key points and components is adopted, that is, a dual-branch reasoning module consisting of a parallel high-confidence key point reasoning branch and a component reasoning branch is designed to achieve all-round information integration. The method of the present invention makes full use of all available information, including diversified data such as image visual features, predicted key point heat maps, predicted component heat maps, etc., effectively improving the feature representation ability of the network. Compared with most existing methods that only use one of the image visual features and key point features, the method of the present invention does not discard any potentially valuable clues in the reasoning process, especially in the case of severe occlusion and uncertainty in initial predictions. It is believed that any available information may be beneficial to subsequent global reasoning, and human key point estimates that conform to the logic of human structure are made under limited observation conditions. This strategy helps to enhance the robustness and generalization ability of the network in complex occlusion environments.

[0048] Second, a confidence-based gating module is introduced into the high-confidence key point reasoning branch of the method of the present invention, realizing a guiding mechanism for reasoning invisible key points based on high-confidence visible key points. In occlusion scenarios, the prediction confidence of the backbone network for invisible key points is low, resulting in a large deviation between the prediction results and the actual results, while the prediction confidence of visible key points is high, and the prediction results are relatively accurate. The existing network has never used the effective information of prediction confidence. In the method of the present invention, the gating module gives higher weights to high-confidence (visible) key points, while weakening the information of low-confidence (invisible) key points, thereby eliminating the interference of uncertain information and obtaining the deterministic posture of the human body. The subsequent module can distinguish between visible (determined) key points and invisible (uncertain) key points, and based on the global structural relationship, update the heat map features of all key points, reconstruct the heat map features of invisible key points, and the accuracy of the reconstructed human key points is high.

[0049] Third, a human posture model is constructed based on the global attention mechanism of Transformer to achieve global perception of human posture and in-depth reasoning of structural relationships. In occluded scenes, human body parts information and visible key point information play a vital role, and can provide the model with richer and more deterministic information clues. Different from the conventional method of applying attention pixel by pixel and block by block on feature maps or images, the method of the present invention uses heat maps corresponding to different key points and different parts as input tokens of Transformer in two parallel high-confidence key point reasoning branches and component reasoning branches, respectively, and constructs human body structure models based on human body key points and components, thereby performing global reasoning in two different dimensions and understanding human body structure relationships. Finally, the initial prediction results are optimized by fusing the dual-branch information, especially when locating occluded human body key points, a more reasonable position estimate can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of the composition structure of a 2D human posture estimation network for dual-branch human structure modeling constructed by the method of the present invention;

[0051] Figure 2 The human pose estimation results obtained by predicting each human body in the first human image selected in the validation set of the dataset MS COCO using the human pose estimation method using the HRNet-W32 network;

[0052] Figure 3 The human pose estimation results obtained by predicting each human body in the first human image selected in the validation set of the dataset MS COCO using the human pose estimation method using the TokenPose-L / D6 network;

[0053] Figure 4 The human posture estimation result obtained by predicting each human body in the first human body image selected in the validation set of the data set MS COCO using the method of the present invention;

[0054] Figure 5 The human pose estimation results obtained by predicting each human body in the second human image selected in the validation set of the dataset MS COCO using the human pose estimation method using the HRNet-W32 network;

[0055] Figure 6 The human pose estimation results obtained by predicting each human body in the second human image selected in the validation set of the dataset MS COCO using the human pose estimation method using the TokenPose-L / D6 network;

[0056] Figure 7 The human body posture estimation result obtained by predicting each human body in the second human body image selected in the validation set of the data set MS COCO using the method of the present invention;

[0057] Figure 8 The human pose estimation results obtained by predicting each human body in the third human image selected in the validation set of the dataset MS COCO using the human pose estimation method using the HRNet-W32 network;

[0058] Fig. 9 The human pose estimation results obtained by predicting each human body in the third human image selected in the validation set of the dataset MS COCO using the human pose estimation method using the TokenPose-L / D6 network;

[0059] Fig.10The human body posture estimation result is obtained by predicting each human body in the third human body image selected in the validation set of the data set MS COCO using the method of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below with reference to the accompanying drawings.

[0061] The present invention proposes a 2D human body posture estimation method based on dual-branch human body structure modeling, which comprises the following steps:

[0062] Step 1: Select a data set that contains several human images in natural scenes, where the human images are three-channel color images and the coordinates of the visible human key points in the human images are known; then divide the data set into a training set, a validation set, and a test set.

[0063] In this embodiment, the coordinates of the key points of the human body are determined based on the image coordinate system of the human body image in which they are located. The image coordinate system takes the upper left corner vertex of the human body image as the coordinate origin, and the direction horizontally to the right from the coordinate origin (i.e., the direction of the line connecting the upper left corner vertex and the upper right corner vertex of the human body image) is defined as the positive horizontal coordinate, and the direction vertically downward from the coordinate origin (i.e., the direction of the line connecting the upper left corner vertex and the lower left corner vertex of the human body image) is defined as the positive vertical coordinate.

[0064] In the simulation experiment, a public dataset MS COCO for 2D human pose estimation tasks can be obtained from the official website https: / / cocodataset.org / #keypoints-2019. A total of 175,000 human images and the coordinates of the visible human key points in each human image are obtained from the public dataset MS COCO. These 175,000 human images are divided into three categories: training set, validation set, and test set. Among them, there are 150,000 human images in the training set, 5,000 human images in the validation set, and 20,000 human images in the test set. Human images are images containing people.

[0065] Step 2: For each human image in the training set and the validation set, obtain 17 key point heat maps and 9 corresponding component heat maps corresponding to each human body in each human image. When there is no occlusion, each human body has 17 human key points and 9 components. The 17 human key points are left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle. The 9 components are the left upper arm composed of the left shoulder and left elbow key point pair, the left lower arm composed of the left elbow and left wrist key point pair, the right upper arm composed of the right shoulder and right elbow key point pair, the right lower arm composed of the right elbow and right wrist key point pair, the left hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the left hip composed of the right hip composed of the right hip composed of the left hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the left hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the left hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the right hip composed of the left hip composed of the right hip composed of the left hip composed of the left elbow key point pair. The left thigh is composed of the left hip and right knee key point pair, the right thigh is composed of the right hip and right knee key point pair, the left calf is composed of the left knee and left ankle key point pair, the right calf is composed of the right knee and right ankle key point pair, and the torso is determined by the shoulders and hips. The key point pair is two connected human key points. The shoulders refer to the left shoulder and the right shoulder, and the hips refer to the left hip and the right hip. Then, each human image in the training set and the validation set and the 17 key point heat maps and the corresponding 9 component heat maps corresponding to each human body in each human image are stretched to adjust their size to 256×192; then the key point heat map with a size of 256×192 is used as the key point label, and the component heat map with a size of 256×192 is used as the component label.

[0066] Further defined, in step 2, the first k The process of obtaining a key point heat map is as follows:

[0067] Step 2.1a: Design a blank image of the same size as the human image. Each pixel in the human image corresponds to the pixel at the same position in the blank image.

[0068] Step 2.2a: If the first k If the key points of the individual body have been marked, then proceed to step 2.3a, where k =1, 2, ..., 17; if the first k If the key points of the human body are not marked, the value 0 is marked at all pixels in the blank image, where the value 0 indicates that it is not the first k After the correct position of each key point of the human body is determined, proceed to step 2.5a.

[0069] Step 2.3a: The first k The coordinates of the key points of the individual body are defined as the central coordinates, and the horizontal and vertical coordinates of the central coordinates are recorded as and ; Then take the center coordinate as the center of the circle, A circular area is determined in this human body image as the radius. In this embodiment, =3cm.

[0070] Step 2.4a: In the blank image, mark the pixel at the same position as the center coordinate with the value 1, where the value 1 indicates that it is the first pixel. k The correct position of the key points of the human body is obtained; then the two-dimensional Gaussian function is used to calculate the value of each pixel in the circular area except the pixel at the center coordinate, and the calculated value of each pixel is marked on the pixel at the same position in the blank image, where the calculated value of each pixel is greater than 0 and less than 1, and these values ​​represent the same as the first k The correct positions of the key points of the individual bodies are close, but not the k The correct position of the key points of the human body is obtained; and the value 0 is marked at all the unmarked pixels in the blank image, where the value 0 indicates that it is not the first pixel. k The correct position of key points of the individual body.

[0071] The preferred solution is that the two-dimensional Gaussian function is ,in, e represents the base of natural logarithms, e =2.71, and Correspondingly, the horizontal and vertical coordinates of the remaining pixels in the circular area except the pixel at the center coordinate are represented. The coordinates of the pixel points in the human body image are determined based on the image coordinate system of the human body image. For the blank image with coordinates of The value at the pixel for .

[0072] Step 2.5a: After marking the values ​​at all the pixels in the blank image, the first pixel corresponding to the processed human body in this human body image is obtained. k Key point heat map.

[0073] Further defined, in step 2, the first j The process of obtaining a component heat map is as follows:

[0074] Step 2.1b: Design a blank image of the same size as the human image, where each pixel in the human image corresponds to the pixel at the same position in the blank image.

[0075] Step 2.2b: If the first j If the components are visible, then execute step 2.3b, where j =1, 2, ..., 9; if the first jIf all or part of a component is invisible, mark all pixels in the blank image with a value of 0 and execute step 2.4b.

[0076] Here, a component is visible means that, for example, both of the human body key points in a component composed of two connected human body key points are marked, or the four human body key points in the torso determined by the shoulders and the hips are marked; a component is completely invisible means that, for example, both of the human body key points in a component composed of two connected human body key points are not marked, or the four human body key points in the torso determined by the shoulders and the hips are not marked; a component is partially invisible means that, for example, one of the human body key points in a component composed of two connected human body key points is not marked, or one of the human body key points, two of the human body key points, or three of the human body key points in the torso determined by the shoulders and the hips are not marked.

[0077] Step 2.3b: If the first j If the first part is composed of two connected human key points, then the first part of the human body processed in this human image j The coordinates of one of the key points of the human body in the parts are defined as the first center coordinates, and the coordinates of the other key point of the human body are defined as the second center coordinates; then in this human body image, a rectangular area with half the width of 2 pixels is determined with the line connecting the first center coordinate and the second center coordinate as the center line in the length direction, that is, the length of the rectangular area is the length of the line connecting the first center coordinate and the second center coordinate, and the width is 4 pixels; then a two-dimensional Gaussian function is used to calculate the value of each pixel in the rectangular area, and the calculated value of each pixel is marked at the pixel at the same position in the blank image, wherein the calculated value of each pixel is greater than 0 and less than 1; then the value 0 is marked at all unmarked pixels in the blank image; finally, step 2.4b is executed.

[0078] If the first j The first part of the human body in this human body image is the torso defined by the shoulders and hips. j The coordinates of the four key points of the human body in each part are used as the vertex coordinates; then in this human body image, a quadrilateral area is determined by the four vertex coordinates; then the pixel points at the same position as each pixel point in the quadrilateral area in the blank image are marked with the value 1; then the value 0 is marked at all the unmarked pixel points in the blank image; finally, step 2.4b is executed.

[0079] The preferred solution is that the two-dimensional Gaussian function is ,in, is a hyperparameter. In this embodiment, , erepresents the base of natural logarithms, e =2.71, d Represents the vertical distance from the pixel point in the rectangular area to the line connecting the first center coordinate and the second center coordinate. If the pixel point is on the line connecting the first center coordinate and the second center coordinate, then d =0, if the pixel point is not on the line connecting the first center coordinate and the second center coordinate, then d =1 or 2, set the coordinates of any pixel in the rectangular area to , using the two-dimensional Gaussian function Calculate the value at this pixel for ,Will The coordinates of the annotation in the blank image are At the pixel point, and Corresponding to the horizontal coordinate and the vertical coordinate of the pixel point in the rectangular area, the coordinate of the pixel point in the human body image is determined based on the image coordinate system of the human body image.

[0080] Step 2.4b: After marking the values ​​at all the pixels in the blank image, the first pixel corresponding to the processed human body in this human body image is obtained. j A heat map of components.

[0081] Step 3: Construct a convolutional neural network as a 2D human posture estimation network for dual-branch human structure modeling, whose composition structure is as follows Figure 1 As shown, it includes a backbone network (Backbone), a dual-branch reasoning module and a feature fusion decoding module. The dual-branch reasoning module consists of a parallel high-confidence keypoint reasoning branch (High-Confidence Keypoint-based Reasoning Branch) and a component reasoning branch (Part-based Reasoning Branch). The high-confidence keypoint reasoning branch includes the first decoding stage, namely the keypoint decoding stage, and the first reasoning stage consisting of a gate module (Gate), a first linear mapping layer (Linear Projection), and a first attention module (Pose Transformer). The component reasoning branch includes the second decoding stage, namely the component decoding stage, and the second reasoning stage consisting of a second linear mapping layer (LinearProjection) and a second attention module (Pose Transformer). The feature fusion decoding module consists of an MLP (multi-layer perceptron) and a third decoding stage. Here, the first decoding stage, the second decoding stage, the third decoding stage, the first linear mapping layer, the second linear mapping layer, and the MLP all adopt existing technologies.

[0082] The backbone network receives a human image of size 256×192×3, and outputs a visual feature map of size 64×48×32. The backbone network is a three-stage HRNet network, which consists of the first stage, the second stage, and the third stage connected in sequence; the first stage receives a human image of size 256×192×3, and extracts features from the received human image to generate a feature map output of size 64×48×256; the second stage receives the feature map of size 64×48×256 output by the first stage, and extracts features from the received feature map to generate a feature map output of size 64×48×32; the third stage receives the feature map of size 64×48×32 output by the second stage, and generates a visual feature map output of size 64×48×32. The existing HRNet network is composed of a first stage, a second stage, a third stage, and a fourth stage connected in sequence. The existing HRNet network is improved by removing the fourth stage to obtain a three-stage HRNet network used in the present invention. Since only the fourth stage in the existing HRNet network is removed and the first stage, the second stage, and the third stage are kept unchanged, the structural parameters of the first stage, the second stage, and the third stage are not repeated.

[0083] In the high-confidence key point reasoning branch, the first decoding stage receives the visual feature map of size 64×48×32 output by the backbone network and generates an initial predicted key point heat map (KH) of size 64×48×17 output; the gating module receives the initial predicted key point heat map of size 64×48×17 output by the first decoding stage and generates a 17×3072-dimensional gated output matrix output; the first linear mapping layer simultaneously receives the visual feature map of size 64×48×32 output by the backbone network and passes through a The feature map of size 64×48×1 obtained after the global maximum pooling operation and the 17×3072-dimensional gated output matrix output by the gating module are used to generate an 18×192-dimensional first mapping matrix output; the 18×192-dimensional first mapping matrix is ​​positionally encoded to obtain an 18×192-dimensional encoding matrix; the first attention module simultaneously receives the 18×192-dimensional encoding matrix and the randomly initialized 17×192-dimensional key point matrix, and generates 17 192-dimensional first key point tokens (i.e., vectors).

[0084] The preferred solution is that the processing process of the gating module is as follows: a global maximum pooling operation is performed on the initial predicted key point heat map (KH) of size 64×48×17 received by the gating module to obtain a 17×1 dimensional confidence vector, where each element in the confidence vector is the confidence of a human key point; then the 17×1 dimensional confidence vector is input into In the function, The function outputs a 17×1 dimensional vector, where Function , Indicates i The confidence of individual key points, i =1, 2, …, 17, e represents the base of natural logarithms, e =2.71, a and b are all constants. In this embodiment, a =10, b =0.5; then multiply the 17×1 vector by the 64×48×17 predicted key point heat map received by the gating module by channel, and flatten the multiplication result by channel to obtain a 17×3072-dimensional gated output matrix. The purpose of the gating module is to preserve the information on the key point heat map with high confidence, while severely weakening the information on the key point heat map with low confidence.

[0085] In the component reasoning branch, the second decoding stage receives the visual feature map of size 64×48×32 output by the backbone network, and generates a predicted component heat map (PH) output of size 64×48×9; the second linear mapping layer simultaneously receives the visual feature map of size 64×48×32 output by the backbone network, a feature map of size 64×48×1 obtained after a global maximum pooling operation, and the predicted component heat map of size 64×48×9 output by the second decoding stage, and generates a 10×192-dimensional second mapping matrix output; the 10×192-dimensional second mapping matrix is ​​positionally encoded to obtain a 10×192-dimensional encoding matrix; the second attention module simultaneously receives the 10×192-dimensional encoding matrix and the randomly initialized 17×192-dimensional key point matrix, and generates 17 192-dimensional second key point tokens.

[0086] It is further defined that the first attention module and the second attention module have the same structure, both of which are composed of four Transformer layers connected in sequence.

[0087] In the feature fusion decoding module, the 17 192-dimensional first key point tokens output by the high-confidence key point reasoning branch and the 17 192-dimensional second key point tokens output by the component reasoning branch are added; the first layer of perceptrons in the MLP receives the result of the addition and generates a 17×384-dimensional matrix output; the second layer of perceptrons in the MLP receives the result of the 17×384-dimensional matrix output by the first layer of perceptrons after layer normalization and generates a 17×3072-dimensional matrix output; the third decoding stage receives the result of the 17×3072-dimensional matrix output by the second layer of perceptrons in the MLP after layer normalization and generates 17 final predicted key point heat map outputs with a size of 64×48.

[0088] Step 4: Use the training set and the validation set to train the 2D human pose estimation network to obtain a trained 2D human pose estimation network.

[0089] Further defined, the training process of step 4 is:

[0090] Step 4.1: Import the pre-trained parameters for the backbone network.

[0091] Step 4.2: All human images with a size of 256×192 in the training set are randomly divided into multiple batches, so that each batch contains batchsize human images. In this embodiment, batchsize=32.

[0092] In actual processing, if the total number of human images in the training set is divisible by batchsize, they are divided into Num batches, where Num = total number of human images / batchsize; if the total number of human images in the training set is not divisible by batchsize, the excess part is discarded to obtain int(Num) batches, where int() represents the rounding function.

[0093] Step 4.3: Take one of the batches of the training set and perform data augmentation on all human images in this batch using random rotation, where the rotation degree range is greater than 0° and less than or equal to 45°.

[0094] Step 4.4: All human images in this batch are used as the input of the 2D human pose estimation network, and are input into the 2D human pose estimation network for recognition processing. In the middle, 17 initial predicted key point heat maps and 9 corresponding predicted component heat maps corresponding to each human body in each human image in this batch are obtained. Finally, 17 final predicted key point heat maps corresponding to each human body in each human image in this batch are obtained.

[0095] Step 4.5: For each person in each human image in this batch, calculate the initial predicted key point loss , Parts loss and the final predicted key point loss , and the total loss of human pose estimation To supervise the network training, MSE loss is used. , , , , is the two-norm operator symbol, Indicates the first i Zhang initial prediction key point heat map, Indicates the first i Zhang key point labels, Indicates the first j A heat map of predicted components, Indicates the first j Component labels, Indicates the first i Zhang's final predicted key point heat map, It is a device for adjusting and The hyperparameters that affect the training are taken as .

[0096] Step 4.6: After the total loss of human pose estimation corresponding to all human bodies in all human images in this batch is calculated, the parameters of the 2D human pose estimation network are trained using the Adam optimizer with a learning rate of 1e-3 to complete the training of the 2D human pose estimation network for this batch.

[0097] Step 4.7: Repeat the process from step 4.3 to step 4.6 until all batches of the training set have trained the 2D human pose estimation network once.

[0098] Step 4.8: Randomly divide all human images of size 256×192 in the validation set into multiple batches, so that each batch contains batchsize human images. In actual processing, the processing method is the same as that of the training set.

[0099] Step 4.9: Take one of the batches of the validation set; then input all the human images in this batch into the 2D human pose estimation network trained with the training set, and obtain 17 initial predicted key point heat maps and 9 corresponding predicted component heat maps for each human body in each human image in this batch in the middle, and finally obtain 17 final predicted key point heat maps for each human body in each human image in this batch; then follow the process of step 4.5, in the same way, calculate the total loss of human pose estimation corresponding to each human body in each human image in this batch.

[0100] Step 4.10: Repeat the process of step 4.9 until all batches of the validation set have been processed once; then calculate the average loss of the total loss of human posture estimation corresponding to all human bodies in all human images in the validation set.

[0101] Step 4.11: Repeat the process from step 4.2 to step 4.10 until the average loss value of the 2D human pose estimation network trained with the training set on the validation set converges, and finally a trained 2D human pose estimation network is obtained. Experiments have shown that a total of 260 rounds of training can enable the average loss value on the validation set to converge.

[0102] Step 5: Stretch each human image in the test set to adjust its size to 256×192; then input each human image of size 256×192 into the trained 2D human pose estimation network, and output 17 final predicted key point heat maps corresponding to each human body in each human image in the test set, which are the human pose estimation results.

[0103] To further illustrate the feasibility and effectiveness of the method of the present invention, a large number of experiments were carried out on the widely used and large-scale dataset MS COCO.

[0104] In the experiment, the average precision AP and average recall AR are used as evaluation indicators.

[0105] Table 1 shows the average precision AP and average recall AR of the proposed method and the existing methods on the validation set of the MS COCO dataset.

[0106] Table 1 Comparison of the average precision AP and average recall AR of the proposed method and the existing methods on the validation set of the MS COCO dataset

[0107]

[0108] HRNet in Table 1 refers to the human pose estimation method using the HRNet network, which is the method disclosed in Reference 1 (Sun, Ke, et al. "Deep High-Resolution Representation Learning for Human Pose Estimation", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019), HRNet-W32 indicates that the number of channels is 32, and HRNet-W48 indicates that the number of channels is 48; DARK refers to the human pose estimation method using the DARK network, which is the method disclosed in Reference 2 (Zhang, Feng, et al. "Distribution-based Coordinate Representation for Human Pose Estimation", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020.); TokenPose refers to the human pose estimation method using the TokenPose network, which is the method disclosed in Reference 3 (Li, Yanjie, et al. "Tokenpose: Learning Keypoint Symbols for Human Pose Estimation", Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021. ),TokenPose-L / D6 means that the number of transformer layers is 6, and TokenPose-L / D24 means that the number of transformer layers is 24; TransPose-H refers to the human body posture estimation method using the TransPose-H network, which is the method disclosed in reference 4 (Yang, Sen, et al. "TransPose: Localizing key points through Transformer", Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021. ) ; HRFormer-B refers to the human body posture estimation method using the HRFormer-B network, which is the method disclosed in reference 5 (Yuan, Y., Fu, R., Huang,L., Lin, W., Zhang, C., Chen, X.,&Wang, J. (2021). Hrformer: "High-resolution visual transformers for dense prediction", Advances in Neural Information Processing Systems, 34, 7281-7293. ) ; I^2RNet refers to the method using I^ The human body posture estimation method of the 2RNet network is a method disclosed in reference 6 (Ding, Yiwei, et al. "I^2R-Net: "Interpersonal Relationship Network for Multi-Person Pose Estimation", 2206.10892 (2022). ), I^2RNet (HRNET) means that the backbone network used is hrnet, and I^2RNet (Tokenpose) means that the backbone network used is Tokenpose.

[0109] From the data shown in Table 1, it can be seen that the method of the present invention can significantly improve the accuracy of human posture estimation on the validation set of the MS COCO dataset. For example, compared with the HRNet-W48 method, the method of the present invention has achieved a 1.5% improvement on the basis of the HRNet-W48 method. In particular, the method of the present invention surpasses all other existing methods on the validation set of the MS COCO dataset.

[0110] Figure 2 , Figure 3 , Figure 4 The human body posture estimation results obtained by predicting each human body in the first human body image selected in the validation set of the data set MS COCO using the existing human body posture estimation method using the HRNet-W32 network, the existing human body posture estimation method using the TokenPose-L / D6 network, and the method of the present invention are given accordingly. Figure 2 , Figure 3 , Figure 4 It can be seen that the three networks predict the key points of the left and middle human bodies relatively accurately. However, for the right human body, the key points of the right leg of the human body are severely occluded. The two existing networks cannot accurately locate the key points of the right knee and right ankle, and give wrong predictions. The method of the present invention locates these two occluded key points of the human body relatively accurately under this occlusion condition.

[0111] Figure 5 , Figure 6 , Figure 7 The human body posture estimation results obtained by predicting each human body in the second human body image selected in the validation set of the data set MS COCO using the existing human body posture estimation method using the HRNet-W32 network, the existing human body posture estimation method using the TokenPose-L / D6 network, and the method of the present invention are given accordingly. Figure 5 , Figure 6 , Figure 7 It can be seen that the tennis player's right ankle key point is blocked by himself due to jumping. The two existing networks cannot accurately locate the right ankle key point and give an incorrect prediction. The method of the present invention locates the blocked human key point more accurately under this occlusion condition.

[0112] Figure 8 , Fig. 9 , Fig.10 The human body posture estimation results obtained by predicting each human body in the third human body image selected in the validation set of the data set MS COCO using the existing human body posture estimation method using the HRNet-W32 network, the existing human body posture estimation method using the TokenPose-L / D6 network, and the method of the present invention are given accordingly. Figure 8 , Fig. 9 , Fig.10 It can be seen that due to the viewing angle, the left leg key points of the man opening the refrigerator door are blocked by himself. The two existing networks cannot accurately locate the left knee and left ankle key points and give wrong predictions. The method of the present invention locates these two blocked key points more accurately under this occlusion condition.

[0113] Through the above experiments, it is fully verified that when the key points of the human body are severely blocked, the method of the present invention can also accurately locate the blocked key points of the human body.

Claims

1. A 2D human body posture estimation method based on dual-branch human body structure modeling, characterized in that The following steps are involved: Step 1: Select a data set that contains several human images in natural scenes, and the coordinates of the key points of the human body visible in the human images are known; then divide the data set into a training set, a validation set, and a test set; Step 2: For each human image in the training set and the validation set, obtain 17 key point heat maps and 9 corresponding component heat maps corresponding to each human body in each human image; then use the key point heat map as the key point label and the component heat map as the component label; Step 3: Construct a convolutional neural network as a 2D human posture estimation network for dual-branch human structure modeling, which includes a backbone network, a dual-branch reasoning module, and a feature fusion decoding module. The dual-branch reasoning module consists of a parallel high-confidence key point reasoning branch and a component reasoning branch. First, input the human image into the backbone network to obtain a visual feature map corresponding to the human image. Then, the visual feature map is input into the high-confidence key point reasoning branch, the visual feature map is passed through the first decoding stage to generate an initial predicted key point heat map, the initial predicted key point heat map is passed through the gating module to obtain a gated output matrix, the visual feature map and the gated output matrix are passed through the first linear mapping layer together to generate a first mapping matrix, and the encoding matrix obtained by position encoding the randomly initialized key point matrix and the first mapping matrix is ​​passed through the first attention module together to generate a first key point token; At the same time, the visual feature map is input into the component reasoning branch, the visual feature map is passed through the second decoding stage to generate a predicted component heat map, the visual feature map and the predicted component heat map are passed through the second linear mapping layer to obtain a second mapping matrix, and the encoding matrix obtained by position encoding the randomly initialized key point matrix and the second mapping matrix is ​​passed through the second attention module to generate a second key point token; Then the first key point token and the second key point token are input into the feature fusion decoding module, and the fusion generates the final predicted key point heat map; Step 4: Use the training set and the validation set to train the 2D human pose estimation network to obtain a trained 2D human pose estimation network; Step 5: Input each human image in the test set into the trained 2D human pose estimation network, and output 17 final predicted key point heat maps corresponding to each human body in each human image in the test set, which are the human pose estimation results.

2. The 2D human body posture estimation method based on dual-branch human body structure modeling according to claim 1 is characterized in that In step 2, the first k The process of obtaining a key point heat map is as follows: Step 2.1a: Design a blank image of the same size as the human image, where each pixel in the human image corresponds to the pixel at the same position in the blank image; Step 2.2a: If the first k If the key points of the individual body have been marked, then proceed to step 2.3a, where k =1, 2, ..., 17; if the first k If the key points of the human body are not marked, the value 0 is marked at all pixels in the blank image, where the value 0 indicates that it is not the first k The correct position of the key points of the individual body is determined, and then step 2.5a is performed; Step 2.3a: The first k The coordinates of the key points of the individual body are defined as the central coordinates, and the horizontal and vertical coordinates of the central coordinates are recorded as and ; Then take the center coordinate as the center of the circle, Determine a circular region in this human body image for radius; Step 2.4a: In the blank image, mark the pixel at the same position as the center coordinate with the value 1, where the value 1 indicates that it is the first pixel. k The correct position of the key points of the human body is obtained; then the two-dimensional Gaussian function is used to calculate the value of each pixel in the circular area except the pixel at the center coordinate, and the calculated value of each pixel is marked on the pixel at the same position in the blank image, where the calculated value of each pixel is greater than 0 and less than 1, and these values ​​represent the same as the first k The correct positions of the key points of the individual body are close, but not the k The correct position of the key points of the human body is obtained; and the value 0 is marked at all the unmarked pixels in the blank image, where the value 0 indicates that it is not the first pixel. k Correct location of key points of the individual body; Step 2.5a: After marking the values ​​at all the pixels in the blank image, the first pixel corresponding to the processed human body in this human body image is obtained. k Key point heat map.

3. The 2D human body posture estimation method based on dual-branch human body structure modeling according to claim 2 is characterized in that In step 2.4a, the two-dimensional Gaussian function is ,in, e represents the base of natural logarithms, and The corresponding horizontal and vertical coordinates represent the remaining pixel points in the circular area except the pixel point at the center coordinate.

4. A 2D human body posture estimation method based on dual-branch human body structure modeling according to any one of claims 1 to 3, characterized in that In step 2, the first j The process of obtaining a component heat map is as follows: Step 2.1b: Design a blank image of the same size as the human image, where each pixel in the human image corresponds to the pixel at the same position in the blank image; Step 2.2b: If the first j If the components are visible, then execute step 2.3b, where j =1, 2, ..., 9; if the first j If all or part of the components are invisible, mark all pixels in the blank image with a value of 0 and then execute step 2.4b; Step 2.3b: If the first j If the first part is composed of two connected human key points, then the first part of the human body processed in this human image j The coordinates of one of the key points of the human body in the components are defined as the first center coordinates, and the coordinates of the other key point of the human body are defined as the second center coordinates; then in this human body image, a rectangular area with half the width of 2 pixels is determined with the line connecting the first center coordinate and the second center coordinate as the center line in the length direction; then a two-dimensional Gaussian function is used to calculate the value of each pixel in the rectangular area, and the calculated value of each pixel is marked at the pixel at the same position in the blank image, wherein the calculated value of each pixel is greater than 0 and less than 1; then the value 0 is marked at all the pixels that are not marked in the blank image; finally, step 2.4b is executed; If the first j The first part of the human body in this human body image is the torso defined by the shoulders and hips. j The coordinates of the four key points of the human body in the parts are used as the vertex coordinates; then in this human body image, a quadrilateral area is determined by the four vertex coordinates; then the pixel points at the same position as each pixel point in the quadrilateral area in the blank image are marked with a value of 1; then all the unmarked pixel points in the blank image are marked with a value of 0; finally, step 2.4b is executed; Step 2.4b: After marking the values ​​at all the pixels in the blank image, the first pixel corresponding to the processed human body in this human body image is obtained. j A heat map of components.

5. The 2D human body posture estimation method based on dual-branch human body structure modeling according to claim 4 is characterized in that In step 2.3b, the two-dimensional Gaussian function is ,in, is a hyperparameter, e represents the base of natural logarithms, d Indicates the vertical distance from the pixel point in the rectangular area to the line connecting the first center coordinate and the second center coordinate.

6. The 2D human body posture estimation method based on dual-branch human body structure modeling according to claim 1, characterized in that In step 1, the coordinates of the key points of the human body are determined based on the image coordinate system of the human body image in which they are located. The image coordinate system takes the upper left corner vertex of the human body image as the coordinate origin, the direction horizontally to the right from the coordinate origin is defined as the positive horizontal coordinate, and the direction vertically downward from the coordinate origin is defined as the positive vertical coordinate.

7. The 2D human body posture estimation method based on dual-branch human body structure modeling according to claim 1, characterized in that In the step 3, the backbone network is a three-stage HRNet network, and the three-stage HRNet network consists of a first stage, a second stage, and a third stage connected in sequence; the first stage receives a human body image of a size of 256×192×3, and performs feature extraction on the received human body image to generate a feature map output of a size of 64×48×256; the second stage receives a feature map of a size of 64×48×256 output by the first stage, and performs feature extraction on the received feature map to generate a feature map output of a size of 64×48×32; the third stage receives a feature map of a size of 64×48×32 output by the second stage, and generates a visual feature map output of a size of 64×48×32.

8. The 2D human body posture estimation method based on dual-branch human body structure modeling according to claim 7, characterized in that In step 3, the processing process of the gate control module is: The global maximum pooling operation is performed on the initial predicted key point heat map of size 64×48×17 received by the gating module to obtain a 17×1-dimensional confidence vector, in which each element is the confidence of a human key point; then the 17×1-dimensional confidence vector is input into In the function, The function outputs a 17×1 dimensional vector, where Function , Indicates i The confidence of individual key points, i =1, 2, …, 17, e represents the base of natural logarithms, a and b ; are all constants; then the 17×1 dimensional vector is multiplied by the 64×48×17 predicted key point heat map received by the gating module by channel, and the multiplication result is flattened by channel to obtain a 17×3072 dimensional gating output matrix; Before the visual feature map passes through the first linear mapping layer, it is first subjected to a global maximum pooling operation to obtain a feature map of size 64×48×1, and then the feature map of size 64×48×1 and the 17×3072-dimensional gated output matrix output by the gating module are passed through the first linear mapping layer together, and the size of the obtained first mapping matrix is ​​18×192 dimensions; the size of the key point matrix is ​​17×192 dimensions, and the first attention module generates 17 192-dimensional first key point tokens; The size of the predicted component heat map is 64×48×9. Before the visual feature map passes through the second linear mapping layer, the visual feature map is first subjected to a global maximum pooling operation to obtain a feature map of size 64×48×1. The feature map of size 64×48×1 and the predicted component heat map are then passed through the second linear mapping layer together. The size of the obtained second mapping matrix is ​​10×192 dimensions. The second attention module generates 17 192-dimensional second key point tokens. The structures of the first attention module and the second attention module are the same, both of which are composed of four layers of Transformer layers connected in sequence.

9. The 2D human body posture estimation method based on dual-branch human body structure modeling according to claim 8, characterized in that In step 3, the feature fusion decoding module consists of an MLP and a third decoding stage, and adds 17 192-dimensional first key point tokens output by the high-confidence key point reasoning branch and 17 192-dimensional second key point tokens output by the component reasoning branch; The first layer of perceptrons in the MLP receives the result of addition and generates a 17×384-dimensional matrix output; the second layer of perceptrons in the MLP receives the 17×384-dimensional matrix output of the first layer of perceptrons after layer normalization and generates a 17×3072-dimensional matrix output; the third decoding stage receives the 17×3072-dimensional matrix output of the second layer of perceptrons in the MLP after layer normalization and generates 17 final predicted key point heat map outputs with a size of 64×48.

10. A 2D human body posture estimation method based on dual-branch human body structure modeling according to claim 1 or 7 or 8 or 9, characterized in that The training process of step 4 is as follows: Step 4.1: Import pre-trained parameters for the backbone network; Step 4.2: Randomly divide all human images of size 256×192 in the training set into multiple batches, so that each batch contains batchsize human images; Step 4.3: Take one batch of the training set and perform data augmentation on all human images in this batch by random rotation, where the rotation degree range is greater than 0° and less than or equal to 45°; Step 4.4: All human images in this batch are used as inputs of the 2D human pose estimation network, and are input into the 2D human pose estimation network for recognition processing. In the middle, 17 initial predicted key point heat maps and 9 corresponding predicted component heat maps corresponding to each human body in each human image in this batch are obtained, and finally 17 final predicted key point heat maps corresponding to each human body in each human image in this batch are obtained; Step 4.5: For each person in each human image in this batch, calculate the initial predicted key point loss , Parts loss and the final predicted key point loss , and the total loss of human pose estimation To supervise the network training, MSE loss is used. , , , , is the two-norm operator symbol, Indicates the first i Zhang initial prediction key point heat map, Indicates the first i Zhang key point labels, Indicates the first j A heat map of predicted components, Indicates the first j Component labels, Indicates the first i Zhang's final predicted key point heat map, It is a device for adjusting and Hyperparameters that influence training; Step 4.6: After the total loss of human pose estimation corresponding to all human bodies in all human images in this batch is calculated, the parameters of the 2D human pose estimation network are trained using the Adam optimizer with a learning rate of 1e-3 to complete the training of the 2D human pose estimation network for this batch; Step 4.7: Repeat the process from step 4.3 to step 4.6 until all batches of the training set have trained the 2D human pose estimation network once; Step 4.8: Randomly divide all human images of size 256×192 in the validation set into multiple batches, so that each batch contains batchsize human images; Step 4.9: Take one of the batches of the validation set; then input all the human images in this batch into the 2D human pose estimation network trained with the training set, and obtain 17 initial predicted key point heat maps and 9 corresponding predicted component heat maps for each human body in each human image in this batch in the middle, and finally obtain 17 final predicted key point heat maps for each human body in each human image in this batch; then follow the process of step 4.5, in the same way, calculate the total loss of human pose estimation corresponding to each human body in each human image in this batch; Step 4.10: Repeat the process of step 4.9 until all batches of the validation set have been processed once; then calculate the average loss of the total loss of human posture estimation corresponding to all human bodies in all human images in the validation set; Step 4.11: Repeat the process from step 4.2 to step 4.10 until the average loss of the 2D human pose estimation network trained with the training set on the validation set converges, and finally obtains a trained 2D human pose estimation network.

Images