Human body posture estimation method based on spatial perception label

Abstract

The invention discloses a human body posture estimation method based on a spatial perception label. According to the method, a multi-resolution feature map is generated by constructing a HighHRNet network, and a key point heat map is generated by using a Gaussian kernel. Meanwhile, a multi-dimensional embedded label containing spatial information is generated by adopting a mode of combining an initial embedded label and coordinate coding, so that the spatial structure of the key point is ensured to be reserved in a label generation process. In the model training process, multiple loss functions are designed, including label similarity loss and space consistency loss, so as to optimize the performance of the model. And finally, the key points are grouped through a Hungary algorithm, so that accurate key point detection of different human bodies is realized. According to the method, while the accuracy of the model is kept, the spatial information can be better utilized, and the stability and robustness of key point detection are improved. And finally, the validity of the method disclosed by the invention is verified on a CrowdPose data set.

Description

Technical Field

[0001] The present invention belongs to the task of multi-person human posture estimation in the field of computer vision, and is a method for grouping key points in an image by generating embedded labels, and in particular refers to a human posture estimation method based on space-aware labels. Background Art

[0002] The multi-person human pose estimation task is an important branch of computer vision, which aims to locate the key points of the human body (such as hands, head, etc.) from the image and infer the human pose. Research in this field has broad application prospects, including human behavior analysis, human-computer interaction, virtual reality, and monitoring systems. Especially in multi-person scenes, accurately identifying and grouping the key points of different human bodies is a very challenging task. Effectively grouping the key points of each person accurately has become a core issue in multi-person pose estimation.

[0003] The current multi-person pose estimation task is mainly divided into two methods: top-down method and bottom-up method. The top-down method first detects the person instance through the human detector, and then detects the key points of each detected instance. In contrast, the bottom-up method first detects all the key points and then groups these key points into different people. In current practical applications, the use of the bottom-up method often achieves more satisfactory results in terms of real-time and accuracy.

[0004] How to group the key points obtained is the main problem faced when using the bottom-up method. The method based on the part field uses the joint information of the character to determine the distance between the key points for grouping. The center regression method uses the center point of the character to represent the target, and groups the target by estimating the offset relative to the center. The Associative Embedding (AE) method predicts a label for each key point, and then assigns the key points to different human targets through the obtained labels through the grouping algorithm. Specifically, the AE method trains the key point label values ​​of the same human target to be similar, while the label values ​​of different targets are different, thereby achieving accurate grouping of key points. In recent years, the method of using embedded labels has been widely used in human posture estimation tasks, and has achieved excellent results on various data sets.

[0005] Although the method based on embedded labels has achieved excellent results on some data sets, it still encounters the problem of reduced accuracy in some complex scenes. This is because the embedded labels randomly generated by the AE method cannot retain the spatial information of the image, and it is difficult to assist key point grouping through spatial feature information in complex scenes. In order to solve this problem, the present invention proposes a human posture estimation method based on spatial perception labels, which aims to add spatial information to the generated embedded labels and retain it, so that the model can obtain the spatial feature information of the image, thereby improving the accuracy of the posture estimation result. Summary of the invention

[0006] This paper proposes a method for human posture estimation based on spatial perception labels. By generating initial labels and coordinate encoding methods, the generated embedded labels retain the spatial information of the image, and by improving the loss function, the key point grouping task is completed while retaining spatial information during training. Its effectiveness is verified on the HigherHRNet network, which improves the accuracy of posture estimation.

[0007] The technical solution used in the present invention includes the following steps:

[0008] Step 1, generation of initial embedded labels. In order to retain the spatial structural information of human key points in the embedded labels, the present invention assigns initial embedded label values ​​to each key point before model training. These label values ​​are assigned according to the spatial distribution of different parts of the human body to ensure that the label values ​​of key points in the same part are more similar during initialization. This initialization step lays the foundation for subsequent label learning, enabling the model to better retain the spatial consistency of human posture during training. The present invention divides the human target into six different parts: head, torso, left arm, right arm, left leg, and right leg, and assigns a label value within a certain range to each part. These label values ​​are generated by adding a random perturbation value to a given central value, as shown in formula (1):

[0009] T+∈(1)

[0010] Where T is the center point of the joint point set in this paper, and ∈ represents a very small random perturbation to avoid generating exactly the same label value and affecting the training results. The specific value ranges of each part are shown in Table 1.

[0011] Location head trunk Left Arm Right Arm Left leg Right Leg Value range [-5-3] [-3,-1] [-1,1] [1,3] [3,5] [5,7]

[0012] Table 1

[0013] Step 2: Introduction of coordinate coding. In order to further enhance the model's capture of spatial information, the present invention adds coordinate coding to the embedding label of each key point. Specifically, the coordinates of the key points in the image are first normalized, as shown in formula (2):

[0014]

[0015] Where x and y are the horizontal and vertical coordinates of the key point, and W and H are the width and height of the image respectively.

[0016] Next, we define the relative position encoding to reflect the relative distance between the key point and the center point of the image. Assume that the normalized coordinates of the center point of the image are (x c ,y c ), the normalized coordinates of the key points are (x k ,y k ), then the coordinate difference of the key point relative to the center point is as shown in formula (3):

[0017] Δx=x k -x c , Δy=y k -y c (3)

[0018] By putting the two coordinate differences and the initialization label value obtained in step 1 into a vector, we can get an embedded label with a dimension of 3. This is the new multi-dimensional label generated by combining the image spatial information with the embedded label. Coordinate encoding allows the label to not only contain spatial information, but also directly reflect the position distribution of key points in the image, enhancing the model's sensitivity to spatial relationships.

[0019] Step 3, key point grouping. The AE method is trained to make the key point label values ​​of the same human body as similar as possible, while the distances between the key point labels of different human bodies are relatively separated. Under this label similarity principle, the Hungarian algorithm is used to complete the task of key point grouping by minimizing the total matching cost. Based on the inheritance of the idea of ​​the AE method, the present invention makes the label value more spatially structured by setting the initial embedded label, and introduces a coordinate encoding method so that the label not only contains the similarity of the label, but also additionally encodes the location information of the key points.

[0020] Therefore, although the present invention improves the embedding label, since the label still retains clear similarity and spatial information, this structure makes the Hungarian algorithm still applicable in the key point grouping task. The Hungarian algorithm can continue to group the key points according to the improved embedding label by minimizing the matching cost, thereby correctly identifying the posture of each human target. Finally, the present invention successfully groups the key points through the Hungarian algorithm.

[0021] Step 4, design a loss function. The loss function designed by the present invention consists of two parts. The first part is the similarity loss, which ensures that the key point labels of the same person are as similar as possible. The second part is the spatial loss, which retains the spatial information in the initial embedded label to prevent the spatial consistency of the label from being lost during the training process. The loss function can be expressed as formula (4):

[0022] L g = λ1 L x +λ2L k (4)

[0023] L k Represents the spatial loss of the label. The purpose of this part of the loss function is to maintain the spatial information in the initial embedded label and keep the relative position relationship of adjacent key points unchanged. This loss function ensures that the trained label can represent the spatial information of the image. k The expression of is shown in formula (5):

[0024]

[0025] Among them, i and j represent the numbers of different key points, e i and e j Represents the label value of the key point, p i and p j represents the coordinates of the key points in the image, and N represents the number of detected key points.

[0026] L x L is the label similarity loss, which minimizes the difference in embedding labels between different key points of the same person target, ensuring that the model can aggregate the key points of the same person target together during training. x The expression of is shown in formula (6):

[0027]

[0028] Where N is the number of instances, K is the number of key point types, {n,k} is the kth key point of the nth person target, and c n,k represents the coordinates of the corresponding key points, m represents the dimension of the embedded label, and the dimension of the label is fixed to 3 in the present invention.

[0029] λ1 and λ2 are hyperparameters. In actual training, the AE method sets λ1 to 1e-3. In order to ensure the comparability of the experiment, the present invention also sets λ1 to 1e-3. As for the hyperparameter λ2, considering that a smaller weight value can ensure that the model pays moderate attention to spatial loss, and based on experimental considerations, the present invention sets λ2 to 1e-4.

[0030] Compared with the prior art, the present invention has the following advantages and improvements:

[0031] 1. In the traditional AE method, the initial embedding label is randomly generated and then gradually optimized through training. However, the method of the present invention consciously designs the initial embedding label and directly encodes the spatial information of the human joint points into the label, making it easier to identify and group the key points of the same part. The addition of this spatial information makes the model more physically constrained during the learning process, thereby improving the accuracy of grouping.

[0032] 2. The method of the present invention further introduces coordinate encoding based on the initial embedded label. By directly encoding the coordinate information of the joint point into the label, the embedded label not only depends on the similarity of the label, but also contains the position information. This multi-dimensional label design increases the richness of model learning, reduces the risk of overfitting, and retains more detailed spatial structure information.

[0033] 3. The present invention designs a special spatial loss function to ensure that the spatial information encoded in the initial embedded labels can be preserved during the training process. This loss function ensures that the key points of the same human body remain consistent in space by measuring the deviation between the embedded labels output by the model and the initial spatial structure, further enhancing the model's spatial structure perception ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Flowchart for generating spatially aware labels for descriptions;

[0035] Figure 2 Visualization results of pose estimation;

[0036] Figure 3 Visualization results of pose estimation. DETAILED DESCRIPTION

[0037] The present invention will be further described in detail below with reference to examples, but the embodiments of the present invention are not limited to the following steps.

[0038] This example is carried out under the pytorch deep learning framework, and the computer configuration adopts: Intel Xeon Platinum 8352V processor, 120GB memory, NVIDIA GeForce RTX4090 graphics card, Linux operating system. This example discloses a human posture estimation method based on spatial perception labels, and its details are as follows:

[0039] Step 1: Training heatmaps and labels. The training and validation sets of the CrowdPose dataset are used to train and validate human pose estimation. The CrowdPose dataset annotates the information of 14 key points of the human body. Each annotation includes the type of key point and the coordinate information of the key point in the image.

[0040] The model needs to build a HigherHRNet network structure, which is divided into four stages, and different stages process feature maps of different resolutions. The network flowchart is as follows Figure 1 As shown in Figure 2, the network starts with a convolutional layer that reduces the resolution of the input image by half using a 3×3 convolution kernel with a stride of 2. The first stage consists of a high-resolution subnet with four residual blocks, and the output is used as a high-resolution feature input to the subsequent stages.

[0041] In the second stage, the network introduces three parallel branches with different resolutions. Each branch is obtained by downsampling the output of the previous stage (3×3 convolution kernel with a step size of 2). The three branches have different resolutions and each branch contains four residual blocks. Assuming that the highest resolution is 1 / 4 of the original input, the resolutions of the other three branches are 1 / 8, 1 / 16, and 1 / 32 respectively. Upsampling or downsampling is achieved between branches through 1×1 convolution kernels, thereby exchanging information across resolutions and fusing feature information of different resolutions.

[0042] The third stage maintains the four-branch structure of the second stage and adds four residual blocks to each branch to further enhance the feature representation capability. The cross-branch connection mechanism continues to be used for feature fusion, combining high-resolution and low-resolution features to extract richer feature information.

[0043] The fourth stage further increases the network depth and enhances the breadth and depth of feature fusion. Four more residual blocks are added to each branch to keep the resolution unchanged. This structure strengthens the feature expression ability and improves the accuracy of key point detection. By increasing the depth, the fourth stage is more flexible in feature fusion, and can more effectively combine multi-resolution information to accurately predict the location of key points.

[0044] Step 2, generation of initial embedded labels. In order to retain the spatial structural information of the key points of the human body in the embedded labels, the present invention assigns an initial embedded label value to each key point before model training. These label values ​​are assigned according to the spatial distribution of different parts of the human body to ensure that the label values ​​of the key points in the same part are more similar during initialization. This initialization step lays the foundation for subsequent label learning, enabling the model to better retain the spatial consistency of human posture during training. The present invention divides the human target into six different parts: head, torso, left arm, right arm, left leg, and right leg, and assigns a label value within a certain range to each part. These label values ​​are generated by adding a given central value to a random perturbation value, as shown in formula (1):

[0045] T+∈(1)

[0046] Where T is the center point of the joint point set in this paper, and ∈ represents a very small random perturbation to avoid generating exactly the same label value and affecting the training results. The specific value range is shown in Table 1.

[0047] Step 2: Use the coordinate encoding method. In order to further enhance the model's capture of spatial information, coordinate encoding is added to the embedded label of each key point. Specifically, the coordinates of the key points in the image are first normalized, as shown in formula (2):

[0048]

[0049] Where x and y are the horizontal and vertical coordinates of the key point, and W and H are the width and height of the image respectively.

[0050] Define relative position encoding to reflect the relative distance between the key point and the center point of the image. Assume that the normalized coordinates of the center point of the image are (x c ,y c ), the normalized coordinates of the key points are (x k ,y k ), then the coordinate difference of the key point relative to the center point is as shown in formula (3):

[0051] Δx=x k -x c , Δy=y k -y c (3)

[0052] By putting the two interpolated coordinates and the initialization label value obtained in step 1 into a vector, we can get an embedded label with a dimension of 3. This is the new multi-dimensional label generated by combining the image spatial information with the embedded label. Coordinate encoding allows the label to not only contain spatial information, but also directly reflect the position distribution of key points in the image, enhancing the model's sensitivity to spatial relationships.

[0053] Step 3, design the loss function. The loss function designed by the invention consists of two parts. The first part is the similarity loss, which ensures that the key point labels of the same person are as similar as possible. The second part is the spatial loss, which retains the spatial information in the initial embedded label to prevent the spatial consistency of the label from being lost during the training process. The loss function can be expressed as formula (4):

[0054] L g = λ1 L x +λ2L k (4)

[0055] L k Represents the spatial loss of the label. The purpose of this part of the loss function is to maintain the spatial information in the initial embedded label and keep the relative position relationship of adjacent key points unchanged. This loss function ensures that the trained label can represent the spatial information of the image. k The expression of is shown in formula (5):

[0056]

[0057] Among them, i and j represent the numbers of different key points, e i and e j Represents the label value of the key point, p i and p j represents the coordinates of the key points in the image, and N represents the number of detected key points.

[0058] L x L is the label similarity loss, which minimizes the difference in embedding labels between different key points of the same person target, ensuring that the model can aggregate the key points of the same person target together during training. x The expression of is shown in formula (6):

[0059]

[0060] Where N is the number of instances, K is the number of key point types, {n,k} is the kth key point of the nth person target, and c n,k represents the coordinates of the corresponding key points, m represents the dimension of the embedded label, and the dimension of the label is fixed to 3 in the present invention.

[0061] λ1 and λ2 are hyperparameters. In actual training, the AE method sets λ1 to 1e-3. In order to ensure the comparability of the experiment, the present invention also sets λ1 to 1e-3. As for the hyperparameter λ2, considering that a smaller weight value can ensure that the model pays moderate attention to spatial loss, and based on experimental considerations, the present invention sets λ2 to 1e-4.

[0062] Step 4: Construct a human posture instance. HigherHRNet is a multi-person posture estimation network. It is necessary to assign the detected key points to the corresponding human targets. The key points are grouped and connected to each other using the labels obtained through network training using the Hungarian algorithm. A complete human posture instance is constructed to obtain the posture estimation result predicted by the network. The posture estimation result is shown in the figure below. Figure 2 , Figure 3 As shown. Finally, the prediction results are output in json file format.

[0063] The comparison of the indicators of HigherHRNet and the present invention in the CrowdPose validation set is shown in Table 2. It can be seen that the average accuracy index of the present invention has been further improved.

[0064] method Pre-training Input image resolution AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AP E ]]> HigherHRNet have 512*512 65.9 86.4 70.6 73.3 The present invention have 512*512 66.7 86.9 71.7 74.3

[0065] Table 2

[0066] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A human posture estimation method based on spatial perception labels, characterized by The following steps are involved: Step 1: Training heatmaps and labels. The training and validation sets of the CrowdPose dataset are used to train and validate human pose estimation. The CrowdPose dataset annotates the information of 14 key points of the human body. Each annotation includes the type of key point and the coordinate information of the key point in the image. The model needs to build a HigherHRNet network structure, and its network architecture is divided into four stages, with different stages processing feature maps of different resolutions. The network flowchart is shown in Figure 1. The network starts with a convolutional layer, using a 3×3 convolution kernel with a stride of 2 to reduce the resolution of the input image by half. The first stage consists of a high-resolution subnet, which contains four residual blocks, and the output is used as a high-resolution feature input to the subsequent stage. In the second stage, the network introduces three parallel branches with different resolutions. Each branch is obtained by downsampling the output of the previous stage (3×3 convolution kernel with a step size of 2). The three branches have different resolutions and each branch contains four residual blocks. Assuming that the highest resolution is 1 / 4 of the original input, the resolutions of the other three branches are 1 / 8, 1 / 16, and 1 / 32 respectively. Upsampling or downsampling is achieved between branches through 1×1 convolution kernels, thereby exchanging information across resolutions and fusing feature information of different resolutions. The third stage maintains the four-branch structure of the second stage and adds four residual blocks to each branch to further enhance the feature representation capability. The cross-branch connection mechanism continues to be used for feature fusion, combining high-resolution and low-resolution features to extract richer feature information. The fourth stage further increases the network depth and enhances the breadth and depth of feature fusion. Four more residual blocks are added to each branch to keep the resolution unchanged. This structure strengthens the feature expression ability and improves the accuracy of key point detection. By increasing the depth, the fourth stage is more flexible in feature fusion, and can more effectively combine multi-resolution information to accurately predict the location of key points. Step 2, generation of initial embedded labels. In order to retain the spatial structural information of the key points of the human body in the embedded labels, the present invention assigns an initial embedded label value to each key point before model training. These label values ​​are assigned according to the spatial distribution of different parts of the human body to ensure that the label values ​​of the key points in the same part are more similar during initialization. This initialization step lays the foundation for subsequent label learning, enabling the model to better retain the spatial consistency of human posture during training. The present invention divides the human target into six different parts: head, torso, left arm, right arm, left leg, and right leg, and assigns a label value within a certain range to each part. These label values ​​are generated by adding a given central value to a random perturbation value, as shown in formula (1): T+∈(1) where T is the center point of the joint point set in this paper, and ∈ represents a very small random perturbation to avoid generating exactly the same label value and affecting the training results. The specific value range is shown in Table 1. Step 2: Use the coordinate encoding method. In order to further enhance the model's capture of spatial information, coordinate encoding is added to the embedded label of each key point. Specifically, the coordinates of the key points in the image are first normalized, as shown in formula (2): Where x and y are the horizontal and vertical coordinates of the key point, and W and H are the width and height of the image respectively. Next, we define the relative position encoding to reflect the relative distance between the key point and the center point of the image. Assume that the normalized coordinates of the center point of the image are (x c ,y c ), the normalized coordinates of the key points are (x k ,y k ), then the coordinate difference of the key point relative to the center point is as shown in formula (3): Δx=x k -x c ,Δy=y k -y c (3) By putting the two coordinate differences and the initialization label value obtained in step 1 into a vector, we can get an embedded label with a dimension of 3. This is the new multi-dimensional label generated by combining the image spatial information with the embedded label. Coordinate encoding allows the label to not only contain spatial information, but also directly reflect the position distribution of key points in the image, enhancing the model's sensitivity to spatial relationships. Step 3, design a loss function. The loss function designed for the present invention consists of two parts. The first part is the similarity loss, which ensures that the key point labels of the same person are as similar as possible. The second part is the spatial loss, which retains the spatial information in the initial embedded label to prevent the spatial consistency of the label from being lost during the training process. The loss function can be expressed as formula (4): L g =λ1 L x +λ2L k (4) L k Represents the spatial loss of the label. The purpose of this part of the loss function is to maintain the spatial information in the initial embedded label and keep the relative position relationship of adjacent key points unchanged. This loss function ensures that the trained label can represent the spatial information of the image. k The expression of is shown in formula (5): Among them, i and j represent the numbers of different key points, e i and e j Represents the label value of the key point, p i and p j represents the coordinates of the key points in the image, and N represents the number of detected key points. L x L is the label similarity loss, which minimizes the difference in embedding labels between different key points of the same person target, ensuring that the model can aggregate the key points of the same person target together during training. x The expression of is shown in formula (6): Where N is the number of instances, K is the number of key point types, {n,k} is the kth key point of the nth person target, and c n,k represents the coordinates of the corresponding key points, m represents the dimension of the embedded label, and the dimension of the label is fixed to 3 in the present invention. λ1 and λ2 are hyperparameters. In actual training, the AE method sets λ1 to 1e-3. In order to ensure the comparability of the experiment, the present invention also sets λ1 to 1e-3. As for the hyperparameter λ2, considering that a smaller weight value can ensure that the model pays moderate attention to spatial loss, and based on experimental considerations, the present invention sets λ2 to 1e-4. Step 4: Construct a human posture instance. HigherHRNet is a multi-person posture estimation network. It is necessary to assign the detected key points to the corresponding human targets. The labels obtained through network training using the Hungarian algorithm are used to group the key points and connect them to each other. A complete human posture instance is constructed to obtain the posture estimation results predicted by the network. The results of posture estimation are shown in Figures 2 and 3. Finally, the prediction results are output in the json file format.

Images