A human body posture estimation model and a human body posture estimation method based on the model

Abstract

The present invention proposes a human posture estimation model and a human posture estimation method based thereon. The human posture estimation model is composed of a convolutional neural network and a parallel human region segmentation network, a human joint positioning network and a human limb positioning network; the convolutional neural network is used to extract a human posture feature map from an input two-dimensional human image as the input of the human region segmentation network, the human joint positioning network and the human limb positioning network; the human region segmentation network is used to distinguish the human region from the background region, the human joint positioning network is used to locate 25 joints of the human body, and the human limb positioning network is used to locate 23 limbs between two joints. The present invention combines the human region segmentation network with the human joint positioning network and the human limb positioning network, and uses the cumulative sum of the three network loss functions as the optimization objective function of the entire estimation model, thereby eliminating the need to perform superposition processing on the loss results obtained by multiple iterations.

Description

Technical Field

[0001] The present invention relates to the technical field of human body posture recognition, and in particular to a human body posture estimation model and a human body posture estimation method based thereon. Background Art

[0002] Human pose estimation is one of the important research directions in the field of computer vision and is widely used in human activity analysis, human-computer interaction, and video surveillance. Human pose estimation refers to the use of computer algorithms to locate key points of the human body in an image or video, such as the shoulder, elbow, wrist, hip, knee, ankle and other joints. Human pose estimation has great research significance and practical value. For example, it can be used for fall monitoring and can be used for teaching projects such as fitness, sports, and dance. In the field of robot training, human pose estimation technology enables robots to "learn" to move their joints; in common movie special effects production and interactive games, human pose estimation technology is used to track human movements to achieve synchronization and integration of virtual and real characters. Summary of the invention

[0003] In view of the increasing demand for human body posture estimation, the present invention proposes a human body posture estimation model and a human body posture estimation method based thereon. Compared with the existing human body posture estimation method, the human body posture estimation result map can be obtained quickly and accurately.

[0004] The present invention protects a human body posture estimation model, which is composed of two-level networks connected in series, wherein the first-level network is a convolutional neural network, and the second-level network is a parallel human body region segmentation network, a human body joint positioning network and a human body limb positioning network.

[0005] The convolutional neural network is used to extract a human posture feature map from an input two-dimensional human image as input to the human region segmentation network, the human joint positioning network and the human limb positioning network.

[0006] The human body region segmentation network outputs 1 feature map for distinguishing the human body region from the background region, the human body joint positioning network outputs 25 feature maps for locating the 25 joints of the human body, and the human body limb positioning network outputs 23 feature maps for locating the 23 limbs between two joints.

[0007] The optimization objective function of the entire estimation model is in is the loss function of the human body segmentation network at the sth iteration during training, is the loss function of the human joint positioning network at the sth iteration during training, is the loss function of the human limb localization network at the sth iteration during the training process, H(k) represents the probability that the pixel k in the input two-dimensional human image is a human region, and C i (k) represents the feature map of the i-th joint of the human body, B m (k) represents the characteristic graph of the mth limb of the human body, H s (k) represents the probability that the pixel k in the input two-dimensional human image is a human body region at the sth iteration during the training process, represents the feature map of the i-th joint of the human body at the s-th iteration during training, It represents the feature map of the mth human limb at the sth iteration during the training process, i represents the ith joint, m represents the mth limb, I represents the total number of human joints, and M represents the total number of human limbs.

[0008] Furthermore, during the training of the human joint localization network, the target output of the feature map is C i (k) = max j C i,j (k), where represents the feature map of the i-th joint of the j-th person in the training sample image, K represents the coordinate position of pixel k, x i,j represents the position coordinates of the i-th joint of the j-th person. In the inference process, since the number of people is unknown, the feature map of each joint is directly calculated. w (k) represents the feature map of the w-th joint, which means That is, the probability that pixel k is a joint, K represents the coordinate position of pixel k, x w Represents the position coordinate of the wth joint. Obviously, the closer to the joint, the closer the value of the feature map is to 1, and the farther away from the joint, the closer the value of the feature map is to 0.

[0009] Furthermore, in the human limb localization network, Where v(k) is a two-dimensional unit vector indicating the direction of the mth limb; α(k) is a scalar indicating the confidence that pixel k belongs to the limb, 0≤α(k)≤1.

[0010] The present invention also protects a method for estimating a human body posture, comprising the following steps:

[0011] Step 1, inputting a two-dimensional human body image to be processed into the above-mentioned human posture estimation model;

[0012] Step 2, a convolutional neural network extracts a human posture feature map in the two-dimensional human image, and inputs the feature map into a human region segmentation network, a human joint positioning network, and a human limb positioning network respectively;

[0013] Step 3, the human body region segmentation network outputs a human body region segmentation feature map in the two-dimensional human body image, the human body joint positioning network outputs a feature map of all joints of each person in the two-dimensional human body image, and the human body limb positioning network outputs a direction feature map and a confidence feature map of all limbs of each person in the two-dimensional human body image;

[0014] Step 4: The pixel point k whose probability value of the human body region segmentation feature map is greater than the preset value P is determined to be a human body region, otherwise it is determined to be a non-human body region; the human joint positioning network takes the first N points with the largest probability value of the feature map as the N candidate points of each joint point; the human limb positioning network takes the first Q limbs with the largest probability value of the confidence feature map as the Q candidate ranges of each limb;

[0015] Step 5, the probability value of the human body region segmentation feature map of any pixel point k in the two-dimensional human body image is multiplied by the probability values ​​of the 25 joint point feature maps to represent the probability that the pixel point k is both a human body region and a joint, and the points before the product probability values ​​of each joint feature map are taken to obtain 25*R joint candidate points;

[0016] Step 6, the probability value of the human body region segmentation feature map of any pixel point k in the two-dimensional human body image is multiplied by the probability values ​​of the 23 limb confidence feature maps to represent the probability that the pixel point k is both a human body region and a limb, and the points before the product probability values ​​of each limb feature map are taken to obtain a 23*U limb candidate range;

[0017] Step 7, comparing the candidate joint point with the candidate limb. If the distance between the candidate joint point and the two end points of the limb is less than a preset value, the candidate joint point is the joint point, and the candidate limb is the limb, and finally the joint point, limb and its coordinate information are obtained;

[0018] Step 8, complete the human body posture estimation based on the obtained joint point and limb recognition results, draw the human body posture estimation result map based on the coordinate information, and display and save the result map.

[0019] The beneficial effects of the present invention are as follows: 1. The human body region segmentation network is combined with the human body joint positioning network and the human body limb positioning network, and the cumulative sum of the three network loss functions is used as the optimization objective function of the entire estimation model, so that there is no need to superimpose the loss results obtained from multiple iterations; 2. The extraction of confidence feature maps is added to the human body limb positioning network. Compared with the existing network, more information can be obtained from the same input image, thereby improving the recognition accuracy; 3. The network post-processing process can obtain the posture estimation result according to the extracted features after the network is running, which can reduce the number of model runs to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of human posture estimation model;

[0021] Figure 2 It is a schematic diagram of the key joints and key limbs of the human body;

[0022] Figure 3 Schematic diagram of the input image;

[0023] Figure 4 This is the human body posture estimation result diagram output by the model. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0025] Example 1

[0026] A human pose estimation model, such as Figure 1 As shown, it consists of two levels of networks in series, the first level network is a convolutional neural network, and the second level network is a parallel human body region segmentation network, a human body joint positioning network and a human body limb positioning network. Figure 2 This is a schematic diagram of the key joints and key limbs of the human body. There are 25 key joints and 23 key limbs in the diagram.

[0027] The convolutional neural network is used to extract a human posture feature map from an input two-dimensional human image as input to the human region segmentation network, the human joint positioning network and the human limb positioning network.

[0028] The human body region segmentation network outputs 1 feature map for distinguishing the human body region from the background region; the human body joint positioning network outputs 25 feature maps for positioning the 25 human body joints; the human body limb positioning network outputs 23 feature maps for positioning the 23 limbs between two joints. When the human body in the two-dimensional input image has an occlusion relationship, a person may have less than 25 joint points and less than 23 limbs.

[0029] The optimization objective function of the entire estimation model is in is the loss function of the human body segmentation network at the sth iteration during training, is the loss function of the human joint positioning network at the sth iteration during training, is the loss function of the human limb localization network at the sth iteration during the training process, H(k) represents the probability that the pixel k in the input two-dimensional human image is a human region, and C i (k) represents the feature map of the i-th joint of the human body, B m (k) represents the characteristic graph of the mth limb of the human body, H s (k) represents the probability that the pixel k in the input two-dimensional human image is a human body region at the sth iteration during the training process, represents the feature map of the i-th joint of the human body at the s-th iteration during training, It represents the feature map of the mth human limb at the sth iteration during the training process, i represents the ith joint, m represents the mth limb, I represents the total number of human joints, and M represents the total number of human limbs.

[0030] Compared with the existing human posture estimation model, the present invention combines the human body region segmentation network with the human body joint positioning network and the human body limb positioning network, and takes the cumulative sum of the three network loss functions as the optimization objective function of the entire estimation model, thereby eliminating the need to superimpose the loss results obtained from multiple iterations.

[0031] H(k) represents the probability that the pixel k in the input two-dimensional human body image is a human body region. To ensure that H(k) is between 0 and 1, the output of the previous layer of the neural network is processed by sigmoid, that is, Where H'(k) represents the output of the human body region segmentation network before outputting H(k). The setting of the human body region segmentation network can directly limit the positioning of joints and limbs to the human body region, greatly improving the calculation efficiency.

[0032] During the training process of the human joint localization network, the target output of the feature map is C i (k) = max j C i,j (k), where represents the feature map of the i-th joint of the j-th person, K represents the coordinate position of pixel k, x i,j represents the position coordinates of the i-th joint of the j-th person. During the reasoning process, C w (k) represents the feature map of the w-th joint, which means That is, the probability that pixel k is a joint, K represents the coordinate position of pixel k, x w Represents the position coordinate of the wth joint. Obviously, the closer to the joint, the closer the value of the feature map is to 1, and the farther away from the joint, the closer the value of the feature map is to 0. In the human limb positioning network, Where v(k) is a two-dimensional unit vector indicating the direction of the mth limb; α(k) is a scalar indicating the confidence that pixel k belongs to the limb, 0≤α(k)≤1.

[0033] The human limb localization network outputs 23 feature maps, each feature map B m (k) contains three feature maps, in addition to the feature map representing the confidence α, there are two feature maps v representing the direction x 、v y , where v x represents the coordinate value of the limb vector in the x direction, v y Represents the coordinate value of the limb vector in the y direction. The normalized activation function is used to normalize the vector size to 1. The calculation formula is: where v' x 、v' y Represents the output of the normalized previous human limb localization network.

[0034] By adding the extraction of confidence feature maps to the human limb localization network, compared with the existing network, it is possible to obtain more information from the same input image and improve the recognition accuracy.

[0035] Below Figure 3 The human body posture estimation method based on the human body posture estimation model is described using the input image of the estimation model, including the following steps:

[0036] Step 1: Input the to-be-processed Figure 3 , Figure 3 Five people can be identified in the close-up.

[0037] Step 2: The convolutional neural network extracts the human body posture feature map in the two-dimensional human body image and inputs it into the human body region segmentation network, the human body joint positioning network and the human body limb positioning network respectively.

[0038] Step 3, the human body region segmentation network outputs the human body region segmentation feature map in the two-dimensional human body image (that is, the probability that any pixel point k is a human body region), the human joint positioning network outputs the feature map of all joints of each person in the two-dimensional human body image, and the human limb positioning network outputs the direction feature map and confidence feature map of all limbs of each person in the two-dimensional human body image.

[0039] Step 4: If the probability value of the pixel point k in the human body region segmentation feature map is greater than the preset value P, it is determined to be a human body region, otherwise it is determined to be a non-human body region; the human joint positioning network takes the first N points with the largest probability values ​​in the feature map as the N candidate points for each joint point; the human limb positioning network takes the first Q limbs with the largest probability values ​​in the confidence feature map as the Q candidate ranges for each limb.

[0040] Step 5, the probability value of the human body region segmentation feature map of any pixel point k in the two-dimensional human body image is multiplied by the probability values ​​of the 25 joint point feature maps to represent the probability that the pixel point k is both a human body region and a joint, and the points before R of the product probability values ​​of each joint feature map are taken to obtain 25*R joint candidate points.

[0041] Step 6, the probability value of the human body region segmentation feature map of any pixel point k in the two-dimensional human body image is multiplied with the probability values ​​of the 23 limb confidence feature maps to characterize the probability that the pixel point k is both a human body region and a limb, and the points before U of the product probability values ​​of each limb feature map are taken to obtain a 23*U limb candidate range.

[0042] Step 7, compare the joint candidate point with the limb candidate. If the distance between the joint candidate point and the two end points of the limb is less than the preset value, the joint candidate point is the joint point, and the limb candidate is the limb, and finally the joint point, limb and its coordinate information are obtained.

[0043] Step 8: Based on the obtained joint point and limb recognition results, complete the human body posture estimation, and draw the human body posture estimation result map according to the coordinate information (refer to Figure 4 ) to display and save the resulting graph.

[0044] Based on the post-processing flow of steps 4-7, after the network model is run, the posture estimation result can be obtained according to the extracted features, which can reduce the number of model runs to a certain extent. Note that the above P, Q, R, and U are set according to actual needs, and the present invention does not limit their values.

[0045] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention.

Claims

1. A human body posture estimation model, characterized in that: It is composed of two-stage networks connected in series, the first-stage network is a convolutional neural network, and the second-stage network is a human body region segmentation network, a human body joint positioning network and a human body limb positioning network connected in parallel; the convolutional neural network is used to extract a human body posture feature map from an input two-dimensional human body image as the input of the human body region segmentation network, the human body joint positioning network and the human body limb positioning network; the human body region segmentation network outputs a feature map for distinguishing the human body region from the background region; The human joint positioning network outputs 25 feature maps for positioning 25 human joints; the human limb positioning network outputs 23 feature maps for positioning 23 limbs between two joints; The optimization objective function of the entire estimation model is in is the loss function of the human body segmentation network at the sth iteration during training, is the loss function of the human joint positioning network at the sth iteration during training, is the loss function of the human limb localization network at the sth iteration during the training process, H(k) represents the probability that the pixel k in the input two-dimensional human image is a human region, and C i (k) represents the feature map of the i-th joint of the human body, B m (k) represents the characteristic graph of the mth limb of the human body, H s (k) represents the probability that the pixel k in the input two-dimensional human image is a human body region at the sth iteration during the training process, represents the feature map of the i-th joint of the human body at the s-th iteration during training, It represents the feature map of the mth human limb at the sth iteration during the training process, i represents the ith joint, m represents the mth limb, I represents the total number of human joints, and M represents the total number of human limbs.

2. The human body posture estimation model according to claim 1, characterized in that In the human joint localization network, C i (k) = max j C i，j (k), where represents the feature map of the i-th joint of the j-th person, K represents the coordinate position of pixel k, x i，j Represents the position coordinates of the i-th joint of the j-th person.

3. The human body posture estimation model according to claim 2, characterized in that: In the human limb localization network, Where v(k) is a two-dimensional unit vector indicating the direction of the mth limb; α(k) is a scalar indicating the confidence that pixel k belongs to the limb, 0≤α(k)≤1.

4. A method for estimating a human body posture, characterized in that: The following steps are involved: Step 1, inputting a two-dimensional human body image to be processed into the human body posture estimation model described in claim 3; Step 2, a convolutional neural network extracts a human posture feature map in the two-dimensional human image, and inputs the feature map into a human region segmentation network, a human joint positioning network, and a human limb positioning network respectively; Step 3, the human body region segmentation network outputs a human body region segmentation feature map in the two-dimensional human body image, the human body joint positioning network outputs a feature map of all joints of each person in the two-dimensional human body image, and the human body limb positioning network outputs a direction feature map and a confidence feature map of all limbs of each person in the two-dimensional human body image; Step 4: The pixel point k whose probability value of the human body region segmentation feature map is greater than the preset value P is determined to be a human body region, otherwise it is determined to be a non-human body region; the human joint positioning network takes the first N points with the largest probability value of the feature map as the N candidate points of each joint point; the human limb positioning network takes the first Q limbs with the largest probability value of the confidence feature map as the Q candidate ranges of each limb; Step 5, the probability value of the human body region segmentation feature map of any pixel point k in the two-dimensional human body image is multiplied by the probability values ​​of the 25 joint point feature maps to represent the probability that the pixel point k is both a human body region and a joint, and the points before the product probability values ​​of each joint feature map are taken to obtain 25*R joint candidate points; Step 6, the probability value of the human body region segmentation feature map of any pixel point k in the two-dimensional human body image is multiplied by the probability values ​​of the 23 limb confidence feature maps to represent the probability that the pixel point k is both a human body region and a limb, and the points before the product probability values ​​of each limb feature map are taken to obtain a 23*U limb candidate range; Step 7, comparing the candidate joint point with the candidate limb. If the distance between the candidate joint point and the two end points of the limb is less than a preset value, the candidate joint point is the joint point, and the candidate limb is the limb, and finally the joint point, limb and its coordinate information are obtained; Step 8, complete the human body posture estimation based on the obtained joint point and limb recognition results, draw the human body posture estimation result map based on the coordinate information, and display and save the result map.

Images