Sub-pixel heat map decoding method based on curved surface fitting

Abstract

The invention discloses a sub-pixel heat map decoding method based on curved surface fitting. According to the invention, a High is used as a backbone network. In order to solve the problem of inaccurate decoding coordinates of a human body key point heat map, a curved surface fitting mode is used for optimizing a decoding process, a Taylor series is used for fitting a curved surface of a maximum value point region, and key point sub-pixel coordinates with more accurate positions are obtained through calculation, so that a posture estimation result is more accurate. The heat map decoding mode of the key point sub-pixel coordinates is designed, the accuracy of the decoded coordinates is effectively enhanced, and the precision of the attitude estimation result output by the model is improved. And finally, the validity of the method is verified on a CrowdPose data set.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and is a method for estimating human posture based on a heat map, in particular, it refers to a method for obtaining precise coordinates of key points by using a sub-pixel surface fitting decoding method. Background Art

[0002] With the continuous development of deep learning technology, the research on human pose estimation has also grown, achieving results far beyond traditional algorithms and attracting widespread attention from researchers. Human pose estimation is a computer vision task that aims to detect the position of specific key points of a human target, connect the key points based on prior knowledge, and finally obtain a predicted human skeleton.

[0003] In the task of human posture estimation, there are two common coordinate representation designs: coordinate regression and heat map regression. The coordinate regression method directly uses coordinates as the output of the model. This is a more intuitive method, but in actual application, it is found that the coordinate regression method has some problems. For example, the function learned by the model is highly nonlinear, which is not conducive to network learning. In addition, the coordinate regression method only outputs coordinates, which results in less supervision information, resulting in greater ambiguity between key point positions, which is not conducive to network learning. Currently, only a few methods use coordinate regression for prediction.

[0004] In contrast, the heat map regression method solves the above problems very well. In the heat map regression method, the model extracts features from the input image, and then generates K heat maps with dimensions (W, H) through a Gaussian kernel, where W and H represent the width and height of the image, respectively, and K represents the number of key points. This process of generating a heat map is usually called the heat map encoding process. The position of the key point in the generated heat map corresponds to the vicinity of the peak on the heat map. The maximum value point of the heat map is usually taken as the position of the target key point. This is the process of heat map decoding. Most of the current methods use heat map regression to complete it.

[0005] In actual application, because the scale of the generated heat map is smaller than the scale of the input image, if the coordinates of the maximum point are directly used as the prediction result, the result will only contain the integer part, and then a quantization error will occur. Therefore, the obtained key point coordinates must be further processed to reduce the error. There are currently two main methods: The first method is to offset the predicted coordinate point by a quarter of a pixel in the direction of the second maximum point. This is a method that is completely based on experiments and has no corresponding theoretical basis, but because it has achieved good results in many methods in recent years, it has been widely used. The second method is to perform sub-pixel surface fitting decoding on the obtained heat map, and calculate the coordinates of the key point by fitting the surface around the key point.

[0006] Compared with the first method, the second method for heat map decoding is more convincing in theory, and the results also prove that the coordinates obtained by the second method are more accurate than those obtained by the first method. However, the current methods of decoding using sub-pixel surface fitting are all based on the assumption that the predicted heat map highly obeys the Gaussian distribution. In fact, the heat map obtained after observing the network training can be found to have multiple peaks around the maximum point, so the predicted heat map must be smoothed first, which will have a certain impact on the accuracy of the result. The present invention decodes the heat map without being defined by the above assumptions, performs surface fitting on the predicted heat map through the expansion of the Taylor series, and calculates the fitted equation and its partial derivatives from the eight pixel points around the maximum point on the predicted heat map, and then calculates the decoded key point coordinates. Finally, the method proposed in this article is verified on the CrowdPose target dataset through the HigherHRNet network to verify the effectiveness of the method. Summary of the invention

[0007] The present invention proposes a method for human posture estimation based on sub-pixel surface fitting heat map decoding. Some of the methods used for heat map decoding were relatively simple, and some of the fitting of local area surfaces was not accurate enough, resulting in inaccurate key point coordinates obtained by the final decoding, making it difficult to further improve the accuracy of model prediction. To address the above problems, the present invention improves the heat map decoding method so that the network can obtain more accurate key point coordinates, and verifies its effectiveness on the HigherHRNet network, thereby improving the accuracy of posture estimation.

[0008] The technical solution used in the present invention includes the following steps:

[0009] Step 1: Training heatmap: divide the posture estimation dataset into two parts: training set and validation set, and build a HigherHRNet network to train it to get the output heatmap. For a multi-scale prediction network like HigherHRNet, after getting the predicted heatmap output by the network, the heatmaps of different resolutions must be fused first. This process requires sampling the heatmaps of different resolutions to a uniform resolution and averaging them to get a more accurate heatmap.

[0010] Step 2, key point location. The network will output the corresponding predicted heat map for M human key points. The pixel value of each coordinate point on the heat map represents the probability of the predicted key point appearing at that position. Therefore, after obtaining the heat map, it is necessary to perform local maximum value detection on the heat map of each key point and find the maximum value point (x0, y0) in the map, that is, the coordinate point with the highest probability of the key point appearing. Around each predicted key point, a 3×3 neighborhood containing the point is selected. This neighborhood contains the key point and the surrounding 8 coordinate points. The coordinates and pixel values ​​of these 9 points will be used for subsequent calculations.

[0011] Step 3: Set and solve the equation to find the parameters of the quadratic function. For the binary function f(x,y), the second-order Taylor expansion near the point (x0,y0) can be expressed as formula (1):

[0012]

[0013] Using the activation values ​​of the existing 9 points, a local two-dimensional quadratic function is fitted. The function form is shown in formula (2):

[0014] f(x,y)=ax 2 +by 2 +cxy+dx+ey+f (2)

[0015] By comparing equation (1) with equation (2), it can be seen that the fitting function actually contains all the terms that appear in the second-order expansion of the Taylor series. By substituting the information of the existing coordinate points and setting at least 6 equations, the coefficients a, b, c, d, e, and f of the quadratic function can be solved.

[0016] Step 4, solve the partial derivative equation to obtain the extreme point of the quadratic function. After solving the coefficients of the quadratic function, the next step is to find the extreme point of the quadratic function by solving the partial derivative equations. Specifically, we take the partial derivatives of the function f(x, y) with respect to x and y respectively, obtain two equations and set these two partial derivatives to zero. The partial derivative equations are shown in equations (3) and (4):

[0017]

[0018] Solving this system of equations to obtain the values ​​of x and y, the sub-pixel coordinates of the key points after heat map decoding (x * ,y * ) as shown in formula (5) and formula (6):

[0019] x * =x0+x (5)

[0020] y * =y0+y (6)

[0021] That is, based on the existing maximum value coordinate point obtained by maximum value suppression, the sub-pixel coordinates of the maximum value point are calculated by surface fitting. Through the above steps, we can decode the position of the key points from the heat map with higher accuracy, thereby making the results obtained by the human posture network more accurate.

[0022] Step 5: Construct a human posture instance. Because HigherHRNet is a multi-person posture estimation network, it is necessary to assign the detected key points to the corresponding human targets, use the Hungarian algorithm to obtain the labels obtained through network training to group the key points and connect them to each other, construct a complete human posture instance, and finally obtain the posture estimation result predicted by the network.

[0023] Compared with the prior art, the present invention has the following advantages and improvements:

[0024] 1. The existing method of heat map decoding based on sub-pixel surface fitting takes the predicted heat map to obey Gaussian distribution as a premise and is more dependent on the distribution of the predicted heat map. The present invention combines the pixels in the neighborhood of the key points to perform surface fitting on the predicted heat map through the second-order expansion of the Taylor series, which completes the heat map decoding process more accurately and improves the accuracy of the results of the human posture estimation task.

[0025] 2. Unlike most traditional methods that directly shift the maximum point of the heat map to the second maximum point by a quarter of a pixel, the present invention obtains the sub-pixel coordinates of the key points after heat map decoding by calculation, and can effectively capture the local changes of the surface based on the expansion of the second-order Taylor series, thereby obtaining more accurate predictions in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flowchart describing heatmap decoding.

[0027] Figure 2 , Figure 3 is the decoded key point position image. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below with reference to examples, but the embodiments of the present invention are not limited to the following steps.

[0029] This example is carried out under the pytorch deep learning framework, and the computer configuration adopts: Intel Xeon Platinum 8352V processor, 120GB memory, NVIDIA GeForce RTX4090 graphics card, Linux operating system. This example discloses a human posture estimation method based on surface fitting for heat map decoding, and its details are as follows:

[0030] Step 1: Training heatmap. Use the training set and validation set of the Crowdpose dataset to train and validate human pose estimation. The CrowdPose dataset annotates the label information of 14 key points of the human body. Each annotation information includes the type of key point and the coordinate information of the key point in the image. This information will then be converted into a real heatmap for comparison with the predicted heatmap.

[0031] Next, we build the HigherHRNet network structure, which can be divided into four stages. Different stages contain features of different resolutions. After the last stage, we convolve the feature map with a Gaussian kernel to obtain a key point heat map.

[0032] The network starts with a convolutional layer that halves the resolution of the input image using a 3×3 convolution kernel with a stride of 2. The first stage consists of a high-resolution subnet with four residual blocks, and the output of this stage is provided as high-resolution input to subsequent stages.

[0033] In the second stage, the network will add three branches of different resolutions in parallel. Each branch is obtained by downsampling the output of the previous stage with a 3×3 convolution kernel with a stride of 2. The three branches have different resolutions and contain four residual blocks. Assuming the highest resolution is 1 / 4 of the original input resolution, the resolutions of the other three branches are 1 / 8, 1 / 16 and 1 / 32 respectively. These branches are upsampled or downsampled by a 1×1 convolution kernel to exchange information across resolutions in order to fuse feature information of different resolutions.

[0034] In the third stage, while maintaining the structure of the four branches formed at the end of the second stage, the fusion of depth and enhanced feature information is further increased, and four residual blocks are added to each branch. Each residual block is designed to enhance the feature representation capability of a specific resolution branch. Similar to the second stage, feature fusion in the third stage is also achieved through cross-branch connections. This mode combines high-resolution and low-resolution features to extract richer and more comprehensive feature information.

[0035] The fourth stage further increases the depth of the network and strengthens the breadth and depth of feature fusion. It maintains the structure of four branches and the resolution remains unchanged. Similar to the third stage, four residual blocks are added to each branch, which further enhances the ability to express feature information on each branch and improves the accuracy of key point detection. By increasing the depth, the fourth stage is more flexible in feature information fusion and can more effectively combine information of different resolutions to make more accurate predictions of key point locations. After the fourth stage, 1×1 convolution is used to unify the number of feature maps of each resolution, and then they are connected to the branch with the highest resolution through upsampling to form the final feature map, and the key point heat map is generated through the Gaussian kernel function.

[0036] Step 2, key point location. The network will output the corresponding predicted heat map for M human key points. The pixel value of each coordinate point on the heat map represents the probability of the predicted key point appearing at that position. Therefore, after obtaining the heat map, the heat map of each key point needs to be used for non-maximum suppression to find the maximum value point (x0, y0) in the map, that is, the coordinate point with the highest probability of the key point appearing. Around each predicted key point, a 3×3 neighborhood containing the point is selected. This neighborhood contains the key point and the surrounding 8 coordinate points. The coordinates and pixel values ​​of the 9 points obtained will be used for subsequent calculations.

[0037] Step 3: Set and solve the equation to find the parameters of the quadratic function. For the binary function f(x,y), the second-order Taylor expansion near the point (x0,y0) can be expressed as formula (1):

[0038]

[0039] Next, we select (x0, y0) as the expansion point. The coefficients of the quadratic term and the cross term represent the shape change of the surface around this point, which is compared with the two-dimensional quadratic function set by formula (2):

[0040] f(x,y)=ax 2 +by 2 +cxy+dx+ey+f (2)

[0041] It can be seen that the fitting function actually contains all the terms that appear in the second-order expansion of the Taylor series, where a and b represent the second-order partial derivatives respectively. c represents mixed partial derivatives d and e represent the first-order partial derivative coefficients respectively; f represents the constant term of the function. In this way, we can see that the quadratic polynomial (1) is actually an approximation of the second-order Taylor expansion of the function f(x,y) near the maximum point (x0,y0). From this, it can be concluded that it is mathematically reasonable to use such a quadratic polynomial for fitting. The reason for using the second-order Taylor series instead of the first-order Taylor series when fitting the surface is that the first-order and second-order Taylor series correspond to the linear model and the quadratic model respectively, describing the change of the function in the local area, and the second-order Taylor series is more in line with the properties of the quadratic function of the surface. Compared with the first-order Taylor series, it can better capture the curvature characteristics of the surface, and thus can provide a more accurate fitting effect.

[0042] Then, by inputting the coordinate information and pixel values ​​of the existing coordinate points, at least 6 equations are obtained and the coefficients a, b, c, d, e, and f of the quadratic function are solved.

[0043] Step 4, solve the partial derivative equation to obtain the extreme point of the quadratic function. After solving the coefficients of the quadratic function, the next step is to find the extreme point of the quadratic function by solving the partial derivative equations. Specifically, we take the partial derivatives of the function f(x, y) with respect to x and y respectively, obtain two equations and set these two partial derivatives to zero. The partial derivative equations are shown in equations (3) and (4):

[0044]

[0045] Solving this system of equations to obtain the values ​​of x and y, the sub-pixel coordinates of the key points after heat map decoding (x * ,y * ) as shown in formula (5) and formula (6):

[0046] x * =x0+x (5)

[0047] y * =y0+y (6)

[0048] Through the above steps, we can decode the position of the key points from the heat map with higher accuracy, thereby making the results obtained by the human posture network more accurate. The decoded key point position is represented as follows Figure 2 , Figure 3 shown.

[0049] Step 5: Construct a human posture instance. Because HigherHRNet is a multi-person posture estimation network, it is necessary to assign the detected key points to the corresponding human targets, use the Hungarian algorithm to obtain the labels obtained through network training to group the key points and connect them to each other, construct a complete human posture instance, and obtain the posture estimation result predicted by the network. Finally, output the prediction result in json file format.

[0050] The comparison of the indicators of HigherHRNet and the present invention on the CrowdPose validation set is shown in Table 1. It can be seen that the present invention has further improved the two accuracy indicators of average accuracy AP and average recall AR.

[0051] method Pre-training Input image resolution AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> AR HigherHRNet have 512*512 64.8 85.7 69.5 71.2 The present invention have 512*512 65.4 85.9 69.7 71.5

[0052] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. Sub-pixel heat map decoding method based on surface fitting, characterized by The following steps are involved: Step 1: Training heatmap. Use the training set and validation set of the Crowdpose dataset to train and validate human pose estimation. The CrowdPose dataset annotates the label information of 14 key points of the human body. Each annotation information includes the type of key point and the coordinate information of the key point in the image. This information will then be converted into a real heatmap for comparison with the predicted heatmap. Next, we build the HigherHRNet network structure, which can be divided into four stages. Different stages contain features of different resolutions. After the last stage, we convolve the feature map with a Gaussian kernel to obtain a key point heat map. The network starts with a convolutional layer that halves the resolution of the input image using a 3×3 convolution kernel with a stride of 2. The first stage consists of a high-resolution subnet with four residual blocks, and the output of this stage is provided as high-resolution input to subsequent stages. In the second stage, the network will add three branches of different resolutions in parallel. Each branch is obtained by downsampling the output of the previous stage with a 3×3 convolution kernel with a stride of 2. The three branches have different resolutions and contain four residual blocks. Assuming the highest resolution is 1 / 4 of the original input resolution, the resolutions of the other three branches are 1 / 8, 1 / 16 and 1 / 32 respectively. These branches are upsampled or downsampled by a 1×1 convolution kernel to exchange information across resolutions in order to fuse feature information of different resolutions. In the third stage, while maintaining the structure of the four branches formed at the end of the second stage, the fusion of depth and enhanced feature information is further increased, and four residual blocks are added to each branch. Each residual block is designed to enhance the feature representation capability of a specific resolution branch. Similar to the second stage, feature fusion in the third stage is also achieved through cross-branch connections. This mode combines high-resolution and low-resolution features to extract richer and more comprehensive feature information. The fourth stage further increases the depth of the network and strengthens the breadth and depth of feature fusion. It maintains the structure of four branches and the resolution remains unchanged. Similar to the third stage, four residual blocks are added to each branch, which further enhances the ability to express feature information on each branch and improves the accuracy of key point detection. By increasing the depth, the fourth stage is more flexible in feature information fusion and can more effectively combine information of different resolutions to make more accurate predictions of key point locations. After the fourth stage, 1×1 convolution is used to unify the number of feature maps of each resolution, and then they are connected to the branch with the highest resolution through upsampling to form the final feature map, and the key point heat map is generated through the Gaussian kernel function. Step 2, key point location. The network will output the corresponding predicted heat map for M human key points. The pixel value of each coordinate point on the heat map represents the probability of the predicted key point appearing at that position. Therefore, after obtaining the heat map, the heat map of each key point needs to be used for non-maximum suppression to find the maximum value point (x0, y0) in the map, that is, the coordinate point with the highest probability of the key point appearing. Around each predicted key point, a 3×3 neighborhood containing the point is selected. This neighborhood contains the key point and the surrounding 8 coordinate points. The coordinates and pixel values ​​of the 9 points obtained will be used for subsequent calculations. Step 3: Set and solve the equation to find the parameters of the quadratic function. For the binary function f(x,y), the second-order Taylor expansion near the point (x0,y0) can be expressed as formula (1): Next, we select (x0, y0) as the expansion point. The coefficients of the quadratic term and the cross term represent the shape change of the surface around this point, which is compared with the two-dimensional quadratic function set by formula (2): f(x,y)=ax 2 +by 2 +cxy+dx+ey+f (2) It can be seen that the fitting function actually contains all the terms that appear in the second-order expansion of the Taylor series, where a and b represent the second-order partial derivatives respectively. c represents mixed partial derivatives d and e represent the first-order partial derivative coefficients respectively; f represents the constant term of the function. In this way, we can see that the quadratic polynomial (1) is actually an approximation of the second-order Taylor expansion of the function f(x,y) near the maximum point (x0,y0). From this, it can be concluded that it is mathematically reasonable to use such a quadratic polynomial for fitting. The reason for using the second-order Taylor series instead of the first-order Taylor series when fitting the surface is that the first-order and second-order Taylor series correspond to the linear model and the quadratic model respectively, describing the change of the function in the local area, and the second-order Taylor series is more in line with the properties of the quadratic function of the surface. Compared with the first-order Taylor series, it can better capture the curvature characteristics of the surface, and thus can provide a more accurate fitting effect. Then, by inputting the coordinate information and pixel values ​​of the existing coordinate points, at least 6 equations are obtained and the coefficients a, b, c, d, e, and f of the quadratic function are solved. Step 4, solve the partial derivative equation to obtain the extreme point of the quadratic function. After solving the coefficients of the quadratic function, the next step is to find the extreme point of the quadratic function by solving the partial derivative equations. Specifically, we take the partial derivatives of the function f(x, y) with respect to x and y respectively, obtain two equations and set these two partial derivatives to zero. The partial derivative equations are shown in equations (3) and (4): Solving this system of equations to obtain the values ​​of x and y, the sub-pixel coordinates of the key points after heat map decoding (x * ,y * ) as shown in formula (5) and formula (6): x * =x0+x (5) and * =y0+y (6) Through the above steps, we can decode the positions of key points from the heat map with higher accuracy, thereby making the results obtained by the human posture network more accurate. The decoded key point positions are shown in Figures 2 and 3. Step 5: Construct a human posture instance. Because HigherHRNet is a multi-person posture estimation network, it is necessary to assign the detected key points to the corresponding human targets, use the Hungarian algorithm to obtain the labels obtained through network training to group the key points and connect them to each other, construct a complete human posture instance, and obtain the posture estimation result predicted by the network. Finally, output the prediction result in json file format.

Images