A RefineF algorithm for improving the accuracy of key points of human skeleton based on deep learning thermal values

Abstract

The present invention relates to the field of deep learning technology, specifically a RefineF human skeleton key point accuracy improvement algorithm based on deep learning thermal value, comprising: obtaining an input image to obtain original image input data; flipping the input image data to obtain flipped image input data; inputting the image data into a deep neural network to obtain its thermal value output; integrating the two outputs to obtain the final thermal value output of the network; obtaining the size of all maximum values ​​of the output and the corresponding maximum value coordinates; continuously executing the Refine method with the maximum value coordinate as the center; summarizing the coordinates of the skeleton key points at all positions to obtain the final joint point position. The present invention has strong anti-interference ability against foreign objects near the human skeleton key points in the image, can quickly and more accurately obtain the output coordinate positions of the human skeleton key points, and can greatly improve the accuracy of the model's human posture capture results, and the output results also have certain robustness and stability.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a RefineF human skeleton key point accuracy improvement algorithm based on deep learning thermal values. Background Art

[0002] With the rapid development of society, economy, and technology, people's material, cultural, and spiritual lives have been greatly enriched, and their demand for a better life is growing. The government and all sectors of society are paying close attention to this growing demand and are committed to contributing to ensuring public safety, enriching cultural civilization, and improving the quality of life. Against this backdrop, video image acquisition technology has rapidly developed. Not only are cameras installed in important public places such as highways, office buildings, train stations, and subway stations, but some smart homes also have video acquisition capabilities.

[0003] Through intelligent analysis, people hope to extract more information from videos for application in various areas of life and work, such as human-computer interaction and athlete-assisted training. In the field of intelligent recognition, it is hoped that the posture, gestures, and language of people in videos can be analyzed, and further, it is hoped that machines can imitate this information to communicate and interact with people. In sports, dance training, and other sports, it is hoped that the parameters of athletes' joint motion can be analyzed to improve training methods and enhance training results.

[0004] Currently, mainstream video image analysis methods are all based on deep learning. There are two main human skeleton key point extraction schemes based on deep learning neural networks. One is to use a fully connected output layer to regress the coordinates of skeleton key points, and the other is to use the deep learning heat map maximum direct output method.

[0005] Regarding the method of using fully connected layers to regress coordinates, early human skeleton key point extraction all adopted similar methods. The paper "DeepPose: Human Pose Estimation via DeepNeural Networks" published by Alexander Toshev et al. in 2013 is a typical example. It uses fully connected layers to regress the coordinates of human skeleton key points. Because the fully connected layer is included in the network structure, parameter training is required, and the algorithm process is complex, which will lead to a significant increase in computing time.

[0006] Alejandro Newell et al. applied the heatmap maximum direct output method in their 2013 paper "Stacked Hourglass Networks for Human Pose Estimation." This network uses a funnel-shaped deep learning convolutional neural network for structural repetition, ultimately using heatmap direct output to obtain the final coordinates of key human skeletal points. However, because it relies too much on the feature extraction results of the convolutional neural network, it often deviates significantly from the true value when there are foreign objects around the human body. However, compared with the fully connected layer regression method, heatmaps are more accurate in obtaining key human skeletal points. Summary of the Invention

[0007] In order to address the common shortcomings of existing algorithms such as low regression accuracy, cumbersome calculations, slow speed, and susceptibility to interference, the present invention provides a faster, more accurate, and parameter-training-free algorithm for improving the accuracy of key points of the human skeleton based on deep learning thermal values. The algorithm can quickly and accurately obtain the coordinate positions of key points of the human skeleton at extremely low computing power costs, and the results obtained have high robustness and stability.

[0008] The above objectives are achieved through the following technical solutions:

[0009] A RefineF algorithm for improving the accuracy of key points of human skeleton based on deep learning thermal values ​​includes the following steps:

[0010] Step 1: Get the input image to obtain the original image input data input;

[0011] Step 2: Flip the input image to obtain the flipped image input data flipped_input;

[0012] Step 3: Input the original image input data input and the flipped image input data flipped_input into the deep neural network to obtain their corresponding network thermal value outputs, named noneflip_output and flipped_output respectively;

[0013] Step 4: Integrate noneflip_output and flipped_output to get the network's final thermal output output;

[0014] Step 5: Obtain the sizes of all maximum values ​​max_values ​​of the final thermal value output of the network and the corresponding maximum value coordinates max_coordinates;

[0015] Step 6: Continue executing the Refine method with the maximum value coordinate max_coordinates as the center;

[0016] Step 7: Summarize the coordinates of the skeleton key points at all positions to get Final_Coordinates.

[0017] Furthermore, the format of the input image in step 1 is a three-channel RGB image, and the size of the input image is H*W, where H represents the height of the image and W represents the width of the image; the original image input data input is an NdArray matrix of size [1*3*H*W].

[0018] Furthermore, the flipped input image in step 2 is flipped horizontally, that is, the flipping symmetry axis is the x-axis, and the flipped image input data flipped_input is an NdArray matrix of size [1*3*H*W], where H represents the height of the image and W represents the width of the image.

[0019] Furthermore, the network thermal value output in step 3 includes two output results: one is the output thermal value obtained by directly inputting the original image input data input into the deep neural network, named noneflip_output, and the other is the output obtained by inputting the flipped image input data flipped_input into the network, named flipped_output; the sizes of both outputs are [1*M*I*J], where M represents the number of key points of the human skeleton, I represents the height of the thermal map, and J represents the width of the thermal map.

[0020] Furthermore, the step 4 specifically includes the following steps:

[0021] Step 401: Sort flipped_output from 0-(M-1), that is, index a heat map of size I*J each time, and go to step 402;

[0022] Step 402: Reorder the outputs according to Joint_Pairs to obtain flipback_output. At this time, noneflip_output and flipped_output are in one-to-one correspondence, and go to step 403;

[0023] Step 403: Integrate noneflip_output and flipped_output to obtain the final thermal value output output. The integration formula is as follows:

[0024]

[0025] Furthermore, the step 5 specifically includes the following steps:

[0026] Step 501: Reorganize the network's final thermal value output into an NdArray matrix of size [I*J*M], and go to step 502;

[0027] Step 502: Initialize the heat map index variable m=0, and go to step 503;

[0028] Step 503: Initialize single heat map index variables i=0, j=0, max=x 00 ;

[0029] Then use the for loop to execute the traversal comparison algorithm to obtain the maximum value max_values ​​and corresponding coordinates max_coordinates of each heat map. The specific pseudo code is as follows:

[0030]

[0031] Furthermore, the step 6 specifically includes the following steps:

[0032] Step 601: Extract the heat map data of position n=0, and go to step 602;

[0033] Step 602: Reorganize the heat map into a matrix of size I*J, and go to step 603;

[0034] Step 603: Combine the result in step 5 to index the position of max_coordinate corresponding to the heat map (p y , p x ), perform the following judgment:

[0035] if 1<p y <I-1&&1<p x <J-1 (0.8)

[0036] Among them, p y Represents the row where max_coordinate is located, p x Represents the column where the coordinates are located;

[0037] If the inequality holds, go to step 604;

[0038] If the inequality does not hold, go to step 601 and execute n=n+1 to index the next heat map;

[0039] Step 604: Get the max_value corresponding to the point. This value has been stored in step 5. Execute the judgment:

[0040] if max_value>0.2 (0.9)

[0041] If the inequality holds, go to step 605;

[0042] If the inequality does not hold, go to step 601 and execute n=n+1 to index the next heat map;

[0043] Step 605: With (p y , p x ) as the center, obtain the thermal values ​​hm_value of the eight points around it (which can be expanded), and go to step 606;

[0044] Step 606: Calculate the Refine value using the following formula:

[0045]

[0046] The Refine value obtained by concatenating is a 1*2 matrix, which represents the y , p x ) is the comprehensive vector difference of the thermal values ​​on the x-axis and y-axis of the coordinate origin, and it is stipulated that the positive direction of the x-axis is to the right and the positive direction of the y-axis is downward;

[0047] When calculating the vector difference of the thermal value point on the x-axis or y-axis, the calculation result does not need to be decomposed and is directly used as one of the Refine values ​​of the axis;

[0048] When calculating the vector difference of thermal value points at diagonally opposite angles, the calculation results need to be decomposed onto the axis according to the vector decomposition principle. Taking the case of 8 points as an example, depending on the direction of the vector calculation results, the difference needs to be multiplied by the absolute value of cos±45°, sin±45° or cos±135°, sin±135° before performing the axial summation operation;

[0049] After the calculation is completed, go to step 607;

[0050] Step 607: Update the coordinate values ​​of the skeleton key points. The updating formula is as follows:

[0051] refined_coords=max_coordinate+sign(Refine)*0.25 (0.11)

[0052] Among them, refined_coords is the new skeleton key point coordinate set obtained after using the Refine method. It is a 1*M*2 NdArray matrix, where M represents the number of human skeleton key points defined, and 2 represents a single human skeleton key point after the update. Indicates that max_coordinate is the original maximum value coordinate (p y , px ), the sign() function has the following characteristics:

[0053]

[0054] After this step, go to step 601 and execute n=n+1 to index the next heat map until all the skeleton key points are traversed. Finally, the positions of the human skeleton key points are stored in refined_coords.

[0055] Beneficial effects

[0056] The RefineF human skeleton key point precision improvement algorithm based on deep learning thermal values ​​provided by the present invention can obtain the output coordinate positions of human skeleton key points more quickly and accurately, and the results are highly robust and stable. At the same time, it also has the following advantages:

[0057] 1. Image flipping is added to the algorithm and integrated with the normal output, which greatly improves the stability and robustness of the output results;

[0058] 2. A Refine method based on deep learning thermal values ​​is proposed to overcome the disadvantage that the output bone key point positions are easily interfered with by foreign objects. The combination of the two methods produces the RefineF human bone key point accuracy improvement algorithm proposed in this invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The following is a step description of the overall steps of the algorithm of the present invention;

[0060] Figure 2 Detailed description of step 4 in the overall steps of the algorithm of the present invention;

[0061] Figure 3 Detailed description of step 5 in the overall steps of the algorithm of the present invention;

[0062] Figure 4 Detailed description of step 6 in the overall steps of the algorithm of the present invention;

[0063] Figure 5 This is a detailed description of step 606 of the algorithm of the present invention, wherein (a) represents the case where the difference point is on the x-axis and the difference is positive; (b) represents the case where the difference point is on the x-axis and the difference is negative; (c) represents the case where the difference point is on the oblique axis and the difference is positive; (d) represents the case where the difference point is on the oblique axis and the difference is negative;

[0064] Figure 6 The data set used in the specific embodiment of the present invention includes the sequence numbers of key points of the human skeleton and the corresponding joint position descriptions;

[0065] Figure 7 This is a comparison of the accuracy results and annotations of the image in specific embodiment 1 of the present invention under different methods;

[0066] Figure 8 This is a table showing the accuracy (AP) evaluation results of the specific embodiment 1 of the present invention on the COCO Val207 Dataset;

[0067] Figure 9 This is a table of average precision (AP) evaluation results of specific embodiment 2 of the present invention on the COCO Val207 Dataset. DETAILED DESCRIPTION

[0068] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the technical solutions of the present invention are further described in detail below with reference to the accompanying drawings:

[0069] like Figure 1 As shown in FIG, a RefineF algorithm for improving the accuracy of key points of human skeleton based on deep learning thermal values ​​includes the following steps:

[0070] Step 1: Get the input image to obtain the original image input data input;

[0071] Step 2: Flip the input image to obtain the flipped image input data flipped_input;

[0072] Step 3: Input the original image input data input and the flipped image input data flipped_input into the deep neural network to obtain their corresponding network thermal value outputs, named noneflip_output and flipped_output respectively;

[0073] Step 4: Integrate noneflip_output and flipped_output to get the network's final thermal output output;

[0074] Step 5: Obtain the sizes of all maximum values ​​max_values ​​of the final thermal value output of the network and the corresponding maximum value coordinates max_coordinates;

[0075] Step 6: Continue executing the Refine method with the maximum value coordinate max_coordinates as the center;

[0076] Step 7: Summarize the coordinates of the skeleton key points at all positions to get Final_Coordinates.

[0077] As a further illustration of the above method, the following is provided:

[0078] The input image in step 1 is in the form of a three-channel RGB image. The size of the input image is H*W, where H represents the height of the image and W represents the width of the image. The original image input data is an NdArray matrix of size [1*3*H*W].

[0079] The input image is flipped in step 2 in a horizontal flipping mode, that is, the flipping symmetry axis is the x-axis. The flipped image input data flipped_input is an NdArray matrix of size [1*3*H*W], where H represents the height of the image and W represents the width of the image.

[0080] The network thermal value output in step 3 includes two output results: one is the output thermal value obtained by directly inputting the original image input data input into the deep neural network, named noneflip_output, and the other is the output obtained by inputting the flipped image input data flipped_input into the network, named flipped_output; the size of both outputs is [1*M*I*J], where M represents the number of key points of the human skeleton, I represents the height of the thermal map, and J represents the width of the thermal map.

[0081] like Figure 2 As shown, step 4 specifically includes the following steps:

[0082] Step 401: Sort flipped_output from 0-(M-1), that is, index a heat map of size I*J each time, and go to step 402;

[0083] Step 402: Reorder the outputs according to Joint_Pairs to obtain flipback_output. At this time, noneflip_output and flipped_output are in one-to-one correspondence, and go to step 403;

[0084] Step 403: Integrate noneflip_output and flipped_output to obtain the final thermal value output output. The integration formula is as follows:

[0085]

[0086] like Figure 3 As shown, step 5 specifically includes the following steps:

[0087] Step 501: Reorganize the output into an NdArray matrix of size [I*J*M] and go to step 502;

[0088] Step 502: Initialize the heat map index variable m=0, and go to step 503;

[0089] Step 503: Initialize single heat map index variables i=0, j=0, max=x 00 ;

[0090] Then use the for loop to execute the traversal comparison algorithm to obtain the maximum value max_values ​​and corresponding coordinates max_coordinates of each heat map. The specific pseudo code is as follows:

[0091]

[0092] like Figure 4 As shown, step 6 specifically includes the following steps:

[0093] Step 601: Extract the heat map data of position n=0, and go to step 602;

[0094] Step 602: Reorganize the heat map into a matrix of size I*J, and go to step 603;

[0095] Step 603: According to the result in step 5 above, the position of max_coordinate corresponding to the heat map is indexed (p y , p x ), where p y is the row where max_coordinate is located, p x Is the column where the coordinates are located, perform the following judgment:

[0096] if 1<p y <I-1&&1<p x <J-1 (0.14)

[0097] If the inequality holds, go to step 604; otherwise go to step 601 and execute n=n+1 to index the next heat map;

[0098] Step 604: Get the max_value corresponding to the point. This value has been stored in step 5. Execute the judgment:

[0099] if max_value>0.2 (0.15)

[0100] If the inequality is established, go to step 605; otherwise go to step 601 and execute n=n+1 to index the next heat map;

[0101] Step 605: With (p y , p x) as the center, obtain the thermal values ​​hm_value of the eight points around it (which can be expanded), and go to step 606;

[0102] Step 606: Calculate the Refine value using the following formula:

[0103]

[0104] The Refine value obtained by concatenating is a 1*2 matrix, which represents the y , p x ) is the comprehensive vector difference of the thermal values ​​on the x-axis and y-axis of the coordinate origin, and it is stipulated that the positive direction of the x-axis is to the right and the positive direction of the y-axis is downward.

[0105] like Figure 5 As shown in (a) and (b), when calculating vector differences at thermal value points on the x-axis, the result does not need to be decomposed and is directly used as one of the Refine values ​​for that axis. The Refine value calculation result for this axis is in the form of [0, a], where a is a vector with a positive or negative sign and is used for subsequent vector splicing. The same is true when calculating vector differences at thermal value points on the y-axis. The Refine value calculation result for this axis is in the form of [b, 0], where b is a vector with a positive or negative sign and is used for subsequent vector splicing.

[0106] Take the case of 8 points as an example, Figure 5 As shown in (c) and (d), when calculating the vector difference at the thermal value point at the diagonal angle, the calculation result needs to be decomposed onto the axis according to the vector decomposition principle. When the difference calculation result is positive, that is, Figure 5 In the case shown in (c), the difference needs to be decomposed into the x-axis and y-axis respectively by multiplying its absolute value by the angle between the vector and the x-axis, that is, cos45° and sin45°. In this case, the calculation result of the Refine value of the axis is in the form of [c*sin45°, c*cos45°], where c is the absolute value of the difference, without a sign, and the result is used for subsequent vector splicing; when the difference calculation result is negative, that is, Figure 5 In the case shown in (d), the difference needs to be decomposed into the x-axis and y-axis by multiplying its absolute value by the angle between the vector and the x-axis, i.e., cos(-135°) and sin(-135°), respectively. In this case, the calculated Refine value for that axis is in the form of [d*sin(-135°), d*cos(-135°)], where d is the absolute value of the difference, without a sign. The result is used for subsequent vector splicing. The same is true for the other oblique axis. After the calculation is completed, go to step 607;

[0107] Step 607: Update the coordinate values ​​of the skeleton key points. The updating formula is as follows:

[0108] refined_coords=max_coordinate+sign(Refine)*0.25 (0.17)

[0109] Among them, refined_coords is the new skeleton key point coordinate set obtained after using the Refine method. It is a 1*M*2 NdArray matrix, where M represents the number of human skeleton key points defined, and 2 represents a single human skeleton key point after the update. Indicates that max_coordinate is the original maximum value coordinate (p y , p x ), the sign() function has the following characteristics:

[0110]

[0111] After this step, go to step 601 and execute n=n+1 to index the next heat map until all the skeleton key points are traversed. Finally, the positions of the human skeleton key points are stored in refined_coords.

[0112] Example 1

[0113] like Figure 1 As shown in FIG, a RefineF algorithm for improving the accuracy of key points of human skeleton based on deep learning thermal values ​​includes the following steps:

[0114] Step 1: Get the input image to obtain the original image input data input;

[0115] Step 2: Flip the input image to obtain the flipped image input data flipped_input;

[0116] Step 3: Input the original image input data input and the flipped image input data flipped_input into the deep neural network to obtain their corresponding network thermal value outputs, named noneflip_output and flipped_output respectively;

[0117] Step 4: Integrate noneflip_output and flipped_output to get the network's final thermal output output;

[0118] Step 5: Obtain the sizes of all maximum values ​​max_values ​​of the final thermal value output of the network and the corresponding maximum value coordinates max_coordinates;

[0119] Step 6: Continue executing the Refine method with the maximum value coordinate max_coordinates as the center;

[0120] Step 7: Summarize the coordinates of the skeleton key points at all positions to get Final_Coordinates.

[0121] In this embodiment, the network model used is the Resnet18-based deep learning neural network model proposed by Bin Xiao et al. in "Simple Baselines for Human Pose Estimation and Tracking". This model obtains the coordinates of skeletal key points by directly outputting thermal values. The output heat map size is 48*64, where 48 represents the width of the heat map and 64 represents the height of the heat map.

[0122] As a further solution of the present invention, in step 1, the input image format is a three-channel RGB image, and the input image size is H*W, where H is the height of the image, and in this embodiment, H is set to 640, and W is the width of the image, and in this embodiment, W is set to 425. The input data finally obtained is an NdArray matrix of size [1*3*640*425].

[0123] As a further solution of the present invention, in step 2, the input image is flipped to obtain flipped image data, and the flipping method is horizontal flipping, that is, the flipping symmetry axis is the x-axis, and the flipped input image data flipped_input is still an NdArray matrix of size [1*3*640*425].

[0124] As a further embodiment of the present invention, the network thermal output obtained in step 3 includes two output results: one is the thermal output obtained by directly inputting the input into the deep neural network, named noneflip_output, and the other is the output obtained by inputting flipped_input into the network, named flipped_output. The size of both outputs is [1*M*I*J], where M is the number of human skeleton key points, I is the height of the heatmap, and J is the width of the heatmap.

[0125] like Figure 6As shown, the dataset referenced in this embodiment is the COCO dataset, which has 17 key points of the human skeleton. The corresponding relationship between the definition sequence number and position of the key points of the human skeleton is as follows: 0-nose, 1-left_eye, 2-right_eye, 3-left-ear, 4-right-ear, 5-left_shoulder, 6-right_shoulder, 7-left_elbow, 8-right_elbow, 9-left_wrist, 10-right_wrist, 11-left_hip, 12-right_hip, 13-left_knee, 14-right-knee, 15-left_ankle, 16-right_ankle. Therefore, the size of both outputs is [1*17*64*48].

[0126] like Figure 2 As shown, in step 4, the specific steps of outputting the final thermal value are as follows:

[0127] Step 401: Sort flipped_output from 0 to 16, that is, index a heat map of size 64*48 each time, and go to step 402;

[0128] Step 402: Reorder the outputs according to Joint_Pairs to obtain flipback_output. At this time, noneflip_output and flipped_output are in one-to-one correspondence, and go to step 403;

[0129] Step 403: Integrate noneflip_output and flipped_output to obtain the final thermal value output output. The integration formula is as follows:

[0130]

[0131] like Figure 3 As shown, in step 5, the size of all maximum values ​​max_values ​​of output and the corresponding maximum value coordinates max_coordinates are obtained. The specific steps are as follows:

[0132] Step 501: Reorganize the output into an NdArray matrix of size [I*J*M] and go to step 502;

[0133] Step 502: Initialize the heat map index variable m=0, and go to step 503;

[0134] Step 503: Initialize single heat map index variables i=0, j=0, max=x00 ;

[0135] Then use the for loop to execute the traversal comparison algorithm to obtain the maximum value max_values ​​and corresponding coordinates max_coordinates of each heat map. The specific pseudo code is as follows:

[0136]

[0137]

[0138] like Figure 4 As shown, in step 6, the Refine method is continuously executed with max_coordinates as the center, and the specific steps are as follows:

[0139] Step 601: Extract the heat map data of position n=0, and go to step 602;

[0140] Step 602: Reorganize the heat map into a 64*48 matrix, and go to step 603;

[0141] Step 603: According to the result in step 5 above, the position of max_coordinate corresponding to the heat map is indexed (p y , p x ), where p y is the row where max_coordinate is located, p x Is the column where the coordinates are located, perform the following judgment:

[0142] if 1<p y <63&&1<p x <43 (0.20)

[0143] If the inequality holds, go to step 604; otherwise go to step 601 and execute n=n+1 to index the next heat map;

[0144] Step 604: Get the max_value corresponding to the point. This value has been stored in step 5. Execute the judgment:

[0145] if max_value>0.2 (0.21)

[0146] If the inequality is established, go to step 605; otherwise go to step 601 and execute n=n+1 to index the next heat map;

[0147] Step 605: With (p y , p x ) as the center, obtain the thermal values ​​hm_value of the eight points around it (which can be expanded), and go to step 606;

[0148] Step 606: Calculate the Refine value using the following formula:

[0149]

[0150] The Refine value obtained by concatenating is a 1*2 matrix, which represents the y , p x ) is the comprehensive vector difference of the thermal values ​​on the x-axis and y-axis of the coordinate origin, and it is stipulated that the positive direction of the x-axis is to the right and the positive direction of the y-axis is downward.

[0151] like Figure 5 As shown in (a) and (b), when calculating vector differences at thermal value points on the x-axis, the result does not need to be decomposed and is directly used as one of the Refine values ​​for that axis. The Refine value calculation result for this axis is in the form of [0, a], where a is a vector with a positive or negative sign and is used for subsequent vector splicing. The same is true when calculating vector differences at thermal value points on the y-axis. The Refine value calculation result for this axis is in the form of [b, 0], where b is a vector with a positive or negative sign and is used for subsequent vector splicing.

[0152] Take the case of 8 points as an example, Figure 5 As shown in (c) and (d), when calculating the vector difference at the thermal value point at the diagonal angle, the calculation result needs to be decomposed onto the axis according to the vector decomposition principle. When the difference calculation result is positive, that is, Figure 5 In the case shown in (c), the difference needs to be decomposed into the x-axis and y-axis respectively by multiplying its absolute value by the angle between the vector and the x-axis, that is, cos45° and sin45°. In this case, the calculation result of the Refine value of the axis is in the form of [c*sin45°, c*cos45°], where c is the absolute value of the difference, without a sign, and the result is used for subsequent vector splicing; when the difference calculation result is negative, that is, Figure 5 In the case shown in (d), the difference needs to be decomposed into the x-axis and y-axis by multiplying its absolute value by the angle between the vector and the x-axis, i.e., cos(-135°) and sin(-135°), respectively. In this case, the calculated Refine value for that axis is in the form of [d*sin(-135°), d*cos(-135°)], where d is the absolute value of the difference, without a sign. The result is used for subsequent vector splicing. The same is true for the other oblique axis. After the calculation is completed, go to step 607;

[0153] Step 607: Update the coordinate values ​​of the skeleton key points. The updating formula is as follows:

[0154] refined_coords=max_coordinate+sign(Refine)*0.25 (0.23)

[0155] Among them, refined_coords is the new skeleton key point coordinate set obtained after using the Refine method. It is a 1*17*2 NdArray matrix, where M represents the number of human skeleton key points defined, and 2 represents a single human skeleton key point after the update. Indicates that max_coordinate is the original maximum value coordinate (p y , p x ), the sign() function has the following characteristics:

[0156]

[0157] After this step, go to step 601 and execute n=n+1 to index the next heat map until all the skeleton key points are traversed. Finally, the positions of the human skeleton key points are stored in refined_coords.

[0158] like Figure 7 As shown in the figure, we compared the direct method of obtaining the maximum thermal value (Direct method) and the RefineF algorithm proposed in the present invention on a single posture image. It can be seen that after using the algorithm proposed in the present invention, the image has obvious improvements in the left ear, nose, right elbow, left knee, right wrist and other positions, which are more consistent with the actual positions and closer to the annotation results.

[0159] like Figure 8 As shown in the table, we used the COCO Val2017 dataset to obtain the improved comparison results of the model accuracy of this algorithm. It can be seen from the table that compared with the method of directly obtaining the maximum thermal value (Direct method), this algorithm has improved the accuracy of various accuracy indicators, with the highest accuracy exceeding 4%.

[0160] Example 2

[0161] In this embodiment, the network model used in this solution is the deep learning neural network model based on Resnet50 proposed by Bin Xiao et al. in "Simple Baselines for Human Pose Estimation and Tracking". This model relies on the direct output of thermal values ​​to obtain the coordinates of skeletal key points. The output heat map size is still 48*64, 48 represents the width of the heat map, and 64 represents the height of the heat map.

[0162] The rest of the process is exactly the same as that in Specific Example 1.

[0163] like Figure 9As shown in the table, we used the COCO Val2017 dataset to obtain the improved comparison results of the model accuracy of the present invention. It can be seen from the table that compared with the method of directly obtaining the maximum thermal value (Direct method), the present algorithm has improved accuracy in various accuracy indicators, with the highest accuracy exceeding 3%.

[0164] Through the above series of embodiments, the implementation of the algorithm mentioned in the present invention is realized. In order to verify the improvement of the accuracy of the algorithm for the key points of the human skeleton, the present invention finally obtains the advantages of the present invention over other algorithms on the authoritative COCO dataset. At the same time, different algorithms are selected to obtain the results of the key points of the human skeleton of the same image, such as Figure 7 As shown, it can be seen that the RefineF human skeleton key point accuracy improvement algorithm based on deep learning thermal value of the present invention can significantly improve the accuracy of human skeleton key points, and greatly improve the accuracy of the model's human posture capture results.

[0165] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0166] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that those skilled in the art can understand. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A RefineF algorithm for improving the accuracy of key points of human skeleton based on deep learning thermal values, characterized by: The steps include: Step 1: Get the input image to obtain the original image input data input; Step 2: Flip the input image to obtain the flipped image input data flipped_input; Step 3: Input the original image input data input and the flipped image input data flipped_input into the deep neural network to obtain their corresponding network thermal value outputs, named noneflip_output and flipped_output respectively; Step 4: Integrate noneflip_output and flipped_output to get the network's final thermal output output; Step 5: Obtain the sizes of all maximum values ​​max_values ​​of the final thermal value output of the network and the corresponding maximum value coordinates max_coordinates; Step 6: Continue executing the Refine method with the maximum value coordinate max_coordinates as the center; Step 7: Summarize the coordinates of the skeleton key points at all positions to obtain Final_Coordinates; The step 6 specifically includes the following steps: Step 601: Extract the heat map data of position n=0, and go to step 602; Step 602: Reorganize the heat map into a matrix of size I*J, and go to step 603; Step 603: Combine the result in step 5 to index the position of max_coordinate corresponding to the heat map (p y ,p x ), perform the following judgment: if 1<p y <I-1&&1<p x <J-1(1.2) Among them, p y Represents the row where max_coordinate is located, p x Represents the column where the coordinates are located; If the inequality holds, go to step 604; If the inequality does not hold, go to step 601 and execute n=n+1 to index the next heat map; Step 604: Get the max_value corresponding to the point. This value has been stored in step 5. Execute the judgment: if max_value>0.2(1.3) If the inequality holds, go to step 605; If the inequality does not hold, go to step 601 and execute n=n+1 to index the next heat map; Step 605: With (p y ,p x ) as the center, obtain the thermal values ​​hm_value of the 8 points that can be expanded around it, and go to step 606; Step 606: Calculate the Refine value using the following formula: The Refine value obtained by concatenating is a 1*2 matrix, which represents the y ,p x ) is the comprehensive vector difference of the thermal values ​​on the x-axis and y-axis of the coordinate origin, and it is stipulated that the positive direction of the x-axis is to the right and the positive direction of the y-axis is downward; When calculating the vector difference of the thermal value point on the x-axis or y-axis, the calculation result does not need to be decomposed and is directly used as one of the axial Refine values; When calculating the vector difference of thermal value points at diagonally opposite angles, the calculation results need to be decomposed onto the axis according to the vector decomposition principle. Taking the case of 8 points as an example, depending on the direction of the vector calculation results, the difference needs to be multiplied by the absolute value of cos±45°, sin±45° or cos±135°, sin±135° before performing the axial summation operation; After the calculation is completed, go to step 607; Step 607: Update the coordinate values ​​of the skeleton key points. The updating formula is as follows: refined_coords=max_coordinate+sign(Refine)*0.25(1.5) where refined_coords is the new set of skeleton keypoint coordinates obtained after using the Refine method. It is a 1*M*2 NdArray matrix, where M represents the number of defined human skeleton keypoints and 2 represents a single human skeleton keypoint after the update. Indicates that max_coordinate is the original maximum value coordinate (p y ,p x ), the sign() function has the following characteristics: After this step, go to step 601 and execute n=n+1 to index the next heat map until all the skeleton key points are traversed. Finally, the positions of the human skeleton key points are stored in refined_coords.

2. The RefineF human skeleton key point accuracy improvement algorithm based on deep learning thermal values ​​according to claim 1 is characterized by: The input image in step 1 is in the form of a three-channel RGB image. The size of the input image is H*W, where H represents the height of the image and W represents the width of the image. The original image input data is an NdArray matrix of size [1*3*H*W].

3. The RefineF human skeleton key point accuracy improvement algorithm based on deep learning thermal values ​​according to claim 1, characterized in that: The input image is flipped in step 2 in a horizontal flipping mode, that is, the flipping symmetry axis is the x-axis. The flipped image input data flipped_input is an NdArray matrix of size [1*3*H*W], where H represents the height of the image and W represents the width of the image.

4. The RefineF human skeleton key point accuracy improvement algorithm based on deep learning thermal values ​​according to claim 1, characterized in that: The network thermal value output in step 3 includes two output results: one is the output thermal value obtained by directly inputting the original image input data input into the deep neural network, named noneflip_output, and the other is the output obtained by inputting the flipped image input data flipped_input into the network, named flipped_output; the size of both outputs is [1*M*I*J], where M represents the number of key points of the human skeleton, I represents the height of the thermal map, and J represents the width of the thermal map.

5. The RefineF human skeleton key point accuracy improvement algorithm based on deep learning thermal values ​​according to claim 4 is characterized by: The step 4 specifically includes the following steps: Step 401: Sort flipped_output from 0-(M-1), that is, index a heat map of size I*J each time, and go to step 402; Step 402: Reorder the outputs according to Joint_Pairs to obtain flipback_output. At this time, noneflip_output and flipped_output are in one-to-one correspondence, and go to step 403; Step 403: Integrate noneflip_output and flipped_output to obtain the final thermal value output output. The integration formula is as follows:

6. The RefineF human skeleton key point accuracy improvement algorithm based on deep learning thermal values ​​according to claim 5, characterized in that: The step 5 specifically includes the following steps: Step 501: Reorganize the network's final thermal value output into an NdArray matrix of size [I*J*M], and go to step 502; Step 502: Initialize the heat map index variable m=0, and go to step 503; Step 503: Initialize single heat map index variables i=0, j=0, max=x 00 ; Then use the for loop to execute the traversal comparison algorithm to obtain the maximum value max_values ​​and corresponding coordinates max_coordinates of each heat map. The specific pseudo code is as follows:

Images