Gesture key point detection method, device, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a gesture key point detection method, device and equipment and a medium, the method comprising the following steps: acquiring a gesture image to be detected; inputting the gesture image to be detected into a gesture key point heat map prediction network, and acquiring a pair of heat maps from the output of the gesture key point heat map prediction network; wherein each pair of heat maps comprises a position heat map and a depth heat map; the gesture key point heat map prediction network is obtained through supervised training, the supervised training label comprises a pre-set position heat map and a pre-set depth heat map, and the position heat map and the depth heat map are both Gaussian heat maps with sub-pixel information; based on the position heat map and the depth heat map, the expected value of a key point coordinate is calculated through softmax normalization; and the key point coordinate is calculated according to the expected value of the key point coordinate. The embodiment has a faster reasoning speed and can extract more accurate key point coordinate information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a gesture key point method, apparatus, device and medium. Background Art

[0002] The most commonly used gesture key point detection scheme detects 21 key points for each hand, including one key point at the base of the palm and four key points for each finger, such as Figure 1 As shown in the figure, these 21 key points can be used to reconstruct the hand model and perform gesture recognition later. There are two main methods for gesture key point detection in academia. The first method is the key point regression model, which was first proposed by Google scholars Alexander Toshe and Christian Szegedy in 2013. Simply put, the key point regression model first uses a convolutional neural network to extract a feature map, and then uses a fully connected layer to regress the feature map into the coordinates of 21 key points. The advantage of this method is that it can directly output floating-point values ​​without causing loss of accuracy, but the disadvantage is that it is easy to cause overfitting, resulting in poor generalization ability. For example, if the hand is always in the upper left corner of the image when training the neural network, then the trained model will find it difficult to recognize the hand in the lower right corner and its key points.

[0003] The second method is the heatmap regression model, which was proposed in 2014 by New York University scholars Jonathan Tompson, Arjun Jain, Christoph Bregle and the father of neural networks, Yann Lecun. Simply put, the heatmap regression model outputs 21 heatmaps. The value of each pixel in the heatmap is between 0 and 1, representing the probability that the point is a key point. After obtaining 21 heatmaps, find the maximum value of each heatmap, and the location of the maximum value is the key point location. The advantage of the heatmap regression model is its strong generalization ability. Even if the sample distribution is uneven during training, it can still get better results during prediction. The disadvantage is that the amount of calculation is large, because the key point regression model only needs to output 21 key point coordinates (a total of 42 outputs, because each key point contains x, y), while the heatmap regression needs to output 21 heatmaps (a total of 21*h*w outputs, h and w are the height and width of the heatmap). Ideally, the size of the 21 heatmaps output should be the same as the original Figure 1 But in reality, this is usually not feasible due to the limited computing power of hardware. In order to reduce the amount of calculation, the length and width of the heat map need to be reduced to 1 / 4 of the original image, but this introduces another problem, that is, the key point positions found from the heat map have a loss of accuracy and cannot be accurately restored to the original image. This is because after the heat map is reduced, each point on the heat map corresponds to the original image. Figure 1The image is a 4*4 area, so after finding the key point position from the heat map, we can only know that the key point is in a certain area on the original image, but cannot get its specific position.

[0004] So how can we leverage the generalization capabilities of heatmap regression without losing keypoint accuracy? In 2018, Google Scholar proposed a heatmap regression method called heatmap+offset. Simply put, this method uses heatmap regression to find the keypoints on the heatmap. Each keypoint corresponds to a 4*4 region in the original image. The method first takes the center of the 4*4 region as the keypoint location, and then uses the offset to provide the offset of the keypoint relative to the center of the 4*4 region. The final keypoint location is the combination of the offset and the keypoint location.

[0005] Although the heatmap+offset heatmap regression method has strong generalization ability and reduces the problem of precision loss in key point positions, the key point coordinate positions calculated using the heatmap+offset heatmap regression method are still not accurate enough. Summary of the Invention

[0006] In response to the above technical problems, the purpose of this application is to provide a gesture key point detection method, device, and device storage medium, aiming to improve the accuracy of gesture key point coordinate calculation.

[0007] In order to solve the above technical problems, in a first aspect, embodiments of the present application provide a gesture key point detection method, including:

[0008] Get the gesture image to be detected;

[0009] Inputting the gesture image to be detected into a gesture key point heat map prediction network, and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network; wherein each pair of heat maps includes a position heat map and a depth heat map; the gesture key point heat map prediction network is obtained through supervised training, and the supervised training labels include a pre-set position heat map and a pre-set depth heat map, and the pre-set position heat map and the pre-set depth heat map are both Gaussian heat maps with sub-pixel information;

[0010] Calculate the position of the maximum heat of the position heat map;

[0011] Extracting a first matrix from the position heat map with the position of maximum heat in the position heat map as the center;

[0012] Normalizing the first matrix, and calculating expected position values ​​of key point coordinates based on a result of normalizing the first matrix;

[0013] convert the position expectation value of the key point coordinate according to a ratio of a width of the position heat map and a width of the gesture image to be detected and a ratio of a length of the position heat map and a length of the gesture image to be detected, to obtain a position value of the key point coordinate;

[0014] calculate a position of maximum heat of the depth heat map;

[0015] extract a second matrix from the depth heat map with the position of maximum heat of the depth heat map as a center, wherein the second matrix and the first matrix have the same number of columns;

[0016] normalize the second matrix, and calculate a depth expectation value of the key point coordinate according to a result of normalizing the second matrix;

[0017] obtain the key point coordinate according to the position value of the key point coordinate and the depth expectation value of the key point coordinate.

[0018] Further, the pre-set position heat map is a two-dimensional Gaussian distribution, the mean of the two-dimensional Gaussian distribution is (x / 4, y / 4), the standard deviation is σ, and the effective area is m*m, and the part exceeding the effective area is all set to 0; the depth heat map is a one-dimensional Gaussian distribution, the mean of the one-dimensional Gaussian distribution is (z+0.5)*32, the standard deviation is σ, and the effective area is 1*m, and the part exceeding the effective area is all set to 0, wherein (x, y, z) is the coordinate of the labeled gesture key point.

[0019] Further, the normalizing the first matrix and calculating the position expectation value of the key point coordinate according to the result of normalizing the first matrix comprises:

[0020] calculating the position expectation value of the key point coordinate according to the formula i,j In the coordinate position of the position heat map, the matrix h is an n*n matrix, n is a positive integer, is the position of maximum heat of the position heat map, H is the position heat map, τ is the softmax temperature, which is a set value, and (x', y') is the position expectation value of the key point coordinate.

[0021] Further, the normalizing the second matrix and calculating the depth expectation value of the key point coordinate according to the result of normalizing the second matrix comprises:

[0022] calculating the depth expectation value of the key point coordinate according to the formula

[0023] wherein k is an element d in the second matrix d​​k In the coordinate position in the heat map D, the matrix is a 1*n matrix, n is a positive integer, τ is a softmax temperature, and is a set value, is the position of the maximum heat of the depth heat map, and z is the depth expectation value of the key point coordinate.

[0024] Further, the position expectation value of the key point coordinate is converted according to a ratio of a width of the position heat map to a width of the gesture image to be detected and a ratio of a length of the position heat map to a length of the gesture image to be detected, to obtain a position value of the key point coordinate, including:

[0025] According to the formula The position expectation value of the key point coordinate is converted to obtain a position value of the key point coordinate; wherein (x, y) is the position value of the key point coordinate, and (x', y') is the position expectation value of the key point coordinate, s x s is a ratio of a width of the position heat map to a width of the gesture image to be detected, y is a ratio of a length of the position heat map to a length of the gesture image to be detected.

[0026] Further, the gesture key point heat map prediction network is inputted with the gesture image to be detected, and a pair of heat maps is obtained from an output of the gesture key point heat map prediction network, including:

[0027] The first gesture key point heat map prediction network is inputted with the gesture image to be detected, the feature map of the gesture image to be detected is extracted by the first gesture key point heat map prediction network, and M pairs of heat maps are generated based on the feature map of the gesture image to be detected; wherein each pair of heat maps includes a position heat map and a depth heat map; wherein M is a positive integer;

[0028] A pair of heat maps is obtained from the M pairs of heat maps.

[0029] Further, the gesture key point heat map prediction network is inputted with the gesture image to be detected, and a pair of heat maps is obtained from an output of the gesture key point heat map prediction network, including:

[0030] The second gesture key point heat map prediction network is inputted with the gesture image to be detected and the historical key point corresponding to the gesture image to be detected, the feature map of the gesture image to be detected and the feature map of the historical key point are extracted by the second gesture key point heat map prediction network, the feature map of the gesture image to be detected and the feature map of the historical key point are spliced, and M pairs of heat maps are generated based on the spliced feature map.

[0031] In a second aspect, an embodiment of the present application provides a gesture key point detection device, including:

[0032] an acquisition module configured to acquire a gesture image to be detected;

[0033] a heat map acquisition module configured to input the gesture image to be detected into a gesture key point heat map prediction network, and acquire a pair of heat maps from an output of the gesture key point heat map prediction network; wherein each pair of heat maps comprises a position heat map and a depth heat map; the gesture key point heat map prediction network is obtained through supervised training, and a supervised training label comprises a pre-set position heat map and a pre-set depth heat map, both of which are Gaussian heat maps with sub-pixel information;

[0034] a first calculation module configured to calculate a position of maximum heat of the position heat map;

[0035] a first matrix extraction module configured to extract a first matrix from the position heat map with the position of maximum heat of the position heat map as a center;

[0036] a key point coordinate position expectation value calculation module configured to normalize the first matrix, and calculate a key point coordinate position expectation value according to a result of the normalization of the first matrix;

[0037] a conversion module configured to the key point coordinate position expectation value calculation module, and configured to convert the key point coordinate position expectation value according to a ratio of a width of the position heat map to a width of the gesture image to be detected, and a ratio of a length of the position heat map to a length of the gesture image to be detected, to obtain a key point coordinate position value;

[0038] a second calculation module configured to calculate a position of maximum heat of the depth heat map;

[0039] a second matrix extraction module configured to extract a second matrix from the depth heat map with the position of maximum heat of the depth heat map as a center; wherein the second matrix and the first matrix have the same number of columns;

[0040] a key point coordinate depth expectation value calculation module configured to normalize the second matrix, and calculate a key point coordinate depth expectation value according to a result of the normalization of the second matrix;

[0041] a key point coordinate determination module configured to obtain a key point coordinate according to the key point coordinate position value and the key point coordinate depth expectation value.

[0042] In a third aspect, an embodiment of the present application provides a computer device comprising a memory and a processor, and the memory stores a computer program, and the processor implements steps of any of the above methods when executing the computer program.

[0043] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0044] The embodiment of the present application provides a gesture key point detection method, comprising: obtaining a gesture image to be detected;

[0045] The gesture image to be detected is input into the gesture key point heat map prediction network, and a pair of heat maps is obtained from the output of the gesture key point heat map prediction network; wherein, each pair of heat maps includes a position heat map and a depth heat map; the gesture key point heat map prediction network is obtained through supervised training, and the supervised training labels include a pre-set position heat map and a pre-set depth heat map, and the pre-set position heat map and the pre-set depth heat map are both Gaussian heat maps with sub-pixel information; the position of the maximum heat of the position heat map is calculated; with the position of the maximum heat of the position heat map as the center, a first matrix is ​​extracted from the position heat map; the first matrix is ​​normalized, and the result of normalizing the first matrix is ​​calculated. The expected position value of the key point coordinates; converting the expected position value of the key point coordinates according to the ratio of the width of the position heat map and the width of the gesture image to be detected, and the ratio of the length of the position heat map and the length of the gesture image to be detected to obtain the position value of the key point coordinates; calculating the position of the maximum heat of the depth heat map; extracting a second matrix from the depth heat map with the position of the maximum heat of the depth heat map as the center; wherein the second matrix and the first matrix have the same number of columns; normalizing the second matrix, and calculating the depth expected value of the key point coordinates according to the result of normalizing the second matrix; obtaining the key point coordinates according to the position value of the key point coordinates and the depth expected value of the key point coordinates. With this design, the embodiment of the present invention changes the traditional groundtruth (label) design method. Instead of directly using coordinate points as groundtruth or groundtruth heatmap plus offset, it establishes a Gaussian heatmap with sub-pixel information. It also changes the traditional processing method and no longer uses offset to compensate for precision loss. Instead, it successfully extracts sub-pixel information from the Gaussian heatmap by calculating coordinate expectation through softmax normalization. Experiments have shown that this method has a more streamlined neural network structure than the heatmap+offset method, faster inference speed, and can extract more accurate key point coordinate information. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings described in the following embodiments are only some of the embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative labor based on these drawings also belong to the protection scope of the present application.

[0047] Figure 1 The positions of 21 gesture key points;

[0048] Figure 2 A flowchart of a gesture key point detection method provided by an embodiment of the present application

[0049] Figure 3 A discrete Gaussian distribution diagram provided by an embodiment of the present application;

[0050] Figure 4 A flowchart of inputting a gesture image to be detected into a gesture key point heat map prediction network and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network provided by an embodiment of the present application;

[0051] Figure 5 A flowchart of inputting a gesture image to be detected into a gesture key point heat map prediction network and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network provided by an embodiment of the present application;

[0052] Figure 6 A structural diagram of a gesture key point detection device provided by an embodiment of the present application;

[0053] Figure 7 A structural schematic block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor also belong to the protection scope of the present application.

[0055] Embodiment one:

[0056] Please refer to Figure 2 A gesture key point detection method provided by an embodiment of the present application includes steps S1-S10:

[0057] S1, obtaining a gesture image to be detected;

[0058] S2, input the gesture image to be detected into a gesture key point heat map prediction network, and obtain a pair of heat maps from the output of the gesture key point heat map prediction network; wherein each pair of heat maps comprises a position heat map and a depth heat map; the gesture key point heat map prediction network is obtained through supervised training, and the supervised training label comprises a pre-set position heat map and a pre-set depth heat map, both of which are Gaussian heat maps with sub-pixel information;

[0059] S3, calculate the position of the maximum heat of the position heat map;

[0060] S4, extract a first matrix from the position heat map with the position of the maximum heat of the position heat map as the center;

[0061] S5, normalize the first matrix, and calculate the position expected value of the key point coordinate according to the result of normalizing the first matrix;

[0062] S6, convert the position expected value of the key point coordinate according to the ratio of the width of the position heat map to the width of the gesture image to be detected and the ratio of the length of the position heat map to the length of the gesture image to be detected, to obtain the position value of the key point coordinate;

[0063] S7, calculate the position of the maximum heat of the depth heat map;

[0064] S8, extract a second matrix from the depth heat map with the position of the maximum heat of the depth heat map as the center; wherein the number of columns of the second matrix and the first matrix is the same;

[0065] S9, normalize the second matrix, and calculate the depth expected value of the key point coordinate according to the result of normalizing the second matrix;

[0066] S10, obtain the key point coordinate according to the position value of the key point coordinate and the depth expected value of the key point coordinate.

[0067] In step S1, the gesture image to be detected is obtained from an image acquisition device or an electronic device with an image data storage function. The image acquisition device and the electronic device with an image storage function can be a mobile terminal. The mobile terminal can be a smartphone, tablet computer, desktop, laptop, notebook computer, ultra-mobile personal computer (UMPC), handheld computer, PC device, personal digital assistant (PDA), routing device, virtual reality device (VR / AR glasses), etc. When the gesture image to be detected is obtained from a virtual reality device, the gesture image to be detected is a frame of gesture image of a video captured by the virtual reality device. It should be understood that the gesture image necessarily includes a hand.

[0068] For step S2, generally speaking, 21 gesture key points are required to reconstruct the hand model or perform gesture recognition, such as Figure 1 As shown, there is a key point at the base of the palm and 4 key points for each finger (unless otherwise specified, the key points described below refer to gesture key points). The output of the gesture key point heat map prediction network of the present invention is 21 pairs of heat maps with sub-pixel information. Each pair of heat maps includes a position heat map H(x, y) and a depth heat map D(z). Each pair of heat maps corresponds to a gesture key point, that is, a gesture key point can be calculated using a pair of heat maps. For any pair of heat maps output by the gesture key point heat map prediction network, the method of calculating the key points is the same. Therefore, by executing steps S3-S10 on the 21 pairs of heat maps output, 21 gesture key points can be calculated.

[0069] The following is a detailed description of how the gesture key point heat map prediction network of the present invention is generated:

[0070] The first step in training a neural network is to design the groundtruth (label). After obtaining an image of a hand and the 21 key points of the image (21 manually labeled points), the 21 key points need to be converted from digital coordinates to a groundtruth heat map. It should be noted that the coordinates of the key points are (x, y, z). If a three-dimensional heat map is output directly, the amount of calculation will be very large. In order to reduce the amount of calculated data, the embodiment of the present invention splits the heat map into a two-dimensional heat map H (x, y) and a one-dimensional heat map D (z). H (x, y) is a depth heat map used to calculate the x value and y value of the key point coordinates, and D (z) is a depth heat map used to calculate the y value of the key point coordinates.

[0071] Since the length and width of the position heat map output by the final neural network is 1 / n of the original image, and n is greater than or equal to 1, the size of the groundtruth position heat map designed by the present invention must also be 1 / n of the original image. In addition, since each image has 21 key points, it is necessary to design 21 groundtruth position heat maps for each image. For one of the key point coordinates (x1, y1), the embodiment of the present invention establishes a discrete two-dimensional Gaussian distribution. The mean of the two-dimensional Gaussian distribution is (x1 / 4, y1 / 4), the standard deviation is σ, the effective area is m*m, and the part exceeding the effective area is all set to 0. The center (mean) of the Gaussian distribution is not rounded. This two-dimensional Gaussian distribution is the groundtruth heat map of (x1, y1), as shown in Figure 3 As shown, Figure 3 The two-dimensional Gaussian distribution in the figure is for illustration only. Neither the mean nor the variance has an actual value. It should be emphasized that, unlike other groundtruth designs for heatmap regression, the center (mean) of the Gaussian distribution in this invention is not rounded. This is the prerequisite for sub-pixel heatmap regression. Repeating this step for 21 key points can produce 21 groundtruth position heatmaps. In addition, other heatmap regression-based methods also require the design of a groundtruthoffset, but this invention does not require the design of a groundtruthoffset and can also retain the sub-pixel position information of the groundtruth.

[0072] The above describes the design of the position heatmap ground truth. Similarly, for the depth heatmap ground truth, we need to design 21 depth heatmap ground truths for each image. Unlike the position heatmap ground truth, the depth heatmap ground truth has only one dimension (1 x 32). For a given depth z1, we construct a one-dimensional Gaussian distribution with a standard deviation of σ, a mean of (z + 0.5) x 32, and an effective area of ​​1 x m. Similarly, to achieve sub-pixel regression, the mean does not need to be rounded.

[0073] After completing the groundtruth design, the next step is to design the neural network structure. The overall structure of the neural network is a ResNet34 / 18 backbone plus a detection head. The detection head is divided into a position head and a depth head, which output the predicted heat map H(x, y) of the position and the predicted heat map D(z) of the depth, respectively. Unlike other neural network structures for heat map regression, the neural network of this invention does not have an offset output because it can complete sub-pixel keypoint regression without an offset.

[0074] The present application provides two model training methods below:

[0075] The first method: when the gesture key point heat map prediction network is trained, the neural network has one input, i.e. the gesture picture to be trained. After the neural network processing, the neural network outputs two heat maps, i.e. the position heat map and the depth heat map. Then we provide the labels corresponding to the input at this time, i.e. the ground truth position heat map and the ground truth depth heat map. At this time, the MSE loss is made between the neural network output heat map and the corresponding label heat map, and then the gradient back propagation is used to update the parameters of the neural network.

[0076] The second method: when the gesture key point heat map prediction network is trained, the neural network has two inputs, i.e. the gesture picture to be trained and the historical key point position. After the neural network processing, the neural network outputs two heat maps, i.e. the position heat map and the depth heat map. Then we provide the labels corresponding to the input at this time, i.e. the ground truth position heat map and the ground truth depth heat map. At this time, the MSE loss (mean square error loss function) is made between the neural network output heat map and the corresponding label heat map, and then the gradient back propagation is used to update the parameters of the neural network.

[0077] Finally, after the neural network processing, we need to post-process the output heat map to obtain the key point information. The common post-processing method is to find the maximum value of the heat map to locate the approximate position of the key point, and then combine the offset output by the other branch of the neural network to obtain the final position of the key point. Unlike the commonly used method, the present method can recover the lost position information without using offset. The specific method is as shown in steps S3-S10.

[0078] It should be noted that the coordinates of the key points in the present application are (x, y, z). If the three-dimensional heat map is directly output, the calculation amount will be very large. The gesture key point heat map prediction network of the present application splits the three-dimensional heat map into a two-dimensional heat map H(x, y) and a one-dimensional heat map D(z), which can greatly reduce the data amount and thus improve the overall speed of gesture key point detection.

[0079] For step S3, the position of the maximum heat of the position heat map can be calculated according to the formula ; wherein H is the position heat map, is the position of the maximum heat of the position heat map.

[0080] For step S4, the first matrix is extracted from the position heat map with the position of the maximum heat degree of the position heat map as the center, specifically, an n*n matrix is extracted from the position heat map with the position of the maximum heat degree of the position heat map as the center; n is a positive integer. It should be noted that the specific value of n can be determined according to actual needs and test results.

[0081] For step S5, the sub-pixel information of the Gaussian heat map is successfully extracted by the method of coordinate expectation through softmax normalization based on the output heat map. Experiments show that this method has a more simplified neural network structure, a faster inference speed, and can extract more accurate key point coordinate information than the heatmap+offset method.

[0082] For step S6, for the sake of understanding, the following refers to the original image as the gesture image to be detected. Generally, the size of the output heat map should be the same as the original image in an ideal case. Figure 1 However, in reality, it is often not feasible due to the computing power of hardware. In order to reduce the amount of calculation, it is generally necessary to change the length and width of the heat map to 1 / p of the original image, where p is greater than 1; for example, the length and width of the heat map are changed to 1 / 4 of the original image. Therefore, after calculating the position expectation value (x', y') of the key point coordinates using the position heat map, the position expectation value of the key point coordinates needs to be converted according to the ratio of the width of the position heat map to the width of the original image and the ratio of the length of the position heat map to the length of the original image. It should be noted that if the computing power of the hardware device is strong enough, p can also be equal to 1.

[0083] For steps S7-S9, since the key point coordinates are (x, y, z), steps S3-S6 only calculate the x value and y value of the key point, therefore, the z value of the key point coordinates, i.e. the depth value of the key point coordinates, needs to be calculated. Through steps S7-S9, the depth value of the key point coordinates can be calculated.

[0084] For step S10, since the x value and y value of the key point coordinates are calculated through steps S3-S6, and the z value of the key point coordinates is calculated through steps S7-S9, the key point coordinates (x, y, z) can be determined.

[0085] This invention changes the traditional groundtruth design method. Instead of directly using coordinate points as groundtruth or groundtruth heatmap plus offset, it establishes a Gaussian heatmap with sub-pixel information. It also changes the traditional processing method. Instead of using offset to compensate for precision loss, it successfully extracts sub-pixel information from the Gaussian heatmap by calculating coordinate expectation through softmax normalization. Experiments have shown that this method has a more streamlined neural network structure and faster inference speed than the heatmap+offset method, and can extract more accurate key point coordinate information.

[0086] In one embodiment, the pre-set position heat map is a two-dimensional Gaussian distribution, the mean of the two-dimensional Gaussian distribution is (x / 4, y / 4), the standard deviation is σ, the valid area is m*m, and the part exceeding the valid area is all set to 0; the depth heat map is a one-dimensional Gaussian distribution, the mean of the one-dimensional Gaussian distribution is (z+0.5)*32, the standard deviation is σ, the valid area is 1*m, and the part exceeding the valid area is all set to 0, wherein (x, y, z) are the coordinates of the marked gesture key points.

[0087] In the embodiment of the present invention, by designing the position heatmap and the depth heatmap in this way, both the position heatmap and the depth heatmap have sub-pixel information.

[0088] In one embodiment, normalizing the first matrix and calculating the expected position value of the key point coordinates according to the result of normalizing the first matrix includes:

[0089] According to the formula Calculate the expected value of the key point coordinates; where i, j are the elements h in the first matrix h i,j At the coordinate position in the position heat map, the matrix h is an n*n matrix, where n is a positive integer. is the position with the maximum heat in the position heat map, H is the position heat map, τ is the softmax temperature, which is the set value, and (x′, y′) is the expected value of the position of the key point coordinates.

[0090] In the embodiment of the present invention, first, it is necessary to calculate the Calculate the position of the maximum heat of the position heat map, and then extract the n*n matrix from the position heat map with the position of the maximum heat of the position heat map as the center, and then according to the formula Calculate the expected position value of the key point coordinates. It should be noted that the specific value of n can be determined according to actual needs and test results.

[0091] The embodiment of the present invention successfully extracted sub-pixel information of the Gaussian heat map according to the above formula. Experiments have shown that this method has a more streamlined neural network structure and faster inference speed than the heatmap+offset method, and can extract more accurate key point coordinate information.

[0092] In one embodiment, converting the expected position values ​​of the key point coordinates according to the ratio of the width of the position heat map to the width of the gesture image to be detected, and the ratio of the length of the position heat map to the length of the gesture image to be detected to obtain the position values ​​of the key point coordinates includes:

[0093] According to the formula The expected position value of the key point coordinates is converted to obtain the position value of the key point coordinates; wherein (x, y) is the position value of the key point coordinates, (x′, y′) is the expected position value of the key point coordinates, s x is the ratio of the width of the position heat map to the width of the gesture image to be detected, s y is the ratio of the length of the position heat map to the length of the gesture image to be detected.

[0094] In the embodiment of the present invention, the output heat map size should ideally be the same as the original Figure 1 But in reality, it is usually not feasible due to the limitation of hardware computing power. In order to reduce the amount of calculation, it is generally necessary to change the length and width of the heat map to 1 / p of the original image, where p is a positive integer; for example, change the length and width of the heat map to 1 / 4 of the original image. Therefore, when the expected position value of the key point coordinates is calculated using the position heat map, the expected position value of the key point coordinates needs to be converted back to the original image. Therefore, after the expected position value of the key point coordinates is calculated, the expected position value of the key point coordinates is converted according to the ratio of the width of the position heat map to the width of the gesture image to be detected, and the ratio of the length of the position heat map to the length of the gesture image to be detected. Specifically, first, according to the formula Calculate the ratio of the width of the position heat map to the width of the gesture image to be detected, according to the formula Calculate the ratio of the length of the position heat map to the length of the gesture image to be detected, and then use the formula Calculate the position value of the key point in the original image.

[0095] It should be noted that the size of the gesture image to be detected in the gesture key point heat map prediction network is generally set. If the gesture image to be detected does not meet the set size, it needs to be modified to the set size. For example, the gesture image that does not meet the set size can be cropped to the set size. Since the gesture image to be cropped is pre-set, the size of the heat map is also set. Therefore, s can be predetermined.x and s y The value of .

[0096] In one embodiment, normalizing the second matrix and calculating the depth expectation value of the key point coordinates according to the result of normalizing the second matrix includes:

[0097] According to the formula Calculate the expected depth value of the key point coordinates;

[0098] Where k is the element d in the second matrix d k At the coordinate position in the heat map D, the matrix is ​​a 1*n matrix, n is a positive integer, τ is the softmax temperature, which is the set value, is the position of the maximum heat of the depth heat map, and z is the expected depth value of the key point coordinate.

[0099] In the embodiment of the present invention, first, according to the formula Calculate the maximum heat position of the depth heat map D Then take the location of the maximum heat in the depth heat map D As the center, extract the 1*n matrix, and then according to the formula The expected depth value of the key point coordinates is calculated and used as the depth value of the key point coordinates.

[0100] The embodiment of the present invention successfully extracted sub-pixel information of the Gaussian heat map according to the above formula. Experiments have shown that this method has a more streamlined neural network structure and faster inference speed than the heatmap+offset method, and can extract more accurate key point coordinate information.

[0101] See also Figure 4 In one embodiment, inputting the gesture image to be detected into a gesture key point heat map prediction network and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network includes:

[0102] S21, inputting the gesture image to be detected and the historical key points corresponding to the gesture image to be detected into a second gesture key point heat map prediction network, extracting a feature map of the image to be detected and a feature map of the historical key points by the second gesture key point heat map prediction network, and splicing the feature map of the image to be detected and the feature map of the historical key points, and generating M pairs of heat maps based on the spliced ​​feature maps; wherein each pair of heat maps includes a position heat map and a depth heat map, and M is a positive integer;

[0103] S22. Obtain a pair of heat maps from the M pairs of heat maps.

[0104] See also Figure 5In one embodiment, inputting the gesture image to be detected into a gesture key point heat map prediction network and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network includes:

[0105] S31, inputting the gesture image to be detected into a first gesture key point heat map prediction network, extracting a feature map of the image to be detected by the first gesture key point heat map prediction network, and generating M pairs of heat maps based on the feature map of the image to be detected; wherein each pair of heat maps includes a position heat map and a depth heat map, and M is a positive integer;

[0106] S32. Obtain a pair of heat maps from the M pairs of heat maps.

[0107] In the embodiment of the present invention, it should be noted that M is generally 21, representing 21 heat maps corresponding to 21 key points. However, if 21 key point heat maps are not used for training the neural network, M in the embodiment of the present invention may not be limited to 21 key points and can be set according to actual conditions. The embodiment of the present invention does not limit this.

[0108] In the embodiment of the present invention, it should be noted that a video includes multiple frames of images, the gesture image to be detected is the image of the current frame, and the historical key points are the gesture key points detected based on the image before the current image frame (for example, it can be understood as the previous frame of image, the previous two frames of image, the previous three frames of image...).

[0109] The gesture key point heat map prediction network can be understood as a heat map regression neural network. Unlike other heat map regression neural networks that have only one input, the heat map regression neural network of the present invention has two inputs, namely the input image and the historical key point positions. It should be noted that historical key points can provide effective information to help the current image perform key point regression. In addition, unlike other existing technologies, the neural network of the present invention utilizes a feature pyramid, which can extract effective information from feature maps of various scales and improve the final key point position accuracy.

[0110] In one embodiment, the second gesture key point heat map prediction network structure includes a ResNet34 / 18 backbone and a detection head; wherein the detection head includes a position head and a depth head.

[0111] In an embodiment of the present invention, the positionhead outputs a position heatmap, and the depthhead outputs a depth heatmap. The neural network structure used in the embodiment of the present invention is relatively streamlined and has a faster processing speed. It should be noted that in addition to using the neural network structure described in the present invention, other neural network structures can also be used, as long as they can achieve feature map extraction and feature map splicing functions. The present invention is an embodiment and does not limit this.

[0112] See also Figure 6 In an embodiment of the present invention, a gesture key point detection device includes:

[0113] Acquisition module 1 is used to obtain the gesture image to be detected;

[0114] Heatmap acquisition module 2 is used to input the gesture image to be detected into a gesture key point heatmap prediction network, and obtain a pair of heatmaps from the output of the gesture key point heatmap prediction network; wherein each pair of heatmaps includes a position heatmap and a depth heatmap; the gesture key point heatmap prediction network is obtained through supervised training, and the supervised training labels include a pre-set position heatmap and a pre-set depth heatmap, and the pre-set position heatmap and the pre-set depth heatmap are both Gaussian heatmaps with sub-pixel information;

[0115] A first calculation module 3 is used to calculate the position of the maximum heat of the position heat map;

[0116] A first matrix extraction module 4 is configured to extract a first matrix from the position heat map with the position of the maximum heat of the position heat map as the center;

[0117] A position expected value calculation module 5 for key point coordinates, configured to normalize the first matrix and calculate the position expected value of the key point coordinates based on the result of normalizing the first matrix;

[0118] a conversion module 6, configured to convert the expected position values ​​of the key point coordinates according to the ratio of the width of the position heat map to the width of the gesture image to be detected, and the ratio of the length of the position heat map to the length of the gesture image to be detected, to obtain position values ​​of the key point coordinates;

[0119] A second calculation module 7 is used to calculate the position of the maximum heat of the depth heat map;

[0120] A second matrix extraction module 8 is configured to extract a second matrix from the depth heat map with the position of the maximum heat of the depth heat map as the center; wherein the second matrix has the same number of columns as the first matrix;

[0121] The depth expectation value calculation module 9 of the key point coordinates normalizes the second matrix, and calculates the depth expectation value of the key point coordinates according to the result of normalizing the second matrix.

[0122] The key point coordinate determination module 10 is configured to obtain the key point coordinates according to the position value of the key point coordinates and the depth expectation value of the key point coordinates.

[0123] Embodiment three:

[0124] With reference to Figure 7 The computer device can be a server, and the internal structure of the computer device can be as shown in FIG. 6. Figure 7As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data applicable to a gesture key point detection method, etc. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by a processor, it realizes a gesture key point detection, including: obtaining a gesture image to be detected; inputting the gesture image to be detected into a gesture key point heat map prediction network, and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network; wherein each pair of heat maps includes a position heat map and a depth heat map; the gesture key point heat map prediction network is obtained through supervised training, and the supervised training labels include a pre-set position heat map and a pre-set depth heat map, and the pre-set position heat map and the pre-set depth heat map are both Gaussian heat maps with sub-pixel information; calculating the position of the maximum heat of the position heat map; extracting a first matrix from the position heat map with the position of the maximum heat of the position heat map as the center; normalizing the first matrix, The expected position value of the key point coordinates is calculated based on the result of normalizing the first matrix; the expected position value of the key point coordinates is converted according to the ratio of the width of the position heat map and the width of the gesture image to be detected, and the ratio of the length of the position heat map and the length of the gesture image to be detected to obtain the position value of the key point coordinates; the position of the maximum heat of the depth heat map is calculated; the second matrix is ​​extracted from the depth heat map with the position of the maximum heat of the depth heat map as the center; wherein the second matrix and the first matrix have the same number of columns; the second matrix is ​​normalized, and the depth expected value of the key point coordinates is calculated based on the result of normalizing the second matrix; the key point coordinates are obtained according to the position value of the key point coordinates and the depth expected value of the key point coordinates.

[0125] This invention changes the traditional groundtruth design method. Instead of directly using coordinate points as groundtruth or groundtruth heatmap plus offset, it establishes a Gaussian heatmap with sub-pixel information. It also changes the traditional processing method. Instead of using offset to compensate for precision loss, it successfully extracts sub-pixel information from the Gaussian heatmap by calculating coordinate expectation through softmax normalization. Experiments have shown that this method has a more streamlined neural network structure and faster inference speed than the heatmap+offset method, and can extract more accurate key point coordinate information.

[0126] Example 4:

[0127] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements a gesture key point detection method when executed by a processor, including the steps of: obtaining a gesture image to be detected; inputting the gesture image to be detected into a gesture key point heat map prediction network, and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network; wherein each pair of heat maps includes a position heat map and a depth heat map; the gesture key point heat map prediction network is obtained through supervised training, and the supervised training labels include a pre-set position heat map and a pre-set depth heat map, and the pre-set position heat map and the pre-set depth heat map are both Gaussian heat maps with sub-pixel information; calculating the position of the maximum heat of the position heat map; taking the position of the maximum heat of the position heat map as the center, extracting the first A matrix; normalize the first matrix, and calculate the position expected value of the key point coordinates according to the result of normalizing the first matrix; convert the position expected value of the key point coordinates according to the ratio of the width of the position heat map and the width of the gesture image to be detected, and the ratio of the length of the position heat map and the length of the gesture image to be detected to obtain the position value of the key point coordinates; calculate the position of the maximum heat of the depth heat map; extract a second matrix from the depth heat map with the position of the maximum heat of the depth heat map as the center; wherein the second matrix and the first matrix have the same number of columns; normalize the second matrix, and calculate the depth expected value of the key point coordinates according to the result of normalizing the second matrix; obtain the key point coordinates according to the position value of the key point coordinates and the depth expected value of the key point coordinates.

[0128] The above-mentioned execution is applicable to a method for detecting gesture key points. The present invention changes the traditional groundtruth design method. Instead of directly using coordinate points as groundtruth or a groundtruth heatmap plus an offset, a Gaussian heatmap with sub-pixel information is established. The traditional processing method is also changed. Instead of using an offset to compensate for precision loss, the sub-pixel information of the Gaussian heatmap is successfully extracted by calculating the expected coordinates through softmax normalization. Experiments have shown that this method has a more streamlined neural network structure than the heatmap+offset method, faster inference speed, and can extract more accurate key point coordinate information.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, databases, or other media in this application and in examples provided herein, unless specifically stated otherwise, can include non-volatile and / or volatile memory. Non-volatile memory can include, for example, read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include, for example, random access memory (RAM), or external cache memory. As an illustration and not a limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus DRAM (RDRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0130] It should be noted that the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, a device, an article or a method that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, device, article or method. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, device, article or method that includes the element.

[0131] The above description is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A gesture key point detection method, characterized in that: include: Get the gesture image to be detected; Inputting the gesture image to be detected into a gesture key point heat map prediction network, and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network; wherein each pair of heat maps includes a position heat map and a depth heat map; the gesture key point heat map prediction network is obtained through supervised training, and the supervised training labels include a pre-set position heat map and a pre-set depth heat map, and the pre-set position heat map and the pre-set depth heat map are both Gaussian heat maps with sub-pixel information; Calculate the position of the maximum heat of the position heat map; Extracting a first matrix from the position heat map with the position of maximum heat in the position heat map as the center; Normalizing the first matrix, and calculating expected position values ​​of key point coordinates based on a result of normalizing the first matrix; Converting the expected position values ​​of the key point coordinates according to a ratio of a width of the position heat map to a width of the gesture image to be detected, and a ratio of a length of the position heat map to a length of the gesture image to be detected, to obtain position values ​​of the key point coordinates; Calculate the position of the maximum heat of the depth heat map; Extracting a second matrix from the depth heat map with the position of the maximum heat of the depth heat map as the center; wherein the second matrix has the same number of columns as the first matrix; Normalizing the second matrix, and calculating the depth expectation value of the key point coordinates according to the result of normalizing the second matrix; The key point coordinates are obtained according to the position value of the key point coordinates and the expected depth value of the key point coordinates.

2. The gesture key point detection method according to claim 1, characterized in that: The pre-set position heat map is a two-dimensional Gaussian distribution, the mean of the two-dimensional Gaussian distribution is (x / 4, y / 4), the standard deviation is σ, the effective area is m*m, and the part exceeding the effective area is all set to 0; The depth heat map is a one-dimensional Gaussian distribution with a mean of (z+0.5)*32, a standard deviation of σ, a valid area of ​​1*m, and all parts exceeding the valid area are set to 0, where (x, y, z) are the coordinates of the marked gesture key points.

3. The gesture key point detection method according to claim 1, characterized in that: Normalizing the first matrix and calculating the expected position value of the key point coordinates according to the result of normalizing the first matrix includes: According to the formula Calculate the expected value of the key point coordinates; where i, j are the elements h in the first matrix h i,j At the coordinate position in the position heat map, the first matrix h is an n*n matrix, where n is a positive integer. is the position with the maximum heat in the position heat map, H is the position heat map, τ is the softmax temperature, which is the set value, and (x′, y′) is the expected value of the position of the key point coordinates.

4. The gesture key point detection method according to claim 1, characterized in that: Normalizing the second matrix and calculating the depth expectation value of the key point coordinates according to the result of normalizing the second matrix includes: According to the formula Calculate the expected depth value of the key point coordinates; Where k is the element d in the second matrix d k At the coordinate position in the heat map D, the matrix is ​​a 1*n matrix, n is a positive integer, τ is the softmax temperature, which is the set value, is the position of the maximum heat of the depth heat map, and z is the expected depth value of the key point coordinate.

5. The gesture key point detection method according to claim 3, characterized in that: The converting the expected position values ​​of the key point coordinates according to the ratio of the width of the position heat map to the width of the gesture image to be detected, and the ratio of the length of the position heat map to the length of the gesture image to be detected, to obtain the position values ​​of the key point coordinates, includes: According to the formula The expected position value of the key point coordinates is converted to obtain the position value of the key point coordinates; wherein (x, y) is the position value of the key point coordinates, (x′, y′) is the expected position value of the key point coordinates, s x is the ratio of the width of the position heat map to the width of the gesture image to be detected, s y is the ratio of the length of the position heat map to the length of the gesture image to be detected.

6. The gesture key point detection method according to claim 1, characterized in that: Inputting the gesture image to be detected into a gesture key point heat map prediction network, and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network, comprises: Inputting the gesture image to be detected into a first gesture key point heat map prediction network, extracting a feature map of the gesture image to be detected by the first gesture key point heat map prediction network, and generating M pairs of heat maps based on the feature map of the gesture image to be detected; wherein each pair of heat maps includes a position heat map and a depth heat map; and M is a positive integer; Obtain a pair of heatmaps from the M pairs of heatmaps.

7. The gesture key point detection method according to claim 1, characterized in that: Inputting the gesture image to be detected into a gesture key point heat map prediction network, and obtaining a pair of heat maps from the output of the gesture key point heat map prediction network, comprises: Inputting the gesture image to be detected and the historical key points corresponding to the gesture image to be detected into a second gesture key point heat map prediction network, extracting a feature map of the image to be detected and a feature map of the historical key points by the second gesture key point heat map prediction network, and splicing the feature map of the image to be detected and the feature map of the historical key points, generating M pairs of heat maps based on the spliced ​​feature maps; wherein each pair of heat maps includes a position heat map and a depth heat map; and M is a positive integer; Obtain a pair of heatmaps from the M pairs of heatmaps.

8. A gesture key point detection device, characterized in that: include: An acquisition module, used to acquire the gesture image to be detected; A heatmap acquisition module is configured to input the gesture image to be detected into a gesture keypoint heatmap prediction network and obtain a pair of heatmaps from the output of the gesture keypoint heatmap prediction network; wherein each pair of heatmaps includes a position heatmap and a depth heatmap; the gesture keypoint heatmap prediction network is obtained through supervised training, and the supervised training labels include a pre-set position heatmap and a pre-set depth heatmap, and the pre-set position heatmap and the pre-set depth heatmap are both Gaussian heatmaps with sub-pixel information; A first calculation module is used to calculate the position of the maximum heat of the position heat map; A first matrix extraction module is configured to extract a first matrix from the position heat map with the position of the maximum heat of the position heat map as the center; A position expected value calculation module for key point coordinates, configured to normalize the first matrix and calculate the position expected value of the key point coordinates based on a result of normalizing the first matrix; a conversion module, configured to convert the expected position values ​​of the key point coordinates according to a ratio of a width of the position heat map to a width of the gesture image to be detected, and a ratio of a length of the position heat map to a length of the gesture image to be detected, to obtain position values ​​of the key point coordinates; A second calculation module is used to calculate the position of the maximum heat of the depth heat map; A second matrix extraction module, configured to extract a second matrix from the depth heat map with the position of the maximum heat of the depth heat map as the center; wherein the second matrix has the same number of columns as the first matrix; A depth expectation value calculation module for key point coordinates, configured to normalize the second matrix and calculate the depth expectation value of the key point coordinates based on a result of normalizing the second matrix; The key point coordinate determination module is used to obtain the key point coordinates according to the position value of the key point coordinates and the expected depth value of the key point coordinates.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images