Two-stage hand key point identification method based on deep learning

Abstract

The invention provides a two-stage hand key point identification method based on deep learning, and belongs to the technical field of artificial intelligence, and the method specifically comprises the steps: obtaining image data containing a hand, and generating a detection frame training label and a key point training label based on the image data; training a target detection module by using the detection frame training label and the first loss function, and training a key point detection module by using the key point training label and the second loss function; inputting a to-be-detected image into the trained target detection module to obtain a prediction frame; intercepting a corresponding area image in the to-be-detected image according to the coordinate of the prediction frame; and inputting the intercepted image into the trained key point detection module to obtain coordinates of a plurality of key points in the to-be-detected image. Through the processing scheme provided by the invention, the recognition accuracy of the hand key points is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a two-stage hand key point recognition method based on deep learning. Background Art

[0002] Understanding human gestures is one of the important tasks of many real-world artificial intelligence applications. Its purpose is to determine the positions of various joints of the human hand based on the input camera image information. However, for actual images, human hands are often in different positions and postures, which can easily cause occlusion, so a certain degree of prediction and estimation is often required. The process of designing feature description methods through manual modeling is very cumbersome, unable to mine higher-level feature information, and difficult to achieve real-time requirements, which will affect the performance of traditional gesture recognition methods. Summary of the invention

[0003] In view of this, an embodiment of the present application provides a two-stage hand key point recognition method based on deep learning, conducts research on the problem of gesture key point detection in daily scenes, analyzes common feature extraction networks and detection regression algorithms, and designs a two-stage detection algorithm process for this problem on this basis, thereby solving the problem of inaccurate detection of traditional algorithms and improving the positioning accuracy of gesture key points.

[0004] The present application embodiment provides a two-stage hand key point recognition method based on deep learning, the method comprising:

[0005] Acquire image data containing a hand, and generate a detection frame training label and a key point training label based on the image data;

[0006] The object detection module is trained using the detection box training label and the first loss function, and the key point detection module is trained using the key point training label and the second loss function;

[0007] Input the image to be detected into the trained object detection module to obtain the predicted box;

[0008] Intercepting a corresponding area image in the image to be detected according to the coordinates of the prediction box;

[0009] The captured image is input into the trained key point detection module to obtain the coordinates of multiple key points in the image to be detected.

[0010] According to a specific implementation of an embodiment of the present application, generating a detection frame training label and a key point training label based on the image data includes:

[0011] Using image annotation software to annotate and store the rectangular area and key point positions of the hand in the image data;

[0012] Based on the (x, y, w, h) coordinates of each rectangular area, generating the detection box training label, where (x, y) is the coordinate of the center point of the rectangular area, w is the width of the rectangular area, and h is the height of the rectangular area;

[0013] Based on the (X, Y) coordinates of each key point, the key point training labels are generated.

[0014] According to a specific implementation method of an embodiment of the present application, the feature extraction network of the target detection module is set to a Darknet-53 network.

[0015] According to a specific implementation of the embodiment of the present application, the step of inputting the image to be detected into a trained target detection module to obtain a prediction frame includes:

[0016] Input the image to be detected into the trained Darknet-53 network, take the last three feature layer outputs of the network, and obtain three output branches, each of which predicts and outputs multiple prediction boxes to be screened;

[0017] Calculate the center coordinates and width and height of each prediction box to be filtered;

[0018] Based on the calculation results, the final prediction box is screened out from the multiple prediction boxes to be screened by a non-maximum suppression method.

[0019] According to a specific implementation of the embodiment of the present application, the method of selecting the final prediction box from the multiple prediction boxes to be selected by the non-maximum suppression method includes:

[0020] Step 1: For the set B of all prediction boxes to be screened, sort them in descending order according to the confidence level;

[0021] Step 2: Select the prediction box with the highest confidence from set B and put it into result D;

[0022] Step 3: traverse the prediction box to be filtered in set B, calculate the IoU value between it and the prediction box in result D, and if the IoU value is greater than the preset threshold, delete the prediction box to be filtered from set B;

[0023] Step 4: Repeat steps 2 and 3 until there are no more prediction boxes to be screened in set B. At this time, the prediction boxes to be screened in result D are the final prediction boxes.

[0024] According to a specific implementation method of an embodiment of the present application, the first loss function includes coordinate center loss, width and height loss, and confidence loss. The coordinate center loss is used to calculate the loss of the center coordinates of all predicted boxes that match the real box, the width and height loss is used to calculate the loss of the width and height coordinates of all predicted boxes that match the real box, and the confidence loss is used to calculate whether all detection boxes contain real objects.

[0025] 11. According to a specific implementation of the embodiment of the present application, the expression of the first loss function is:

[0026] Loss = L1 + L2 + L3,

[0027]

[0028]

[0029]

[0030] Among them, Loss is the first loss function, L1 is the coordinate center loss, L2 is the width and height loss, and L3 is the confidence loss; and are indicator factors, If the predicted box is assigned to the true box, is 1, otherwise is 0; B is the number of real boxes, S is the output feature map size, S 2 is the number of prediction boxes; is the center coordinate of the real frame, (x i ,y i ) is the center coordinate of the prediction box output by the algorithm; is the width and height of the real frame, The width and height of the predicted box output by the algorithm; Indicates whether the predicted box output by the algorithm matches the real box. If a predicted box i matches the real box j, then otherwise is the confidence of the algorithm output; noobj It is an empirical parameter used to adjust the proportion of the confidence loss of the predicted box that does not match the true box in the total confidence loss.

[0031] According to a specific implementation method of an embodiment of the present application, the feature extraction network of the key point detection module is set to a ResNet50 network.

[0032] According to a specific implementation of an embodiment of the present application, the second loss function is set to an MSE loss function, and the expression of the MSE loss function is:

[0033]

[0034] Among them, Loss' is the MSE loss function, (X1, Y1) is the model output coordinate, and (X2, Y2) is the true value.

[0035] According to a specific implementation method of an embodiment of the present application, the ResNet50 network is divided into five stages, the first stage performs a convolution operation, the second to fifth stages stack different numbers of residual blocks respectively, and an attention module is added to the residual block of each stage; each layer of the ResNet50 network is set as a convolution layer, and 42 output channels are set, and the 42 output channels correspond to the X and Y coordinate positions of 21 key points respectively.

[0036] Beneficial effects:

[0037] The two-stage hand key point recognition method based on deep learning in the embodiment of the present application optimizes the hand key point detection process on the basis of the original technology, and is divided into a target detection part and a key point detection part. In the target detection part, a neural network is constructed to integrate multi-level features to achieve a high-level expression of the original image, and detect and output at different levels. After obtaining the hand image according to the detection results, the improved ResNet50 is used for feature extraction, and the network is adapted according to the key point detection requirements, the loss function is designed, the key point coordinates are output, and the accuracy of gesture key point recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of the overall network architecture of a target detection module according to an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of marking key points of a hand according to an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of a residual block neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0043] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0044] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0045] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0046] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0047] In daily life, there are often multiple hands in an image, and each hand needs to be detected and its key point position determined. Since 2012, image feature extraction methods based on deep learning have become increasingly popular, providing some help for this task. For the problem of hand detection, YOLO (You Only Look Once) is a common single-stage target detection algorithm, which can directly output the two-dimensional position and category information of the detected object based on neural network feature extraction. For the problem of key point detection, the feature extraction network based on ResNet extracts high-level abstract features by stacking multiple feature extraction modules. It achieves better results in the field of image classification. By adapting the output part of the network, it can perform key point coordinate regression tasks and output the key point coordinates of the hands in the image.

[0048] Based on the above content, the embodiment of the present application provides a two-stage hand key point recognition method based on deep learning. Figures 1 to 3 Describe in detail.

[0049] In one embodiment, a two-stage hand key point recognition method based on deep learning is provided, the method comprising the following steps:

[0050] Acquire image data containing a hand, and generate a detection frame training label and a key point training label based on the image data;

[0051] The object detection module is trained using the detection box training label and the first loss function, and the key point detection module is trained using the key point training label and the second loss function;

[0052] Input the image to be detected into the trained object detection module to obtain the predicted box;

[0053] Intercepting a corresponding area image in the image to be detected according to the coordinates of the prediction box;

[0054] The captured image is input into the trained key point detection module to obtain the coordinates of multiple key points in the image to be detected.

[0055] For daily images, human hands are often in random positions and only occupy a small part of the image area. It is difficult to detect key points directly. Therefore, the overall key point detection scheme of this embodiment is divided into two modules for detection. The first module is a target detection module for identifying human hands. The function of this module is to determine the area where the human hands are located in the image and cut out this part of the image. The second module is a key point detection module. The function of this module is to detect the location of the key points in each hand image. Both modules of the algorithm flow include neural network construction, loss function design, and data acquisition and preprocessing.

[0056] Furthermore, the generating of the detection frame training label and the key point training label based on the image data includes:

[0057] Using image annotation software to annotate and store the rectangular area and key point positions of the hand in the image data;

[0058] Based on the (x, y, w, h) coordinates of each rectangular area, generating the detection box training label, where (x, y) is the coordinate of the center point of the rectangular area, w is the width of the rectangular area, and h is the height of the rectangular area;

[0059] Based on the (X, Y) coordinates of each key point, the key point training labels are generated.

[0060] In the specific implementation, for the target detection module, it is necessary to first build a detection training data set. Use the camera to collect and save images of human hands in multiple scenes, use image annotation software to annotate the human hand area and key points in the image, and use x, y, w, h to represent each detection frame, and generate the corresponding training label file. For the key point detection module, use (X, Y) to represent the training label of the key point.

[0061] In one embodiment, the feature extraction network of the target detection module is set to a Darknet-53 network.

[0062] During the specific implementation, in order to maintain the real-time performance of the overall solution, the YOLO v3 algorithm is used as the basic detector. Compared with the previous generation detection algorithm, Darknet-53 is used as the basic feature extraction network in this embodiment. Compared with the Darknet-19 network, it has stronger feature extraction capabilities, and the last output layer network uses a fully convolutional layer to replace the fully connected layer, which reduces the number of parameters and speeds up training and inference. Darknet-53 is mainly composed of multiple stacked residual blocks. The use of residual blocks can accelerate the convergence speed of the network, reduce the problem of small gradients, and better learn the abstract features in the input image. After basic feature extraction, the low-level feature semantic information is relatively small, but the target location information is accurate. The high-level feature semantic information is relatively rich, but the target location is relatively rough. Refer to Figure 1 ,In order to improve the detection ability of the detection algorithm for ,human hands under different scales, three feature layers at different positions of the ,feature extraction network are taken, and the high-level features are ,upsampled at the same time and fused with the low-level feature layers, and ,the algorithm prediction is performed at three scales, so that the algorithm can ,detect both small and large targets at the same time.

[0063] In specific implementation, the output of the original detection algorithm for each predicted bounding box is x, y, w, h, confidence and class, where x and y represent the center point of the detection box, w and h represent the width and height of the detection box, confidence is the confidence, and class is the class probability. For this application, the detection category is a single class, so the network output layer can be simplified and the class probability class information is no longer output.

[0064] Therefore, in this embodiment, the first loss function includes coordinate center loss, width and height loss, and confidence loss. The coordinate center loss is used to calculate the loss of the center coordinates of all predicted boxes that match the real box, the width and height loss is used to calculate the loss of the width and height coordinates of all predicted boxes that match the real box, and the confidence loss is used to calculate whether all detection boxes contain real objects.

[0065] 12. Furthermore, the expression of the first loss function is:

[0066] Loss = L1 + L2 + L3,

[0067]

[0068]

[0069]

[0070] Among them, Loss is the first loss function, L1 is the coordinate center loss, L2 is the width and height loss, and L3 is the confidence loss; and are indicator factors, If the predicted box is assigned to the true box, is 1, otherwise is 0; B is the number of real boxes, S is the output feature map size, S 2 is the number of prediction boxes; is the center coordinate of the real frame, (x i ,y i ) is the center coordinate of the prediction box output by the algorithm; is the width and height of the real frame, The width and height of the predicted box output by the algorithm; Indicates whether the predicted box output by the algorithm matches the real box. If a predicted box i matches the real box j, then otherwise is the confidence of the algorithm output; noobj It is an empirical parameter used to adjust the proportion of the confidence loss of the predicted box that does not match the true box in the total confidence loss.

[0071] Specifically, the first item L1 is the coordinate center loss, which is calculated for the center coordinate loss of all predicted boxes that match the real box. is the coordinate center error weight, which is used to adjust the proportion of different size detection box losses in the total coordinate center loss. The second term L2 is the width and height loss, which calculates the width and height coordinate losses of all predicted boxes that match the real box. The third term L3 is the confidence loss, which calculates the loss of whether all detection boxes contain real objects. Since most predicted boxes cannot match the real box, we set λ noobj It is used to adjust the proportion of the confidence loss of the predicted box that does not match the real box in the total confidence loss. The value is 0.5 in the actual experiment.

[0072] The above loss functions take into account the position, size and confidence of the detection box. By calculating the total loss and returning the gradient, the network parameters are updated based on the gradient descent method to optimize the final target detection algorithm performance.

[0073] In one embodiment, the feature extraction network of the key point detection module is set to a ResNet50 network.

[0074] In the specific implementation, the key point detection module takes as input the detection frame area image obtained by the target detection module. For each hand image, a total of 21 key points are defined and annotated, such as Figure 2 As shown in Figure 1, each point is represented by its pixel coordinates x and y. Key point prediction can be treated as a regression problem, where the network predicts the output position of each key point.

[0075] ResNet50 is used as the feature extraction network. The network is divided into five stages and also uses residual block stacking for feature extraction. Figure 3 , assuming that the input image is a, the output is H(a), and the output after the convolution operation is a nonlinear function of F(a), then the final output is H(a)=F(a)+a, and the network only needs to fit F(a)=H(a)-a, which is easier to optimize. For the last fully connected layer of the original network, it is changed to a convolutional layer with a total of 42 channel outputs, corresponding to the x and y coordinate positions of 21 annotated key points. In order to further improve the feature extraction capability of the original network, an attention mechanism module is added to the residual block, which has a small number of parameters and calculations and can be plug-and-play. By using the channel relationship between features to generate a channel attention map, the network can focus on the meaningful target category of a given input image. By using the spatial relationship between features to generate a spatial attention map, the network can focus on the location of the target.

[0076] For the final output of ResNet50, the MSE loss function is used to calculate the difference between the predicted coordinate value and the true value, and gradient descent optimization is performed. For a certain output coordinate (X1, Y1) and true value (X2, Y2), the calculation method is as follows:

[0077]

[0078] Finally, the accurate coordinate values ​​of the hand key points are obtained.

[0079] In one embodiment, the second loss function is set to an MSE loss function, and the expression of the MSE loss function is:

[0080]

[0081] Among them, Loss' is the MSE loss function, (X1, Y1) is the model output coordinate, and (X2, Y2) is the true value.

[0082] In one embodiment, the ResNet50 network is divided into five stages. The first stage performs convolution operation, and different numbers of residual blocks are stacked in the second to fifth stages respectively, and an attention module is added to the residual block in each stage; each layer of the ResNet50 network is set as a convolution layer, and 42 output channels are set, and the 42 output channels correspond to the X and Y coordinate positions of 21 key points respectively.

[0083] In one embodiment, the step of inputting the image to be detected into a trained target detection module to obtain a prediction frame includes:

[0084] Input the image to be detected into the trained Darknet-53 network, take the last three feature layer outputs of the network, and obtain three output branches, each of which predicts and outputs multiple prediction boxes to be screened;

[0085] Calculate the center coordinates and width and height of each prediction box to be filtered;

[0086] Based on the calculation results, the final prediction box is screened out from the multiple prediction boxes to be screened by a non-maximum suppression method.

[0087] Furthermore, the method of selecting the final prediction box from the multiple prediction boxes to be selected by the non-maximum suppression method includes:

[0088] Step 1: For the set B of all prediction boxes to be screened, sort them in descending order according to the confidence level;

[0089] Step 2: Select the prediction box with the highest confidence from set B and put it into result D;

[0090] Step 3: traverse the prediction box to be filtered in set B, calculate the IoU value between it and the prediction box in result D, and if the IoU value is greater than the preset threshold, delete the prediction box to be filtered from set B;

[0091] Step 4: Repeat steps 2 and 3 until there are no more prediction boxes to be screened in set B. At this time, the prediction boxes to be screened in result D are the final prediction boxes.

[0092] Taking a specific image to be detected as an example, the two-stage hand key point recognition method based on deep learning is described in detail below, including the following steps:

[0093] Step 1: Use a camera to collect image data containing hands in daily scenes. For the collected image data, use image annotation software to annotate and store the rectangular area and key point positions. For the object detection module, use (x, y, w, h) to represent the label of the detection box. For the key point detection module, use (X, Y) to represent the label of the key point. According to the first loss function and the second loss function, the two modules are trained.

[0094] Step 2: Reference Figure 1 In the module use phase, the image to be detected is scaled to 416x416 and fed into the DarkNet-53 neural network of the target detection module. The number of residual blocks in the network is 1, 2, 8, 8, and 4. For the 416-size image fed in, the last three feature layer outputs of the network are taken, with sizes of 52x52, 26x26, and 13x13, respectively. For the 13x13-size feature map, it is first used as the first output branch, then fused with the 26x26 feature map through upsampling, as the second output branch, and then upsampled again and fused with the 52x52 feature map, as the third output branch.

[0095] Step 3: The predicted output of the three output branches includes the confidence and offset of each detection box. The number of output channels is 3x(1+4), which represents the confidence and offset of each prior box. x , c y ) is the coordinate of the current grid, (P w , P h ) is the width and height of the prior frame. If the offset of the currently predicted prior frame is (t x ,t y ,t w ,t h ), the coordinates of the prediction box (b x ,b y ) and width and height (b w ,b h ), the position and size of the actual detection frame can be calculated by the following formula:

[0096]

[0097] Step 4: Use the non-maximum suppression (NMS) method to eliminate redundant prediction boxes and retain high-quality prediction boxes as the final detection results. The specific steps are:

[0098] Step 1: For the set B of all prediction boxes to be screened, sort them in descending order according to the confidence level;

[0099] Step 2: Select the prediction box with the highest confidence from set B and put it into result D;

[0100] Step 3: traverse the prediction box to be filtered in set B, calculate the IoU value between it and the prediction box in result D, and if the IoU value is greater than the preset threshold, delete the prediction box to be filtered from set B;

[0101] Step 4: Repeat steps 2 and 3 until there are no more prediction boxes to be screened in set B. At this time, the prediction boxes to be screened in result D are the final prediction boxes.

[0102] Step 5: After obtaining the detection frame information, the corresponding area image is intercepted from the original image according to the predicted detection frame coordinates and scaled to 416x416 size. For each intercepted image, it is sent to the key point detection network ResNet50 for forward propagation. The improved ResNet50 is divided into five stages. The first stage is the basic convolution operation, and the next four stages stack different numbers of residual blocks. Add an attention module to the residual block of each stage. The network has a total of 49 convolutional layers. For the last fully connected layer of the original network, it is changed to a convolutional layer with 42 output channels, corresponding to the x-coordinates and y-coordinates of 21 key points. At the same time, according to the position of the hand image in the original image, the key point coordinates are restored to the scale range of the original image, so as to obtain the key point positions of multiple hands in the original image.

[0103] The embodiment provided by the present invention optimizes the hand key point detection process based on the original technology, which is divided into a target detection part and a key point detection part. In the target detection part, a neural network is constructed to integrate multi-level features to achieve a high-level expression of the original image, and detection output is performed at different levels. After obtaining the hand image according to the detection results, the improved ResNet50 is used for feature extraction, and the network is adapted according to the key point detection requirements, the loss function is designed, the key point coordinates are output, and the accuracy of gesture key point recognition is improved.

[0104] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A two-stage hand key point recognition method based on deep learning, characterized in that: The method comprises: Acquire image data containing a hand, and generate a detection frame training label and a key point training label based on the image data; The object detection module is trained using the detection box training label and the first loss function, and the key point detection module is trained using the key point training label and the second loss function; Input the image to be detected into the trained object detection module to obtain the predicted box; Intercepting a corresponding area image in the image to be detected according to the coordinates of the prediction box; The captured image is input into the trained key point detection module to obtain the coordinates of multiple key points in the image to be detected.

2. The two-stage hand key point recognition method based on deep learning according to claim 1 is characterized in that: The generating of the detection frame training label and the key point training label based on the image data comprises: Using image annotation software to annotate and store the rectangular area and key point positions of the hand in the image data; Based on the (x, y, w, h) coordinates of each rectangular area, generating the detection box training label, where (x, y) is the coordinate of the center point of the rectangular area, w is the width of the rectangular area, and h is the height of the rectangular area; Based on the (X, Y) coordinates of each key point, the key point training labels are generated.

3. The two-stage hand key point recognition method based on deep learning according to claim 2 is characterized in that: The feature extraction network of the target detection module is set to the Darknet-53 network.

4. The two-stage hand key point recognition method based on deep learning according to claim 3 is characterized in that: The step of inputting the image to be detected into the trained target detection module to obtain a prediction frame includes: Input the image to be detected into the trained Darknet-53 network, take the last three feature layer outputs of the network, and obtain three output branches, each of which predicts and outputs multiple prediction boxes to be screened; Calculate the center coordinates and width and height of each prediction box to be filtered; Based on the calculation results, the final prediction box is screened out from the multiple prediction boxes to be screened by a non-maximum suppression method.

5. The two-stage hand key point recognition method based on deep learning according to claim 4 is characterized in that: The method of selecting the final prediction box from the multiple prediction boxes to be selected by the non-maximum suppression method includes: Step 1: For the set B of all prediction boxes to be screened, sort them in descending order according to the confidence level; Step 2: Select the prediction box with the highest confidence from set B and put it into result D; Step 3: traverse the prediction box to be filtered in set B, calculate the IoU value between it and the prediction box in result D, and if the IoU value is greater than the preset threshold, delete the prediction box to be filtered from set B; Step 4: Repeat steps 2 and 3 until there are no more prediction boxes to be screened in set B. At this time, the prediction boxes to be screened in result D are the final prediction boxes.

6. The two-stage hand key point recognition method based on deep learning according to claim 3 is characterized in that: The first loss function includes coordinate center loss, width and height loss and confidence loss. The coordinate center loss is used to calculate the loss of the center coordinates of all predicted boxes that match the real boxes, the width and height loss is used to calculate the loss of the width and height coordinates of all predicted boxes that match the real boxes, and the confidence loss is used to calculate whether all detection boxes contain real objects.

7. The two-stage hand key point recognition method based on deep learning according to claim 6, characterized in that: The expression of the first loss function is: Loss = L1 + L2 + L3, Among them, Loss is the first loss function, L1 is the coordinate center loss, L2 is the width and height loss, and L3 is the confidence loss; and are indicator factors, If the predicted box is assigned to the true box, is 1, otherwise is 0; B is the number of real boxes, S is the output feature map size, S 2 is the number of prediction boxes; is the center coordinate of the real frame, (x i ,y i ) is the center coordinate of the prediction box output by the algorithm; is the width and height of the real frame, The width and height of the predicted box output by the algorithm; Indicates whether the predicted box output by the algorithm matches the real box. If a predicted box i matches the real box j, then otherwise is the confidence of the algorithm output; noobj It is an empirical parameter used to adjust the proportion of the confidence loss of the predicted box that does not match the true box in the total confidence loss.

8. The two-stage hand key point recognition method based on deep learning according to claim 2, characterized in that: The feature extraction network of the key point detection module is set to a ResNet50 network.

9. The two-stage hand key point recognition method based on deep learning according to claim 8, characterized in that: The second loss function is set to the MSE loss function, and the expression of the MSE loss function is: Among them, Loss' is the MSE loss function, (X1, Y1) is the model output coordinate, and (X2, Y2) is the true value.

10. The two-stage hand key point recognition method based on deep learning according to claim 8, characterized in that: The ResNet50 network is divided into five stages. The first stage performs convolution operations. From the second stage to the fifth stage, different numbers of residual blocks are stacked respectively, and an attention module is added to the residual blocks in each stage. Each layer of the ResNet50 network is set as a convolutional layer, and 42 output channels are set, and the 42 output channels correspond to the X and Y coordinate positions of 21 key points respectively.

Images