Hand key point detection method based on contour features

Abstract

The invention provides a hand key point detection method based on contour features, and belongs to the technical field of machine vision, and the method specifically comprises the steps: obtaining a hand image sample set; preprocessing the hand image sample set to respectively obtain a first training set and a second training set; training the first model based on the first training set to obtain a hand contour feature detection model; training the second model based on the second training set to obtain a hand key point detection model; detecting a to-be-detected hand image by using the hand contour feature detection model to obtain an accurate hand contour feature map; and detecting the accurate hand contour feature map by using a hand key point detection model to obtain hand key points of the to-be-detected hand image. Through the processing scheme provided by the invention, the hand contour feature extraction precision is improved, and the hand key point detection adaptability and precision are improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a method for detecting key points of a hand based on contour features. Background Art

[0002] Hand key point detection is the core part of gesture recognition technology. It generally refers to the detection of the positions of the 21 skeletal joints of the hand. The meaning of the gesture can be determined by the changes in the positions of the hand key points. The gesture semantic understanding technology based on hand key points has a wide range of applications in human-computer interaction, autonomous driving, virtual reality and other fields.

[0003] At present, there are two main methods for detecting hand key points: the first is the traditional method based on texture feature extraction. This method has a small amount of computation, but the feature extraction is easily affected by changes in the external environment and cannot adapt to texture feature extraction when wearing gloves; the second is to extract features based on deep neural networks and directly infer the position coordinates of hand key points. This end-to-end one-stage neural network method indiscriminately extracts hand features, the clarity and interpretability of feature extraction are poor, and the convergence of network training is poor, resulting in low accuracy of hand key point detection. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a hand key point detection method based on contour features, which combines traditional feature extraction methods with image generation technology. It can generate clear and distinct hand contour features in complex situations, and infer the positions of hand key points through contour features, thereby accelerating the convergence speed of the inference network. It can also adapt to hand key point detection in special situations such as wearing gloves, thereby improving the adaptability and accuracy of hand key point detection.

[0005] The present application provides a method for detecting key points of a hand based on contour features, the method comprising:

[0006] Obtain a sample set of hand images;

[0007] Preprocessing the hand image sample set to obtain a first training set and a second training set respectively;

[0008] Training a first model based on the first training set to obtain a hand contour feature detection model;

[0009] Training the second model based on the second training set to obtain a hand key point detection model;

[0010] Using the hand contour feature detection model to detect the hand image to be detected, and obtaining an accurate hand contour feature map;

[0011] The hand key point detection model is used to detect the precise hand contour feature map to obtain the hand key points of the hand image to be detected.

[0012] According to a specific implementation of the embodiment of the present application, the preprocessing of the hand image sample set to obtain a first training set and a second training set respectively includes:

[0013] Performing edge detection on the hand image sample set based on an edge detection method to generate a binary rough contour feature map as a first training sample;

[0014] Manually annotating the hand contour in the original image of the hand image sample accurately to obtain a binary accurate contour feature map, and using it as a first sample label;

[0015] The first training sample and the first sample label constitute the first training set;

[0016] Converting the binarized precise contour feature map into a spatial feature map and using it as a second training sample;

[0017] Manually marking the position coordinates of the key points of the hand in the original image of the hand image sample, and converting the position coordinates of the key points of the hand into pixel ranking values ​​in the original image, wherein the pixel ranking values ​​are used as the second sample labels;

[0018] The second training samples and the second sample labels constitute the second training set.

[0019] According to a specific implementation method of an embodiment of the present application, the first model is set to a GAN network model, and the second model is set to a lightweight convolutional neural network model.

[0020] According to a specific implementation of the embodiment of the present application, the training of the first model based on the first training set includes:

[0021] The GAN network model includes a generator and a discriminator, and the parameters of the generator and the discriminator are initialized;

[0022] Inputting the first training sample into the generator to generate a new feature map;

[0023] Inputting the first sample label and the new feature map into the discriminator in turn to obtain a discrimination result of the discriminator;

[0024] According to the discrimination result and based on the first loss function, the parameters of the generator and the discriminator are iteratively optimized to complete the training of the GAN network model.

[0025] According to a specific implementation of the embodiment of the present application, the parameters of the generator and the discriminator are iteratively optimized based on the discrimination result and the first loss function to complete the training of the GAN network model, including:

[0026] According to the discrimination result, the generator loss value is calculated based on the first loss function, the gradient is back-propagated and the parameters of the generator are updated to improve the generation ability of the generator;

[0027] According to the discrimination result, a discriminator loss value is calculated based on the first data loss function, a gradient is back-propagated and the parameters of the discriminator are updated to improve the discrimination ability of the discriminator;

[0028] The training is repeated until the generator and the discriminator reach a convergence state or a predetermined number of training rounds are reached, thereby completing the training of the GAN network model.

[0029] According to a specific implementation of the embodiment of the present application, the expression of the first loss function is:

[0030] m G inm D axV(D,G)=E x [logD(x)]+E z [log(1-D(G(x)))],

[0031] Where G is the generator, D is the discriminator, x is the first sample label input to the discriminator, z is the first training sample input to the generator, G(z) represents the data generated based on z in the generator, D(x) represents the result of the discriminator's judgment on the first sample label; D(G(z)) represents the result of the discriminator's judgment on G(z), which represents the probability that the input discriminator data is real data, E x [log D(x)] means calculating the expected value of log D(x) over all data x, E z [log(1-(D(G(x)))] means calculating the expected value of log(1-(D(G(x))) over all data z.

[0032] According to a specific implementation of the embodiment of the present application, the training of the second model based on the second training set includes:

[0033] Initializing parameters of the second model;

[0034] Loading the second training sample from the second training set, inputting it into the second model, and outputting the position information of the key points of the hand;

[0035] According to the output hand key point position information and the second sample label, a model loss value is calculated by a second loss function, and the gradient is back-propagated to update the parameters of the second model;

[0036] Repeat the training until the second model reaches a convergence state or reaches a predetermined number of training rounds, thereby completing the training of the second model.

[0037] According to a specific implementation of the embodiment of the present application, the second loss function is a mean square error loss function, which is expressed as:

[0038]

[0039] Where k represents the number of hand key points output by the model; y i represents the position of the i-th hand key point output by the model, t i represents the position of the i-th key point annotated by the second sample label.

[0040] According to a specific implementation method of an embodiment of the present application, the lightweight convolutional neural network model includes a convolution layer, an activation layer, a pooling layer, a spatial information layer and a fully connected layer. The second training sample is input into the model and then connected to the convolution layer, the convolution kernel size is 5*5, and the convolution step is 2; the convolution layer output is connected to the pooling layer, the maximum pooling method is adopted, and the step is 2; the pooling layer output is connected to four convolution layers, the convolution kernel size is 3*3, and the convolution step is 1; the feature map after the convolution layer output is consistent with the output of the spatial information layer The results are added and fused, where the output of the spatial information layer is a feature map containing the relative positions of contour pixels; the fused output feature map is then connected to four convolutional layers with a convolution kernel size of 3*3 and convolution steps of 2, 1, 2, and 1 respectively; the convolutional layer outputs an N*7*7 feature map, which is then connected to the average pooling layer to be converted into a 1*N one-dimensional vector, and finally the one-dimensional vector is input into the fully connected layer to output a 1*21 one-dimensional vector, in which the 21 values ​​represent the pixel sorting positions of the 21 hand key points in the original image.

[0041] According to a specific implementation of an embodiment of the present application, the edge detection method adopts Sobel algorithm, Laplace algorithm or Canny algorithm.

[0042] Beneficial effects:

[0043] The hand key point detection method based on contour features in the embodiment of the present application combines the traditional feature extraction method with the image generation technology, can adapt to the contour feature extraction in complex environments, and improves the accuracy of contour feature extraction; and designs a lightweight neural network, which integrates the contour spatial position information and maps and infers the position of the hand key points through contour features. The network has the advantages of fast training convergence and small reasoning calculation amount, and can adapt to the detection of hand key points in special situations such as wearing gloves, thereby improving the adaptability and accuracy of hand key point detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of a method for detecting hand key points based on contour features according to an embodiment of the present invention;

[0046] Figure 2 A hand key point labeling diagram according to an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of hand contour generation based on a GAN network according to an embodiment of the present invention;

[0048] Figure 4 4 is a structural diagram of a hand key point detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0050] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0051] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0052] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0053] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0054] The present application embodiment provides a method for detecting key points of a hand based on contour features. Figures 1 to 4 Describe in detail.

[0055] In this embodiment, the hand key point detection method based on contour features specifically includes the following steps:

[0056] Obtain a sample set of hand images;

[0057] Preprocessing the hand image sample set to obtain a first training set and a second training set respectively;

[0058] Training a first model based on the first training set to obtain a hand contour feature detection model;

[0059] Training the second model based on the second training set to obtain a hand key point detection model;

[0060] Using the hand contour feature detection model to detect the hand image to be detected, and obtaining an accurate hand contour feature map;

[0061] The hand key point detection model is used to detect the precise hand contour feature map to obtain the hand key points of the hand image to be detected.

[0062] In specific implementation, hand images are collected by a camera to form a sample set. The sample set contains hand images collected under different conditions such as different brightness, wearing gloves and bare hands. A rich sample set can enhance the training of the hand contour generation network and adapt to the contour feature generation in different environments. Since the efficiency of contour feature generation training on the original image collected by the camera is low, the original image needs to be preprocessed. Traditional Sobel, Laplace, Canny and other edge detection methods can be used to perform edge detection on the image to generate a binary rough contour feature map as a training sample, and then the hand contour in the corresponding original image is accurately manually annotated and saved as a binary precise contour feature map as a sample label to form a training set for GAN network learning; the binary precise contour feature map is converted into a spatial feature map, and then the 21 key point position coordinates are manually standardized on the corresponding original image. The 21 key point positions of the hand are referenced. Figure 2 The two-dimensional coordinate values ​​of the key point positions are converted into pixel sorting values ​​in the original image. The spatial feature map and the key point pixel sorting values ​​constitute the training set of the hand key point detection model.

[0063] In one embodiment, the preprocessing of the hand image sample set to obtain a first training set and a second training set respectively includes:

[0064] Performing edge detection on the hand image sample set based on an edge detection method to generate a binary rough contour feature map as a first training sample;

[0065] Manually annotating the hand contour in the original image of the hand image sample accurately to obtain a binary accurate contour feature map, and using it as a first sample label;

[0066] The first training sample and the first sample label constitute the first training set;

[0067] Converting the binarized precise contour feature map into a spatial feature map and using it as a second training sample;

[0068] Manually marking the position coordinates of the key points of the hand in the original image of the hand image sample, and converting the position coordinates of the key points of the hand into pixel ranking values ​​in the original image, wherein the pixel ranking values ​​are used as the second sample labels;

[0069] The second training samples and the second sample labels constitute the second training set.

[0070] Specifically, the first model is set to a GAN network model, and the second model is set to a lightweight convolutional neural network model.

[0071] Furthermore, the training of the first model based on the first training set includes:

[0072] The GAN network model includes a generator and a discriminator, and the parameters of the generator and the discriminator are initialized;

[0073] Inputting the first training sample into the generator to generate a new feature map;

[0074] Inputting the first sample label and the new feature map into the discriminator in turn to obtain a discrimination result of the discriminator;

[0075] According to the discrimination result and based on the first loss function, the parameters of the generator and the discriminator are iteratively optimized to complete the training of the GAN network model.

[0076] Furthermore, the iterative optimization of the parameters of the generator and the discriminator based on the first loss function according to the discrimination result to complete the training of the GAN network model includes:

[0077] According to the discrimination result, the generator loss value is calculated based on the first loss function, the gradient is back-propagated and the parameters of the generator are updated to improve the generation ability of the generator;

[0078] According to the discrimination result, a discriminator loss value is calculated based on the first data loss function, a gradient is back-propagated and the parameters of the discriminator are updated to improve the discrimination ability of the discriminator;

[0079] The training is repeated until the generator and the discriminator reach a convergence state or a predetermined number of training rounds are reached, thereby completing the training of the GAN network model.

[0080] Furthermore, the expression of the first loss function is:

[0081]

[0082] Where G is the generator, D is the discriminator, x is the first sample label input to the discriminator, z is the first training sample input to the generator, G(z) represents the data generated based on z in the generator, D(x) represents the result of the discriminator's judgment on the first sample label; D(G(z)) represents the result of the discriminator's judgment on G(z), which represents the probability that the input discriminator data is real data, E x [log D(x)] means calculating the expected value of log D(x) over all data x, E z [log(1-(D(G(x)))] means calculating the expected value of log(1-(D(G(x))) over all data z.

[0083] In the specific implementation, the adversarial generative network model GAN ​​is used to generate accurate hand contour features. The GAN model is mainly composed of two neural network modules: the generator and the discriminator. The generator and the discriminator are composed of multiple layers of convolutional layers and batch normalization layers. A set of data is input to the generator model to generate a new set of data; the discriminator input is the data generated by the generator or the real label data, and finally a discrimination score is output to indicate the authenticity of the input data. The generator and the discriminator are continuously trained with data to make the generator's generation ability stronger and make the discriminator more accurate. The discriminative ability of the discriminator is getting stronger and stronger. Through the competition between the generator and the discriminator, the data generated by the generator is made consistent with the label data. In this paper, the binarized rough contour feature map in the GAN network training set is used as a training sample to input the generator and generate a new feature map. The manually labeled binarized precise contour feature map in the training set and the new feature map generated by the generator are input into the discriminator in turn. After multiple rounds of training, the parameters of the generator and the discriminator are continuously iterated to make the contour feature map generated by the generator consistent with the manually labeled contour feature map. After the training is completed, the hand image is input into the generator to extract the complete hand contour feature.

[0084] In one embodiment, the training of the second model based on the second training set includes:

[0085] Initializing parameters of the second model;

[0086] Loading the second training sample from the second training set, inputting it into the second model, and outputting the position information of the key points of the hand;

[0087] According to the output hand key point position information and the second sample label, a model loss value is calculated by a second loss function, and the gradient is back-propagated to update the parameters of the second model;

[0088] Repeat the training until the second model reaches a convergence state or reaches a predetermined number of training rounds, thereby completing the training of the second model.

[0089] Furthermore, the second loss function is a mean square error loss function, which is expressed as:

[0090]

[0091] Where k represents the number of hand key points output by the model; y i represents the position of the i-th hand key point output by the model, t i represents the position of the i-th key point annotated by the second sample label.

[0092] Furthermore, the lightweight convolutional neural network model includes a convolution layer, an activation layer, a pooling layer, a spatial information layer and a fully connected layer. After the second training sample is input into the model, it is connected to the convolution layer, the convolution kernel size is 5*5, and the convolution step size is 2; after the output of the convolution layer, it is connected to the pooling layer, using the maximum pooling method with a step size of 2; after the output of the pooling layer, it is connected to four convolution layers, the convolution kernel size is 3*3, and the convolution step size is 1; the feature map after the output of the convolution layer is added and fused with the output result of the spatial information layer, wherein the output of the spatial information layer is a feature map containing the relative positions of contour pixels; the feature map of the fused output is then connected to four convolution layers, the convolution kernel size is 3*3, and the convolution step sizes are 2, 1, 2, and 1 respectively; the convolution layer outputs an N*7*7 feature map, which is then connected to the average pooling layer to be converted into a 1*N one-dimensional vector, and finally the one-dimensional vector is input into the fully connected layer, and a 1*21 one-dimensional vector is output, wherein the 21 numerical values ​​in the one-dimensional vector represent the pixel sorting positions of the 21 hand key points in the original image.

[0093] In this embodiment, a lightweight convolutional neural network model is designed to detect hand key points. Since the hand contour features are strongly correlated with the positions of the hand key points, a lightweight neural network model can be used to satisfy the reasoning of the hand contour features to the positions of the hand key points. The core part of the model is composed of a convolutional layer, an activation layer, a pooling layer, a spatial information layer, and a fully connected layer, wherein the activation layer uses the Relu activation function. The model input data is a single-channel information map containing the position of contour pixels, with a resolution of 112*112. After the input, it is connected to the convolution layer, with a convolution kernel size of 5*5 and a convolution step size of 2; after the output of the convolution layer, it is connected to the pooling layer, using the maximum pooling method with a step size of 2; after the output of the pooling layer, it is connected to four convolution layers, with a convolution kernel size of 3*3 and a convolution step size of 1; the feature map after the output of the convolution layer is added and fused with the output result of the spatial information layer, where the output of the spatial information layer is a feature map containing the relative position of contour pixels; the fused output feature map is then connected to four convolution layers, with a convolution kernel size of 3*3 and convolution step sizes of (2, 1, 2, 1) respectively; the convolution layer outputs an N*7*7 feature map, which is then connected to the average pooling layer to be converted into a 1*N one-dimensional vector, and finally the one-dimensional vector is input into the fully connected layer to output a 1*21 one-dimensional vector, in which the 21 values ​​represent the pixel sorting positions of the 21 hand key points in the original image, which can be converted into two-dimensional coordinates through the original image resolution information. The specific network model structure is as follows Figure 4 shown.

[0094] In one embodiment, the edge detection method uses a Sobel algorithm, a Laplace algorithm or a Canny algorithm.

[0095] The following is a specific example to describe the hand key point detection method based on contour features of the present application. Figures 1 to 4 , including the following steps:

[0096] Step 1: Collect hand image samples of gloved hands and bare hands under different lighting and background environments, perform edge detection on all images based on the Canny operator method, and generate a binary rough contour feature map as a training sample, where the contour pixel value is 255 and the other pixel values ​​are 0; then manually annotate the hand contour of the original image corresponding to each sample and save it as a binary precise contour feature map as a sample label; the generated training samples and sample labels constitute the training set for GAN network learning.

[0097] Step 2: Take the original image corresponding to the precise contour feature map in step 1 to mark the positions of 21 key points of the hand, and convert the two-dimensional position coordinate values ​​into pixel ranking values ​​in the original image. Assuming the two-dimensional coordinate value of the key point is (x, y), and the resolution of the original image is (w, h), the pixel ranking value s of the key point in the original image is calculated as follows:

[0098] s=y*w+x.

[0099] Then convert the binary precise contour feature map into a spatial feature map, and replace the pixel value of 255 with the pixel ranking value of the pixel in the feature map. The converted feature map contains the spatial position information of the contour. Suppose the precise contour feature map is represented as P(x), then the spatial feature map K(x) is calculated as follows:

[0100]

[0101] In the formula, x represents the ranking value of the pixel, K(x) represents the pixel value of the pixel, w and h represent the width and height of the feature map respectively. The spatial feature map K(x) is used as the training sample, and the key point pixel ranking value s is used as the training label to form the training set of the hand key point detection model.

[0102] Step 3: Train the generator and discriminator in the GAN network model. The loss function used in GAN network training is as follows:

[0103]

[0104] Among them, x represents the training label of the input discriminator, z represents the training sample of the input generator, G(z) represents the data generated in the generator based on z, and D(x) represents the result of the discriminator's judgment on the real training label, and D(G(z)) represents the result of the discriminator's judgment on G(z), which represents the probability that the input discriminator data is real data. During the training process, the generator G is first fixed, and the loss is maximized from the perspective of the discriminator D. Then D is fixed and the loss is minimized from the perspective of the generator G. This allows the discriminator and the generator to achieve adversarial training under the condition of shared loss. The specific training method is as follows:

[0105] (1) Initialize the GAN network generator and discriminator parameters;

[0106] (2) Load the rough contour feature map from the created GAN network learning training set and input it into the generator as a training sample to obtain the generated feature map;

[0107] (3) The generated feature map and the accurate contour feature map in the training set are input to the discriminator respectively;

[0108] (4) Obtain the discriminator’s discrimination result, calculate the generator’s loss value based on the loss function, back-propagate the gradient and update the generator’s parameters to improve the generator’s generation capability;

[0109] (5) According to the discriminant results, the discriminator loss value is calculated according to the loss function, the gradient is back-propagated and the discriminator parameters are updated to improve the discriminant ability;

[0110] (6) Repeat the above steps until the generator and discriminator reach convergence or reach the predetermined number of training rounds.

[0111] Step 4: Train the hand key point detection model, which uses the mean square error loss function as shown below:

[0112]

[0113] Where k represents the number of data output by the network model, that is, the number of key points of the hand; y i represents the position of the i-th key point output by the network model, t i Represents the position of the i-th key point marked by the training label. The specific training method is as follows:

[0114] (1) Initialize the hand key point detection model parameters;

[0115] (2) Load the spatial feature map from the hand key point detection training set and input it into the network model as a training sample. The network model outputs the position information of 21 key points.

[0116] (3) According to the network model output results and training labels, the model loss value is calculated through the loss function, the gradient is back-propagated and the model parameters are updated;

[0117] (4) Repeat the above steps until the network model reaches a convergence state or reaches a predetermined number of training rounds.

[0118] Step 5: After completing the training of the GAN network model and the hand key point detection model, the entire algorithm can be forward inferred. First, take a hand image for canny operator edge detection, generate a binary feature map and input it into the GAN network. The GAN network generator generates a binary accurate hand contour feature map; convert the feature map into a spatial feature map according to the method in step 2, and calculate the center coordinates of the maximum circumscribed rectangular box of the contour composed of non-zero pixels in the spatial feature map. Convert the center coordinates into pixel ranking values ​​in the feature map. Let the spatial feature map be represented by K(x), and the ranking value of the center pixel of the rectangular box be q. Then calculate the feature map I(x) containing relative position information. The calculation method is as follows:

[0119]

[0120] In the formula, x represents the sorting value of the pixel, K(x) represents the pixel value of the pixel at that point, w and h represent the width and height of the feature map respectively; the spatial feature map K(x) is used as the input of the hand key point detection model, and the output of the model spatial information layer is calculated as I(x). Finally, the hand key point detection result is output through forward reasoning of the model. The result is a one-dimensional vector of 1*21. The values ​​represent the pixel sorting positions of the 21 hand key points in the original image, which can be converted into two-dimensional coordinates through the original image resolution information to complete the detection of the hand key points.

[0121] The embodiment provided by the present invention designs a hand contour extraction method with strong adaptability and high accuracy, and proposes a method for inferring the position of hand key points based on hand contour feature information. This method mainly includes three parts: processing of hand image sample set, hand contour generation based on GAN network, and hand key point detection based on convolutional neural network. It combines traditional feature extraction methods with image generation technology, which can adapt to contour feature extraction in complex environments and improve the accuracy of contour feature extraction; and designs a lightweight neural network that integrates contour spatial position information and maps and infers the position of hand key points through contour features. The network has the advantages of fast training convergence and small inference calculation amount, and can adapt to hand key point detection in special cases such as wearing gloves, thereby improving the adaptability and accuracy of hand key point detection.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting key points of a hand based on contour features, characterized in that: The method comprises: Obtain a sample set of hand images; Preprocessing the hand image sample set to obtain a first training set and a second training set respectively; Training a first model based on the first training set to obtain a hand contour feature detection model; Training the second model based on the second training set to obtain a hand key point detection model; Using the hand contour feature detection model to detect the hand image to be detected, and obtaining an accurate hand contour feature map; The hand key point detection model is used to detect the precise hand contour feature map to obtain the hand key points of the hand image to be detected.

2. The hand key point detection method based on contour features according to claim 1 is characterized in that: The preprocessing of the hand image sample set to obtain a first training set and a second training set respectively includes: Performing edge detection on the hand image sample set based on an edge detection method to generate a binary rough contour feature map as a first training sample; Manually annotating the hand contour in the original image of the hand image sample accurately to obtain a binary accurate contour feature map, and using it as a first sample label; The first training sample and the first sample label constitute the first training set; Converting the binarized precise contour feature map into a spatial feature map and using it as a second training sample; Manually marking the position coordinates of the key points of the hand in the original image of the hand image sample, and converting the position coordinates of the key points of the hand into pixel ranking values ​​in the original image, wherein the pixel ranking values ​​are used as the second sample labels; The second training samples and the second sample labels constitute the second training set.

3. The hand key point detection method based on contour features according to claim 2 is characterized in that: The first model is set to a GAN network model, and the second model is set to a lightweight convolutional neural network model.

4. The hand key point detection method based on contour features according to claim 3 is characterized in that: The training of the first model based on the first training set includes: The GAN network model includes a generator and a discriminator, and the parameters of the generator and the discriminator are initialized; Inputting the first training sample into the generator to generate a new feature map; Inputting the first sample label and the new feature map into the discriminator in turn to obtain a discrimination result of the discriminator; According to the discrimination result and based on the first loss function, the parameters of the generator and the discriminator are iteratively optimized to complete the training of the GAN network model.

5. The hand key point detection method based on contour features according to claim 4 is characterized in that: The step of iteratively optimizing the parameters of the generator and the discriminator based on the discrimination result and the first loss function to complete the training of the GAN network model includes: According to the discrimination result, the generator loss value is calculated based on the first loss function, the gradient is back-propagated and the parameters of the generator are updated to improve the generation ability of the generator; According to the discrimination result, a discriminator loss value is calculated based on the first data loss function, a gradient is back-propagated and the parameters of the discriminator are updated to improve the discrimination ability of the discriminator; The training is repeated until the generator and the discriminator reach a convergence state or a predetermined number of training rounds are reached, thereby completing the training of the GAN network model.

6. The hand key point detection method based on contour features according to claim 4 is characterized in that: The expression of the first loss function is: Where G is the generator, D is the discriminator, x is the first sample label input to the discriminator, z is the first training sample input to the generator, G(z) represents the data generated based on z in the generator, D(x) represents the result of the discriminator's judgment on the first sample label; D(G(z)) represents the result of the discriminator's judgment on G(z), which represents the probability that the input discriminator data is real data, E x [log D(x)] means calculating the expected value of log D(x) over all data x, E z [log(1-(D(G(x)))] means calculating the expected value of log(1-(D(G(x))) over all data z.

7. The hand key point detection method based on contour features according to claim 3 is characterized in that: The training of the second model based on the second training set includes: Initializing parameters of the second model; Loading the second training sample from the second training set, inputting it into the second model, and outputting the position information of the key points of the hand; According to the output hand key point position information and the second sample label, a model loss value is calculated by a second loss function, and the gradient is back-propagated to update the parameters of the second model; Repeat the training until the second model reaches a convergence state or reaches a predetermined number of training rounds, thereby completing the training of the second model.

8. The hand key point detection method based on contour features according to claim 7 is characterized in that: The second loss function is the mean square error loss function, which is expressed as: Where k represents the number of hand key points output by the model; y i represents the position of the i-th hand key point output by the model, t i represents the position of the i-th key point annotated by the second sample label.

9. The hand key point detection method based on contour features according to claim 3 is characterized in that: The lightweight convolutional neural network model includes a convolution layer, an activation layer, a pooling layer, a spatial information layer and a fully connected layer. After the second training sample is input into the model, it is connected to the convolution layer, the convolution kernel size is 5*5, and the convolution step size is 2; after the convolution layer outputs, it is connected to the pooling layer, using the maximum pooling method with a step size of 2; after the pooling layer outputs, it is connected to four convolution layers, the convolution kernel size is 3*3, and the convolution step size is 1; the feature map after the convolution layer output is added and fused with the output result of the spatial information layer, wherein the output of the spatial information layer is a feature map containing the relative positions of contour pixels; the feature map of the fused output is then connected to four convolution layers, the convolution kernel size is 3*3, and the convolution step sizes are 2, 1, 2, and 1 respectively; the convolution layer outputs an N*7*7 feature map, which is then connected to the average pooling layer to be converted into a 1*N one-dimensional vector, and finally the one-dimensional vector is input into the fully connected layer, and a 1*21 one-dimensional vector is output, wherein the 21 numerical values ​​in the one-dimensional vector represent the pixel sorting positions of the 21 hand key points in the original image.

10. The hand key point detection method based on contour features according to claim 2, characterized in that: The edge detection method adopts Sobel algorithm, Laplace algorithm or Canny algorithm.

Images