A training method for a hand-encoding-based gesture recognition algorithm

Abstract

This invention discloses a training method for a gesture recognition algorithm based on hand encoding. A simple hand encoder, or more precisely, a finger encoder, is constructed. Since the palm is not easily deformed, only the fingers participate in the encoding operation (different states of each finger can combine to create various hand movements). Each finger's state is categorized into three types: not extended, partially extended, and fully extended. This encoding method allows the algorithm to focus on the state of each finger without considering the overall hand-to-hand relationships, thus simplifying the task. This encoding method also requires collaboration with hand keypoints, using a dual-branch, multi-label supervised training approach. After training, the keypoint branches are pruned, retaining only the hand encoder branches. Without increasing the amount of data, a lightweight network structure can achieve performance unattainable using the keypoint-only method.

Description

Technical Field

[0001] This invention relates to the field of deep learning machine vision algorithm technology, and in particular to a training method for a gesture recognition algorithm based on hand encoding. Background Technology

[0002] With the development of artificial intelligence technology in recent years, non-contact human-computer interaction methods such as gestures, voice, and body movements have become widely popular. Gestures are one of the more natural and user-friendly methods, and are used in entertainment, education, healthcare, security, and other fields. To achieve more natural and convenient gesture interaction, higher requirements are placed on gesture recognition technology. Depending on whether additional data collection devices are worn, gesture recognition can be further divided into methods based on wearable devices and methods based on deep learning algorithms. This invention applies to the latter.

[0003] Among existing gesture recognition algorithms, two common types exist: The first type first crops the hand area, then performs cropping and feature classification on the cropped hand region. Since the human hand is a non-rigid body, it can make various complex gestures, and many gestures are not significantly different. Especially when the hand is transitioning between actions, the actions in consecutive video frames are continuous and do not have obvious differences. Direct classification is not very effective. This method can only effectively recognize simple actions and highly differentiated actions, thus its versatility is poor. The second type obtains the skeletal keypoints of each key position of the hand through a skeletal keypoint algorithm, and judges different hand actions by the relative position changes between keypoints. This method can adapt to various complex scenes and hand action changes, and its adaptability is stronger than the first method. However, due to the severe occlusion of the hand itself, to accurately identify occluded keypoints, the complexity of the algorithm structure needs to be increased or the amount of data needs to be increased. The former increases the computational cost, and the latter increases the cost of data acquisition and annotation. Summary of the Invention

[0004] This invention's gesture recognition method differs from any previous method. It constructs a simple hand encoder, or more precisely, a finger encoder. Since the palm is not easily deformed, only the fingers participate in the encoding operation (different states of each finger can combine to create various hand movements). It categorizes the state of each finger into three types: not extended, partially extended, and fully extended. This encoding operation allows the algorithm to focus on the state of each finger without considering the overall hand-to-hand relationships, thus simplifying the task.

[0005] This encoding method also requires collaboration with hand keypoints, using a dual-branch, multi-label supervised training approach. After training, the keypoint branches are pruned, retaining only the hand encoder branches. Without increasing the amount of data, a lightweight network structure can achieve performance that the keypoint-only approach cannot.

[0006] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention discloses a training method for a gesture recognition algorithm based on hand encoding, the method comprising the following steps:

[0007] Step 1: In the data preprocessing stage of the gesture recognition algorithm, a binary code number needs to be generated when collecting and labeling hand data;

[0008] Step 2: Training with the hand keypoint algorithm. Supervised training is performed by combining the encoding number with the hand keypoint algorithm. The algorithm is trained using both hand keypoint and hand encoding data labels.

[0009] Furthermore, step 1 further includes: defining the finger state into three states: not extended, half extended, and fully extended, which are represented by three sequence numbers: 100, 010, and 001, respectively; encoding different gestures according to the finger state using coded numbers; and generating a corresponding code number for each collected gesture action using the code corresponding to a specific gesture.

[0010] Furthermore, the "finger not extended" state is between fully extended and not extended, used to represent bent fingers.

[0011] Furthermore, the encoding needs to take into account the visibility of each finger. The visibility value is in the range of 0 to 1. The calculation method is: the total number of pixels visible to the naked eye / the total number of pixels of each finger. After obtaining the visibility value of each finger, it is multiplied by the respective finger code to obtain the final 15-bit encoding label. This takes into account the hand occlusion situation and has stronger universality.

[0012] Furthermore, the hand keypoint algorithm in step 2 adopts a Gaussian heatmap-based approach to preserve the spatial information of hand features. The heatmap-based network construction includes downsampling layers and upsampling layers. After the last downsampling layer, a new branch is created, which outputs the hand encoding. Supervised training is performed using the hand encoding labels.

[0013] Furthermore, the newly created hand encoding branch shares a downsampling layer with the heatmap-based branch, and the parameters adopt a weight-sharing mode. The supervised data labels for the two are different during training. The former is supervised by the encoding number obtained from the hand encoder, while the latter is supervised by the original hand keypoint coordinates in the data.

[0014] Furthermore, the specific construction and training methods for the hand coding branch are as follows:

[0015] Step 201: Stretch the feature map to transform it from a four-dimensional feature map into a two-dimensional feature plane map.

[0016] Step 202: The flattened features are put into a fully connected layer to be transformed into a feature vector of 128 to 1024 dimensions, and then normalized. The higher the dimension, the higher the accuracy, but the slower the speed.

[0017] Step 203: The normalized feature vector is fed into a fully connected layer to obtain a 15-dimensional output that is numerically aligned with the 15-bit code obtained by the hand encoder.

[0018] Step 204: Finally, the outputs are activated by the sigmoid function. The values ​​of each of the three outputs are compared sequentially to obtain the maximum value of each finger code. The mean absolute error between the maximum value and the actual hand code label is calculated and denoted as L1.

[0019] Furthermore, the feature vectors obtained in step 202 are multiplied by matrix to calculate their cosine similarity. This reduces the similarity between vectors of the same gesture and reduces the similarity between vectors of different gestures. The specific calculation formula is as follows:

[0020]

[0021] Where B represents the training batch size, f i Let f represent each feature vector. + Indicates with f i The feature vectors of the same gesture, f - Indicates with f i Feature vectors of different gestures.

[0022] Furthermore, when upsampling the downsampled feature map, the Gaussian heatmap obtained by upsampling to 1.5 to 2 times the size of the original image contains more spatial information of key points. Then, it is decoded according to the hand encoding training method to obtain the key point coordinates, and the mean absolute error is calculated using the real key point labels, denoted as L3.

[0023] Furthermore, for the original keypoint branches, the final loss function formula is:

[0024] L = L1 + 0.01 * L2 + 0.2 * L3

[0025] After training, the original keypoint upsampling branch is completely removed, leaving only the hand encoding branch.

[0026] Compared with existing technologies, the advantages of this invention are as follows: It constructs a simple hand encoder, or more precisely, a finger encoder. Since the palm is not easily deformed, only the fingers participate in the encoding operation (different states of each finger can combine to perform various hand movements). It categorizes the state of each finger into three types: not extended, partially extended, and fully extended. This encoding operation allows the algorithm to focus on the state of each finger without considering the overall hand-to-hand relationships, thus simplifying the task.

[0027] This encoding method also requires collaboration with hand keypoints, using a dual-branch, multi-label supervised training approach. After training, the keypoint branches are pruned, retaining only the hand encoder branches. Without increasing the amount of data, a lightweight network structure can achieve performance that the keypoint-only approach cannot. Attached Figure Description

[0028] The invention will be further understood from the following description taken in conjunction with the accompanying drawings. The components in the drawings are not necessarily drawn to scale, but rather the emphasis is on illustrating the principles of the embodiments. In the drawings, the same reference numerals designate corresponding parts in different views.

[0029] Figure 1 This is a flowchart of a training method for a gesture recognition algorithm based on hand encoding according to the present invention. Detailed Implementation

[0030] Example 1

[0031] This embodiment mainly includes two aspects: a novel hand encoding method and a training method for a gesture recognition algorithm.

[0032] like Figure 1As shown, hand encoding is mainly used in the data preprocessing stage of gesture recognition algorithms. A binary code number needs to be generated during hand data collection and annotation. The specific generation method is as follows: Finger states are defined as three states: not extended, half-extended, and fully extended, represented by the sequence numbers 100, 010, and 001 respectively. With the encoded numbers, we can encode different gestures according to finger states. Taking the "V" sign as an example, its five finger states are: thumb not extended, index finger fully extended, middle finger fully extended, ring finger not extended, and little finger not extended. The binary codes for each finger are 100, 001, 001, 100, and 100 respectively. Concatenating these codes according to the thumb order yields a 15-bit code number like 100001001100100. This number represents both the "V" sign and the state decomposition of each finger. Similarly, a corresponding code number is generated for each collected gesture.

[0033] Furthermore, the "finger not extended" state is between fully extended and not extended, used to represent bent fingers, such as the index finger state in the "nine" gesture, all finger states in the "claw" gesture, and the thumb and index finger states in the "OK" gesture.

[0034] Furthermore, unlike traditional gesture classification data collection tasks, this hand encoder can collect data arbitrarily, even actions that are not used in some algorithm application scenarios. This is because this hand encoder only cares about the state of each finger and judges the overall hand action category by the state of each finger. The more diverse the hand actions collected, the better the effect of the next step of algorithm training.

[0035] Furthermore, the above encoding needs to take into account the visibility of each finger. The visibility value is in the range of 0 to 1, and is calculated as: the total number of pixels visible to the naked eye / the total number of pixels per finger. After obtaining the visibility value of each finger, it is multiplied by its respective finger code to obtain the final 15-bit encoded label. This takes into account hand occlusion and has stronger versatility.

[0036] Next, algorithm training can begin. Training requires the use of hand keypoint algorithms. Supervised training using only hand codes is insufficient; only by combining it with keypoint training methods, using both hand keypoints and hand codes as data labels to train the algorithm, can better results be achieved.

[0037] Furthermore, the hand keypoint algorithm must adopt a Gaussian heatmap-based approach (hereinafter referred to as heatmap-based) to maximize the preservation of spatial information of hand features. The heatmap-based network construction includes downsampling layers and upsampling layers. We create a new branch after the last downsampling layer, which outputs the hand encoding. Supervised training is then performed using the hand encoding labels.

[0038] It should be noted that the newly created hand encoding branch shares the downsampling layer with the heatmap-based branch, and the parameters adopt a weight-sharing mode. The supervised data labels for the two are different during training. The former is supervised by the encoding number obtained from the hand encoder, while the latter is supervised by the original hand keypoint coordinates in the data.

[0039] The specific construction and training methods for the hand coding branch are as follows:

[0040] The feature map is stretched from the original four-dimensional feature map into a two-dimensional feature plane map.

[0041] The flattened features are put into a fully connected layer and transformed into a feature vector of 128 to 1024 dimensions (the higher the dimension, the higher the accuracy, but the slower the speed; 256 dimensions are recommended), and then normalized.

[0042] The normalized feature vectors are fed into a fully connected layer to obtain a 15-dimensional output (aligned in number with the 15-bit code obtained by the hand encoder).

[0043] Finally, the outputs are activated by the sigmoid function. The values ​​of each of the three outputs are compared sequentially to obtain the maximum value of each finger's code. The mean absolute error between this value and the actual hand code label is calculated and denoted as L1.

[0044] The feature vectors obtained in the second step are used to calculate the cosine similarity between them through matrix multiplication. This reduces the similarity between vectors of the same gesture and reduces the similarity between vectors of different gestures. The specific calculation formula is as follows:

[0045]

[0046] Where B represents the training batch size, f i Let f represent each feature vector. + Indicates with f i The feature vectors of the same gesture, f - Indicates with f i Feature vectors of different gestures.

[0047] The above steps calculate the loss function for the hand-coded branch. For the original keypoint branch, when upsampling the downsampled feature map, it is recommended to upsample it to 1.5 to 2 times the size of the original image. This will result in a Gaussian heatmap containing more spatial information about the keypoints. The keypoint coordinates are then decoded using the original method, and the mean absolute error (MAE) is calculated using the actual keypoint labels, denoted as L3.

[0048] The final loss function formula is:

[0049] L = L1 + 0.01 * L2 + 0.2 * L3

[0050] After training, the original keypoint upsampling branch is completely removed, leaving only the hand encoding branch.

[0051] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0052] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] While the invention has been described above with reference to various embodiments, it should be understood that many changes and modifications can be made without departing from the scope of the invention. Therefore, the detailed description above is intended to be illustrative rather than restrictive, and it should be understood that the following claims (including all equivalents) are intended to define the spirit and scope of the invention. These embodiments should be understood as illustrative only and not as limiting the scope of protection of the invention. After reading the description of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent changes and modifications also fall within the scope defined by the claims of this invention.

Claims

1. A training method of a gesture recognition algorithm based on hand coding, characterized in that, The method comprises the following steps: Step 1: In the data preprocessing stage of the gesture recognition algorithm, a binary code number needs to be generated when collecting and labeling hand data; Step 2: Training by means of a hand key point algorithm, supervised training is performed by combining the code number with the hand key point algorithm, and the algorithm is trained by using two kinds of data labels of hand key points and hand codes, wherein the hand key point algorithm in step 2 adopts a heatmap-based mode to retain the spatial information of the hand features, the heatmap-based network construction comprises a down-sampling layer and an up-sampling layer, a branch is newly built after the last down-sampling layer, the branch outputs a hand code, and supervised training is performed by using a hand code label, the newly built hand code branch shares the down-sampling layer with the heatmap-based, the parameters adopt a weight sharing mode, and the data labels supervised in training are different, the former is supervised and trained by using the code number obtained by the hand code, and the latter is supervised and trained by using the original hand key point coordinates of the data; The step 2 further comprises the following specific construction and training method of the hand code branch: Step 201: Stretching operation is performed on the feature map to stretch the original four-dimensional feature map into a two-dimensional feature plane map; Step 202: The flattened feature is placed into a fully connected layer to become a 128-1024-dimensional feature vector, and normalization is performed, wherein the higher the dimension, the higher the accuracy, and the slower the speed; Step 203: The normalized feature vector is placed into a fully connected layer to obtain a 15-dimensional output which is aligned in quantity with the 15-bit code obtained by the hand code; Step 204: Finally, the sigmoid activation function is used to sequentially compare the size of each three outputs to obtain the maximum value of each finger code, and the average absolute error is calculated with the real hand code label, which is recorded as L1; Wherein, the cosine similarity between the feature vectors obtained in step 202 is calculated by matrix multiplication to narrow the similarity between the vectors of the same gesture and to widen the similarity between the vectors of different gestures, and the specific calculation formula is as follows: where B denotes the training batch size, f i represents each feature vector, f + represents feature vectors of the same gesture as f i represents feature vectors of the same gesture as f - represents feature vectors of a different gesture than f i represents feature vectors of a different gesture than f 2. The method of claim 1, wherein the hand-based gesture recognition algorithm is trained based on a plurality of hand gestures. The step 1 further comprises: defining the finger state into three states of not stretched, half stretched and fully stretched, and using three sequence numbers 100, 010 and 001 to represent them respectively; different gestures are encoded according to the finger state by using the code number, and each gesture action collected is generated a corresponding code number by using the code corresponding to a specific gesture.

3. The method of claim 2, wherein the hand-based gesture recognition algorithm is trained based on a plurality of hand gestures, each of the plurality of hand gestures being associated with a respective one of a plurality of hand gestures. The finger not stretched state is between fully stretched and not stretched, and is used to represent the curved finger.

4. The method of claim 3, wherein the hand-based gesture recognition algorithm is trained based on a plurality of hand gestures. The encoding needs to consider the visible state of each finger, and the visible value is in the range of 0-1, and the calculation method is: total number of pixels visible to the naked eye / total number of pixels of each finger, after obtaining the visible value of each finger, the final 15-bit code label is obtained by multiplying the visible value with the finger code of each finger, which considers the hand occlusion and has stronger general ability.

5. The method of claim 4, wherein the hand-based gesture recognition algorithm is trained based on a plurality of hand gestures, each of the plurality of hand gestures being associated with a respective one of a plurality of hand gestures. When upsampling the down-sampled feature map, the Gaussian heat map obtained by upsampling to 1.5-2 times the size of the original map contains more key point spatial information, and then it is decoded according to the hand encoding training mode to obtain the key point coordinates, and the mean absolute error is calculated with the real key point label, denoted as L3.

6. The method of claim 5, wherein the hand-based gesture recognition algorithm is trained based on a plurality of hand gestures, each of the plurality of hand gestures being associated with a respective one of a plurality of hand gestures. For the original key point branch, the final loss function formula is: After training, the original key point upsampling branch is entirely cut off, and only the hand encoding branch is retained.

Images