An intelligent gesture recognition method based on computer vision

Abstract

This invention discloses an intelligent gesture recognition method based on computer vision, comprising the following steps: S1, providing a depth image, wherein the depth image contains a gesture; S2, image preprocessing the depth image to obtain the portion of the depth image containing the gesture; S3, providing a neural network model, using the portion of the depth image containing the gesture as input to the neural network model, and using the neural network model to regress the spatial positions of each joint in the gesture; S4, determining the finger bending state and finger spatial direction of the gesture; S5, providing a gesture type database; and S6, determining the gesture type of the gesture. The advantages of this invention are: providing a systematic solution to the gesture recognition problem, high recognition accuracy, recognition of multiple gesture types, and easy expansion of recognition types.

Description

Technical Field

[0001] The present invention relates to the field of gesture recognition, and in particular to an intelligent gesture recognition method based on computer vision. Background Art

[0002] In recent years, with the development of related technologies such as machine learning and computer vision, human-computer interaction technology has made rapid progress, and new human-computer interaction technologies have gradually become a hot topic of research. As one of the most convenient auxiliary interaction methods in daily life, gestures have the characteristics of intuitiveness, naturalness and flexibility, and have attracted the attention of many researchers.

[0003] Researchers in the field of gesture recognition have proposed many different gesture recognition methods. For example, the traditional method uses manual design to extract image features, and then post-processes the extracted features to determine the gesture category. Specifically, there are methods based on the Hog feature algorithm, the directional histogram algorithm, and the boundary contour feature algorithm. This method has low accuracy in complex environments, and different manual features need to be designed for different tasks. There are also methods based on deep learning, which use neural networks to directly classify gesture images. This algorithm requires a large number of corresponding training images to be collected for different gesture types, and the model needs to be retrained. In addition, the generalization ability is insufficient, resulting in this method being able to only recognize a few fixed gestures and having low recognition accuracy. In view of the shortcomings of these algorithms, it is necessary to design a gesture recognition method with high recognition accuracy, rich gesture types, and easy expansion. Summary of the invention

[0004] The purpose of the present invention is to overcome the problems in the prior art and to provide a novel intelligent gesture recognition method based on computer vision.

[0005] In order to achieve this purpose, the technical solution of the present invention is as follows: an intelligent gesture recognition method based on computer vision, comprising:

[0006] Step S1, providing a depth image, wherein the depth image contains a gesture;

[0007] Step S2, image preprocessing the depth image to obtain a portion of the depth image containing the gesture;

[0008] Step S3, providing a neural network model, taking the portion of the depth image containing the gesture as input of the neural network model, and using the neural network model to regress the spatial position of each joint point in the gesture;

[0009] Step S4, determining the bending state and spatial direction of the fingers of the gesture according to the spatial position of each joint point in the gesture;

[0010] Step S5, providing a gesture type database, wherein the gesture type database contains data on finger bending states and finger spatial directions corresponding to various gesture types; and

[0011] Step S6, determining the gesture type of the gesture according to the bending state of the finger of the gesture, the spatial direction of the finger and the gesture type library.

[0012] As a preferred solution of a computer vision-based intelligent gesture recognition method, in step S1, the gesture is captured by a depth camera, wherein the gesture is the object closest to the depth camera.

[0013] As a preferred solution of an intelligent gesture recognition method based on computer vision, in step S2, a gesture depth value is set, and the portion of the depth image containing the gesture is segmented from the depth image according to the minimum depth value of the depth image and the gesture depth value.

[0014] As a preferred solution of an intelligent gesture recognition method based on computer vision, in step S2, noise points in the portion of the depth image containing the gesture are eliminated.

[0015] As a preferred solution of an intelligent gesture recognition method based on computer vision, in step S2, the portion of the depth image containing the gesture is normalized: the depth value of the centroid of the portion of the depth image containing the gesture is determined; the depth value of each pixel in the portion of the depth image containing the gesture after normalization is the difference between the depth value of each pixel in the portion of the depth image containing the gesture before normalization and the depth value of the centroid.

[0016] As a preferred solution of a computer vision-based intelligent gesture recognition method, in step S2, the portion of the depth image containing the gesture is adjusted to a specified size, preferably using a linear interpolation method.

[0017] As a preferred solution of an intelligent gesture recognition method based on computer vision, in step S3, the backbone network of the neural network model adopts a ResNet50 network model with the top fully connected layer removed, wherein the number of neural network branches is 6, and the corresponding outputs are 4 joints on the palm joints, thumb, index finger, middle finger, ring finger and little finger, respectively. The outputs of each of the neural network branches are merged to obtain the three-dimensional spatial positions of all the joints of the gesture; further, each of the neural network branches is composed of a convolutional layer, a global average pooling layer and a fully connected layer.

[0018] As a preferred solution of the intelligent gesture recognition method based on computer vision, in step S3, the neural network model is pre-trained using the public data set cvpr15 and trained by the Adam optimizer.

[0019] As a preferred solution of the intelligent gesture recognition method based on computer vision, in step S3, the pre-training of the neural network model is:

[0020] Training cycle 1-50: The backbone network ResNet50 loads the parameters pre-trained by the ImageNet dataset, and the parameters are frozen;

[0021] The branch network parameters are randomly initialized, and the loss function is the sum of the Euclidean distances between the predicted and actual positions of all nodes;

[0022] The initial value of the learning rate is 1e -4 ,The decay rate per cycle is 0.96;

[0023] Training cycle 50-80: The backbone network and branch network are trained simultaneously, and the loss function is the same as training cycle 1-50; the initial value of the learning rate is 1e -5 , the decay rate per cycle is 0.96.

[0024] Compared with the prior art, the beneficial effects of the present invention are at least that: a systematic solution is provided for the problem of gesture recognition, the recognition accuracy is high, a plurality of gesture types can be recognized, and the recognition types are easily expandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a system schematic diagram of the present invention.

[0026] Figure 2 Schematic diagram of the hand gesture joint recognition model of the present invention. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below by means of specific embodiments connected with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] See also Figures 1 to 2 ,The figure shows an intelligent gesture recognition method based on computer vision.

[0029] Step 1: Image preprocessing:

[0030] A depth camera is used to obtain a depth image containing gesture depth information.

[0031] The system defaults to the gesture being the object closest to the camera, which is also in line with most application scenarios. By calculating the minimum depth value mindepth of the depth image, the gesture depth is set to 175mm, and the depth values ​​greater than mindepth+175mm are set to 0, thereby segmenting the gesture from the depth image.

[0032] In order to further eliminate the noise information in the image, the gesture area is the largest connected area in the segmented image, and the contour of the segmented image is found. The largest contour is the gesture contour, forming a mask to segment the gesture image more accurately.

[0033] The gesture depth map is further normalized, the centroid of the gesture image is calculated, and the normalized depth value is generated by subtracting the centroid depth value from the depth value of each pixel in the gesture image.

[0034] The gesture image is further resized to 224×224 size by linear interpolation.

[0035] Step 2: Joint point identification

[0036] Step 2-1: Build a neural network model

[0037] The backbone network uses the ResNet50 network model to remove the top fully connected layer. The input is the 224×224 depth image in step 1, and the output is a 7×7×2048 feature map.

[0038] Different neural network branches are designed for the positions of different joints. The number of neural network branches is 6. Each branch consists of a 3×3 convolution layer, a global average pooling layer and two fully connected layers. Among them, the output of the first neural network branch is the palm joint. The second to sixth neural network branches output 4 joints on the thumb, index finger, middle finger, ring finger and little finger respectively. The outputs of each branch are combined to obtain the three-dimensional spatial position of all the joints of the gesture.

[0039] The parameters of the first neural network branch are described as follows:

[0040] Conv1: The convolution kernel size is 3×3, the number of input channels is 2048, and the number of output channels is 2048.

[0041] GlobalAveragePooling1: Output is a 2048-dimensional vector.

[0042] FC1: The number of input neurons is 2048 and the number of output neurons is 128.

[0043] FC1: The number of input neurons is 128 and the number of output neurons is 3.

[0044] The parameters of the second to sixth neural network branches are described as follows:

[0045] Conv1: The convolution kernel size is 3×3, the number of input channels is 2048, and the number of output channels is 2048.

[0046] GlobalAveragePooling1: Output is a 2048-dimensional vector.

[0047] FC1: The number of input neurons is 2048 and the number of output neurons is 128.

[0048] FC1: The number of input neurons is 128 and the number of output neurons is 12.

[0049] Step 2-2: Network model training:

[0050] The network model uses the public dataset cvpr15_MSRAHandGestureDB and uses the Adam optimizer to train the model.

[0051] Training cycle 1 to 50:

[0052] The backbone network ResNet50 loads the parameters of the ImageNet dataset pre-training, and the parameters are frozen

[0053] The branch network parameters are randomly initialized, and the loss function is the sum of the Euclidean distances between the predicted positions and the actual positions of all nodes.

[0054] The initial value of the learning rate is 1e-4, and the decay rate per cycle is 0.96.

[0055] Training cycle 50-80:

[0056] The backbone network and branch networks are trained simultaneously, and the loss function is the same as the training cycle 1-50.

[0057] The initial value of the learning rate is 1e-5, and the decay rate per cycle is 0.96.

[0058] Step 2-3: Joint point prediction:

[0059] Load the neural network model and its parameters trained in step 2-2.

[0060] The 224×224 depth image segmented in step 1 is input into the model, and the model outputs a 63-dimensional vector, which is the spatial position of the gesture joint point.

[0061] Step 3: Determine the finger bending state

[0062] Each finger has 4 joints, and the positions of these four joints can be used to calculate whether the finger is bent. Taking the thumb as an example, the 4 joints are divided into two groups: No. 1 (j1) and No. 2 (j2), No. 3 (j3) and No. 4 (j4), and the vector is obtained according to the coordinates of each point. With vector The angle between the two vectors is then calculated to determine the degree of bending of each finger:

[0063]

[0064] Affected by the joint judgment error, the threshold of cosα is generally set to 0.6. If it is higher than this value, the finger can be judged as not bent, and if it is lower than this value, the finger is considered bent.

[0065] Determine the bending state of each finger and output the finger bending state vector.

[0066] Step 4: Gesture type determination

[0067] Step 4-1, set the mode:

[0068] When the system is in setup mode, the system stores the state vector in step 3 into the gesture library and records the user-defined gesture type.

[0069] Step 4-2, prediction model:

[0070] When the system is in prediction mode, the system compares the state vector in step 3 with the gesture state vector in the gesture type library. The gesture type with equal vectors is the system output, otherwise the system does not output.

[0071] The above only expresses the implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. An intelligent gesture recognition method based on computer vision, characterized in that: Contains: Step S1, providing a depth image, wherein the depth image contains a gesture; Step S2, image preprocessing the depth image to obtain a portion of the depth image containing the gesture; Step S3, providing a neural network model, taking the portion of the depth image containing the gesture as input of the neural network model, and using the neural network model to regress the spatial position of each joint point in the gesture; Step S4, determining the bending state and spatial direction of the fingers of the gesture according to the spatial position of each joint point in the gesture; Step S5, providing a gesture type database, wherein the gesture type database contains data on finger bending states and finger spatial directions corresponding to various gesture types; and Step S6, determining the gesture type of the gesture according to the bending state of the finger of the gesture, the spatial direction of the finger and the gesture type library; In step S3, the backbone network of the neural network model adopts the ResNet50 network model by removing the top fully connected layer, wherein the number of neural network branches is 6, and the corresponding outputs are 4 joints on the palm joints, thumb, index finger, middle finger, ring finger and little finger respectively. The outputs of each of the neural network branches are merged to obtain the three-dimensional spatial positions of all the joints of the gesture; each of the neural network branches is composed of a convolutional layer, a global average pooling layer and a fully connected layer.

2. The method for intelligent gesture recognition based on computer vision according to claim 1, characterized in that: In step S1, the gesture is captured by using a depth camera, wherein the gesture is the object closest to the depth camera.

3. The method for intelligent gesture recognition based on computer vision according to claim 1, characterized in that: In step S2, a gesture depth value is set, and a portion of the depth image containing the gesture is segmented from the depth image according to a minimum depth value of the depth image and the gesture depth value.

4. The method for intelligent gesture recognition based on computer vision according to claim 3, characterized in that: In step S2, noise points in the portion of the depth image containing the gesture are eliminated.

5. The method for intelligent gesture recognition based on computer vision according to claim 3, characterized in that: In step S2, the portion of the depth image containing the gesture is normalized: the depth value of the centroid of the portion of the depth image containing the gesture is determined; the depth value of each pixel in the portion of the depth image containing the gesture after normalization is the difference between the depth value of each pixel in the portion of the depth image containing the gesture before normalization and the depth value of the centroid.

6. The method for intelligent gesture recognition based on computer vision according to claim 3, characterized in that: In step S2, the portion of the depth image containing the gesture is adjusted to a specified size by using a linear interpolation method.

7. The method for intelligent gesture recognition based on computer vision according to claim 1, characterized in that: In step S3, the neural network model is pre-trained using the public dataset cvpr15 and trained using the Adam optimizer.

8. The method for intelligent gesture recognition based on computer vision according to claim 7, characterized in that: In step S3, the neural network model is pre-trained: Training cycle 1-50: The backbone network ResNet50 loads the parameters pre-trained by the ImageNet dataset, and the parameters are frozen; The branch network parameters are randomly initialized, and the loss function is the sum of the Euclidean distances between the predicted and actual positions of all nodes; The initial value of the learning rate is 1e -4 ,The decay rate per cycle is 0.96; Training cycle 50-80: The backbone network and branch network are trained simultaneously, and the loss function is the same as training cycle 1-50; The initial value of the learning rate is 1e -5 , the decay rate per cycle is 0.96.

Images