Gesture recognition, gesture interaction method and device

Abstract

The present invention provides a method and apparatus for gesture recognition and gesture interaction. The gesture recognition method includes: determining hand key point data to be recognized; inputting the hand key point data to be recognized into a gesture recognition model to obtain a target gesture category corresponding to the hand key point data to be recognized; wherein the gesture recognition model is trained based on sample hand key point data and gesture category labels corresponding to the sample hand key point data, wherein the sample hand key point data is obtained by normalizing the initial sample hand key point data. The present invention can achieve accurate and rapid gesture recognition, as well as multi-functional human-computer interaction, with high stability and robustness.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a gesture recognition and gesture interaction method and device. Background Art

[0002] Glasses-free 3D display technology can reproduce three-dimensional images with spatial depth information through stereoscopic display devices. Viewers can observe the stereoscopic images with their naked eyes without any auxiliary equipment, greatly reducing the difficulty of observation for users of 3D displays. Therefore, glasses-free 3D display technology is considered a key development direction in the future of the display industry. When interacting with glasses-free 3D displays, traditional interaction methods such as keyboards and mice are unable to meet users' needs for information exchange and a more novel experience. New interaction methods such as voice interaction, eye tracking, and gesture interaction are becoming increasingly popular. Because gestures can convey a large amount of information between people and achieve high-speed communication and interaction, gesture interaction is a key interaction technology for 3D light fields, and gesture recognition is a key research topic for experts and scholars in the field of human-computer interaction.

[0003] Traditional gesture recognition methods often use RGBD image sequences to classify gestures through deep learning. However, this method has problems such as long image processing time and low image clarity. As a result, traditional methods cannot guarantee the recognition accuracy of complex gestures, which has a negative impact on subsequent naked-eye 3D light field human-computer interaction based on gesture recognition results. Summary of the Invention

[0004] The present invention provides a gesture recognition and gesture interaction method and device to address the defects of the existing technology such as low accuracy in complex gesture recognition and difficulty in human-computer interaction, achieve accurate and fast gesture recognition, and can also realize human-computer interaction with multiple functions, with high stability and robustness.

[0005] The present invention provides a gesture recognition method, comprising:

[0006] Determine the key point data of the hand to be identified;

[0007] Inputting the hand key point data to be identified into a gesture recognition model to obtain a target gesture category corresponding to the hand key point data to be identified;

[0008] The gesture recognition model is trained based on sample hand key point data and gesture category labels corresponding to the sample hand key point data. The sample hand key point data is obtained after posture normalization of the initial sample hand key point data.

[0009] According to a gesture recognition method provided by the present invention, determining the sample hand key point data includes:

[0010] Determine the target coordinate system;

[0011] The initial sample hand key point data is normalized based on the target coordinate system to obtain the sample hand key point data.

[0012] According to a gesture recognition method provided by the present invention, the pose normalization of the initial sample hand key point data based on the target coordinate system to obtain the sample hand key point data includes:

[0013] Normalizing the pose of the initial sample hand key point data based on the target coordinate system to obtain normalized hand key point data;

[0014] The intra-class data difference of the normalized hand key point data of each gesture category is obtained, and the normalized hand key point data with the intra-class data difference being less than or equal to a first threshold is determined as the sample hand key point data.

[0015] According to a gesture recognition method provided by the present invention, the pose normalization of the initial sample hand key point data based on the target coordinate system to obtain normalized hand key point data includes:

[0016] Obtaining the palm key point, palm direction vector, and finger direction vector of the initial sample hand key point data;

[0017] The palm key point is displaced to the origin of the target coordinate system, and the palm direction vector and the finger direction vector are rotated to the direction of the coordinate axis of the target coordinate system to obtain the normalized hand key point data.

[0018] According to a gesture recognition method provided by the present invention, the gesture recognition model is trained in the following manner:

[0019] Identify the multilayer perceptron neural network;

[0020] Inputting the sample hand key point data into the multilayer perceptron neural network to obtain a predicted gesture category label corresponding to the sample hand key point data;

[0021] According to the gesture category label and the predicted gesture category label, the model parameters of the multilayer perceptron neural network are updated to train and obtain the gesture recognition model.

[0022] The present invention also provides a gesture interaction method, comprising:

[0023] Acquire key point data of the hand to be identified, and determine the target gesture category corresponding to the key point data of the hand to be identified based on the gesture recognition method;

[0024] Based on the predefined correspondence between gesture categories and human-computer interaction functions, a target human-computer interaction function corresponding to the target gesture category is determined.

[0025] The present invention also provides a gesture recognition device, comprising:

[0026] The acquisition module is used to determine the key point data of the hand to be identified;

[0027] A recognition module, configured to input the hand key point data to be recognized into a gesture recognition model to obtain a target gesture category corresponding to the hand key point data to be recognized;

[0028] The gesture recognition model is trained based on sample hand key point data and gesture category labels corresponding to the sample hand key point data. The sample hand key point data is obtained after posture normalization of the initial sample hand key point data.

[0029] According to a gesture recognition device provided by the present invention, the acquisition module adopts a somatosensory controller, and the hand key point data to be recognized includes the three-dimensional coordinates of the hand key points and the instantaneous movement direction vector of the hand.

[0030] The present invention also provides a gesture interaction device, comprising:

[0031] A gesture recognition module, configured to obtain key point data of a hand to be recognized, and determine a target gesture category corresponding to the key point data of the hand to be recognized based on the gesture recognition method;

[0032] The gesture interaction module is used to determine the target human-computer interaction function corresponding to the target gesture category based on the predefined correspondence between gesture categories and human-computer interaction functions.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-described gesture recognition methods or gesture interaction methods is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements any of the above-mentioned gesture recognition methods or gesture interaction methods.

[0035] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned gesture recognition methods or gesture interaction methods.

[0036] The gesture recognition and gesture interaction methods and devices provided by the present invention improve the accuracy and robustness of gesture data and gesture classification and recognition by collecting hand key point data instead of traditional images. Furthermore, because the gesture recognition model is trained using sample hand key point data and the gesture category labels corresponding to the sample hand key point data, the gesture recognition model has excellent recognition effects on complex gesture data. Furthermore, the sample hand key point data used in training is obtained by normalizing the initial sample hand key point data, effectively improving the accuracy and robustness of gesture classification and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 is a flowchart of a gesture recognition method provided by an embodiment of the present invention;

[0039] Figure 2 Schematic diagram of the principle of deep learning of MLP neural network provided by an embodiment of the present invention;

[0040] Figure 3 is a structural diagram of a gesture recognition device provided by an embodiment of the present invention;

[0041] Figure 4 is a flowchart of a gesture interaction method provided by an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of light field display and human-computer interaction provided by an embodiment of the present invention;

[0043] Figure 6 is a structural diagram of a gesture recognition device provided by an embodiment of the present invention;

[0044] Figure 7 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0046] The following combination Figure 1-Figure 2 Describe the gesture recognition method of the present invention, as Figure 1 As shown, an embodiment of the present invention discloses a gesture recognition method, which includes at least the following steps:

[0047] Step 101: Determine the key point data of the hand to be identified;

[0048] Step 102: Input the key point data of the hand to be identified into the gesture recognition model to obtain the target gesture category corresponding to the key point data of the hand to be identified;

[0049] Among them, the gesture recognition model is trained based on the sample hand key point data and the gesture category labels corresponding to the sample hand key point data. The sample hand key point data is obtained after the initial sample hand key point data is normalized in terms of posture.

[0050] It should be noted that the data of the key points of the hand to be identified is collected through the somatosensory controller (Leap Motion). The somatosensory controller can collect the three-dimensional coordinates of the key points of the human hand skeleton within the effective identifiable range and the instantaneous motion direction of the hand when the acquisition device samples. The instantaneous motion direction specifically refers to the instantaneous motion direction vector of the hand. The somatosensory controller can track the key point data of the 10 fingers of both hands. Specifically, the three-dimensional coordinates of the key points of the skeleton of each hand include the center point of the wrist, the proximal phalanx, middle phalanx, and distal phalanx end of the thumb, the metacarpal bone, proximal phalanx, middle phalanx, and distal phalanx end of the index finger, the metacarpal bone, proximal phalanx, middle phalanx, and distal phalanx end of the middle finger, the metacarpal bone, proximal phalanx, middle phalanx, and distal phalanx end of the ring finger, the metacarpal bone, proximal phalanx, middle phalanx, and distal phalanx end of the little finger, etc. The three-dimensional coordinates of the key points. As an example, the instantaneous motion direction vector of the hand can include the direction vectors of each hand with the palm facing down and the palm of the palm to the base of the middle finger.

[0051] In the initial sample hand key point data, due to differences in gesture collection positions and initial rotation angles, data for the same gesture category may vary greatly. If such data is directly fed into a neural network for deep learning gesture recognition, insufficient feature learning for a particular gesture will result, and accurate deep learning gesture recognition results cannot be obtained. However, the gesture recognition model of the embodiment of the present invention uses a traditional neural network model as the base model and is trained based on a large amount of sample hand key point data. Since the sample key point data is obtained after posture normalization of the initial sample hand key point data, the data used to train the model is valid input after posture normalization, which greatly reduces the difference in data under each gesture category label and effectively improves the reliability of the input training data and the model recognition accuracy.

[0052] Because the data collected by this invention is hand keypoint data, the gesture pose corresponding to the currently collected data can be determined based on the positions of all keypoints. In this embodiment of the present invention, pose normalization refers to unifying the poses corresponding to the initial sample hand keypoint data into the same coordinate system, reducing gesture differences caused by acquisition discrepancies. After inputting the hand keypoint data to be identified into a trained gesture recognition model, the detected target gesture type is output. This target gesture type can then be applied to the light field for light field human-computer interaction.

[0053] It should be noted that this embodiment of the present invention defines gestures as various positions and movements of the human hand. The gesture recognition model primarily recognizes the positions corresponding to static gestures. Gesture categories can include common human-computer interaction gestures, such as making a fist and spreading five fingers. They can also be user-defined gestures to adapt to complex human-computer interaction scenarios, such as a C-shaped gesture or extending a single finger.

[0054] Compared to traditional methods that rely on collecting gesture images for image recognition to determine gesture categories, the gesture recognition method of the present invention collects hand key point data to obtain three-dimensional information about each gesture key point. This eliminates the need for a large number of images as a training dataset, resulting in a smaller amount of gesture key point data, which can effectively speed up model training. Furthermore, the present invention offers improved recognition accuracy for complex gestures and can be widely applied in subsequent human-computer interaction scenarios.

[0055] In some embodiments, determining sample hand key point data includes:

[0056] Determine the target coordinate system;

[0057] The initial sample hand key point data is normalized based on the target coordinate system to obtain the sample hand key point data.

[0058] It should be noted that the initial sample hand key point data can be obtained by calling an existing database, or by manually collecting the initial sample hand key point data and customizing the corresponding different gesture category labels. When the initial sample hand key point data is collected, a set of initial sample hand key points corresponds to one gesture, and the target coordinate system is determined by the initial sample hand key points corresponding to any collected gesture, for example, it can be the coordinate system where the initial sample hand key points corresponding to the first collected gesture are located. Pose normalization based on the target coordinate system means unifying the initial sample hand key point data corresponding to each gesture into the target coordinate system, which can avoid errors caused by differences in the collection space position between each gesture.

[0059] Specifically, the target coordinate system uses the key point of the palm position among the initial sample hand key points in the group as the coordinate origin. Since the palm direction vector and the finger base direction vector on each hand maintain a vertical relationship, the palm direction vector is a direction vector perpendicular to the palm plane. Therefore, the directions of the palm direction vector and the finger base direction vector are used as the two coordinate axes of the target coordinate system to obtain the target coordinate system.

[0060] The gesture recognition method of the embodiment of the present invention can normalize the posture by setting a target coordinate system and unifying the initial sample hand key point data of each gesture collected into the same coordinate system, thereby reducing the intra-group fluctuation of the training data and improving the model recognition accuracy.

[0061] In some embodiments, the initial sample hand key point data is normalized based on the target coordinate system to obtain the sample hand key point data, including:

[0062] Normalize the pose of the initial sample hand key point data based on the target coordinate system to obtain normalized hand key point data;

[0063] The intra-class data difference of the normalized hand key point data of each gesture category is obtained, and the normalized hand key point data with the intra-class data difference being less than or equal to a first threshold is determined as the sample hand key point data.

[0064] It should be noted that although the normalized hand keypoint data has been normalized, it may still contain data due to jitter during acquisition or distortion of the captured gestures. To prevent this data from affecting the model training results, this embodiment of the present invention sets a first threshold to filter out data that does not meet the normalization requirements before acquiring the sample data required for training. The first threshold is the maximum difference between each gesture category.

[0065] As an example, for any gesture, the coordinate difference of the key points at the same position between each two groups of data in the multiple groups of normalized hand key point data corresponding to the gesture can be calculated, and the coordinate difference of the key points at the same position between the multiple groups of normalized hand key point data corresponding to the gesture and the standard key point data of the gesture can also be calculated. The average value of the coordinate difference of all key points in each group of data is taken, and the average value of the coordinate difference is used as the intra-class data difference of the gesture category and compared with the first threshold. If the intra-class data difference of the gesture category is less than or equal to the first threshold, all groups of normalized hand key point data of the gesture category are determined as sample hand key point data of the gesture category. The normalized hand key point data of all gesture categories are filtered using the first threshold in turn, and finally the sample hand key point data is obtained.

[0066] Specifically, the first threshold may be set to 5 mm to 1 cm.

[0067] The gesture recognition method of an embodiment of the present invention further filters the normalized hand key point data by setting a first threshold. The resulting sample hand key point data has smaller data fluctuations within each gesture category and can effectively improve the accuracy of classification after subsequent model training.

[0068] In some embodiments, the initial sample hand key point data is normalized based on the target coordinate system to obtain normalized hand key point data, including:

[0069] Obtain the palm key points, palm direction vectors, and finger direction vectors of the initial sample hand key point data;

[0070] The palm key point is displaced to the origin of the target coordinate system, and the palm direction vector and the finger direction vector are rotated to the direction of the coordinate axis of the target coordinate system to obtain the normalized hand key point data.

[0071] It should be noted that, since the initial sample hand key point data contains the three-dimensional coordinate information of all key points, as well as the palm direction vector and the finger direction vector, wherein the palm direction vector includes the downward direction vector perpendicular to the palm plane at the palm key point, and the finger direction vector includes the direction vector from the palm key point to the key point of the base of any finger in the same palm. Generally, the palm direction vector and the finger direction vector are perpendicular to each other. The embodiment of the present invention normalizes the collected initial sample hand key point data through displacement transformation and rotation transformation. Since the collected key point skeleton data are all based on the same right-handed Cartesian coordinate system, the data of the same gesture are constrained by performing posture transformation on the gestures in space, thereby achieving the effect of preprocessing the hand key point skeleton data.

[0072] Specifically, all key points corresponding to each gesture are first subjected to coordinate displacement. After the displacement, all gestures in the space are near the origin of the target coordinate system (XYZ), and the coincidence of the palm key point and the origin of the coordinate system is used as a sign of the end of the displacement. The position adjustment of the three-dimensional coordinate point is completed, but because the posture has not been adjusted, the same gesture still retains different postures in the space, and the significant difference between the data has not been reduced, and it is necessary to continue to process the posture change of the gesture rotation. An embodiment of the present invention is to rotate the palm direction vector and the finger direction vector to the same direction of the coordinate axis of the target coordinate system, for example, the palm direction vector is rotated to the positive axis direction of the X axis, and the finger direction vector is rotated to the positive axis direction of the Z axis. Or the palm direction vector is rotated to the negative axis direction of the X axis, and the finger direction vector is rotated to the positive axis direction of the Y axis.

[0073] The gesture recognition method of an embodiment of the present invention subjects the initial sample hand key point data to displacement changes and posture transformations, so that the palm key point coordinates of the normalized hand key point data corresponding to the same gesture are located at the origin of the coordinate axis of the target coordinate system. In multiple sets of data for the same gesture, the skeleton data of the key points except the palm fluctuate within a small range, thereby realizing the normalization of the same gesture. In the collected data of the same gesture, the significant differences between gestures disappear, thereby improving the reliability of the training data.

[0074] In some embodiments, the gesture recognition model is trained as follows:

[0075] Identify the multilayer perceptron neural network;

[0076] Input the sample hand key point data into the multi-layer perceptron neural network to obtain the predicted gesture category label corresponding to the sample hand key point data;

[0077] According to the gesture category label and the predicted gesture category label, the model parameters of the multilayer perceptron neural network are updated to train a gesture recognition model.

[0078] It should be noted that the multilayer perceptron (MLP) neural network of the embodiment of the present invention is as follows Figure 2 As shown in the figure, the Input layer is the network's input layer, responsible for inputting data. The Hidden layer is the network's hidden layer, fully connected to the input layer and simulating neurons. The Output layer is the network's output layer, outputting the classification results. P0 is any key point in the sample hand key point data. The MLP neural network of the present invention learns the jump relationship between data points and uses this data jump relationship as a gesture feature, which is ultimately used for gesture classification and recognition.

[0079] Since neural networks emulate and simulate animal neuronal systems, the basic structure of a multilayer perceptron (MLP) can be derived from biological neuron models. The most typical MLP consists of three layers: input layer, hidden layer, and output layer. The different layers of an MLP neural network are fully connected, meaning that any neuron in the previous layer is connected to all neurons in the next layer. An MLP consists of multiple layers of nodes, each fully connected to the next. With the exception of the input node, each node is a neuron with a nonlinear activation function. The multilayer perceptron works similarly to human neurons and can simulate the changes in neurons during learning. It first learns, then uses weights to store data. An algorithm adjusts the weights and reduces bias during training, ultimately achieving data prediction.

[0080] Specifically, the number of nodes in the gesture output layer of the MLP neural network in this embodiment of the present invention represents the number of gesture types to be recognized and can be set by the user. For each gesture, the hand key points are collected, and each key point can be expanded into a set of three x, y, and z sets. The total number of x, y, and z sets of all collected hand key points corresponds to the number of nodes in the neural network's Input Layer. The number of nodes in the Output Layer equals the number of gesture types to be classified and recognized.

[0081] The gesture recognition method according to the embodiment of the present invention can improve the reliability of training data by normalizing gesture data, thereby enabling feature learning of subsequent MLP neural networks.

[0082] The following describes the gesture recognition device provided by the present invention. The gesture recognition device described below and the gesture recognition method described above can be referred to in correspondence with each other. Figure 3 As shown, the gesture recognition device according to an embodiment of the present invention includes:

[0083] The acquisition module 301 is used to determine the key point data of the hand to be identified;

[0084] The recognition module 302 is used to input the hand key point data to be recognized into the gesture recognition model to obtain the target gesture category corresponding to the hand key point data to be recognized;

[0085] Among them, the gesture recognition model is trained based on the sample hand key point data and the gesture category labels corresponding to the sample hand key point data. The sample hand key point data is obtained after the initial sample hand key point data is normalized in terms of posture.

[0086] Compared to traditional methods that rely on collecting gesture images for image recognition to determine gesture classification, the gesture recognition device of the present invention achieves improved recognition accuracy for complex gestures by collecting hand keypoint data to obtain three-dimensional information about each gesture keypoint. Furthermore, the sample hand keypoint data used in training is obtained by normalizing the initial sample hand keypoint data, effectively improving the accuracy and robustness of gesture classification and recognition.

[0087] In some embodiments, the acquisition module 301 uses a somatosensory controller, and the hand key point data to be identified includes the three-dimensional coordinates of the hand key points and the instantaneous movement direction vector of the hand.

[0088] It should be noted that the somatosensory controller can collect three-dimensional information of key points of the human skeleton. Compared with the RGBD images obtained by traditional image acquisition methods, it can collect more accurate gesture information in the presence of shadows and occlusions.

[0089] At the same time, although traditional gesture recognition devices also have solutions for gesture recognition using somatosensory controllers, they often achieve different gesture recognition by collecting information such as finger angles in gestures. When the recognition is not available, the gestures need to be defined in advance. It is difficult to handle gestures with complex finger spatial structures during the customization process. The device of the present invention is further provided with a recognition module 302 after the acquisition module 301. Various gestures are recognized through deep learning methods, which can avoid the tedious steps of customizing gestures when gesture recognition is performed using only somatosensory controllers, and achieve accurate classification of complex gestures.

[0090] The embodiment of the present invention also discloses a gesture interaction method, such as Figure 4 As shown, it at least includes the following steps:

[0091] Step 401: Acquire key point data of a hand to be identified, and determine a target gesture category corresponding to the key point data of the hand to be identified based on the gesture recognition method of the above embodiment;

[0092] Step 402: Determine a target human-computer interaction function corresponding to a target gesture category based on a predefined correspondence between gesture categories and human-computer interaction functions.

[0093] It should be noted that the correspondence between the predefined gesture categories and human-computer interaction functions is set in advance by the user and pre-stored in the controller. Usually, the gesture categories and human-computer interaction functions correspond one to one, such as Figure 5 As shown, the fist gesture type in the first row can be defined to correspond to the display function of 3D model A, and the C-shaped gesture type in the second row can be defined to correspond to the display function of 3D model B. In addition, other gesture types can be defined, such as five fingers spread to correspond to the zoom function. After obtaining the key point data of the hand to be recognized, this method uses deep learning to perform gesture recognition and sends the recognition results to the light field for application in light field human-computer interaction.

[0094] Specifically, step 401 includes:

[0095] The key point data of the hand to be identified is obtained and input into the gesture recognition model determined by the gesture recognition method. The target gesture category is output in the form of a label through the output layer of the gesture recognition model; different gestures correspond to different labels.

[0096] Step 402 includes:

[0097] The target gesture category corresponding to the label information is input into the light field device through data transmission;

[0098] After receiving the tag information, the light field device compares it with the pre-set tag table and triggers different human-computer interaction functions based on different tags;

[0099] The label table stores the correspondence between gesture categories and human-computer interaction functions.

[0100] The gesture interaction method of the present invention addresses issues such as the lengthy image processing time and image clarity limitations of existing gesture recognition solutions. A high-precision gesture recognition method is designed to identify target gesture categories, improving the functional expansion and recognition accuracy of subsequent naked-eye 3D light field human-computer interaction.

[0101] The following describes the gesture interaction device provided by the present invention. The gesture interaction device described below and the gesture interaction method described above can be referred to in correspondence with each other. Figure 6 As shown, the gesture interaction device of an embodiment of the present invention includes:

[0102] The gesture recognition module 601 is used to obtain the key point data of the hand to be recognized, and determine the target gesture category corresponding to the key point data of the hand to be recognized based on the gesture recognition method of the above embodiment;

[0103] The gesture interaction module 602 is configured to determine a target human-computer interaction function corresponding to a target gesture category based on a predefined correspondence between gesture categories and human-computer interaction functions.

[0104] The gesture interaction device of the present invention addresses issues such as the lengthy image processing time and image clarity limitations of existing gesture recognition solutions. A high-precision gesture recognition method is designed to determine the target gesture category, improving the functional expansion and recognition accuracy of subsequent naked-eye 3D light field human-computer interaction.

[0105] In some embodiments, the apparatus further includes a light field device, such as a light field display, which can realize human-computer interaction function.

[0106] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the gesture recognition method, which includes:

[0107] Determine the key point data of the hand to be identified;

[0108] Input the hand key point data to be identified into the gesture recognition model to obtain the target gesture category corresponding to the hand key point data to be identified;

[0109] Among them, the gesture recognition model is trained based on the sample hand key point data and the gesture category labels corresponding to the sample hand key point data. The sample hand key point data is obtained after the initial sample hand key point data is normalized in terms of posture.

[0110] Or executing a gesture interaction method, the method comprising:

[0111] Obtaining key point data of the hand to be identified, and determining the target gesture category corresponding to the key point data of the hand to be identified based on any gesture recognition method;

[0112] Based on the predefined correspondence between gesture categories and human-computer interaction functions, a target human-computer interaction function corresponding to the target gesture category is determined.

[0113] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0114] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the gesture recognition method provided by the above methods, which includes:

[0115] Determine the key point data of the hand to be identified;

[0116] Input the hand key point data to be identified into the gesture recognition model to obtain the target gesture category corresponding to the hand key point data to be identified;

[0117] Among them, the gesture recognition model is trained based on the sample hand key point data and the gesture category labels corresponding to the sample hand key point data. The sample hand key point data is obtained after the initial sample hand key point data is normalized in terms of posture.

[0118] Or executing a gesture interaction method, the method comprising:

[0119] Obtaining key point data of the hand to be identified, and determining the target gesture category corresponding to the key point data of the hand to be identified based on any gesture recognition method;

[0120] Based on the predefined correspondence between gesture categories and human-computer interaction functions, a target human-computer interaction function corresponding to the target gesture category is determined.

[0121] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the gesture recognition method provided by the above methods is implemented, and the method includes:

[0122] Determine the key point data of the hand to be identified;

[0123] Input the hand key point data to be identified into the gesture recognition model to obtain the target gesture category corresponding to the hand key point data to be identified;

[0124] Among them, the gesture recognition model is trained based on the sample hand key point data and the gesture category labels corresponding to the sample hand key point data. The sample hand key point data is obtained after the initial sample hand key point data is normalized in terms of posture.

[0125] Or executing a gesture interaction method, the method comprising:

[0126] Obtaining key point data of the hand to be identified, and determining the target gesture category corresponding to the key point data of the hand to be identified based on any gesture recognition method;

[0127] Based on the predefined correspondence between gesture categories and human-computer interaction functions, a target human-computer interaction function corresponding to the target gesture category is determined.

[0128] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0129] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A gesture recognition method, characterized in that: include: Determine the key point data of the hand to be identified; Inputting the hand key point data to be identified into a gesture recognition model to obtain a target gesture category corresponding to the hand key point data to be identified; The gesture recognition model is trained based on sample hand key point data and gesture category labels corresponding to the sample hand key point data, and the sample hand key point data is obtained after the initial sample hand key point data is normalized in posture; Determining the sample hand key point data includes: Determine the target coordinate system; Obtaining the palm key point, palm direction vector and finger direction vector of the initial sample hand key point data; The palm key point is displaced to the origin of the target coordinate system, and the palm direction vector and the finger direction vector are rotated to the direction of the coordinate axis of the target coordinate system to obtain normalized hand key point data; The intra-class data difference of the normalized hand key point data of each gesture category is obtained, and the normalized hand key point data with the intra-class data difference being less than or equal to a first threshold is determined as the sample hand key point data.

2. The gesture recognition method according to claim 1, characterized in that: The gesture recognition model is trained in the following way: Identify the multilayer perceptron neural network; Inputting the sample hand key point data into the multilayer perceptron neural network to obtain a predicted gesture category label corresponding to the sample hand key point data; According to the gesture category label and the predicted gesture category label, the model parameters of the multilayer perceptron neural network are updated to train and obtain the gesture recognition model.

3. A gesture interaction method, characterized in that: include: Acquire key point data of the hand to be identified, and determine the target gesture category corresponding to the key point data of the hand to be identified based on the gesture recognition method according to claim 1 or 2; Based on the predefined correspondence between gesture categories and human-computer interaction functions, a target human-computer interaction function corresponding to the target gesture category is determined.

4. A gesture recognition device, characterized in that: include: The acquisition module is used to determine the key point data of the hand to be identified; A recognition module, configured to input the hand key point data to be recognized into a gesture recognition model to obtain a target gesture category corresponding to the hand key point data to be recognized; The gesture recognition model is trained based on sample hand key point data and gesture category labels corresponding to the sample hand key point data, and the sample hand key point data is obtained after the initial sample hand key point data is normalized in posture; The identification module is specifically used for: Determine the target coordinate system; Obtaining the palm key point, palm direction vector and finger direction vector of the initial sample hand key point data; The palm key point is displaced to the origin of the target coordinate system, and the palm direction vector and the finger direction vector are rotated to the direction of the coordinate axis of the target coordinate system to obtain normalized hand key point data; The intra-class data difference of the normalized hand key point data of each gesture category is obtained, and the normalized hand key point data with the intra-class data difference being less than or equal to a first threshold is determined as the sample hand key point data.

5. The gesture recognition device according to claim 4, characterized in that: The acquisition module adopts a somatosensory controller, and the hand key point data to be identified includes the three-dimensional coordinates of the hand key point and the instantaneous movement direction vector of the hand.

6. A gesture interaction device, characterized in that: include: A gesture recognition module, configured to obtain key point data of a hand to be recognized, and determine a target gesture category corresponding to the key point data of the hand to be recognized based on the gesture recognition method according to claim 1 or 2; The gesture interaction module is used to determine the target human-computer interaction function corresponding to the target gesture category based on the predefined correspondence between gesture categories and human-computer interaction functions.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the gesture recognition method according to claim 1 or 2 or the gesture interaction method according to claim 3 is implemented.

Images