A dynamic gesture recognition method and apparatus

Abstract

The method comprises the following steps: S100, collecting gesture image and generating gesture video data; S200, judging whether there is a gesture in a current frame of the gesture video data, if there is a gesture, executing step S300; otherwise, returning to step S100; S300, performing skeleton key point detection on the gesture, and judging whether the gesture is a dynamic gesture based on the skeleton key point, if the gesture is a dynamic gesture, executing step S400; otherwise, returning to step S100; S400, identifying the dynamic gesture to obtain a category of the dynamic gesture.

Description

Technical Field

[0001] This disclosure belongs to the fields of computer vision, pattern recognition, and human-computer interaction, and specifically relates to a dynamic gesture recognition method and device. Background Technology

[0002] Gestures, as an innate and natural form of interaction, serve as a crucial bridge for communication between people, between people and machines, and even between humanoid intelligent machines. There is an urgent need for them in many fields, such as communication for the deaf and mute, smart homes, robotics, healthcare, and national defense. Achieving high-precision and high-accuracy gesture recognition has become a key focus of gesture interaction research.

[0003] As a mobile space carrying essential living spaces, automobiles are currently undergoing a transformation, with traditional cars being replaced by intelligent vehicles. Innovative in-car functions and interactive experiences are likely to become key differentiators and highlights of intelligent vehicles' product differentiation and innovation. For in-car interaction, gestures can minimize cognitive and visual communication costs. Combined with voice, facial, and motion recognition technologies, gestures can facilitate natural in-car interaction, transforming intelligent vehicles not only into daily transportation tools but also into new mobile smart homes and smart offices. Currently, the widespread application of gesture recognition technology in intelligent vehicles faces several challenges. These include insufficient dynamic gesture recognition rates, the ability to recognize only specific gestures, difficulty in accurately recognizing dynamic gestures, the impact of environmental factors and vehicle vibrations on recognition accuracy, recognition of gestures from multiple occupants, long-distance gesture recognition, and the varying speeds of gestures from different individuals. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this disclosure is to propose a dynamic gesture recognition method that can recognize various dynamic gestures in real time with high accuracy.

[0005] A dynamic gesture recognition method includes the following steps:

[0006] S100: Acquires gesture images and generates gesture video data;

[0007] S200: Determine whether there is a gesture in the current frame of the gesture video data. If a gesture is determined to exist, proceed to step S300; otherwise, return to step S100.

[0008] S300: Perform skeletal key point detection on the gesture and determine whether the gesture is a dynamic gesture based on the skeletal key points. If it is determined to be a dynamic gesture, proceed to step S400; otherwise, return to step S100.

[0009] S400: Recognizes dynamic gestures to obtain the category of the dynamic gesture.

[0010] Preferably, according to the method of claim 1, step S200 includes the following steps:

[0011] S201: Buffer the gesture video data to obtain an N-frame gesture image sequence;

[0012] S202: Detect N frames of gesture image sequence to obtain N gesture image detection boxes;

[0013] S203: Make decisions on the N gesture image detection boxes to obtain the N best gesture image detection boxes corresponding to the N frame gesture image sequence;

[0014] S204: Crop the N best gesture image detection boxes to obtain N gesture image regions.

[0015] Preferably, according to the method of claim 1, step S300 includes the following steps:

[0016] S301: Extract the key points of the gesture skeleton in each gesture image region and obtain the coordinates of each key point;

[0017] S302: Optimize the gesture skeleton key points of each gesture image region to obtain the optimized gesture skeleton key points;

[0018] S303: Determine whether the gesture in each gesture image region is a dynamic gesture or a static gesture based on whether the optimized gesture skeleton key points have shifted.

[0019] Preferably, in step S302, optimizing the gesture skeletal key points of the N-frame gesture image region includes the following steps:

[0020] S3021: Based on the temporal outlier detection algorithm, the key points of the gesture skeleton in each gesture image region are decomposed temporally to obtain the periodic part, the trend part and the residual part.

[0021] S3032: Perform box plot outlier detection on the residual part to remove key points that deviate too far, and replace the removed point with the average value of the coordinates of the key points on the adjacent fingers that are closest to the palm.

[0022] Preferably, step S303 includes the following steps:

[0023] S3031: Calculate the offset of key points in the gesture skeleton;

[0024] S3032: Compare the average offset of the skeletal key points of the gesture in two adjacent gesture image regions with a set threshold, and determine whether the gesture is a dynamic gesture or a static gesture based on the comparison result.

[0025] Preferably, in step S400, dynamic gestures are recognized using a dynamic gesture classification network, which includes:

[0026] The feature extraction module is used to extract C-dimensional features from K sets of key point location sequences to represent the probability that a dynamic gesture belongs to C categories;

[0027] The normalized exponential function is used to normalize probabilities to the range [0, 1].

[0028] This disclosure also provides a dynamic gesture recognition device, including:

[0029] The acquisition module is used to acquire gesture images and generate gesture video data;

[0030] The first discrimination module is used to determine whether there is a gesture in the gesture video data;

[0031] The second discrimination module is used to determine whether the gesture is a dynamic gesture or a static gesture;

[0032] The recognition module is used to identify dynamic gestures in order to obtain the category of the dynamic gesture.

[0033] Preferably, the first discrimination module includes:

[0034] The caching submodule is used to cache gesture video data to obtain an N-frame gesture image sequence;

[0035] The detection submodule is used to detect N frames of gesture image sequence to obtain N gesture image detection boxes;

[0036] The decision submodule is used to make decisions on N gesture image detection boxes to obtain N optimal gesture image detection boxes corresponding to the N gesture images.

[0037] The cropping submodule is used to crop the N best gesture image detection boxes to obtain N gesture image regions.

[0038] Preferably, the second discrimination module includes:

[0039] The extraction submodule is used to extract the key points of the gesture skeleton in each gesture image region and obtain the coordinates of each key point;

[0040] The optimization submodule is used to optimize the gesture skeletal key points of each gesture image region to obtain optimized gesture skeletal key points;

[0041] The discrimination submodule is used to determine whether the gesture in each gesture image region is a dynamic gesture or a static gesture based on whether the optimized gesture skeletal key points have shifted.

[0042] Preferably, the identification module includes:

[0043] The recognition submodule is used to classify dynamic gestures using a dynamic gesture classification network to obtain category probabilities;

[0044] The category decision submodule is used to make decisions on the category probabilities to obtain the category of the dynamic gesture.

[0045] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0046] 1. This disclosure can recognize a variety of dynamic gestures in real time, with short latency (i.e., a gesture category result can be obtained at the end of every 20 frames, while the camera frame rate is 15-20 frames per second, which means that the entire action is completed and the result is output within 1-1.3 seconds), high recognition accuracy, and can accurately recognize dynamic gestures of different speeds.

[0047] 2. This disclosure is applicable not only to visible light RGB video streams, but also to dynamic gesture detection and recognition in infrared IR video streams, RGB-Depth video streams, or IR-Depth video streams, and can be widely used for natural interaction in fields such as smart cars, smart homes, and robots. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of a dynamic gesture recognition method provided in one embodiment of the present disclosure;

[0049] Figure 2 This is a schematic diagram of a dynamic gesture classification network structure provided in another embodiment of this disclosure;

[0050] Figure 3 This is a schematic diagram of a dynamic gesture detection frame and key detection points provided in another embodiment of this disclosure. Detailed Implementation

[0051] The following is in conjunction with the appendix Figures 1 to 3 The present invention will now be described in further detail.

[0052] In one embodiment, such as Figure 1 As shown, this disclosure proposes a dynamic gesture recognition method, which includes the following steps:

[0053] S100: Captures gesture images and generates gesture video data;

[0054] S200: Determine whether the gesture in the gesture video data is a dynamic gesture or a static gesture. If it is determined to be a dynamic gesture, proceed to step S300; otherwise, return to step S100.

[0055] S300: Divide the gesture video data into different number of frames (for example, the video data can be divided into 5 frames, 10 frames, 15 frames, 20 frames, and other division methods are also possible) to recognize the dynamic gesture and obtain the category of the dynamic gesture.

[0056] The above embodiments constitute the complete technical solution of this disclosure. Generally, within the time range of 0.3s-1.5s, each dynamic gesture can complete an action. When the speed of dynamic gestures varies, if a fixed frame number division method (taking 20 frames as an example) is used to detect and classify gestures, for a camera with 15 frames per second, all actions will have a fixed duration of 1.33s. However, by using different frame number division methods, the duration of the same action is 0.33s, 0.67s, 1.00s, and 1.33s, and the time range with the highest probability, i.e., the most likely, is selected to determine the gesture category. For gestures with an unknown end, four rates are used to approximate the duration, thus significantly improving the accuracy compared to a fixed 1.33s. Therefore, this embodiment improves the accuracy of gesture recognition by dividing dynamic gesture videos of different speeds according to different frame numbers for recognition.

[0057] In another embodiment, step S200 includes the following steps:

[0058] S201: Buffer the gesture video data to obtain an N-frame gesture image sequence;

[0059] In this step, the cached gesture video data can be continuous frames or image frames acquired at intervals. N is an integer and N>4, and the cached N-frame gesture image sequence is updated in a first-in-first-out manner.

[0060] S202: Detect N frames of gesture image sequence to obtain N gesture image detection boxes;

[0061] In this step, if a gesture is detected in both the first and last frames of the N-frame gesture image sequence, the gesture image region is bounded out and the gesture image detection box is output; otherwise, the process returns to step S201 to re-cachise the gesture video data to update the existing N-frame gesture image sequence, and then rereads the updated N-frame gesture image sequence and performs the detection again.

[0062] It should be noted that gesture detection networks can be used to detect gesture images frame by frame. Considering the need for lightweight deployment, lightweight object detection networks, including RetinaNet, YOLO-Fastest, and CenterNet, can be used as gesture detection networks here.

[0063] S203: Make decisions on the N gesture image detection boxes to obtain the N best gesture image detection boxes corresponding to the N frame gesture image sequence.

[0064] In this step, the decision is made by calculating the intersection-union ratio (CIU) of the areas of the bounding boxes corresponding to two adjacent frames in the N-frame gesture image sequence. The formula for calculating the CIU is as follows:

[0065]

[0066] Here, A and B represent the areas of the detection boxes corresponding to two adjacent frames of gesture images, respectively. If the Intersection over Union (IOU) is greater than the threshold S, the detection box corresponding to the previous frame of gesture image is selected as the common detection box for the two adjacent frames; otherwise, the actual detection box is used for the subsequent frame of gesture image. This process continues, making decisions based on the detection boxes corresponding to each frame of gesture image to achieve a stable display of the detection boxes.

[0067] S204: Crop the N best gesture image detection boxes to obtain N gesture image regions.

[0068] In another embodiment, step S300 includes the following steps:

[0069] S301: Extract the key points of the gesture skeleton in each gesture image region (each finger contains 4 key points of the bone joints, plus one key point of the palm, for a total of 21 key points), and obtain the coordinates of each key point (each key point uses two-dimensional pixel coordinates, representing the horizontal and vertical distances from the image origin).

[0070] In this step, commonly used heatmap regression network models or other lightweight keypoint detection models can be used to extract gesture skeleton keypoints.

[0071] S302: Optimize the gesture skeleton key points of each gesture image region to obtain the optimized gesture skeleton key points;

[0072] This step involves optimizing the key points of the gesture skeleton, specifically including the following steps:

[0073] a. The STL algorithm (Temporal Outlier Detection Algorithm) is used to decompose the temporal sequence of each gesture skeleton keypoint into a periodic component, a trend component, and a residual component. (STL consists of an inner loop and an outer loop. The inner loop mainly performs trend fitting and periodic component calculation; the outer loop is mainly used to adjust the robustness weights. The STL decomposition specifically includes detrending, smoothing of periodic subsequences, low-throughput filtering of periodic subsequences, removing and smoothing the trend of periodic subsequences, detrending, trend smoothing, etc. The specific decomposition process is well known to those in the field and will not be described in detail here.)

[0074] b. Use a box plot method to remove key points that deviate too far from the target value from the residual portion of each frame. The specific removal process is as follows: Figure 3 As shown in the figure, this graph displays the maximum, minimum, median, and upper and lower quartiles of a set of data. Values ​​outside the upper and lower boundaries are considered outliers and displayed on the graph.

[0075] c. If the key point to be removed is the key point at the joint closest to the palm of each finger, then the average of the x and y coordinates of the joints closest to the palm of the adjacent fingers is used to replace the coordinate of that point. If the key point is at a joint in another position, then the average of the x and y coordinates of the adjacent joints of the same finger is used to replace it, so as to remove the key point that is far from the palm and provide a more effective and accurate key point sequence for subsequent gesture recognition.

[0076] S303: Determine whether the gesture in each gesture image region is a dynamic gesture or a static gesture based on whether the optimized gesture skeleton key points have shifted.

[0077] In this step, if the offset of the optimized gesture skeleton keypoints (the offset formula is specifically implemented by calculating the average of the Euclidean distances of 21 keypoint pairs as the offset) is greater than or equal to the set offset threshold (the selection of the offset threshold depends on the sensitivity requirements of the gesture recognition system in the specific application scenario; for example, 1 / 15 of the original video image frame size can be taken as the offset threshold), then the gesture is considered to have an offset (for example, as mentioned above, if the offset threshold is 1 / 15 of the original video image frame size, then the gesture is considered to have moved), and the gesture action contained in the current N-frame gesture image sequence is considered to be a dynamic gesture; otherwise, if there is no offset, the gesture action contained in the current N-frame gesture image sequence is considered to be a static gesture, and then it is necessary to return to step S201 to update the gesture video data cache and repeat steps S202 to S302.

[0078] In another embodiment, step S400 includes the following steps:

[0079] S401: Use a dynamic gesture classification network to classify dynamic gestures to obtain category probabilities; the gesture categories specifically include eight categories: waving upwards, waving downwards, waving to the left, waving to the right, pressing forward, snapping fingers, waving hands left and right, and clenching and unfolding fists.

[0080] In this step, the N frames of images cached from the preceding video stream (N being a multiple of 4) are divided into the first N / 4 frames, the first 2N / 4 frames, the first 3N / 4 frames, and the first N frames. Four pre-trained dynamic gesture classification networks are then used to classify and recognize these different frame numbers, resulting in four category probabilities P1, P2, P3, and P4. These four dynamic gesture classification networks have the same structure; the only difference lies in the number of input frames.

[0081] S402: Decisions are made based on the four category probabilities P1, P2, P3, and P4. Specifically: First, P1 is judged. If P1 is greater than or equal to the threshold S, the category corresponding to P1 obtained in the first N / 4 frames is considered valid; if it is less than the threshold S, the category is considered invalid. This process is repeated for P2, P3, and P4. If no valid category exists, the output is "No gesture". In online testing, for the eight gestures of varying speeds, specifically left, right, up, down, and left / right hand movements, the prediction accuracy improved by using the four category probabilities compared to using only P4. For other gestures, the accuracy improved by 0.1.

[0082] In another embodiment, in step S401, as Figure 2 As shown, the dynamic gesture classification network includes:

[0083] The feature extraction module is used to extract C-dimensional features from K sets of key point location sequences to represent the probability that a dynamic gesture belongs to C categories;

[0084] The normalized exponential function is used to normalize probabilities to the range [0, 1].

[0085] In this embodiment, the feature extraction module is composed of a fully connected layer, a batch normalization layer, and a non-linear activation layer stacked together. The input of the dynamic gesture classification network is a sequence of K key point positions, and the output is a C-dimensional feature (C represents the number of categories), which represents the probability that the gesture belongs to each of the C categories. At the same time, in order to facilitate the comparison between the maximum output probability and the set threshold, the probability needs to be normalized to the range [0, 1] using an exponential normalization function.

[0086] Furthermore, the training process of the dynamic gesture classification network is described below:

[0087] 1. The total amount of dynamic gesture video data collected is 8144, including approximately 2000 sets each of 5-frame, 10-frame, 15-frame, and 20-frame videos, containing different subjects, different gestures, and different backgrounds. The videos of each frame rate are divided into training, validation, and test sets in an 8:1:1 ratio.

[0088] 2. During the training process of the network using the training set, the loss value will continue to decrease. When the loss value stops decreasing, the training is complete. After each round of training, the network is validated using the validation set. When the validation result exceeds 0.85, it indicates that the trained network has been obtained.

[0089] 3. Test the trained network using the test set, select the network with the highest test accuracy, convert it, and apply it to the embedded platform.

[0090] The recognition accuracies for eight gestures at different speeds using a fixed frame count division method are 0.65, 0.75, 0.70, 0.80, 0.70, 0.75, 0.80, and 0.70, respectively. However, the recognition accuracies for dynamic gestures trained using different frame count division methods are 0.80, 0.85, 0.85, 0.90, 0.80, 0.85, 0.95, and 0.85, respectively. It is evident that the method described in this disclosure can improve the recognition accuracy of dynamic gestures compared to existing methods.

[0091] In another embodiment, this disclosure also proposes a dynamic gesture recognition device, comprising:

[0092] The acquisition module is used to acquire gesture images and generate gesture video data;

[0093] The acquisition module includes any one of a visible light RGB camera, an infrared IR camera, and an RGBD depth camera (structured light depth camera, ToF depth camera). The output video data can be any one of an RGB video stream, an IR video stream, an RGB-Depth video stream, and an IR-Depth video stream.

[0094] The first discrimination module is used to determine whether there is a gesture in the gesture video data;

[0095] The second discrimination module is used to determine whether the gesture is a dynamic gesture or a static gesture;

[0096] The recognition module is used to identify dynamic gestures in order to obtain the category of the dynamic gesture.

[0097] In another embodiment, the first discrimination module includes:

[0098] The caching submodule is used to cache gesture video data to obtain an N-frame gesture image sequence;

[0099] The detection submodule is used to detect N frames of gesture image sequence to obtain N gesture image detection boxes;

[0100] The decision submodule is used to make decisions on N gesture image detection boxes to obtain N optimal gesture image detection boxes corresponding to the N gesture images.

[0101] The cropping submodule is used to crop the N best gesture image detection boxes to obtain N gesture image regions.

[0102] In another embodiment, the second discrimination module includes:

[0103] The extraction submodule is used to extract the key points of the gesture skeleton in each gesture image region and obtain the coordinates of each key point;

[0104] The optimization submodule is used to optimize the gesture skeletal key points of each gesture image region to obtain optimized gesture skeletal key points;

[0105] The discrimination submodule is used to determine whether the gesture in each gesture image region is a dynamic gesture or a static gesture based on whether the optimized gesture skeletal key points have shifted.

[0106] In another embodiment, the identification module includes:

[0107] The recognition submodule is used to classify dynamic gestures using a dynamic gesture classification network to obtain category probabilities;

[0108] The category decision submodule is used to make decisions on the category probabilities to obtain the category of the dynamic gesture.

[0109] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A dynamic gesture recognition method, comprising the following steps: S100: Acquires gesture images and generates gesture video data; S200: Determine whether there is a gesture in the current frame of the gesture video data. If there is a gesture, execute S300; otherwise, return to S100. The gesture video data is cached to obtain an N-frame sequence of gesture images; Detection is performed on an N-frame sequence of gesture images to obtain N gesture image detection boxes; The decision is made by calculating the intersection-union ratio (IUR) of the areas of the detection boxes of two adjacent frames in an N-frame gesture image sequence. If the IUR is greater than a threshold S, the detection box corresponding to the previous frame gesture image is selected as the common detection box of the two adjacent frames gesture images; otherwise, the actual detection box is used for the next frame gesture image, resulting in N optimal gesture image detection boxes. The N optimal gesture image detection boxes are then cropped to obtain N gesture image regions. S300: Perform skeletal keypoint detection on the gesture and determine whether the gesture is dynamic based on the skeletal keypoints. If it is determined to be a dynamic gesture, proceed to step S400; otherwise, return to step S100; where, The key points of the gesture skeleton in each gesture image region are extracted, and the coordinates of each key point are obtained. The STL algorithm is used to decompose the temporal sequence of each gesture skeleton key point into a periodic part, a trend part, and a residual part. The residual part of each frame is used to remove key points that are far off-center using the box plot method. If the key point removed is the key point of the joint of each finger that is closest to the palm, the coordinates of the joints of adjacent fingers that are closest to the palm are used to replace the coordinates of the point. If the key point is the key point of other joints, the average coordinates of the adjacent joints of the same finger are used to replace it, thus obtaining the optimized gesture skeleton key points. The gesture in each gesture image region is determined to be a dynamic gesture or a static gesture based on whether the optimized gesture skeleton key points have shifted. S400: Recognizes dynamic gestures to obtain the category of the dynamic gesture.

2. The method according to claim 1, wherein, Step S400 specifically includes: A dynamic gesture classification network is used to classify dynamic gestures to obtain category probabilities; For the N frames of video cache, the dynamic gestures are divided into the first N / 4 frames, the first 2N / 4 frames, the first 3N / 4 frames, and the first N frames. The four pre-trained dynamic gesture classification networks are used to classify and recognize the four different frame numbers of the images to obtain the four category probabilities P1, P2, P3, and P4. Decisions are made on the four category probabilities P1, P2, P3, and P4. First, P1 is judged. If P1 is greater than or equal to the threshold S, the category corresponding to P1 obtained in the first N / 4 frames is considered valid. If it is less than the threshold S, the category is considered invalid. Similarly, threshold judgments are made on P2, P3, and P4. If there is no valid category, no gesture is output.

3. The method according to claim 1, wherein, To determine whether a gesture in each gesture image region is dynamic or static, based on whether the optimized gesture skeleton key points have shifted, the following steps are included: Calculate the offset of key points in the gesture skeleton; The average offset of the skeletal key points of the gesture in two adjacent gesture image regions is compared with a set threshold, and the gesture is determined to be a dynamic gesture or a static gesture based on the comparison result.

4. The method according to claim 1, wherein, Dynamic gestures are identified using a dynamic gesture classification network, which includes: The feature extraction module is used to extract features from K sets of key point location sequences to output the probability that the dynamic gesture belongs to C categories; The normalized exponential function is used to normalize probabilities to the range [0, 1].

5. An apparatus for implementing the dynamic gesture recognition method according to any one of claims 1 to 4, comprising: The acquisition module is used to acquire gesture images and generate gesture video data; The first discrimination module is used to determine whether there is a gesture in the gesture video data; The second discrimination module is used to determine whether the gesture is a dynamic gesture or a static gesture; The recognition module is used to identify dynamic gestures in order to obtain the category of the dynamic gesture.

6. The apparatus according to claim 5, wherein, The first discrimination module includes: The caching submodule is used to cache gesture video data to obtain an N-frame gesture image sequence; The detection submodule is used to detect N frames of gesture image sequence to obtain N gesture image detection boxes; The decision submodule is used to make decisions on N gesture image detection boxes to obtain N optimal gesture image detection boxes corresponding to the N gesture images. The cropping submodule is used to crop the N best gesture image detection boxes to obtain N gesture image regions.

7. The apparatus according to claim 5, wherein, The second discrimination module includes: The extraction submodule is used to extract the key points of the gesture skeleton in each gesture image region and obtain the coordinates of each key point; The optimization submodule is used to optimize the gesture skeletal key points of each gesture image region to obtain optimized gesture skeletal key points; The discrimination submodule is used to determine whether the gesture in each gesture image region is a dynamic gesture or a static gesture based on whether the optimized gesture skeletal key points have shifted.

8. The apparatus according to claim 5, wherein, The identification module includes: The recognition submodule is used to classify dynamic gestures using a dynamic gesture classification network to obtain category probabilities; The category decision submodule is used to make decisions on the category probabilities to obtain the category of the dynamic gesture.

Images