Long-distance gesture recognition method and device
Through multimodal gesture recognition model and keyframe sampling technology, combined with RGB and deep video streaming, the problems of low accuracy and slow speed of long-distance gesture recognition are solved, high-precision and fast long-distance gesture recognition are achieved, and anti-interference is enhanced ability.
Patent Information
- Application Number
- CN202210225062.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-03-09
AI Technical Summary
Existing gesture recognition technology has low accuracy, slow speed and difficult to recognize gestures at long distances, especially in scenarios with high background interference, making it difficult to capture gesture details.
Using multimodal gesture recognition model combined with RGB and deep video stream, through gesture area estimation and keyframe sampling technology, the YOLO V4 Tiny detection model and 3D ResNeXt-101 convolutional neural network are used for gesture recognition, and robust video features are extracted and Perform accurate identification.
It improves the accuracy and speed of long-distance gesture recognition, enhances anti-interference ability, and can quickly locate gestures at long distances and detect the specific locations of gestures, adapting to the duration and speed differences of different types of gestures.
Smart Images

Figure CN114613006B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and gesture recognition, and in particular to a long-distance gesture recognition method and device. Background Art
[0002] Gesture recognition plays an important role in communication between people and interaction between people and machines, and has broad application prospects in the fields of sign language recognition and natural human-computer interaction. Due to the complexity and diversity of gestures, the great uncertainty of their duration, coupled with the influence of shooting angle, distance and lighting conditions, gesture detection and recognition is a very challenging task.
[0003] Gesture recognition needs to consider the changes in hand shape and position. It is difficult for manually designed feature descriptors to cover the detailed features of gestures. Deep neural networks have good feature representation capabilities and show their strong advantages in image and video vision tasks. Therefore, the current mainstream gesture recognition method is based on deep neural networks to learn and express the complex spatial morphological features and temporal motion features of gestures. Convolutional Neural Networks (CNNs) are widely used to extract spatial features of images. There are three main methods for representing temporal motion features: the first is based on optical flow and motion vectors. This method has a very large amount of calculation, is easily affected by lighting and occlusion, and has poor robustness; the second is to use recurrent neural networks (RNNs) to extract temporal features. This method inputs the image features extracted by the convolutional neural network into the recurrent neural network to extract motion features. The model is large and complex, difficult to optimize, and often requires a large downsampling of the original video, which easily loses key information; the third is based on 3D convolution, using a three-dimensional convolution kernel to perform convolution in two spatial dimensions and one temporal dimension, and extracting spatial features and temporal features at the same time. This method can model and fuse spatiotemporal features well, but it requires the design of appropriate convolution kernels and network structures to improve prediction speed while ensuring recognition accuracy.
[0004] Existing gesture recognition methods focus on close-range interaction scenarios such as face-to-face communication and gesture-controlled driving, such as Chinese invention patents CN108932500A and CN113255602A. In these scenarios, the person making the gesture is very close to the camera, so the hands are prominent and easily recognized in the captured images. However, in many scenarios, long-distance control and interaction are required. For example, in a meeting scenario, participants want to control the large-screen all-in-one conference machine to show slides through gestures, and use gestures to adjust the playback progress and sound when watching movies at home. In the long-distance case, the area where the gesture occurs accounts for a small proportion of the camera's field of view, the gesture details are insufficient, and the background will also bring more interference, making gesture recognition more difficult. Summary of the invention
[0005] The purpose of the present invention is to provide a long-distance gesture recognition method and device to address the problems of low gesture recognition accuracy, slow speed and inability to recognize long-distance gestures in the above-mentioned prior art, which can extract more robust video features and capture long-distance gestures for accurate recognition.
[0006] In order to achieve the above object, the present invention adopts the following technical scheme:
[0007] A long-distance gesture recognition method, the steps of which include:
[0008] Obtaining a hand position h1 of a frame p1 in a target video, and calculating a gesture estimation area q1 of the frame p1 based on the hand position h1;
[0009] Get the frame p in the target video i The hand position h i , and when the hand position h i Falling in frame p i-1 The hand gesture estimation area q j When the gesture estimation area q j As the frame p i The hand position h is the estimated area of the hand. i , calculate frame p i The hand gesture estimation area q j+1 ;
[0010] Based on the gesture estimation area q j Split the target video to obtain several video streams s t ;
[0011] For each of the video streams t Perform gesture recognition to obtain a gesture recognition result of the target video.
[0012] Furthermore, the step of obtaining the hand position h1 of the frame p1 in the target video includes:
[0013] The YOLO V4 Tiny detection model is supervised trained on the hand position training set to obtain a hand detector;
[0014] The image of the frame p1 is input into the hand detector to obtain the hand position h1.
[0015] Furthermore, the calculation of the gesture estimation area q1 of the frame p1 based on the hand position h1 includes: taking the hand position h1 as the center, expanding outwards r w times the hand width and r h A rectangular area that is times the hand height.
[0016] Furthermore, the gesture-based estimation area q j Split the target video to obtain video stream s t ,include:
[0017] In each frame p i Get several key frames from
[0018] Use the same gesture to estimate the area q j The key frames in the video stream s are constructed t .
[0019] Furthermore, in each frame p i Get key frames from , including:
[0020] For the same gesture estimation area q j Frame p i With frame p i-1 , respectively, the gesture estimation area q j Convert to grayscale image F cur and grayscale image F pre ;
[0021] Calculate the grayscale image F cur With the grayscale image F pre Frame difference map of
[0022] Based on a set pixel value threshold, converting the frame difference image into a binary image;
[0023] Based on the binary image, in the gesture estimation area q j Count the number of pixels that are greater than the pixel value threshold;
[0024] Calculate the number of pixels occupying the gesture estimation area q j The ratio of the total number of pixels of the frame p is determined according to the ratio. i Whether it is a key frame.
[0025] Furthermore, the video streams st Performing gesture recognition to obtain a gesture recognition result of the target video includes:
[0026] Using a sliding window, obtain the video stream s t Several windows;
[0027] Input the window video stream into a multimodal gesture recognition model based on a 3D ResNeXt-101 convolutional neural network to predict the gesture category of the window, wherein each ResNeXt residual module of the gesture recognition model performs weighted fusion on feature maps from different modal video streams;
[0028] When the gesture categories of n consecutive windows are all predicted as gesture category L c When the gesture category L c As the video stream t A prediction result in
[0029] Statistics of the video stream s t The prediction result in the target video is obtained.
[0030] Furthermore, the different modality video streams include: an RGB video stream and a depth video stream.
[0031] Furthermore, when the gesture category of m consecutive windows is predicted as the non-gesture category L c When the gesture category L is determined c Finished.
[0032] A storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when running.
[0033] An electronic device, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the above methods.
[0034] The beneficial effects of the present invention compared with the prior art are:
[0035] (1) Multimodal data driven: The multimodal gesture recognition model is designed to integrate the data features of RGB and depth for prediction. The two data features complement each other and have stronger anti-interference and higher recognition accuracy;
[0036] (2) Long-distance recognition: A gesture region estimation method is proposed to quickly locate gestures at long distances;
[0037] (3) Fast recognition speed and high accuracy: A key frame sampling method is proposed, which can use fewer frames for accurate gesture prediction and can cope with the influence of different gesture durations and individual gesture speed differences;
[0038] (4) Detecting the specific location where the gesture appears: While identifying the gesture category, the specific location where the gesture appears in the video can also be detected, and multiple gestures appearing in the video can be detected and identified. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a flow chart of the intelligent system for long-distance gesture recognition of the present invention.
[0040] Figure 2 This is the structure diagram of the multimodal gesture recognition network model. DETAILED DESCRIPTION
[0041] The present invention will be further described in detail below with reference to the accompanying drawings and a specific embodiment, but the present invention is not limited to this embodiment.
[0042] The present invention performs long-distance gesture recognition based on RGB-D data and 3D convolutional neural network. First, it is necessary to construct a long-distance gesture data set to provide training data for the model. In this embodiment, the Kinect V4 device is used to collect RGB-D video data. The resolutions of the RGB stream and the depth stream are 1280×720 and 640×570, respectively, and are collected synchronously at a frame rate of 30fps. 30 subjects were randomly assigned to 5 scenes for gesture collection. The collection distance is between 1m and 4m. After the collection is completed, the category of each gesture and its start and end frames in the video are marked, and the position of the hand in each video frame (x, y, w, h) is marked. (x, y) is the coordinate of the center point of the hand's rectangular bounding box, and w and h are the width and height of the rectangular box, respectively.
[0043] In order to estimate the gesture area, a hand detector needs to be trained. This embodiment uses the YOLO V4 Tiny detection model, performs supervised training on the constructed data set using hand position annotations, and saves the model weights with the best performance.
[0044] Under long-distance conditions, gestures are not obvious enough in the camera field of view. Therefore, the method proposed in the present invention estimates the gesture occurrence area based on the hand position to quickly narrow the gesture range and perform more accurate recognition. The specific steps are as follows:
[0045] 1) Use the trained hand detector to detect the position of the hand in the current frame, denoted as R hand =(x, y, w, h), where (x, y) is the rectangular area surrounding the hand R hand The center point coordinates of , w and h are the width and height of the area respectively;
[0046] 2) Estimate the possible gesture area as centered on the hand and expanding outward r w times (e.g. 5 times) the width of the hand, r h A rectangular area of times (e.g., 4 times) the hand height is denoted by R ges =(x, y, r w ×w,r h ×h);
[0047] 3) Repeat step 1) for each new frame. If the hand position in the current frame falls within the gesture area estimated in the previous frame, keep the original estimated gesture area unchanged. Otherwise, proceed to step 2) to estimate a new gesture area.
[0048] Different gestures have different durations, and the speed of gestures performed by each individual may also be different. Long-duration gestures require long-distance temporal features to be recognized. Simply increasing the size of the feature window will make the model computationally intensive and the recognition speed slow. The present invention proposes a key frame sampling method to remove similar redundant frames, so that the model can obtain long-distance temporal features with less computational effort, thereby ensuring the accuracy and speed of recognition. The key frame sampling steps are as follows:
[0049] 1) Crop the corresponding gesture estimation area from the current frame and the previous frame and convert it into a grayscale image, denoted as F cur and F pre , calculate the frame difference map of the two gesture estimation areas:
[0050] F diff =|F cur -F pre |
[0051] 2) Take the pixel value threshold as 25 and convert the frame difference image F diff Transformed into a binary image F bin , that is, the position where the pixel value is greater than 25 is assigned a value of 1, indicating that the change between the two frames at this position is large enough, otherwise it is assigned a value of 0;
[0052] 3) Count the proportion of pixels with a value of 1 in the binary image to the total number of pixels:
[0053]
[0054] Where W and H are binary images F bin When r>0.3, the current frame is considered as a key frame and retained for gesture recognition; otherwise, it is considered as a redundant frame and discarded directly.
[0055] The gesture recognition model is designed based on the ResNeXt-101 convolutional neural network, such as Figure 2As shown in the figure, two 3DResNeXt-101 networks are used to extract the features of the RGB video stream and the depth video stream respectively, and the feature map is weighted and corrected after each ResNeXt residual module. The fusion and weight redistribution of the features of the two modal data make the model focus on more effective and robust features. The model training steps are as follows:
[0056] 1) Preprocessing: perform region estimation and key frame sampling on the gestures in the dataset, and then divide the data into a training set and a validation set in a ratio of 7:3;
[0057] 2) Training: Train the multimodal gesture recognition model on the preprocessed RGB-D video data, using the gesture category and gesture start position annotation data as supervision signals;
[0058] 3) Testing: Test the model performance on the validation set and save the model weights with the best performance.
[0059] For a given video or a video stream captured in real time by a camera, the long-distance gesture recognition method of the present invention has the following specific steps:
[0060] 1) obtaining a hand position h1 of a frame p1 in a target video, and calculating a gesture estimation area q1 of the frame p1 based on the hand position h1;
[0061] 2) Get the frame p in the target video i The hand position h i , and when the hand position h i Falling in frame p i-1 The hand gesture estimation area q j When the gesture estimation area q j As the frame p i The hand position h is the estimated area of the hand. i , calculate frame p i The hand gesture estimation area q j+1 ;
[0062] 3) Estimation area q based on gesture j Split the target video to obtain several video streams s t ;
[0063] 4) Using a sliding window, obtain the video stream s t Several windows;
[0064] 5) Inputting the window video stream into a multimodal gesture recognition model based on a 3D ResNeXt-101 convolutional neural network to predict the gesture category of the window, wherein each ResNeXt residual module of the gesture recognition model performs weighted fusion on feature maps from different modal video streams;
[0065] 6) When the gesture categories of n consecutive windows are all predicted as gesture category L c When the gesture category L c As the video stream t A prediction result in
[0066] 7) Count the video stream s t The prediction result in the target video is obtained.
[0067] The method proposed in the present invention estimates the gesture occurrence area based on the hand position, greatly narrows the range of gesture recognition, reduces a large amount of background interference in long-distance situations, and focuses on gesture features, thereby improving the accuracy of long-distance gesture recognition. The use of key frame sampling reduces the impact of gesture duration and speed, while reducing the amount of model calculation, ensuring the accuracy and speed of model recognition. The designed gesture recognition model based on 3D convolutional neural network comprehensively utilizes multimodal data information for recognition, with stronger anti-interference and higher accuracy.
[0068] The above is only a preferred specific embodiment of the present invention. It should be pointed out that all solutions obtained by making modifications, equivalent substitutions, etc. within the basic principle and method of the present invention are included in the protection scope of the present invention.
Claims
1. A method for long-distance gesture recognition, comprising the steps of: Obtaining a hand position h1 of a frame p1 in a target video, and calculating a gesture estimation area q1 of the frame p1 based on the hand position h1; Get the frame p in the target video i The hand position h i , and when the hand position h i Falling in frame p i-1 The hand gesture estimation area q j When the gesture estimation area q j As the frame p i The hand position h is the estimated area of the hand. i , calculate frame p i The hand gesture estimation area q j+1 ; Based on the gesture estimation area q j Split the target video to obtain several video streams s t ; For each of the video streams t Perform gesture recognition to obtain the gesture recognition result of the target video, wherein: use a sliding window to obtain the video stream s t The window video stream is input into a multimodal gesture recognition model based on a 3D ResNeXt-101 convolutional neural network to predict the gesture category of the window, wherein each ResNeXt residual module of the gesture recognition model performs weighted fusion on feature maps from different modal video streams; when the gesture categories of n consecutive windows are all predicted to be gesture category L c When the gesture category L c As the video stream t A prediction result in ; counting the video stream s t The prediction result in the target video is obtained.
2. The method according to claim 1, characterized in that The step of obtaining the hand position h1 of the frame p1 in the target video includes: The YOLO V4 Tiny detection model is supervised trained on the hand position training set to obtain a hand detector; The image of the frame p1 is input into the hand detector to obtain the hand position h1.
3. The method according to claim 1, characterized in that The step of calculating the gesture estimation area q1 of the frame p1 based on the hand position h1 includes: taking the hand position h1 as the center, expanding outwards r w times the hand width and r h A rectangular area that is times the hand height.
4. The method according to claim 1, characterized in that The gesture-based estimation area q j Split the target video to obtain video stream s t ,include: In each frame p i Get several key frames from Use the same gesture to estimate the area q j The key frames in the video stream s are constructed t .
5. The method according to claim 4, characterized in that In each frame p i Get key frames from , including: For the same gesture estimation area q j Frame p i With frame p i-1 , respectively, the gesture estimation area q j Convert to grayscale image F cur and grayscale image F pre ; Calculate the grayscale image F cur With the grayscale image F pre Frame difference map of Based on a set pixel value threshold, converting the frame difference image into a binary image; Based on the binary image, in the gesture estimation area q j Count the number of pixels that are greater than the pixel value threshold; Calculate the number of pixels occupying the gesture estimation area q j The ratio of the total number of pixels of the frame p is determined according to the ratio. i Whether it is a key frame.
6. The method according to claim 1, characterized in that The different modality video streams include: RGB video stream and depth video stream.
7. The method according to claim 1, characterized in that When the gesture category of m consecutive windows is predicted as the non-gesture category L c When the gesture category L is determined c Finished.
8. A storage medium storing a computer program, wherein: The computer program is configured to execute the method according to any one of claims 1 to 7 when executed.
9. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.
