Gesture interaction method based on multi-technology fusion

Abstract

This invention relates to the field of image processing technology. To improve the user experience, it provides a gesture interaction method based on multi-technology fusion, including a dynamic gesture pre-recording stage and a dynamic gesture recognition stage. By introducing facial recognition technology, a dynamic gesture classification model is trained separately for each operator, ensuring the accuracy of action classification while meeting real-time dynamic gesture classification requirements using lightweight algorithms and models. The introduction of human body recognition technology allows for automatic restoration of the main operator's lock in complex application environments, even if hand detection is lost. Through the design of the dynamic gesture feature extraction process, the number of times a user needs to pre-record a particular dynamic gesture is significantly reduced while maintaining action classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically a gesture interaction method based on the fusion of multiple technologies. Background Technology

[0002] In recent years, gestures, as an input channel, have been widely used in human-computer interaction, virtual reality, and other fields, attracting widespread attention. Especially with the emergence of advanced human-computer interaction technologies and the rapid development of computer technology (particularly deep learning and GPU parallel computing), gesture understanding and interaction methods have achieved groundbreaking results, sparking a research boom. In particular, the introduction of concepts and topics related to the metaverse has brought about a new round of technological outlook and development. The biggest change is the shift in interaction philosophy; it is generally believed that the interaction method in the metaverse is a shift from keyboard and mouse-based interaction to gesture interaction. Its interaction method will be based on spatial interaction using sensing devices (monocular cameras, multi-camera systems, and depth cameras), and it will be the focus of the next round of technological development. In addition, gesture interaction can be widely applied in virtual reality, automotive user interfaces, human-robot interaction, biomedicine, and other fields.

[0003] However, current gesture interaction technologies still have many problems, such as the "golden finger" problem, which makes it difficult to distinguish between intentional and subconscious actions during continuous hand movements; dynamic gesture interaction recognition suffers from latency issues; hand pose reconstruction accuracy is low; deep learning-based gesture classification training takes a long time; and hand tracking is prone to loss. Summary of the Invention

[0004] To improve the user experience, a gesture interaction method based on the integration of multiple technologies is provided.

[0005] The technical solution adopted by the present invention to solve the above problems is:

[0006] A gesture interaction method based on multi-technology fusion includes: a dynamic gesture pre-recording stage and a dynamic gesture recognition stage, wherein the dynamic gesture pre-recording stage includes:

[0007] Step 11: Extract keyframes from the raw video stream data, perform image preprocessing on the extracted keyframes, and delineate a preset region on the preprocessed image.

[0008] Step 12: Perform human detection on the preset area, select the case with the highest detection score and closest to the center of the detection area as the subject of the action pre-recording, obtain its border and perform image cropping.

[0009] Step 13: Locate the face and hands in the cropped human body image area, and crop the image data of the face and hands according to the preset size;

[0010] Step 14: Perform face recognition on the acquired face region, output the face features, and store them in the database;

[0011] Step 15: Perform 3D key point detection on the acquired hand region. Select the process based on whether pre-recording is enabled: If pre-recording is not enabled, predict and acquire the hand image region of the next frame based on the current and past 3D key points and their projected 2D key point positions, and repeat step 15 based on the newly acquired hand image region; If pre-recording is enabled, add the 3D key point features of the hand from multiple frames corresponding to the complete gesture to the data queue.

[0012] Step 16: Process the data queue using Gaussian Mixture Model (GMM) to obtain the GMM parameters, and then convert the 3D key point features of the hands in multiple frames into Fisher features; calculate the Fisher features of various dynamic gestures of the pre-recorded subject, and use the SVM model for classification training to obtain the SVM model; combine the trained GMM model and SVM model with the facial recognition features of the pre-recorded subject and store them in the database.

[0013] The dynamic gesture recognition stage includes:

[0014] Step 21: Delineate a preset area in the same way as in Step 11, perform human detection within the preset area, and then perform human recognition on the main operator's human body.

[0015] Step 22: Perform 3D key point detection and face recognition on the hand area corresponding to the main operator's human body;

[0016] Step 23: Based on the facial feature vectors extracted by facial recognition, retrieve the corresponding GMM model and SVM model from the database;

[0017] Step 24: Extract features from the 3D key points of the hand in consecutive frames, and combine them with the pre-trained GMM model to extract Fisher features. Classify the generated Fisher features into action types using the SVM classification model and output the classification results.

[0018] Furthermore, the image preprocessing includes noise filtering, automatic contrast adjustment, and size scaling.

[0019] Furthermore, in step 12, a target detection algorithm is used to detect human bodies. The target detection algorithm includes CenterNet, SSD, or Faster R-CNN algorithms.

[0020] Furthermore, step 13 uses 2D key point detection to locate the face and hands.

[0021] Furthermore, in step 15, hrnet, hourglass, or mediapipe algorithms are used to detect 3D key points of the hand.

[0022] Furthermore, in step 15, after adding the 3D key point features of the hand to the data queue, the method further includes: performing multi-frequency sampling on the data queue to form subsequences of the same action.

[0023] Furthermore, step 15 also includes: manually extracting features from the data in the subsequence: setting one key point at the wrist and four key points for each finger; calculating the spatial position of the remaining key points relative to the wrist with the wrist as the root node, and normalizing the data according to the size of the palm; calculating the spatial vector of each pair of key points as manual features based on a preset set of five non-repeating key points.

[0024] Step 16 uses a Gaussian Mixture Model (GMM) to process the handcrafted features, obtains the Gaussian Mixture Model parameters, and then converts the handcrafted features from multiple frames into Fisher features.

[0025] Furthermore, step 21, after performing human body recognition on the main operator, also includes caching its recognition feature vector;

[0026] In step 22, when performing 3D key point detection on the hand region corresponding to the main human body, if no key points are found or the key point score is below the threshold, human body detection is performed on the entire image of the next frame. Features of all human bodies are identified and extracted, and matched with the cached main human body features to determine the main human body region, and then its hand region is determined.

[0027] The advantages of this invention compared to the prior art are:

[0028] This dynamic gesture recognition solution incorporates facial recognition technology, allowing for the training of a dynamic gesture classification model for each user individually. This ensures accurate action classification while using lightweight algorithms and models, and meets the requirements for real-time dynamic gesture classification, thus providing users with a better interactive experience.

[0029] This dynamic gesture recognition solution incorporates human body detection and recognition. In complex application environments (insufficient light; hands obscured; motion blur), if hand detection is lost, the lock on the main operator can be automatically restored, so that the user can achieve the effect of being unaware of loss.

[0030] In the pre-recording stage of this dynamic gesture recognition scheme, manual features, GMM model, Fisher features, and SVM classifier are introduced. This can greatly reduce the number of times users need to pre-record a certain dynamic gesture while ensuring that the classification accuracy of subsequent actions remains unchanged, so as to achieve a simple interaction effect. Attached Figure Description

[0031] Figure 1 Flowchart for the gesture pre-recording stage;

[0032] Figure 2 This is a flowchart of the gesture recognition stage. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0034] A gesture interaction method based on multi-technology fusion includes: a dynamic gesture pre-recording stage and a dynamic gesture recognition stage.

[0035] like Figure 1 As shown, the dynamic gesture pre-recording stage includes:

[0036] Step 11: For the raw video stream data acquired by the camera system, first decode it into continuous frame image data, and then extract frames based on the key frame information during decoding to reduce the amount of computation while ensuring that the subsequent algorithm processing effect remains unchanged; perform image preprocessing on the extracted image data, including noise filtering, automatic contrast adjustment and size scaling, and delineate a preset area for the detection of the subject in the next step of motion pre-recording.

[0037] Step 12: Perform human detection within the preset area of ​​the image data. Target detection algorithms such as CenterNet, SSD, and Faster R-CNN can be used. Select the case with the highest detection score and closest to the center of the detection area as the subject of the action pre-recording, obtain its bounding box, and perform image cropping.

[0038] Step 13: Locate the face and hands in the cropped human image region, and crop the face and hand image data according to a preset size. In this embodiment, 2D keypoint detection is performed on the cropped human image region from the previous step. Algorithms such as HRNet and Hourglass can be used, or a multi-task objective algorithm can be used to fuse human detection, simultaneously outputting human detection and 2D keypoints, such as the CenterNet algorithm. Based on the detected 2D keypoint results, the positions of the human face and hands are located respectively. The image cropping size can be set according to actual needs and is not limited here.

[0039] Step 14: Perform face recognition on the acquired face region, output the face features, and store them in the database; algorithms such as ArcFace and FaceNet can be used for face recognition.

[0040] Step 15: Perform 3D keypoint detection on the acquired hand region. Algorithms such as HRNet, Hourglass, or MediaPipe can be used for hand 3D keypoint detection. The process selection depends on whether pre-recording is enabled: If pre-recording is not enabled, predict and acquire the hand image region of the next frame based on the current and past 3D keypoints and their projected 2D keypoint positions, and then repeat step 15 based on the newly acquired image; if pre-recording is enabled, add the hand 3D keypoint feature results of multiple frames corresponding to the complete gesture to the data queue. In this embodiment, pre-recording is enabled by manually triggering the pre-recording switch.

[0041] Since the duration of each gesture simulation by the pre-recorded subject varies, the data queue is sampled at multiple frequencies to reduce the number of operations performed by the subject, forming subsequences of identical actions. Subsequences can then be performed based on these subsequences.

[0042] Step 16: Process the data queue using Gaussian Mixture Model (GMM) to obtain the GMM parameters, and then convert the 3D key point features of the hands in multiple frames into Fisher features; calculate the Fisher features of various dynamic gestures of the pre-recorded subject, and use the SVM model for classification training to obtain the SVM model; combine the trained GMM model and SVM model with the facial recognition features of the pre-recorded subject and store them in the database.

[0043] To quickly extract temporal 3D keypoint features, this embodiment does not use related neural network algorithms, but instead employs hand-designed features. There are 21 hand keypoints: one at the wrist and four for each finger. First, the relative spatial positions of the remaining keypoints are calculated using the wrist as the root node, and the data is normalized according to the hand size. Based on multiple pre-set sets of five non-repeating keypoints, the spatial vectors of each pair of keypoints are calculated as hand-designed features. The selection of these multiple sets of keypoints can be obtained using a sequential forward floating search algorithm or through manual prior setting. Hand movements, such as clenching a fist or opening the palm, can be determined by the relative positions of the five set keypoints.

[0044] Correspondingly, when processing features using a Gaussian Mixture Model (GMM) in step 16, manual features are processed.

[0045] like Figure 2As shown, the dynamic gesture recognition stage includes:

[0046] Step 21: Define a preset area in the same way as in Step 11. After human detection is performed within the preset area, human recognition is performed on the main human body. Tripleloss can be trained using CenterNet.

[0047] Step 22: Perform 3D key point detection and face recognition on the hand area corresponding to the main operator's human body;

[0048] Step 23: Based on the facial feature vectors extracted by facial recognition, retrieve the corresponding GMM model and SVM model from the database;

[0049] Step 24: Extract features from the 3D key points of the hand in consecutive frames, and combine them with the pre-trained GMM model to extract Fisher features. Classify the generated Fisher features into action types using the SVM classification model and output the classification results.

[0050] Furthermore, step 21, after performing human body recognition on the main operator, also includes caching its recognition feature vector;

[0051] In step 22, when performing 3D keypoint detection on the hand region corresponding to the main operator's human body, if no keypoints are found or the keypoint score is below a threshold, human body detection is performed on the entire image of the next frame. Features of all human bodies are identified and extracted, and matched with the cached features of the main operator's human body to determine the main operator's human body region, and thus its hand region. Human body recognition can then be used to recover and lock the hand location after it has been lost.

[0052] This application introduces facial recognition technology to train a dynamic gesture classification model for each operator individually. This ensures the accuracy of action classification while meeting real-time dynamic gesture classification requirements using lightweight algorithms and models. By introducing human body recognition technology, the main operator can be automatically locked in complex application environments if hand detection is lost. Through the design of the dynamic gesture feature extraction process, the number of times a user needs to pre-record a certain dynamic gesture is greatly reduced while maintaining the accuracy of action classification.

Claims

1. A gesture interaction method based on multi-technology fusion, characterized in that, include: The dynamic gesture pre-recording stage and the dynamic gesture recognition stage, wherein the dynamic gesture pre-recording stage includes: Step 11: Extract keyframes from the raw video stream data, perform image preprocessing on the extracted keyframes, and delineate a preset region on the preprocessed image. Step 12: Perform human detection on the preset area, select the case with the highest detection score and closest to the center of the detection area as the subject of the action pre-recording, obtain its border and perform image cropping. Step 13: Locate the face and hands in the cropped human body image area, and crop the image data of the face and hands according to the preset size; Step 14: Perform face recognition on the acquired face region, output the face features, and store them in the database; Step 15: Perform 3D key point detection on the acquired hand region. Select the process based on whether pre-recording is enabled: If pre-recording is not enabled, predict and acquire the hand image region of the next frame based on the current and past 3D key points and their projected 2D key point positions, and repeat step 15 based on the newly acquired hand image region; If pre-recording is enabled, add the 3D key point features of the hand from multiple frames corresponding to the complete gesture to the data queue. Step 16: Process the data queue using Gaussian Mixture Model (GMM) to obtain the GMM parameters, and then convert the 3D key point features of the hands in multiple frames into Fisher features; calculate the Fisher features of various dynamic gestures of the pre-recorded subject, and use the SVM model for classification training to obtain the SVM model; combine the trained GMM model and SVM model with the facial recognition features of the pre-recorded subject and store them in the database. The dynamic gesture recognition stage includes: Step 21: Delineate a preset area in the same way as in Step 11, perform human detection within the preset area, and then perform human recognition on the main operator's human body. Step 22: Perform 3D key point detection and face recognition on the hand area corresponding to the main operator's human body; Step 23: Based on the facial feature vectors extracted by facial recognition, retrieve the corresponding GMM model and SVM model from the database; Step 24: Extract features from the 3D key points of the hand in consecutive frames, and combine them with the pre-trained GMM model to extract Fisher features. Classify the generated Fisher features into action types using the SVM classification model and output the classification results.

2. The gesture interaction method based on multi-technology fusion according to claim 1, characterized in that, The image preprocessing includes noise filtering, automatic contrast adjustment, and size scaling.

3. The gesture interaction method based on multi-technology fusion according to claim 1, characterized in that, In step 12, a target detection algorithm is used to detect human bodies. The target detection algorithm includes CenterNet, SSD, or Faster R-CNN.

4. The gesture interaction method based on multi-technology fusion according to claim 1, characterized in that, Step 13 uses 2D key point detection to locate the face and hands.

5. The gesture interaction method based on multi-technology fusion according to claim 1, characterized in that, In step 15, hrnet, hourglass, or mediapipe algorithms are used to detect 3D key points of the hand.

6. The gesture interaction method based on multi-technology fusion according to claim 1, characterized in that, In step 15, after adding the 3D key point features of the hand to the data queue, the method further includes: performing multi-frequency sampling on the data queue to form subsequences of the same action.

7. The gesture interaction method based on multi-technology fusion according to claim 6, characterized in that, Step 15 further includes: manually extracting features from the data in the subsequence: setting one key point at the wrist and four key points for each finger; calculating the spatial position of the remaining key points relative to the wrist with the wrist as the root node, and normalizing the data according to the size of the palm; calculating the spatial vector of each pair of key points as manual features based on multiple sets of five non-repeating key points. Step 16 uses a Gaussian Mixture Model (GMM) to process the handcrafted features, obtains the Gaussian Mixture Model parameters, and then converts the handcrafted features from multiple frames into Fisher features.

8. The gesture interaction method based on multi-technology fusion according to any one of claims 1-7, characterized in that, After performing human body recognition on the main human body, step 21 also includes caching its recognition feature vector. In step 22, when performing 3D key point detection on the hand region corresponding to the main human body, if no key points are found or the key point score is below the threshold, human body detection is performed on the entire image of the next frame. Features of all human bodies are identified and extracted, and matched with the cached main human body features to determine the main human body region, and then its hand region is determined.

Images