Dynamic gesture recognition method based on multi-modal fusion

Abstract

The invention provides a dynamic gesture recognition method based on multi-modal fusion. The method comprises the following steps: firstly, acquiring a dynamic gesture data set as a training sample, including video streams of two modalities of an RGB image and a depth image; preprocessing the training sample; constructing a multi-modal gesture classification and recognition model; a dynamic gradient modulation method optimization algorithm is adopted, and the preprocessed training samples are input into the constructed multi-modal gesture classification recognition model for training; and obtaining to-be-recognized dynamic gesture data, preprocessing the to-be-recognized dynamic gesture data, and inputting the preprocessed to-be-recognized dynamic gesture data into the trained multi-modal gesture classification recognition model to obtain a classification recognition result. According to the dynamic gesture recognition method based on multi-modal fusion provided by the invention, heterogeneity between modals is effectively reduced by balancing optimization speeds of different modals, complementary information of an RGB image and a depth image can be fully utilized, rich feature information can be obtained, and the performance of a multi-modal dynamic gesture recognition model is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, machine learning and human-computer interaction, and specifically provides a gesture recognition technology in a dynamic environment. Background Art

[0002] Gesture recognition has broad application prospects in many fields such as virtual reality, smart home, and assisted communication. At present, dynamic gesture recognition technology mainly relies on gesture video data captured by visual sensors, and analyzes these data to understand the user's gesture intention. Most of the existing dynamic gesture recognition methods are based on single-modal data, such as using only RGB images or depth images. Although these methods can achieve certain recognition effects in some cases, they have different limitations. Methods that use only RGB images may not accurately recognize gestures under complex backgrounds or lighting changes, while methods that use only depth images may not fully capture the texture and color information of gestures. RGB images provide color and texture information of gestures, while depth images provide depth information of gestures. The two provide complementary information for gesture recognition. Therefore, designing a multimodal model that can effectively fuse RGB image or depth image information has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the object of the present invention is to provide a dynamic gesture recognition optimization algorithm based on multimodal fusion, which improves the accuracy and robustness of gesture recognition by fusing RGB and depth images.

[0004] The present invention proposes a dynamic gesture recognition optimization algorithm based on multimodal fusion, which includes the following specific steps:

[0005] Step 1: Use the realsense3D camera to collect a dynamic gesture dataset as training samples, including video streams in two modes: RGB image and depth image.

[0006] Step 2, preprocessing the training samples;

[0007] Step 3, construct a multimodal gesture classification and recognition model;

[0008] Step 4, using the dynamic gradient modulation method to optimize the algorithm, inputting the preprocessed training samples into the constructed multimodal gesture classification and recognition model for training;

[0009] Step 5: obtain the dynamic gesture data to be recognized, and input it into the trained multimodal gesture classification and recognition model after preprocessing to obtain the classification and recognition result.

[0010] Preferably, the preprocessing of the acquired training samples in step 2 includes:

[0011] Given an RGB video stream containing T frames and deep video stream in Represents the nth frame RGB image in the video stream, representing The nth depth image of the video stream is obtained by using a content adaptive sampler to remove video frames without gestures and then rearranging the remaining video frames. And N is much smaller than T;

[0012] Preferably, the multimodal gesture classification and recognition model in step 3 includes an R(2+1)D feature extraction module, a Transformer encoder module, a feature fusion module, and a classification module:

[0013] The feature extraction module uses two pre-trained R(2+1)D models f R (w R ,·),f D (w D , ·) as the gesture feature extractor for RGB video stream and depth video stream, from the preprocessed RGB video stream X R and Deep Video StreamX D Extract the spatiotemporal feature vector sequence F from R 、F D , where w R Feature extraction network f R The model parameters, w D Feature extraction network f R Model parameters of

[0014] The Transformer encoder module extracts the spatiotemporal feature vector sequence F of the two modalities through the R(2+1)D model. R 、F D The fixed position encoding is performed using sine and cosine functions based on different frequencies to obtain the position information of the elements in the sequence, and then the encoder is composed of multiple multi-head self-attention modules and fully connected feedforward neural network layers stacked together. Capture the gesture relationship between the previous and next frames in different sequences, and form a feature vector z with the same dimension R , z D , where θ R For feature extraction network The model parameters, θ D For feature extraction network Model parameters of

[0015] The feature fusion module uses a splicing method to fuse the feature vector z extracted from the RGB video stream and the depth video stream. R and zD , generate the fused feature vector [z R +z D ];

[0016] The classification module uses a multilayer perceptron as the classifier F C , classify the fused features and generate prediction results

[0017] Preferably, in step 4, the dynamic gradient modulation method is used to optimize the algorithm, and the preprocessed training samples are input into the constructed multimodal gesture classification and recognition model for training, which specifically includes the following steps:

[0018] Step 4.1, define the model loss function as the cross entropy function L, where the RGB image loss function is L R , the depth image loss function is L D , the optimization function uses the common stochastic gradient descent algorithm:

[0019]

[0020]

[0021] Where y(i) is the true label of the i-th training sample; is the eigenvector z R The prediction result after the classifier iterates for the tth time, is the eigenvector z D The prediction result after the classifier iterates for the tth time; To predict the results The probability of being the same as the true outcome y(i); To predict the results is the probability of the kth gesture category; N is the number of training samples, and M is the number of gesture categories;

[0022] Step 4.2, calculate the Transformer encoder parameters θ corresponding to the RGB image R Gradient Transformer encoder parameters θ corresponding to the depth image D Gradient

[0023]

[0024]

[0025] Step 4.3, calculate the two-modal loss function L of RGB image and depth image R , L D The slope difference ratio at different iteration numbers

[0026]

[0027]

[0028] Step 4.4, calculate the adjustment factor during RGB image training Adjustment factors in the deep image training process

[0029]

[0030]

[0031] In the formula, α is a hyperparameter that controls the degree of adjustment;

[0032] Step 4.5: Update the gradients of the two modes using the dynamic gradient adjustment algorithm

[0033]

[0034]

[0035] Where η is the learning rate.

[0036] Beneficial Effects

[0037] The present invention proposes a dynamic gesture recognition method based on multimodal fusion. The method not only utilizes the hand color and texture information provided by the RGB image, but also utilizes the depth information of the hand depth image. By balancing the optimization speeds of different modalities, the heterogeneity between the modalities is effectively reduced. The complementary information of the RGB image and the depth image can be fully utilized to obtain rich feature information, thereby improving the performance of the multimodal dynamic gesture recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a dynamic gesture recognition method based on multimodal fusion according to the present invention;

[0039] Figure 2 This is a structural diagram of the multimodal gesture recognition model; DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0041] Embodiment 1:

[0042] like Figure 1 As shown, the embodiment of the present invention provides a dynamic gesture recognition optimization algorithm based on multimodal fusion, comprising the following steps:

[0043] Step 1: Obtain 80% of the dynamic gesture data set collected by the realsense 3D camera as training samples, including video streams in two modes: RGB image and depth image;

[0044] Step 2, preprocessing the training samples;

[0045] Step 3: Construct a multimodal gesture classification and recognition model for the subjects. Its structure is as follows: Figure 2 As shown;

[0046] Step 4, using the dynamic gradient modulation method to optimize the algorithm, inputting the preprocessed training samples into the constructed multimodal gesture classification and recognition model for training;

[0047] Step 5: Obtain the remaining 20% ​​of the dynamic gesture data to be recognized, and after preprocessing, input it into the trained multimodal gesture classification and recognition model to obtain the classification and recognition results.

[0048] Preferably, preprocessing the training samples includes the following specific steps:

[0049] Capture an RGB video stream containing T frames and deep video stream in Represents the nth frame RGB image in the video stream, representing The depth image of the nth frame in the video stream; set the maximum number of frames per video sample to 16 and the stride of the sampler to 3; use a center crop of size 224×224 pixels for all frames and perform data augmentation, including random cropping, color jittering, and random horizontal flipping; use a center crop of size 224×224 pixels for all frames to create a shape x∈R B×3×16×224×224 batches, where B represents the batch size; use the content adaptive sampler to remove video frames without gestures, and then rearrange the remaining video frames to obtain And N is much smaller than T;

[0050] Preferably, the multimodal gesture classification and recognition model in step 3 includes an R(2+1)D feature extraction module, a Transformer encoder module, a feature fusion module, and a classification module:

[0051] The feature extraction module uses two pre-trained R(2+1)D models f R (w R ,·),f D (wD , ·) as the gesture feature extractor for RGB video stream and depth video stream, from the preprocessed RGB video stream X R and Deep Video StreamX D Extract the spatiotemporal feature vector sequence F from R 、F D , where w R Feature extraction network f R The model parameters, w D Feature extraction network f R Model parameters of

[0052] The Transformer encoder module extracts the spatiotemporal feature vector sequence F of the two modalities through the R(2+1)D model. R 、F D The fixed position encoding is performed using sine and cosine functions based on different frequencies to obtain the position information of the elements in the sequence, and then the encoder is composed of multiple multi-head self-attention modules and fully connected feedforward neural network layers stacked together. Capture the gesture relationship between the previous and next frames in different sequences to form a feature vector z of the same dimension R , z D , where θ R For feature extraction network The model parameters, θ D For feature extraction network Model parameters of

[0053] The feature fusion module uses a splicing method to fuse the feature vector z extracted from the RGB video stream and the depth video stream. R and z D , generate the fused feature vector [z R +z D ];

[0054] The classification module uses a multilayer perceptron as the classifier F C , classify the fused features and generate prediction results

[0055] Preferably, in step 4, the dynamic gradient modulation method is used to optimize the algorithm, and the preprocessed training samples are input into the constructed multimodal gesture classification and recognition model for training, which specifically includes the following steps:

[0056] Step 4.1, define the model loss function as the cross entropy function L, where the RGB image loss function is L R , the depth image loss function is L D , the optimization function uses the common stochastic gradient descent algorithm:

[0057]

[0058]

[0059] Where y(i) is the true label of the i-th training sample; is the eigenvector z R The prediction result after the classifier iterates for the tth time, is the eigenvector z D The prediction result after the classifier iterates for the tth time; To predict the results The probability of being the same as the true outcome y(i); To predict the results is the probability of the kth gesture category; N is the number of training samples, M is the number of gesture categories; Step 4.2, calculate the Transformer encoder parameter θ corresponding to the RGB image R Gradient Transformer encoder parameters θ corresponding to the depth image D Gradient

[0060]

[0061]

[0062] Step 4.3, calculate the two-modal loss function L of RGB image and depth image R , L D The slope difference ratio at different iteration numbers

[0063]

[0064]

[0065] Step 4.4, calculate the adjustment factor during RGB image training Adjustment factors in the deep image training process

[0066]

[0067]

[0068] In the formula, α is a hyperparameter that controls the degree of adjustment and is set to 0.6;

[0069] Step 4.5: Update the gradients of the two modes using the dynamic gradient adjustment algorithm

[0070]

[0071]

[0072] Where η is the learning rate, which is 0.001.

Claims

1. A dynamic gesture recognition method based on multimodal fusion, characterized in that: The steps include: Step 1: Use the realsense3D camera to collect a dynamic gesture dataset as training samples, including video streams in two modes: RGB image and depth image. Step 2, preprocessing the training samples; Step 3, construct a multimodal gesture classification and recognition model; Step 4, using the dynamic gradient modulation method to optimize the algorithm, inputting the preprocessed training samples into the constructed multimodal gesture classification and recognition model for training; Step 5: obtain the dynamic gesture data to be recognized, and input it into the trained multimodal gesture classification and recognition model after preprocessing to obtain the classification and recognition result.

2. The method for dynamic gesture recognition based on multimodal fusion according to claim 1, characterized in that: In step 2, the preprocessing of the training samples includes: Given an RGB video stream containing T frames and deep video stream in Represents the nth frame RGB image in the video stream, representing The nth depth image of the video stream is obtained by using a content adaptive sampler to remove video frames without gestures and then rearranging the remaining video frames. And N is much smaller than T.

3. The method for dynamic gesture recognition based on multimodal fusion according to claim 1 is characterized in that In step 3, a multimodal gesture classification and recognition model is constructed, including an R(2+1)D feature extraction module, a Transformer encoder module, a feature fusion module, and a classification module; The feature extraction module uses two pre-trained R(2+1)D models f R (w R ,·),f D (w D , ·) as the gesture feature extractor for RGB video stream and depth video stream, from the preprocessed RGB video stream X R and Deep Video StreamX D Extract the spatiotemporal feature vector sequence F from R 、F D , where w R Feature extraction network f R The model parameters, w D Feature extraction network f R Model parameters of The Transformer encoder module extracts the spatiotemporal feature vector sequence F of the two modalities through the R(2+1)D model. R 、F D The fixed position encoding is performed using sine and cosine functions based on different frequencies to obtain the position information of the elements in the sequence, and then the encoder is composed of multiple multi-head self-attention modules and fully connected feedforward neural network layers stacked together. Capture the gesture relationship between the previous and next frames in different sequences to form a feature vector z of the same dimension R , z D , where θ R For feature extraction network The model parameters, θ D For feature extraction network Model parameters of The feature fusion module uses an additive method to fuse the feature vector z extracted from the RGB video stream and the depth video stream. R and z D , generate the fused feature vector [z R +z D ]; The classification module uses a multilayer perceptron as the classifier F C , classify the fused features and generate prediction results 4. The method for dynamic gesture recognition based on multimodal fusion according to claim 1, characterized in that: In step 4, the method of using the dynamic gradient modulation method to optimize the algorithm and inputting the preprocessed training samples into the constructed multimodal gesture classification and recognition model for training specifically includes the following steps: Step 4.1, define the model loss function as the cross entropy function L, where the RGB image loss function is L R , the depth image loss function is L D , the optimization function uses the common stochastic gradient descent algorithm: Where y(i) is the true label of the i-th training sample; is the eigenvector z R The prediction result after the classifier iterates for the tth time, is the eigenvector z D The prediction result after the classifier iterates for the tth time; To predict the results The probability of being the same as the true outcome y(i); To predict the results is the probability of the kth gesture category; N is the number of training samples, and M is the number of gesture categories; Step 4.2, calculate the Transformer encoder parameters θ corresponding to the RGB image R Gradient Transformer encoder parameters θ corresponding to the depth image D Gradient Step 4.3, calculate the slope difference ratio of the two modal numbers of RGB image and depth image Step 4.4, calculate the adjustment factor during RGB image training Adjustment factors in the deep image training process In the formula, α is a hyperparameter that controls the degree of adjustment; Step 4.5: Update the gradients of the two modes using the dynamic gradient adjustment algorithm Where η is the learning rate.

Images