Intelligent Gesture Recognition Method and System Based on Multimodal Large Model

By fusing displacement and depth features of key hand points using a multimodal large model and combining them with dynamic driving condition information, the problem of misrecognition in complex environments for intelligent gesture recognition has been solved, achieving higher recognition accuracy and robustness.

CN122867256BActive Publication Date: 2026-11-10JIAJIE TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611343179.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-09-01
Publication Date
2026-11-10
Estimated Expiration
2046-09-01

AI Technical Summary

Technical Problem

Existing intelligent gesture recognition methods cannot accurately identify observable differences in key hand points under complex lighting conditions and different driving conditions, resulting in gesture recognition errors and low confidence.

Method used

A multimodal large model is adopted. By acquiring the displacement sequence and depth feature vector of key hand points and combining them with dynamic driving condition information, a multimodal gesture recognition model is constructed. The gesture recognition results are weighted and fused to improve the model's adaptability and robustness to different driving conditions.

Benefits of technology

It significantly improves the accuracy and robustness of gesture recognition, enhances the model's adaptability to complex environments, and improves recognition accuracy in boundary conditions and mixed scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122867256B_ABST
    Figure CN122867256B_ABST
Patent Text Reader

Abstract

This application relates to the field of data processing, and particularly to an intelligent gesture recognition method and system based on a multimodal large model. The method includes: acquiring video data, gesture labels, and state labels for each historical gesture execution process, and identifying the recognition weights and displacement feature vectors of key hand points; acquiring depth images of each historical gesture execution process and obtaining depth feature vectors for that historical gesture execution process; obtaining a multimodal gesture recognition model based on the displacement feature vectors and depth feature vectors; obtaining the discrimination level of each preset dimension; acquiring the real-time sequence of the current gesture, and obtaining the occurrence probability distribution of each state label based on the discrimination level; obtaining the gesture recognition result of the current gesture execution process based on the multimodal gesture recognition model, and weighting and fusing the gesture recognition result according to the occurrence probability distribution to obtain the recognized gesture. The technical solution of this application can improve the accuracy of gesture recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing. In particular, it relates to an intelligent gesture recognition method and system based on a multimodal large model. Background Technology

[0002] As the core interactive carrier of intelligent driving, the intelligent cockpit allows for functional interaction through specific gestures during driving. Gesture recognition technology, due to its natural, intuitive, and non-contact characteristics, has become an important means to improve driving safety and user experience.

[0003] Existing gesture recognition methods are mainly divided into three categories: rule-based template matching, traditional machine learning, and deep learning. They employ globally fixed feature extraction methods, relying on fixed keypoint topology and skin color segmentation extracted by static classification models, as well as artificial features such as HOG and SIFT. These methods do not fully consider the impact of dynamic changes in driving conditions on the same gesture expression. Under different driving conditions, such as high-speed cruising, low-speed congestion, fatigued driving, and slippery conditions in rain, the amplitude, speed, degree of tremor, and observability of key points of a driver's gestures vary significantly. This leads to large differences in the feature distribution of the same gesture under different conditions, making the model prone to misidentification or low confidence. Summary of the Invention

[0004] This application provides an intelligent gesture recognition method and system based on a multimodal large model, which solves the technical problem that existing intelligent gesture recognition methods rely on pixel features and cannot accurately identify the observable differences of key hand points under different driving conditions in complex lighting and driving environments. This leads to hand amplitude offset errors caused by driving conditions affecting actual gesture recognition and resulting in model recognition errors.

[0005] In the first aspect, the intelligent gesture recognition method based on a multimodal large model includes: Acquire video data, gesture tags, and status tags for each historical gesture execution process, and obtain the displacement sequence of each key hand point based on the video data; Based on the displacement sequence and the gesture label, the recognition weight of each hand key point under each state label is obtained; Acquire a depth image of each of the historical gesture execution processes, and fuse the depth images according to the recognition weights to obtain a depth feature vector of each of the historical gesture execution processes; The displacement feature vector of each of the key hand points is obtained based on the displacement sequence, and a multimodal gesture recognition model is constructed and trained by combining the depth feature vector. Obtain the historical sequence of each preset dimension during the execution of each historical gesture, and obtain the degree of distinction of each preset dimension based on the historical sequence; The real-time sequence of each preset dimension during the current gesture execution process is obtained. The historical sequence and the real-time sequence are fused according to the degree of differentiation to obtain the motion similarity. The occurrence probability distribution of each state label is obtained based on the motion similarity. The gesture recognition result of the current gesture execution process is obtained based on the trained multimodal gesture recognition model, and the gesture recognition result is weighted and fused according to the occurrence probability distribution to obtain the recognized gesture.

[0006] Preferably, acquiring video data, gesture tags, and status tags for each historical gesture execution process, and obtaining the displacement sequence of each hand key point based on the video data includes: Video data of each historical gesture execution process is collected by a depth camera, and the gesture label and status label of each historical gesture execution process are obtained through the central system. The acquired video data is extracted frame by frame to obtain multiple RGB images. A multi-person pose estimation model is used to extract hand key points and their 3D coordinates. Based on the Euclidean distance between the 3D coordinates of hand key points in adjacent RGB images, the displacement sequence of each hand key point is obtained.

[0007] Preferably, obtaining the recognition weight of each hand key point under each state label based on the displacement sequence and the gesture label includes: Calculate the normalized DTW distance between the displacement sequences of each hand key point during the execution of each historical gesture, and fuse the normalized DTW distances to obtain the recognition score of each hand key point under each state label; The recognition scores are summed and normalized to obtain the recognition weight of each hand key point under each state label.

[0008] Preferably, acquiring a depth image of each historical gesture execution process and fusing the depth images according to the recognition weights to obtain a depth feature vector for each historical gesture execution process includes: Depth video of each historical gesture execution process is acquired by a depth camera, and depth image of each historical gesture execution process is obtained by extracting the depth video frame by frame. Based on the depth value of each hand key point in each depth image, a depth sequence of each hand key point during each historical gesture execution is obtained. The depth sequences are then fused according to the recognition weights to obtain a depth feature vector for each historical gesture execution process.

[0009] Preferably, obtaining the displacement feature vector of each of the hand key points based on the displacement sequence, and constructing and training a multimodal gesture recognition model by combining the depth feature vector includes: The displacement sequence is fused according to the recognition weight to obtain the displacement feature vector of each historical gesture execution process; A multimodal gesture recognition model is obtained by establishing a displacement temporal coding branch, a depth feature temporal coding branch, and a fusion classification network. The displacement feature vector and depth feature vector are used as model inputs to train the constructed multimodal gesture recognition model, resulting in a trained multimodal gesture recognition model.

[0010] Preferably, obtaining the historical sequence of each preset dimension during the execution of each historical gesture, and obtaining the distinguishability of each preset dimension based on the historical sequence includes: By using multiple sensors to collect historical sequences of each preset dimension during the execution of each historical gesture, the DTW distance is calculated based on the historical sequences to obtain the discrimination score of each preset dimension, and the discrimination scores are fused to obtain the discrimination degree of each preset dimension.

[0011] Preferably, acquiring the real-time sequence of each preset dimension during the current gesture execution process, and fusing the historical sequence and the real-time sequence according to the degree of differentiation to obtain motion similarity includes: By using multiple sensors to collect real-time sequences of each preset dimension during the current gesture execution process, calculating the distance mapping value between the historical sequence and the real-time sequence, and fusing the distance mapping value using the distinguishability, the motion similarity between the current gesture execution process and each of the historical gesture execution processes is obtained.

[0012] Preferably, obtaining the probability distribution of each state label based on the motion similarity includes: The motion similarity between the current gesture execution process and each of the historical gesture execution processes is fused to obtain the working condition similarity between the current gesture execution process and each of the state labels. Based on the working condition similarity, the probability distribution of the occurrence of each of the state labels during the current gesture execution process is obtained.

[0013] Preferably, the gesture recognition result of the current gesture execution process is obtained based on the trained multimodal gesture recognition model, and the gesture recognition result is weighted and fused according to the occurrence probability distribution to obtain the recognized gesture, including: Based on the recognition weights of each hand key point under each state label, the displacement feature vector and depth feature vector of the current gesture execution process under each state label are constructed respectively, and input into the trained multimodal gesture recognition model to obtain the gesture recognition result under each state label; The probability of occurrence of each gesture label in the gesture recognition result is weighted and fused according to the probability distribution of occurrence of each state label during the current gesture execution process to obtain the recognized gesture of the current gesture execution process.

[0014] Secondly, an intelligent gesture recognition system based on a multimodal large model includes: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the intelligent gesture recognition method based on a multimodal large model described in any one of the claims is implemented.

[0015] This application has the following effects: 1. This application calculates the recognition weight of each hand key point under each state label based on displacement sequence and gesture label, and adaptively fuses depth features and displacement features. This significantly enhances the model's ability to perceive differences in the contribution of key points under different driving conditions, effectively suppresses noise interference in complex environments, and improves the accuracy and robustness of gesture recognition. By introducing motion similarity calculations between historical and real-time sequences of dynamic features such as vehicle speed, acceleration, steering torque, and yaw rate, the probability distribution of each state label is obtained. This achieves a driving condition probability-weighted fusion of gesture recognition results, enabling the system to dynamically adjust its decision-making strategy according to the current driving condition, significantly improving the accuracy of gesture recognition in boundary conditions and mixed scenarios.

[0016] 2. This application constructs a multimodal architecture combining displacement temporal coding and depth feature temporal coding branches. Utilizing dynamic trajectory information from video data and 3D structural information from depth images, combined with motion similarity calculations between historical and real-time sequences, it effectively captures the continuity and temporal dependencies of gestures. This solves the problem of poor recognition performance for incomplete or continuous complex gestures in existing technologies, improving the system's adaptability to natural interaction scenarios. Furthermore, by learning the discriminative power of each preset dimension through historical data, it enhances the model's adaptability to variations across different working conditions and users. Compared to existing methods that rely on single-modality approaches or simple splicing and fusion, this application improves generalization ability in scenarios with occlusion, lighting changes, and long-term gestures, reduces data dependence, and enhances model interpretability and gesture recognition accuracy.

[0017] The above and other objects, advantages and features of this application will become more apparent to those skilled in the art from the following detailed description of specific embodiments of this application in conjunction with the accompanying drawings. Attached Figure Description

[0018] The following sections will describe some specific embodiments of this application in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic flowchart of an intelligent gesture recognition method based on a multimodal large model according to an embodiment of this application; Figure 2 This is a schematic structural diagram of an intelligent gesture recognition system based on a multimodal large model according to an embodiment of this application. Detailed Implementation

[0019] The following reference Figures 1 to 2 This application describes an intelligent gesture recognition method and system based on a multimodal large model, according to embodiments of the present application. In the description of these embodiments, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of that feature, that is, include one or more of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. When a feature "includes or contains" one or more of the features it encompasses, unless otherwise specifically described, this indicates that other features are not excluded and may be further included.

[0020] In the description of this embodiment, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0021] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0022] Please see Figure 1 The schematic flowchart of the intelligent gesture recognition method based on a multimodal large model includes steps S1-S7, as follows: S1: Obtain video data, gesture tags, and status tags for each historical gesture execution process, and obtain the displacement sequence of each hand key point based on the video data.

[0023] Due to the complex and ever-changing driving environment, vehicle motion status labels (driving condition labels) can provide contextual information about the occurrence of gestures. The expression, motion characteristics, and underlying intentions of the same gesture vary significantly under different driving conditions. For example, during high-speed cruising, the vehicle speed is high, acceleration is stable, and steering torque fluctuations are small; the gesture amplitude is usually small but easily produces micro-tremors due to vehicle vibrations. In low-speed congested traffic or fatigued driving conditions, frequent vehicle starts and stops, increased steering torque fluctuations, and higher acceleration variance can lead to gestures with reduced amplitude, incomplete trajectories, or unconscious movements. In complex road conditions such as wet and slippery conditions, hand tremor noise increases significantly. Video data is the core visual input source for gesture recognition. Video data collected through in-vehicle RGB-D or multi-camera systems can provide rich information about the hand's appearance, spatial posture, and temporal dynamics, including hand trajectory, motion amplitude, joint angle changes, micro-tremors, and the relative positional relationship between the hand and the cabin environment such as the steering wheel and seat. This information is crucial for extracting skeletal keypoint sequences and reconstructing complete gesture motion patterns. Without video data, the system will struggle to perform reliable visual perception and feature extraction.

[0024] An RGB-D depth camera is installed in the center of the roof, with the shooting angle facing the driver's upper body, to collect video data of each historical gesture execution process. The status label and gesture label of each historical gesture execution process are obtained through the central system.

[0025] Specifically, the OpenPose multi-person pose estimation model adopts a fully convolutional multi-stage network architecture. First, the region of interest of the hand determined by the wrist position detected by the human model is cropped from the RGB image and scaled to 368×368×3 as input; the input is passed through the first 9×9 convolutional layer and the max pooling layer to 184×184×64, and then downsampled to 92×92×128 through the 7×7 convolutional layer. Subsequently, it is passed through 3×3 convolution and 1×1 convolution to construct 6 cascaded stages. Stage 1 outputs an initial confidence heatmap (size 46×46×22, ​​21 hand key points + 1 background channel). Stages 2 to 6 concatenate the original feature map with the 22-channel heatmap predicted in the previous stage along the channel dimension, and then refine it through convolutional layers to output a 46×46×22 confidence heatmap of the same size. L2 intermediate supervision is applied at the end of each stage to prevent gradient vanishing. Finally, by performing non-maximum suppression and sub-pixel peak localization on the heatmap output by Stage 6, the coordinates (x,y) of 21 key points of one hand and their confidence scores are obtained. The network is called independently for detection of the left and right hands respectively.

[0026] The video data of each historical gesture execution process is extracted frame by frame to obtain multiple RGB images of each historical gesture execution process. The RGB images are input into the OpenPose multi-person pose estimation model to obtain 21 hand key points and the two-dimensional coordinates of the hand key points. The two-dimensional coordinates are mapped to three-dimensional coordinates through depth alignment to obtain the three-dimensional coordinates of each hand key point.

[0027] Taking one of the hand key points as an example, let the hand key point be the target key point. Calculate the Euclidean distance between the three-dimensional coordinates of the target key point in any two adjacent RGB images to obtain the displacement sequence of the target key point in each historical gesture execution process.

[0028] Based on the above steps for obtaining the displacement sequence, the displacement sequence of each hand key point in each historical gesture execution process is obtained.

[0029] Specifically, the status tags include high-speed cruising, low-speed congestion, fatigued driving, wet and slippery weather, and normal driving; the gesture tags include eight tags: horizontal waving left and right, vertical waving up and down, drawing a circle clockwise or counterclockwise, and pinching and opening two fingers. These tags are combined in pairs to map media track switching, volume or temperature adjustment, and map zoom control, respectively.

[0030] S2: Based on the displacement sequence and the gesture label, obtain the recognition weight of each hand key point under each state label.

[0031] Taking one of the state labels as an example, let the state label be the target state. Calculate the normalized DTW distance between the displacement sequences of target key points during the historical gesture execution process of any two different gesture labels in the target state. Calculate the average value of the normalized DTW distance to obtain the recognition score of the target key points in the target state.

[0032] Specifically, the normalized DTW distance is implemented as follows: The optimal alignment path is found through dynamic programming to minimize the cumulative distance between corresponding points in two sequences, thus obtaining the standard DTW distance. This distance is then divided by the length of the alignment path or the total length of the sequence to eliminate the influence of sequence length and the number of path points on the distance metric, thereby obtaining a scale-independent normalized DTW distance value. The normalized distance value reflects the degree of difference in the motion trajectories of key points between different gestures. A larger value indicates a more significant difference in the motion of the key point when distinguishing different gestures, and a higher recognition importance; conversely, a smaller value indicates a lower importance. This serves as the basis for subsequent calculation of key point recognition weights.

[0033] Based on the above steps for obtaining recognition scores, the recognition score of each hand key point in the target state is obtained, and the recognition scores are summed and normalized to obtain the recognition weight of each hand key point in the target state.

[0034] Specifically, the summation and normalization method is as follows: the sum of the recognition scores of each hand key point in the target state is used as the denominator, the recognition score of the target key point in the target state is used as the numerator, and the ratio is calculated as the recognition weight of the target key point in the target state to achieve summation and normalization.

[0035] Based on the above steps for obtaining recognition weights, the recognition weights of each hand key point under each state label are obtained.

[0036] S3: Obtain the depth image of each of the historical gesture execution processes, and fuse the depth images according to the recognition weight to obtain the depth feature vector of each of the historical gesture execution processes.

[0037] Depth video of each historical gesture execution process is acquired using an RGB-D depth camera, and multiple depth images are obtained by extracting the depth video of each historical gesture execution process frame by frame.

[0038] Taking one historical gesture execution process as an example, let this process be the target process. The depth sequence of the target key points is obtained based on the depth value of each depth image in the target process. Following the above depth sequence acquisition steps, the depth sequence of each hand key point in the target process is obtained.

[0039] The product of the recognition weight of the target key points under the state label of the target process and each element in the depth sequence of the target key points is calculated to obtain the depth feature sequence of the target key points in the target process. Based on the above steps for obtaining the depth feature sequence, the depth feature sequence of each hand key point in the target process is obtained.

[0040] The depth feature sequence of each of the hand key points is concatenated to obtain the depth feature vector of the target process. Similarly, the depth feature vector of each historical gesture execution process is obtained.

[0041] S4: Obtain the displacement feature vector of each of the key hand points according to the displacement sequence, and combine it with the depth feature vector to construct and train a multimodal gesture recognition model.

[0042] The product of the identification weight of the target key point under the state label in the target process and each element in the displacement sequence of the target key point is calculated to obtain the displacement feature sequence of the target key point in the target process. Based on the above displacement feature sequence acquisition steps, the displacement feature sequence of each hand key point in the target process is obtained.

[0043] The displacement feature sequence of each of the aforementioned hand key points is concatenated to obtain the displacement feature vector of the target process. Similarly, the displacement feature vector of each historical gesture execution process is obtained.

[0044] A multimodal gesture recognition model is obtained by establishing a displacement temporal encoding branch, a deep feature temporal encoding branch, and a fusion classification network. The displacement feature vector of the target process is input into the displacement temporal encoding branch, and the displacement temporal features are extracted through a long short-term memory network. The deep feature vector of the target process is input into the deep feature temporal encoding branch, and the deep temporal features are extracted through a one-dimensional convolutional neural network. The displacement temporal features and the deep temporal features are concatenated along the feature dimension and input into a pooling layer to obtain a fixed-dimensional fusion feature vector, which is then input into a multi-classifier to output the probability of each gesture label.

[0045] Specifically, the displacement temporal coding branch takes the displacement feature vector as input and feeds it into a recurrent encoder consisting of two layers of Long Short-Term Memory (LSTM) networks. The first LSTM layer has 128 hidden units and outputs the complete time-series hidden states, while the second LSTM layer has 64 hidden units and outputs only the hidden state vector of the last frame. A temporal max-pooling layer is then connected after the LSTM layer to maximize the feature response in the time dimension, resulting in a fixed-dimensional displacement temporal feature vector used to characterize the overall trend of the gesture motion trajectory.

[0046] Furthermore, the deep feature temporal coding branch takes the deep feature vector as input and feeds it into a deep encoder consisting of three layers of one-dimensional convolutional neural networks. The first layer has a kernel size of 3 and 64 output channels, the second layer has a kernel size of 3 and 32 output channels, and the third layer has a kernel size of 3 and 16 output channels. Each layer is followed by a ReLU activation function and a batch normalization layer. After the convolutional layers, a global average pooling layer is connected to compress the temporal dimension into a single deep feature vector, which is then mapped to the same feature dimension as the displacement temporal feature through a fully connected layer to obtain a deep temporal feature vector, which is used to characterize the spatial depth distribution characteristics of the gesture.

[0047] Furthermore, the fusion classification network takes the concatenation result of displacement temporal feature vector and depth temporal feature vector as input. First, it concatenates the two types of vectors into a joint feature vector in the feature dimension and inputs it into a classifier consisting of three fully connected layers. The input dimension of the first fully connected layer is equal to the dimension of the joint feature vector, and the output dimension is 512. It is followed by a ReLU activation function and a Dropout regularization layer to prevent overfitting. The output dimension of the second fully connected layer is 256, followed by a ReLU activation function. The third fully connected layer is the output layer, and its output dimension is equal to the total number of predefined gesture categories. The Softmax activation function is used to map the output to the probability distribution of each gesture. During the training phase, the cross-entropy loss function is used to measure the difference between the predicted probability distribution and the real gesture label, and the network parameters are iteratively updated through the backpropagation algorithm. During the inference phase, the gesture category with the highest probability value is selected as the final recognition result.

[0048] The displacement and depth feature vectors of each historical gesture execution process are used as training data. Cross-entropy is selected as the model loss function. The network parameters are iteratively updated until the set number of iterations is reached and training stops. At the same time, if the validation loss is detected to be continuously increasing during training, an early stopping mechanism is triggered to prevent overfitting. The model weights at the point where the validation loss is the lowest are used as the final model parameters after training, thus obtaining the trained multimodal gesture recognition model.

[0049] S5: Obtain the historical sequence of each preset dimension during the execution of each historical gesture, and obtain the degree of distinction of each preset dimension based on the historical sequence.

[0050] Different driving conditions alter the force, occlusion, and degrees of freedom of the hand, leading to systematic changes in the observability and discrimination contribution of key points under the same gesture. For example, during high-speed cruising or sharp turns, the driver's arm is restricted by inertial forces and steering operations, reducing the range of motion of the little and ring fingers and making them prone to occlusion. This makes the thumb, index finger, and wrist key points the primary source of information for gesture recognition. In contrast, during low-speed, stable driving, the finger joints move more freely, significantly increasing the discrimination contribution of distal phalanges. By analyzing the probability distribution of the current gesture execution under different driving conditions using dynamic data, and applying anisotropic confidence levels to the recognition weights of key points under different driving conditions, noise interference caused by vibration, occlusion, and posture distortion can be effectively suppressed, improving the robustness and accuracy of gesture recognition in complex driving environments.

[0051] Specifically, the preset dimensions include vehicle speed, acceleration, steering torque, and yaw rate.

[0052] The vehicle uses multiple built-in sensors to collect time-series data for each preset dimension during the execution of each historical gesture. The time-series data for each preset dimension is then Z-score standardized to obtain the historical sequence for each preset dimension. The acquisition of time-series data and video data follow strict time synchronization based on a timestamp synchronization mechanism.

[0053] Specifically, the Z-score standardization process is implemented by calculating the standard deviation of the sequence to which the value to be standardized belongs as the denominator, calculating the difference between the value to be standardized and the mean of the sequence as the numerator, and using the ratio of the two as the standard value to achieve standardization.

[0054] Taking one of the preset dimensions as an example, let the preset dimension be the target dimension. Calculate the DTW distance between the historical sequences of the target dimension during the execution of each two historical gestures with different state labels, and calculate the mean of the DTW distance to obtain the discrimination score of the target dimension.

[0055] Specifically, the DTW distance is implemented as follows: calculate the Euclidean distance between each point in the two historical sequences point by point, recursively find the optimal alignment path from the start point to the end point of the sequence, such that the sum of the cumulative local distances along the path is minimized, and the minimum cumulative distance obtained is the DTW distance.

[0056] Following the steps outlined above for obtaining the discrimination scores, a discrimination score for each preset dimension is obtained. The discrimination scores for each preset dimension are then summed and normalized to obtain the discrimination weight for each preset dimension.

[0057] Specifically, the implementation of the summation normalization is as follows: the sum of the discrimination scores of each preset dimension is used as the denominator, the discrimination score of each preset dimension is used as the numerator, and the ratio is calculated as the recognition weight of each preset dimension to achieve summation normalization.

[0058] S6: Obtain the real-time sequence of each preset dimension during the current gesture execution process, fuse the historical sequence and the real-time sequence according to the degree of differentiation to obtain motion similarity, and obtain the occurrence probability distribution of each state label based on the motion similarity.

[0059] Based on the above steps for obtaining historical sequences, the real-time sequence of each preset dimension during the current gesture execution process is obtained. The DTW distance between the real-time sequence of the target dimension during the current gesture execution process and the historical sequence of the target dimension during the target process is calculated. The DTW distance is subjected to negative exponential mapping. The sum of the products between the negative exponential mapping value of the DTW distance on each preset dimension and the distinguishing weight of each preset dimension is calculated to obtain the motion similarity between the current gesture execution process and the target process.

[0060] Specifically, the motion similarity satisfies the following relationship: In the formula, This indicates the motion similarity between the current gesture execution process and the target process. Indicates the number of preset dimensions. Indicates the first Distinguishing weights for each preset dimension This indicates that the current gesture execution process is related to the target process in the [number]th step. The negative exponential mapping value of the DTW distance between preset dimensions.

[0061] Based on the above steps for obtaining the similarity of working conditions, the motion similarity between the current gesture execution process and each historical gesture execution process is obtained. The mean of the motion similarity between the current gesture execution process and each historical gesture execution process under the same state label is calculated to obtain the working condition similarity between the current gesture execution process and each state label. The working condition similarity is summed and normalized to obtain the probability distribution of the occurrence of each state label in the current gesture execution process.

[0062] Specifically, the probability distribution is implemented as follows: the sum of the similarities of all working conditions is calculated as the denominator, and the similarity of the current gesture execution process with each of the states is calculated as the numerator to obtain the probability of each state label occurring in the current gesture execution process. Based on the probability, the probability distribution of each state label occurring in the current gesture execution process is constructed.

[0063] S7: Obtain the gesture recognition result of the current gesture execution process based on the trained multimodal gesture recognition model, and perform weighted fusion on the gesture recognition result according to the occurrence probability distribution to obtain the recognized gesture.

[0064] Based on the above steps for obtaining displacement feature vectors and depth feature vectors, the recognition weights of each hand key point under each state label are used to obtain the displacement feature vectors and depth feature vectors when each state label occurs during the current gesture execution process. These are then input into the trained multimodal gesture recognition model, and the gesture recognition results under each state label during the current gesture execution process are output.

[0065] Specifically, the gesture recognition result is a probability distribution composed of the occurrence probability of each gesture label in the current gesture execution process output by the model.

[0066] Taking one of the gesture tags as an example, let the gesture tag be the target gesture. Calculate the product between the probability of each of the states occurring during the current gesture execution process and the probability of the target gesture occurring under each state tag. Calculate the sum of the products to obtain the recognition confidence of the target gesture occurring during the current gesture execution process.

[0067] Based on the above steps for obtaining recognition confidence, the recognition confidence of each gesture label occurring during the current gesture execution process is obtained, and the gesture label with the highest recognition confidence is selected as the recognition gesture of the current gesture execution process.

[0068] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0069] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0070] This application also provides an intelligent gesture recognition system based on a multimodal large model. For example... Figure 2 As shown, the system includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the intelligent gesture recognition method based on a multimodal large model according to the first aspect of this application. The system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface. Their configuration and functions are known in the art and will not be described in detail here.

[0071] Therefore, those skilled in the art should recognize that although many exemplary embodiments of this application have been shown and described in detail herein, many other variations or modifications conforming to the principles of this application can be directly determined or derived from the disclosure of this application without departing from the spirit and scope of this application. Thus, the scope of this application should be understood and construed as covering all such other variations or modifications.

Claims

1. A method for intelligent gesture recognition based on a multimodal large model, characterized in that, include: The system acquires video data, gesture tags, and status tags for each historical gesture execution process. Based on the video data, it acquires the displacement sequence of each key hand point. The gesture tags include horizontal left and right waving, vertical up and down waving, clockwise or counterclockwise circle drawing, and two-finger pinching and opening. The status tags include high-speed cruising, low-speed congestion, fatigued driving, wet and slippery weather, and normal driving. Based on the displacement sequence and the gesture label, the recognition weight of each hand key point under each state label is obtained; Acquire a depth image of each of the historical gesture execution processes, and fuse the depth images according to the recognition weights to obtain a depth feature vector of each of the historical gesture execution processes; The displacement feature vector of each of the key hand points is obtained based on the displacement sequence, and a multimodal gesture recognition model is constructed and trained by combining the depth feature vector. The historical sequence of each preset dimension during the execution of each historical gesture is obtained, and the degree of differentiation of each preset dimension is obtained based on the historical sequence, wherein the preset dimension includes vehicle speed, acceleration, steering torque, and yaw rate; The real-time sequence of each preset dimension during the current gesture execution process is obtained. The historical sequence and the real-time sequence are fused according to the degree of differentiation to obtain the motion similarity. The occurrence probability distribution of each state label is obtained based on the motion similarity. The gesture recognition result of the current gesture execution process is obtained based on the trained multimodal gesture recognition model, and the gesture recognition result is weighted and fused according to the occurrence probability distribution to obtain the recognized gesture.

2. The intelligent gesture recognition method based on a multimodal large model according to claim 1, characterized in that, The process of acquiring video data, gesture tags, and status tags for each historical gesture execution process, and acquiring the displacement sequence of each hand key point based on the video data, includes: Video data of each historical gesture execution process is collected by a depth camera, and the gesture label and status label of each historical gesture execution process are obtained through the central system. The acquired video data is extracted frame by frame to obtain multiple RGB images. A multi-person pose estimation model is used to extract hand key points and their 3D coordinates. Based on the Euclidean distance between the 3D coordinates of hand key points in adjacent RGB images, the displacement sequence of each hand key point is obtained.

3. The intelligent gesture recognition method based on a multimodal large model according to claim 1, characterized in that, The step of obtaining the recognition weight of each hand key point under each state label based on the displacement sequence and the gesture label includes: Calculate the normalized DTW distance between the displacement sequences of each hand key point during the execution of each historical gesture, and fuse the normalized DTW distances to obtain the recognition score of each hand key point under each state label; The recognition scores are summed and normalized to obtain the recognition weight of each hand key point under each state label.

4. The intelligent gesture recognition method based on a multimodal large model according to claim 1, characterized in that, The step of acquiring a depth image for each historical gesture execution process and fusing the depth images according to the recognition weights to obtain a depth feature vector for each historical gesture execution process includes: Depth video of each historical gesture execution process is acquired by a depth camera, and depth image of each historical gesture execution process is obtained by extracting the depth video frame by frame. Based on the depth value of each hand key point in each depth image, a depth sequence of each hand key point during each historical gesture execution is obtained. The depth sequences are then fused according to the recognition weights to obtain a depth feature vector for each historical gesture execution process.

5. The intelligent gesture recognition method based on a multimodal large model according to claim 1, characterized in that, The step of obtaining the displacement feature vector of each of the key hand points based on the displacement sequence, and constructing and training a multimodal gesture recognition model by combining the depth feature vector, includes: The displacement sequence is fused according to the recognition weight to obtain the displacement feature vector of each historical gesture execution process; A multimodal gesture recognition model is obtained by establishing a displacement temporal coding branch, a depth feature temporal coding branch, and a fusion classification network. The displacement feature vector and depth feature vector are used as model inputs to train the constructed multimodal gesture recognition model, resulting in a trained multimodal gesture recognition model.

6. The intelligent gesture recognition method based on a multimodal large model according to claim 1, characterized in that, The step of obtaining the historical sequence of each preset dimension during the execution of each historical gesture, and obtaining the distinguishability of each preset dimension based on the historical sequence, includes: By using multiple sensors to collect historical sequences of each preset dimension during the execution of each historical gesture, the DTW distance is calculated based on the historical sequences to obtain the discrimination score of each preset dimension, and the discrimination scores are fused to obtain the discrimination degree of each preset dimension.

7. The intelligent gesture recognition method based on a multimodal large model according to claim 1, characterized in that, The step of acquiring the real-time sequence of each preset dimension during the current gesture execution process, and fusing the historical sequence and the real-time sequence according to the degree of distinction to obtain motion similarity includes: By using multiple sensors to collect real-time sequences of each preset dimension during the current gesture execution process, calculating the distance mapping value between the historical sequence and the real-time sequence, and fusing the distance mapping value using the distinguishability, the motion similarity between the current gesture execution process and each of the historical gesture execution processes is obtained.

8. The intelligent gesture recognition method based on a multimodal large model according to claim 1, characterized in that, The process of obtaining the probability distribution of each state label based on the motion similarity includes: The motion similarity between the current gesture execution process and each of the historical gesture execution processes is fused to obtain the working condition similarity between the current gesture execution process and each of the state labels. Based on the working condition similarity, the probability distribution of the occurrence of each of the state labels during the current gesture execution process is obtained.

9. The intelligent gesture recognition method based on a multimodal large model according to claim 1, characterized in that, The step of obtaining the gesture recognition result of the current gesture execution process based on the trained multimodal gesture recognition model, and weighting and fusing the gesture recognition result according to the occurrence probability distribution to obtain the recognized gesture, includes: Based on the recognition weights of each hand key point under each state label, the displacement feature vector and depth feature vector of the current gesture execution process under each state label are constructed respectively, and input into the trained multimodal gesture recognition model to obtain the gesture recognition result under each state label; The probability of occurrence of each gesture label in the gesture recognition result is weighted and fused according to the probability distribution of occurrence of each state label in the current gesture execution process to obtain the recognized gesture of the current gesture execution process.

10. An intelligent gesture recognition system based on a multimodal large model, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the steps of the intelligent gesture recognition method based on a multimodal large model according to any one of claims 1-9.

Citation Information

Patent Citations

  • Dynamic interface interaction system based on multi-modal gesture recognition

    CN120762574A

  • Three-dimensional dynamic gesture recognition method and system based on image recognition

    CN121330766A