Dynamic gesture recognition method based on double-branch fusion, model training method and equipment

Abstract

The invention provides a double-branch fusion-based dynamic gesture recognition method, a model training method and equipment, and the recognition method comprises the steps: extracting a plurality of target images containing hand targets from target video data, and generating a first image and a second image corresponding to each target image to form a target image sequence, the resolution of the first image is smaller than that of the second image; inputting the target image sequence into a dynamic gesture recognition model to enable the image feature extraction branch, the hand feature extraction branch and the recognition head to obtain gesture category prediction result data corresponding to the target image sequence; and outputting gesture category prediction result data. According to the method and the device, the problems that the model feature extraction capability is reduced and the gesture recognition precision is not high due to the fact that the distance between the hand and the camera is changed to cause the scale inconsistency of the hand region in the existing dynamic gesture recognition mode can be solved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and artificial intelligence, and in particular to a dynamic gesture recognition method, model training method and device based on dual-branch fusion. Background Technology

[0002] Dynamic gesture recognition is an important research direction in computer vision, aiming to accurately classify hand movements appearing in videos. Unlike static gesture recognition, which classifies only a single image of a hand, dynamic gesture recognition requires classifying a sequence of images composed of a series of hand images. This technology has broad practical significance in applications such as intelligent human-computer interaction, sign language recognition, and virtual reality. Compared with conventional behavior recognition tasks, dynamic gesture recognition places higher demands on the representation of fine-grained spatial features and the modeling of temporal dynamic information. The network not only needs to accurately capture the shape features of the hand but also effectively model the temporal patterns of hand movements. Only by accurately distinguishing both the static form and dynamic changes of the gesture can correct gesture recognition be achieved. In complex environments such as varying distances between the gesture and the camera, and inconsistent hand area sizes, existing methods often struggle to maintain good robustness.

[0003] Current mainstream dynamic gesture recognition algorithms can be broadly categorized into those based on 2D convolutional neural networks (2D CNN), 3D convolutional neural networks (3D CNN), recurrent neural networks (RNN), and hybrid network structures. 2D CNN methods model spatial features using static images or motion-enhanced frames, making them suitable for modeling gestures with low temporal dependence. 3D CNNs can capture both spatial and temporal features simultaneously, but are sensitive to scale changes. RNNs and their variants improve the understanding of action temporality through sequence modeling, but are highly dependent on the spatial representation ability of input features. Hybrid networks attempt to integrate multiple structures to improve overall performance.

[0004] However, most existing methods implicitly assume that the hand size in the input video is relatively fixed, ignoring the issue of hand size variations caused by changes in the distance between the hand and the camera in real-world application scenarios. This problem weakens the spatial representation ability of deep networks during feature extraction, thus affecting the final classification performance. Therefore, existing gesture recognition methods are insufficient in dealing with hand size variations and distance adaptation. Summary of the Invention

[0005] In view of this, embodiments of this application provide a dynamic gesture recognition method, model training method and device based on dual-branch fusion, so as to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of this application provides a dynamic gesture recognition method based on dual-branch fusion, comprising: Extract multiple target images containing hand targets from the target video data, and generate a first image and a second image corresponding to each of the target images to form a target image sequence, wherein the resolution of the first image is smaller than the resolution of the second image; The target image sequence is input into a dynamic gesture recognition model, so that the image feature extraction branch of the dynamic gesture recognition model extracts the image feature sequence corresponding to the target image sequence based on each of the first images, and the hand feature extraction branch of the dynamic gesture recognition model extracts the hand feature sequence corresponding to the target image sequence based on each of the second images; and the recognition head of the dynamic gesture recognition model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence. Output the gesture category prediction results data.

[0007] In some embodiments of this application, the extraction of multiple target images containing a hand from the target video data, and the generation of a target image sequence based on each of the target images, includes: Hand target detection is performed on all image frames in the target video data. If a hand is detected, uniform sampling is performed within the range of the image frames in which the hand is detected to obtain multiple target images. Each of the target images is resolution adjusted to obtain a first image and a second image corresponding to each target image, thereby generating a target image sequence containing a first image sequence and a second image sequence; wherein, the first image sequence contains each of the first images; and the second image sequence contains each of the second images; Correspondingly, the image feature extraction branch is used to extract image features from each of the first images in the first image sequence to obtain an image feature sequence containing the image feature vectors corresponding to each of the first images. The hand feature extraction branch is used to extract hand features from each of the second images in the second image sequence, to obtain a hand feature sequence containing the hand feature vectors corresponding to each of the second images.

[0008] In some embodiments of this application, the image feature extraction branch includes: The ResNet50 network is used to extract features from each of the first images in the first image sequence to obtain the feature map corresponding to each of the first images. A spatial global average pooling layer is used to process each of the feature maps by taking the average value of all pixels in each channel, so as to obtain the first feature vector corresponding to each of the first images. A linear transformation layer is used to reduce the dimensionality of each of the first feature vectors to obtain the image feature vectors corresponding to each of the first images, so as to form an image feature sequence containing each of the image feature vectors.

[0009] In some embodiments of this application, the hand feature extraction branch includes: The YOLO11 model is used to perform hand target detection on each of the second images in the second image sequence, and obtain hand target detection result data corresponding to each of the second images; wherein, the hand target detection result data includes: the confidence score and position coordinates of the hand target; The result filtering module is used to sort the hand targets in the hand target detection result data according to their respective position coordinates in a top-down and left-to-right order if the number of hand targets in the hand target detection result data is greater than a number threshold, and select multiple hand targets from the sorted hand targets as filtered hand targets; if the number of hand targets in the hand target detection result data is less than or equal to the number threshold, all hand targets in the hand target detection result data are used as filtered hand targets. The stitching module is used to stitch together the confidence scores and position coordinates of each of the filtered hand targets corresponding to the hand target detection result data if the hand target detection result data corresponds to multiple filtered hand targets, so as to obtain the target feature vector corresponding to the hand target detection result data; if the hand target detection result data corresponds to one filtered hand target, then the confidence score and position coordinates of the filtered hand target are used as the target feature vector corresponding to the hand target detection result data. A multilayer perceptron is used to perform dimensionality upscaling on each of the target feature vectors to obtain hand feature vectors corresponding to each of the second images, so as to form a hand feature sequence containing each of the hand feature vectors.

[0010] In some embodiments of this application, the identification head includes: A dual-modal fusion module is used to perform two cross-attention enhancement processes on the image feature sequence and the hand feature sequence to obtain a secondary enhanced image feature sequence corresponding to the image feature sequence and a secondary enhanced hand feature sequence corresponding to the hand feature sequence; and to fuse the secondary enhanced image feature sequence and the secondary enhanced hand feature sequence to output a corresponding fused feature sequence; wherein the image feature sequence and the hand feature sequence have the same dimension; The temporal modeling module is used to add preset position encoding data to each fusion feature vector in the fusion feature sequence to obtain the target fusion feature sequence, and input the target fusion feature sequence into a multi-layer stacked Transformer encoder so that the multi-layer stacked Transformer encoder outputs the temporal enhancement feature sequence corresponding to the target fusion feature sequence; The classification module is used to perform global temporal average pooling on the temporal enhancement feature sequence to obtain a compressed feature vector corresponding to the temporal enhancement feature sequence; and input the compressed feature vector into a fully connected classifier so that the fully connected classifier outputs the gesture category prediction result data corresponding to the target image sequence.

[0011] In some embodiments of this application, the dual-modal fusion module includes: The first cross-attention unit is used to perform feature enhancement processing on the image feature sequence based on a preset gating mechanism, using the image feature sequence as a query marker and the hand feature sequence as a key marker and a value marker, to obtain a first-order enhanced image feature sequence corresponding to the image feature sequence; then, using the hand feature sequence as a query marker and the first-order enhanced image feature sequence as a key marker and a value marker, it performs feature enhancement processing on the hand feature sequence based on a preset gating mechanism to obtain a first-order enhanced hand feature sequence corresponding to the hand feature sequence; The second cross-attention unit is used to perform feature enhancement processing on the first-enhanced image feature sequence based on a preset gating mechanism, using the first-enhanced image feature sequence as a query marker and the first-enhanced hand feature sequence as a key marker and a value marker, to obtain the second-enhanced image feature sequence corresponding to the first-enhanced image feature sequence; and then, using the first-enhanced hand feature sequence as a query marker and the second-enhanced image feature sequence as a key marker and a value marker, perform feature enhancement processing on the first-enhanced hand feature sequence based on a preset gating mechanism to obtain the second-enhanced hand feature sequence corresponding to the first-enhanced hand feature sequence; The feature fusion unit is used to add the secondary enhanced image feature sequence and the secondary enhanced hand feature sequence element by element, and output the corresponding fused feature sequence.

[0012] A second aspect of this application also provides a method for training a dynamic gesture recognition model, including: In the current iteration, the target image sequence is input into the deep learning model, so that the image feature extraction branch of the deep learning model extracts the image feature sequence corresponding to the target image sequence, and the hand feature extraction branch of the deep learning model extracts the hand feature sequence corresponding to the target image sequence; and the recognition head of the deep learning model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence. Based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, the total loss of the deep learning model in the current iteration is determined, and the parameters of the deep learning model are updated based on the total loss. If the current deep learning model meets the preset training termination condition, the deep learning model with updated parameters is output as a dynamic gesture recognition model, wherein the dynamic gesture recognition model is used to execute the dynamic gesture recognition method based on dual-branch fusion described in the first aspect above.

[0013] In some embodiments of this application, determining the total loss of the deep learning model in the current iteration based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, and updating the parameters of the deep learning model based on the total loss, includes: Based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, the classification loss of the deep learning model in the current iteration is calculated; The center loss and supervised contrast loss corresponding to the target image sequence are calculated; Based on the classification loss, the center loss, the supervised comparison loss, the preset weight hyperparameters corresponding to the center loss and the preset weight hyperparameters corresponding to the supervised comparison loss, the total loss of the deep learning model in the current iteration is calculated. The model parameters in the deep learning model, excluding the frozen modules, are updated based on the total loss. The frozen modules include the YOLO11 model in the hand feature extraction branch and the other convolutional blocks in the ResNet50 network in the image feature extraction branch, excluding the last two convolutional blocks.

[0014] A third aspect of this application provides a dynamic gesture recognition device based on dual-branch fusion, comprising: The data acquisition module is used to extract multiple target images containing hand targets from target video data, and generate a first image and a second image corresponding to each of the target images to form a target image sequence, wherein the resolution of the first image is smaller than the resolution of the second image; The model recognition module is used to input the target image sequence into a dynamic gesture recognition model, so that the image feature extraction branch of the dynamic gesture recognition model extracts the image feature sequence corresponding to the target image sequence based on each first image, and the hand feature extraction branch of the dynamic gesture recognition model extracts the hand feature sequence corresponding to the target image sequence based on each second image; and the recognition head of the dynamic gesture recognition model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence. The result output module is used to output the gesture category prediction result data.

[0015] A fourth aspect of this application provides a dynamic gesture recognition model training device, comprising: The iterative training module is used to input the target image sequence into the deep learning model in the current iteration, so that the image feature extraction branch of the deep learning model extracts the image feature sequence corresponding to the target image sequence, and the hand feature extraction branch of the deep learning model extracts the hand feature sequence corresponding to the target image sequence; and the recognition head of the deep learning model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence. The model optimization module is used to determine the total loss of the deep learning model in the current iteration based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, and to update the parameters of the deep learning model based on the total loss. The model output module is used to output the deep learning model with updated parameters as a dynamic gesture recognition model if the current deep learning model meets the preset training termination condition. The dynamic gesture recognition model is used to execute the dynamic gesture recognition method based on dual-branch fusion described in the first aspect above.

[0016] The fifth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dynamic gesture recognition method based on dual-branch fusion described in the first aspect, and / or the dynamic gesture recognition model training method described in the second aspect.

[0017] The sixth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dynamic gesture recognition method based on dual-branch fusion described in the first aspect above, and / or the dynamic gesture recognition model training method described in the second aspect above.

[0018] The seventh aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dynamic gesture recognition method based on dual-branch fusion described in the first aspect above, and / or the dynamic gesture recognition model training method described in the second aspect above.

[0019] The dynamic gesture recognition method based on dual-branch fusion provided in this application extracts multiple target images containing hand targets from target video data, and generates a first image and a second image corresponding to each of the target images to form a target image sequence, wherein the resolution of the first image is smaller than the resolution of the second image; the target image sequence is input into a dynamic gesture recognition model, so that the image feature extraction branch of the dynamic gesture recognition model extracts the image feature sequence corresponding to the target image sequence based on each of the first images, and the hand feature extraction branch of the dynamic gesture recognition model extracts the hand feature sequence corresponding to the target image sequence based on each of the second images; and the recognition head of the dynamic gesture recognition model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence; the gesture category prediction result data is output, and then... By fusing image appearance information and hand position information, a more comprehensive understanding of gestures in videos can be achieved. By integrating visual stream (image features) and hand keypoint stream information, the complementary advantages of both types of information are fully preserved. The image stream provides rich scene and gesture appearance information, while the hand keypoint stream provides precise hand skeletal kinematic information. The combination of these two elements makes the model more robust to complex backgrounds, lighting changes, and different gesture shapes, thus significantly improving the accuracy and robustness of dynamic gesture recognition. This better meets the high recognition rate requirements of practical applications such as assistive robots. Furthermore, by preparing inputs of different resolutions for different tasks (high resolution for accurate detection, low resolution for efficient feature extraction), the model can adapt to hand regions of different scales, overcoming the performance degradation caused by changes in shooting distance. This makes the model more suitable for real-world, complex applications such as intelligent human-computer interaction and sign language translation. In addition, the model structure design of this application is simple and efficient. By performing dual-modal fusion through simple frame-by-frame addition, the computational burden caused by complex fusion mechanisms can be avoided, which is beneficial for deployment on embedded or edge computing platforms such as assistive robots.

[0020] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0021] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a schematic diagram of the first process of a dynamic gesture recognition method based on dual-branch fusion in one embodiment of this application.

[0023] Figure 2 This is a schematic diagram of the second process of a dynamic gesture recognition method based on dual-branch fusion in one embodiment of this application.

[0024] Figure 3 This is a schematic diagram of the overall architecture of a dynamic gesture recognition model in one embodiment of this application.

[0025] Figure 4 This is a schematic diagram of the specific architecture of the dynamic gesture recognition model in one embodiment of this application.

[0026] Figure 5 This is a flowchart illustrating the dynamic gesture recognition model training method in one embodiment of this application.

[0027] Figure 6 The overall network structure diagram of the dynamic gesture recognition model provided for the application example of this application.

[0028] Figure 7 The structure diagram of the ResNet50 network provided for the application example of this application.

[0029] Figure 8 The structural diagram of the graph feature size transformation module provided for the application example of this application.

[0030] Figure 9 A structural diagram of the hand feature size transformation module provided for the application example of this application.

[0031] Figure 10 The structure diagram of the cross-attention module provided for the application example of this application.

[0032] Figure 11 The structure diagram of the timing modeling module provided for the application example of this application.

[0033] Figure 12 A structural diagram of the classification module provided for the application example of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0035] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0036] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0037] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0038] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0039] To address the problems of inconsistent hand region scale due to changes in the distance between the hand and the camera, leading to decreased model feature extraction capability and low gesture recognition accuracy in existing dynamic gesture recognition methods based on two-branch fusion, this application provides a dynamic gesture recognition method based on two-branch fusion, a dynamic gesture recognition device based on two-branch fusion for executing the method, a dynamic gesture recognition model training method, a dynamic gesture recognition model training device for executing the training method, an electronic device, a computer-readable storage medium, and a computer program product. By fusing image appearance information and hand position information, the model can more comprehensively understand gestures, thereby significantly improving the accuracy and robustness of dynamic gesture recognition. It can adapt to hand regions of different scales and overcome the performance degradation caused by changes in shooting distance, making the model more suitable for real and complex application scenarios (such as intelligent human-computer interaction and sign language translation).

[0040] The following examples will provide a detailed description.

[0041] Based on this, embodiments of this application provide a dynamic gesture recognition method based on dual-branch fusion, which can be implemented by a dynamic gesture recognition device based on dual-branch fusion. See [link to relevant documentation]. Figure 1 The dynamic gesture recognition method based on dual-branch fusion specifically includes the following: Step 100: Extract multiple target images containing hand targets from the target video data, and generate a first image and a second image corresponding to each target image to form a target image sequence, wherein the resolution of the first image is smaller than the resolution of the second image.

[0042] In step 100, the target video data can refer to an RGB video stream lasting several seconds with a resolution of 1920x1080. Extracting multiple target images containing the hand target from the target video data can be achieved using existing target detection methods (such as the YOLO11 model). Each frame of the video can be preliminarily analyzed to determine whether it contains a range of frames. If so, multiple original frames are uniformly sampled within that range to serve as target images. In one example, 16 frames can be acquired. That is, given an input video, hand target detection is first performed on all frames of the video, and 16 frames are uniformly sampled within the range where the hand is detected, serving as the basis for subsequent feature extraction.

[0043] For each sampled target image, two versions with different resolutions can be generated simultaneously, for example: (1) First image: can refer to an RGB image that has been cropped from the center of the target image and scaled to 336 pixels × 336 pixels; (2) Second image: can refer to an RGB image that has been cropped from the center of the target image and scaled to 960 pixels × 960 pixels.

[0044] These multiple pairs (first image, second image) are then arranged in chronological order to form a target image sequence.

[0045] Step 200: Input the target image sequence into the dynamic gesture recognition model, so that the image feature extraction branch in the dynamic gesture recognition model extracts the image feature sequence corresponding to the target image sequence according to each of the first images, and the hand feature extraction branch in the dynamic gesture recognition model extracts the hand feature sequence corresponding to the target image sequence according to each of the second images; and the recognition head in the dynamic gesture recognition model recognizes the gesture category prediction result data corresponding to the target image sequence according to the image feature sequence and the hand feature sequence.

[0046] In step 200, the prepared target image sequence is fed into the trained dynamic gesture recognition model. The dynamic gesture recognition model can be a deep learning model. The dynamic gesture recognition model includes parallel image feature sequences and hand feature extraction branches, and also includes a recognition head that is connected to the output of the image feature sequences and hand feature extraction branches respectively.

[0047] The image feature extraction branch extracts image feature sequences based on each of the first images. ResNet50 can be used as the backbone network to extract features from each first image, resulting in a 1024-dimensional feature vector for each image, which can be called an image feature vector. The image feature vectors are arranged in order to obtain an image feature sequence, which mainly encodes the appearance, shape, and texture information of the gesture.

[0048] The hand feature extraction branch extracts hand feature sequences based on each of the second images. The YOLO11 model can be used to detect hand targets in each second image, resulting in feature vectors for each second image, which can be called hand feature vectors. The hand feature vectors are arranged in order to obtain a hand feature sequence, which explicitly encodes the motion trajectory and spatial position information of the gesture.

[0049] In addition, the recognition head is the post-processing part of the model, responsible for receiving and fusing image feature sequences and hand feature sequences, performing temporal modeling, and finally outputting classification results, i.e., gesture category prediction results data.

[0050] The gesture category prediction result data can be a probability vector, representing the probability that the input video belongs to each preset gesture category. The gesture categories can be set according to actual application needs. In one example, the gesture categories can include 11 types, specifically: still, swipe left, swipe right, swipe up, swipe down, clench fist, open fist, draw a circle clockwise, draw a circle counterclockwise, wave, and tap.

[0051] Step 300: Output the gesture category prediction result data.

[0052] Understandably, gesture category prediction data can also be passed to the application, allowing the application to perform corresponding operations based on this gesture category prediction data (such as recognizing the "swipe right" gesture), such as flipping through a PPT to the next page.

[0053] As described above, the dynamic gesture recognition method based on dual-branch fusion provided in this application can more comprehensively understand gestures in videos by fusing image appearance information and hand position information. By fusing visual stream (image features) and hand keypoint stream information, it can fully retain the complementary advantages of the two types of information. The image stream provides rich scene and gesture appearance information, while the hand keypoint stream provides accurate hand skeletal kinematic information. The combination of the two makes the model more robust to complex backgrounds, lighting changes, and different gesture shapes, thereby significantly improving the accuracy and robustness of dynamic gesture recognition and better meeting the high recognition rate requirements of practical application scenarios such as assistive robots. Furthermore, by preparing inputs of different resolutions for different tasks (high resolution for accurate detection and low resolution for efficient feature extraction), it can adapt to hand regions of different scales and overcome the problem of performance degradation caused by changes in shooting distance, making the model more suitable for real and complex application scenarios, such as intelligent human-computer interaction and sign language translation. In addition, the model structure design of this application is simple and efficient. By performing dual-modal fusion through simple frame-by-frame addition, it can avoid the computational burden caused by complex fusion mechanisms, which is conducive to the deployment of applications on embedded or edge computing platforms such as assisted robots.

[0054] To further prepare input data efficiently and specifically for dual-feature extraction branches, and to address the issue that directly using image sequences of uniform resolution may not simultaneously achieve both high efficiency in image feature extraction and high accuracy in hand detection, this application provides a dynamic gesture recognition method based on dual-branch fusion, see [link to relevant documentation]. Figure 2 Step 100 in the dynamic gesture recognition method based on dual-branch fusion specifically includes the following: Step 110: Perform hand target detection on all image frames in the target video data. If a hand is detected, perform uniform sampling within the range of the image frames where the hand is detected to obtain multiple target images.

[0055] Step 120: Adjust the resolution of each of the target images to obtain a first image and a second image corresponding to each of the target images, so as to generate a target image sequence containing a first image sequence and a second image sequence; wherein, the first image sequence contains each of the first images; and the second image sequence contains each of the second images.

[0056] Correspondingly, the image feature extraction branch is used to extract image features from each of the first images in the first image sequence to obtain an image feature sequence containing the image feature vectors corresponding to each of the first images. The hand feature extraction branch is used to extract hand features from each of the second images in the second image sequence, to obtain a hand feature sequence containing the hand feature vectors corresponding to each of the second images.

[0057] As can be seen from the above description, the dynamic gesture recognition method based on dual-branch fusion provided in this application can provide a lower resolution image for the image feature branch to ensure the computational efficiency of feature extraction; and can provide a higher resolution image for the hand detection branch to ensure the accuracy of hand position and confidence detection; and the two branches process different versions of the same set of target images, which can ensure temporal and content alignment and provide a reliable and effective data foundation for subsequent modality fusion.

[0058] To further efficiently and fully extract discriminative spatial features from single-frame images, a dynamic gesture recognition method based on dual-branch fusion is provided in this application embodiment, see [link to relevant documentation]. Figure 3 The image feature extraction branch in the dynamic gesture recognition method based on dual-branch fusion specifically includes the following: (1) A ResNet50 network is used to extract features from each of the first images in the first image sequence to obtain the feature map corresponding to each of the first images.

[0059] (2) Spatial global average pooling layer, used to process each feature map by taking the average value of all pixels in each channel, to obtain the first feature vector corresponding to each of the first images.

[0060] (3) A linear transformation layer is used to reduce the dimensionality of each of the first feature vectors to obtain the image feature vectors corresponding to each of the first images, so as to form an image feature sequence containing each of the image feature vectors.

[0061] Specifically, the image feature extraction branch uses ResNet50 as the backbone network to extract features from each frame of the first image (input resolution 336×336×3). In one example, a feature map with a maximum layer size of 11×11×2048 can be selected, and a 2048-dimensional first feature vector is obtained through spatial global average pooling. A linear transformation layer is then used to reduce the dimensionality of the first feature vector for each frame to 1024 dimensions, resulting in the image feature vector corresponding to each of the first images. Thus, 16 frames of the first image can form a 16×1024 image feature sequence.

[0062] As can be seen from the above description, the dynamic gesture recognition method based on dual-branch fusion provided in this application adopts the mature backbone network ResNet50, which can extract rich and multi-level hand shape and appearance features; through pooling and linear transformation layers, the features can be unified to a fixed dimension, which is convenient for subsequent fusion with hand feature sequences.

[0063] To convert the unstructured, variable-length detection results (bounding boxes and confidence scores) output by the object detector into fixed-dimensional feature vectors usable in temporal models, a dynamic gesture recognition method based on dual-branch fusion is provided in this application embodiment. (See also...) Figure 3 The hand feature extraction branch in the dynamic gesture recognition method based on dual-branch fusion specifically includes the following: (1) YOLO11 model, used to perform hand target detection on each of the second images in the second image sequence, and obtain the hand target detection result data corresponding to each of the second images; wherein, the hand target detection result data includes: the confidence score and position coordinates of the hand target.

[0064] (2) Result filtering module, used to sort the hand targets in the hand target detection result data in a top-down and left-to-right order according to the position coordinates of each hand target in the hand target detection result data if the number of hand targets in the hand target detection result data is greater than the number threshold, and select multiple hand targets from the sorted hand targets as filtered hand targets; if the number of hand targets in the hand target detection result data is less than or equal to the number threshold, all hand targets in the hand target detection result data are used as filtered hand targets.

[0065] (3) A splicing module is used to splice the confidence and position coordinates of each of the filtered hand targets corresponding to the hand target detection result data if there are multiple filtered hand targets corresponding to the hand target detection result data, so as to obtain the target feature vector corresponding to the hand target detection result data; if there is one filtered hand target corresponding to the hand target detection result data, the confidence and position coordinates of the filtered hand target are used as the target feature vector corresponding to the hand target detection result data.

[0066] (4) A multilayer perceptron is used to perform dimensionality upscaling on each of the target feature vectors to obtain the hand feature vectors corresponding to each of the second images, so as to form a hand feature sequence containing each of the hand feature vectors.

[0067] Specifically, the hand feature extraction branch is designed to meet the specific needs of this task. The YOLO11 model is used to detect hand targets in each frame of the second image (input resolution 960×960), obtaining the confidence score (conf) and location results for each hand. , , , The confidence and location results are normalized to the [0,1] interval. In one example, the quantity threshold is 5. If the number of hand targets in the hand target detection result data is greater than the quantity threshold, in order to handle multi-hand scenes, all hands detected in each frame are sorted from top to bottom and from left to right, the first 5 results are retained, and the corresponding 5 5-dimensional vectors are concatenated into a 25-dimensional target feature vector. Subsequently, a multilayer perceptron (MLP, which can be composed of Linear, GeLU activation and Linear respectively) is used to map the target feature vector to 1024 dimensions to achieve dimensional alignment with the image features. Finally, a 16×1024 hand feature sequence is constructed. This application uses a detection network with a large capacity to compensate for the insufficient accuracy caused by the limited scale of hand detection data and obtain more reliable hand location information.

[0068] As can be seen from the above description, the dynamic gesture recognition method based on dual-branch fusion provided in this application, through sorting, filtering and splicing, regularizes the detection results of each frame into a fixed 25-dimensional vector, which can effectively solve the problem of inaccurate recognition results caused by the variable number of hand targets; it adopts a high-performance YOLO11 model and trains it on a high-quality dataset, providing reliable hand position and confidence information, and can provide accurate attention guidance for image feature branches; by using MLP to increase the dimensionality of the hand detection vector, it can interact and fuse with image features in the feature space.

[0069] In other words, the two feature sequences mentioned above provide rich information support for the dynamic gesture recognition task: the image feature sequence mainly focuses on the shape of the hand in each frame, while the hand feature sequence explicitly encodes the position of the hand in each frame and the overall motion trajectory. To fully integrate the information from the two modalities, in a dynamic gesture recognition method based on dual-branch fusion provided in this application embodiment, see [link to relevant documentation]. Figure 3 The recognition head in the dynamic gesture recognition method based on dual-branch fusion specifically includes the following components: (1) A dual-modal fusion module is used to perform two cross-attention enhancement processes on the image feature sequence and the hand feature sequence to obtain a secondary enhanced image feature sequence corresponding to the image feature sequence and a secondary enhanced hand feature sequence corresponding to the hand feature sequence; and to perform feature fusion on the secondary enhanced image feature sequence and the secondary enhanced hand feature sequence to output the corresponding fused feature sequence; wherein the image feature sequence and the hand feature sequence have the same dimension.

[0070] (2) Temporal modeling module, used to add preset position encoding data to each fusion feature vector in the fusion feature sequence to obtain the target fusion feature sequence, and input the target fusion feature sequence into a multi-layer stacked Transformer encoder so that the multi-layer stacked Transformer encoder outputs the temporal enhancement feature sequence corresponding to the target fusion feature sequence.

[0071] (3) Classification module, used to perform global temporal average pooling on the temporal enhancement feature sequence to obtain the compressed feature vector corresponding to the temporal enhancement feature sequence; and input the compressed feature vector into a fully connected classifier so that the fully connected classifier outputs the gesture category prediction result data corresponding to the target image sequence.

[0072] Specifically, to fully integrate information from both modalities, the bimodal fusion module can employ a cross-attention module with a gating mechanism. Specifically, using the image feature sequence as the query and the hand feature sequence as the key and value, the cross-attention mechanism obtains an enhanced image feature sequence (16×1024) guided by hand features. In the reverse operation, the hand feature sequence is used as the query marker, and the enhanced image feature sequence as the key and value marker, resulting in an enhanced hand feature sequence (16×1024) guided by image features. This bidirectional cross-attention module is repeated twice to obtain a secondary enhanced image feature sequence and a secondary enhanced hand feature sequence, both enhanced by the other modality. The introduction of the gating mechanism allows the model to selectively incorporate useful cross-modal information and suppress redundant or invalid features, thereby enhancing the discriminative power of the fusion result.

[0073] Subsequently, the two cross-enhanced feature sequences are added element-wise to obtain the final fused feature sequence (16×1024). Global temporal average pooling is used to compress the fused feature sequence into a single 1024-dimensional feature vector. Random deactivation (with a ratio of 0.1) is then applied to prevent overfitting. Finally, the vector is fed into a fully connected classifier (a linear layer that maps the 1024 dimensions to 11 dimensions, where 11 represents the number of gesture categories), outputting the final gesture category prediction results.

[0074] As described above, the dynamic gesture recognition method based on dual-branch fusion provided in this application, through cross-attention and temporal modeling, can achieve deep temporal interaction and enhancement of two modalities, generating more discriminative fusion features. Employing a Transformer encoder, it can effectively model long-range temporal dependencies in gesture movements and accurately capture the dynamic change patterns of gestures.

[0075] To effectively achieve adaptive information flow between modalities during dual-modal fusion, suppress redundant or noisy information, and effectively filter invalid information, a dynamic gesture recognition method based on dual-branch fusion is provided in this application embodiment. (See also...) Figure 4 The dual-modal fusion module in the dynamic gesture recognition method based on dual-branch fusion specifically includes the following components: (1) A first cross-attention unit is used to perform feature enhancement processing on the image feature sequence based on a preset gating mechanism, using the image feature sequence as a query marker and the hand feature sequence as a key marker and a value marker, to obtain a first-enhanced image feature sequence corresponding to the image feature sequence; and then, using the hand feature sequence as a query marker and the first-enhanced image feature sequence as a key marker and a value marker, perform feature enhancement processing on the hand feature sequence based on a preset gating mechanism to obtain a first-enhanced hand feature sequence corresponding to the hand feature sequence.

[0076] (2) The second cross-attention unit is used to perform feature enhancement processing on the first enhanced image feature sequence based on a preset gating mechanism, using the first enhanced image feature sequence as a query marker and the first enhanced hand feature sequence as a key marker and a value marker, to obtain the second enhanced image feature sequence corresponding to the first enhanced image feature sequence; and then, using the first enhanced hand feature sequence as a query marker and the second enhanced image feature sequence as a key marker and a value marker, perform feature enhancement processing on the first enhanced hand feature sequence based on a preset gating mechanism to obtain the second enhanced hand feature sequence corresponding to the first enhanced hand feature sequence.

[0077] (3) Feature fusion unit, used to add the secondary enhanced image feature sequence and the secondary enhanced hand feature sequence element by element, and output the corresponding fused feature sequence.

[0078] As can be seen from the above description, the dynamic gesture recognition method based on dual-branch fusion provided in this application adopts a bidirectional, two-stage cross-attention mechanism, which enables the two features to be repeatedly enhanced and refined. The introduced gating mechanism can dynamically and selectively control the amount of cross-modal information passing through, effectively suppressing noise or redundant information from another modality, thereby greatly improving the quality and robustness of the fused features.

[0079] To further improve the performance and robustness of the dynamic gesture recognition model, this application also provides an embodiment of a dynamic gesture recognition model training method that can be executed using a dynamic gesture recognition model training device. The dynamic gesture recognition model training method specifically includes the following: Step 010: In the current iteration, the target image sequence is input into the deep learning model, so that the image feature extraction branch in the deep learning model extracts the image feature sequence corresponding to the target image sequence, and the hand feature extraction branch in the deep learning model extracts the hand feature sequence corresponding to the target image sequence; and the recognition head in the deep learning model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence.

[0080] Step 020: Based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, determine the total loss of the deep learning model in the current iteration round, and update the parameters of the deep learning model based on the total loss.

[0081] Step 030: If the current deep learning model meets the preset training termination condition, then the deep learning model with the updated parameters is used as the dynamic gesture recognition model for output, wherein the dynamic gesture recognition model is used to execute the dynamic gesture recognition method based on dual-branch fusion.

[0082] In step 030, the training termination condition may include any of the following: reaching the maximum number of iterations, the validation set performance no longer improving, or the joint loss value converging. For example: the maximum number of iterations (250 rounds) has been reached, the validation set performance has not improved for 10 consecutive rounds, or the joint loss value has converged (rate of change < 1e-5).

[0083] It is understood that the dynamic gesture recognition model refers to a trained deep learning model with fixed parameters, which can be verified and tested before output. For example, if it achieves a Dice coefficient of 71.2% and a classification accuracy of 93.7% on the test set, it can be used for online applications.

[0084] The dynamic gesture recognition model used in step 030 of this application can specifically implement the steps of the dynamic gesture recognition method based on dual-branch fusion in the above embodiments. Its function will not be repeated here, but can be referred to the detailed description of the above embodiments of the dynamic gesture recognition method based on dual-branch fusion.

[0085] As can be seen from the above description, the dynamic gesture recognition model training method provided in this application embodiment can ensure that the final dynamic gesture recognition model has the excellent performance described in the dynamic gesture recognition method based on dual-branch fusion provided in the foregoing embodiment; and through iterative training and loss optimization, the model parameters are converged to an ideal state, thereby ensuring its recognition effect in real-world scenarios.

[0086] To further address the issues of overfitting, intra-class feature discriminative power, and insufficient inter-class feature discriminative power that may result from a single classification loss function, as well as the problems of overfitting and high computational cost during full parameter fine-tuning of large-scale pre-trained models, a dynamic gesture recognition model training method is provided in this application embodiment. (See [link to relevant documentation]). Figure 5 Step 030 in the dynamic gesture recognition model training method specifically includes the following: Step 031: Calculate the classification loss of the deep learning model in the current iteration based on the preset gesture category label corresponding to the target image sequence and the gesture category prediction result data.

[0087] Cross-entropy loss with label smoothing is used as the classification loss, with a smoothing coefficient set to 0.1. Classification Loss The expression is: (1) in, This refers to the batch size. The total number of gesture categories, such as 11. The model represents the first The predicted value for the nth sample is... The probability of a class The gesture category is labeled (1 for the true category, 0 for the rest). After introducing label smoothing, the label probability of the true category is set to... The label probabilities for the remaining categories are all Label smoothing helps prevent the model from developing excessive confidence in the training data, thereby mitigating overfitting and improving the model's generalization ability.

[0088] In one example, the gesture category labels are presupposed to be one-hot encoded. For instance, for the first sample, its true label is "swipe right" (assumed to be category 2), so the label is [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]. Then, the predicted probability distribution of the 32 samples and the true labels are substituted into the label-smoothed cross-entropy loss function. Label smoothing refers to reducing the true category probability from 1 to 0.9, and uniformly increasing the probabilities of other categories from 0 to 0.1 / (11-1) = 0.01. The smoothed gesture category labels become [0.01, 0.9, 0.01, ..., 0.01], and the final scalar value is calculated. = 0.45.

[0089] Step 032: Calculate the center loss and supervised contrast loss corresponding to the target image sequence.

[0090] To promote the clustering of similar samples in the feature space, a center loss is further introduced. This center loss maintains a learnable center vector for each class. ,in is the feature dimension. For the in a batch... compressed feature vectors and gesture category tags Central loss The definition is as follows: (2) Finally, the central loss is multiplied by its corresponding weight hyperparameter. In order to balance its impact on the total loss.

[0091] In one example, for 32 samples, the model outputs a 1024-dimensional compressed feature vector before the classifier. The system maintains a learnable 1024-dimensional center vector for each category. For the first sample (true class 2), calculate its compressed feature vector. With the center vector of category 2 Squared Euclidean distance: || - ||². Perform this calculation on all 32 samples within the batch and take the average, assuming the result is... .

[0092] Furthermore, to further enhance inter-class discriminative power, this application introduces Supervised Contrastive Loss. Its basic idea is to maximize the feature similarity of samples of the same class and minimize the feature similarity between samples of different classes within the same batch. Specific implementations include: (1) Perform L2 normalization on the compressed feature vector, with the shape as follows: ; (2) Calculate the similarity matrix between features ,in This represents the temperature coefficient, and numerical stabilization is performed (the maximum value is subtracted from each row). Indicates the first One compressed feature vector; (3) Construct a mask If and only if and The time is 1; (4) For each positive (anchor) sample The loss is ,in To and A collection of similar samples; (5) Take the mean of all samples and multiply it by the normalization coefficient. ; (6) Finally, supervise and compare losses. Multiply by its corresponding weight hyperparameter This is to control the weight of this item in the total loss.

[0093] In one example, L2 normalization is performed on the 1024-dimensional feature vectors of 32 samples, and the cosine similarity matrix between all sample pairs is calculated. (32x32 matrix) Construct a mask based on the true label. For example, for the first sample, only the mask positions of other samples belonging to the same category 2 (such as the fifth and seventeenth samples) are 1. For the first sample, bring its similarity closer to the fifth and seventh samples, while pushing it further away from the similarity to all other out-of-category samples in the batch. Then calculate and average all 32 samples, multiply by a normalization coefficient, and then assume the following... =0.25.

[0094] Step 033: Based on the classification loss, the center loss, the supervised comparison loss, the preset weight hyperparameters corresponding to the center loss and the preset weight hyperparameters corresponding to the supervised comparison loss, calculate the total loss of the deep learning model in the current iteration round.

[0095] In summary, the total loss function of the model in this application is defined as: (3) By using multi-loss joint optimization, the model can effectively enhance the discriminability and generalization ability of features while improving classification accuracy.

[0096] Step 034: Update the model parameters in the deep learning model, excluding the frozen modules, based on the total loss. The frozen modules include: the YOLO11 model in the hand feature extraction branch, and the other convolutional blocks in the ResNet50 network in the image feature extraction branch, excluding the last two convolutional blocks.

[0097] Specifically, in order to utilize pre-trained knowledge and prevent overfitting when large-scale gesture datasets may be insufficient, during the network training phase, to enhance the model's generalization ability, only the deep layers of ResNet50 are fine-tuned. This allows the model to effectively prevent overfitting while leveraging its powerful feature extraction capabilities, thereby accelerating training and reducing computational resource requirements. Furthermore, by freezing the YOLO11 model, the stability of its detection capabilities can be maintained, and the number of training parameters can be reduced.

[0098] In other words, the dynamic gesture recognition method and dynamic gesture recognition model training method based on dual-branch fusion provided in this application, in order to involve new network structures and fusion strategies to improve the model's adaptability to complex scenarios in real-world applications, uses two completely corresponding sequences as network inputs: one is the original image sequence, and the other is a sequence of hand detection probability and position information. The hand detection position and probability sequence can explicitly indicate the specific position of the hand in each frame of the image, effectively assisting the network to adaptively focus on hand regions at different scales. The two input sequences achieve modal alignment through a bidirectional gated cross-attention mechanism, and complete modal fusion through an addition operation to obtain a single feature sequence. Finally, the fused feature sequence is input into a classifier after global temporal average pooling to achieve gesture category discrimination.

[0099] From a software perspective, this application also provides a dynamic gesture recognition device based on dual-branch fusion for performing all or part of the dynamic gesture recognition method based on dual-branch fusion, wherein the dynamic gesture recognition device specifically includes the following: The data acquisition module is used to extract multiple target images containing hands from the target video data, and generate a target image sequence based on each of the target images.

[0100] The model recognition module is used to input the target image sequence into a dynamic gesture recognition model, so that the image feature extraction branch in the dynamic gesture recognition model extracts the image feature sequence corresponding to the target image sequence, and the hand feature extraction branch in the dynamic gesture recognition model extracts the hand feature sequence corresponding to the target image sequence; and the recognition head in the dynamic gesture recognition model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence.

[0101] The result output module is used to output the gesture category prediction result data.

[0102] The embodiments of the dynamic gesture recognition device provided in this application can be used to execute the processing flow of the embodiment of the dynamic gesture recognition method based on dual-branch fusion in the above embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the embodiment of the dynamic gesture recognition method based on dual-branch fusion in the above embodiments.

[0103] The dynamic gesture recognition portion of the device can be performed on either a server or a client device. The choice depends on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are performed on the client device, the client device may further include a processor for the specific processing of dynamic gesture recognition.

[0104] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0105] The server and the client device can communicate using any suitable network protocol, including those not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Furthermore, such network protocols may also include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer Protocol) protocols used on top of the aforementioned protocols.

[0106] As described above, the dynamic gesture recognition device provided in this application embodiment can more comprehensively understand gestures in videos. By fusing visual stream (image features) and hand keypoint stream information, it can fully retain the complementary advantages of the two types of information. The image stream provides rich scene and gesture appearance information, while the hand keypoint stream provides accurate hand skeletal kinematic information. The combination of the two makes the model more robust to complex backgrounds, lighting changes, and different gesture shapes, thereby significantly improving the accuracy and robustness of dynamic gesture recognition and better meeting the high recognition rate requirements of practical application scenarios such as assistive robots. Furthermore, by preparing inputs of different resolutions for different tasks (high resolution for accurate detection and low resolution for efficient feature extraction), it can adapt to hand regions of different scales and overcome the problem of recognition performance degradation caused by changes in shooting distance, making the model more suitable for real and complex application scenarios, such as intelligent human-computer interaction and sign language translation.

[0107] From a software perspective, this application also provides a dynamic gesture recognition model training device for performing all or part of the dynamic gesture recognition model training method, wherein the dynamic gesture recognition model training device specifically includes the following: The iterative training module is used to input the target image sequence into the deep learning model in the current iteration, so that the image feature extraction branch in the deep learning model extracts the image feature sequence corresponding to the target image sequence, and the hand feature extraction branch in the deep learning model extracts the hand feature sequence corresponding to the target image sequence; and the recognition head in the deep learning model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence.

[0108] The model optimization module is used to determine the total loss of the deep learning model in the current iteration based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, and to update the parameters of the deep learning model based on the total loss.

[0109] The model output module is used to output the deep learning model with updated parameters as a dynamic gesture recognition model if the current deep learning model meets the preset training termination condition. The dynamic gesture recognition model is used to execute the dynamic gesture recognition method based on dual-branch fusion described in the foregoing embodiment.

[0110] The embodiments of the dynamic gesture recognition model training device provided in this application can be used to execute the processing flow of the embodiments of the dynamic gesture recognition model training method described above. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the dynamic gesture recognition model training method described above.

[0111] The dynamic gesture recognition model training device can perform the training of the dynamic gesture recognition model in either a server or a client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations in this regard. If all operations are performed in the client device, the client device may further include a processor for the specific processing of the dynamic gesture recognition model training.

[0112] As can be seen from the above description, the dynamic gesture recognition device provided in this application has a simple and efficient model structure design. By performing dual-modal fusion through simple frame-by-frame addition, it can avoid the computational burden caused by complex fusion mechanisms, which is conducive to deployment and application on embedded or edge computing platforms such as assistive robots.

[0113] To further illustrate the above embodiments, this application also provides an application example of a dynamic gesture recognition method based on dual-branch fusion, which can also be called a dynamic gesture recognition algorithm based on dual-modal fusion or a distance-adaptive dynamic gesture recognition method based on a two-stream network (AttentionLSTM-TwoStream).

[0114] Dynamic gesture recognition has significant application value in intelligent human-computer interaction, sign language translation, and augmented reality. However, the inconsistency in hand scale caused by changes in shooting distance poses a significant challenge to existing methods. To address the issue of feature decay and decreased recognition accuracy in traditional models such as TSM and TimeSformer under these conditions, this application proposes a distance-adaptive dual-stream dynamic gesture recognition algorithm. This algorithm simultaneously inputs an RGB frame sequence and a hand target detection result sequence: the former provides appearance and motion information, while the latter provides hand position and motion information, guiding the network to focus on multi-scale hand regions. The two feature streams are first aligned through a bidirectional gated cross-attention mechanism and then fused by addition. The fused features are then input into a Transformer encoder for temporal modeling. Finally, after global temporal average pooling and a classifier, the corresponding gesture category is obtained.

[0115] See Figure 6 The dynamic gesture recognition model provided in this application example specifically includes the following: (a) Image Feature Extraction Branch A ResNet50 network is used as the backbone network. An RGB image with an input size of 336×336×3 is processed through layer-by-layer convolution and downsampling to obtain feature representations at different levels: 168×168×64, 84×84×256, 42×42×512, 21×21×1024, and 11×11×2048. Finally, the feature map output from the last layer of the network (11×11×2048) is selected, and spatial global average pooling is used to obtain a 2048-dimensional global feature vector, which serves as the representation of that frame image, i.e., the image feature vector.

[0116] Among them, see Figure 7 The ResNet50 can contain five sequentially connected convolutional blocks: Conv1, Conv2_x, Conv3_x, Conv4_x, and Conv5_x. During network training, to enhance the model's generalization ability, only the parameters of the last two convolutional blocks of the ResNet50 are updated, while the rest remain frozen. This is because the ResNet50 loads pre-trained weights from the ImageNet-1K dataset during model initialization. Freezing the shallow structure helps preserve low-level texture and detail information, while training the deep structure enables high-level semantic feature extraction for gesture recognition tasks, thus improving the overall network's generalization performance while ensuring feature richness.

[0117] See Figure 8 To achieve effective alignment and fusion of image features and hand features, this application applies a dimensional transformation to the image feature sequence based on the dimensional difference between the two features: the original size of the image feature sequence is 16×2048, which is reduced to 16×1024 by the linear transformation layer (Linear) as the image feature size transformation module.

[0118] (II) Hand Feature Extraction Branch To accurately obtain the positions of all hands and their corresponding confidence information in each frame of the image, this application introduces probability as an important component of the hand detection output. The introduction of confidence information provides a basis for the subsequent gesture classification network to make judgments when the hand target detection network makes false or false detections, guiding the model to focus more on hand regions with higher detection probabilities, thereby improving the overall robustness of the recognition.

[0119] See Figure 9 To achieve effective alignment and fusion of image features and hand features, this application addresses the dimensionality difference between the two feature streams. The original size of the hand feature sequence is 16×25, which is then increased to 16×1024 using a multilayer perceptron (MLP) as the hand feature size transformation module. The multilayer perceptron can be written as... It consists of sequentially connected linear transformation layers, GeLU activation functions, and linear transformation layers. At this point, the two feature sequences are aligned in terms of temporal length and feature dimension, laying the foundation for subsequent deep fusion.

[0120] This application example uses YOLO11 as a hand detector. The input image resolution is 960×960. It detects the probability and location coordinates of each hand in the image. After sorting all detection results from top to bottom and left to right, the top 5 results are concatenated to obtain a 25-dimensional vector. The detector is trained on a dataset of 11,897 high-quality hand target detection images collected and labeled independently. This detector can adapt to various complex scenes with an input image resolution of 960×960. Performance evaluation of the YOLO11 model on the test set shows a precision of 96.9% (meaning that 96.9% of the hands identified by the model are indeed hands) and a recall of 94.8% (meaning that 94.8% of the real hands were successfully identified by the model). Furthermore, under common object detection evaluation standards, its overall detection performance (mAP@0.5) reaches 97.4%; even under more stringent localization requirements (mAP@0.5:0.95), its performance reaches 84.8%. These data fully demonstrate that the model has extremely high accuracy and strong generalization ability in hand detection tasks.

[0121] (III) Dual-modal fusion module The image feature sequence obtained from the image feature extraction branch and the hand feature sequence obtained from the hand feature extraction branch are subjected to dimensionality transformation. The image feature sequence is reduced from 16×2048 to 16×1024 through a linear layer, and the hand feature sequence is increased from 16×25 to 16×1024 through a multilayer perceptron. Then, a gated cross-attention mechanism is introduced to deeply fuse the feature sequences of the two modalities. First, the hand feature sequence is used as the Query (Q), the image feature sequence is used as the Key (K) and Value (V), and input into the cross-attention module to obtain the first-stage enhanced hand feature sequence. Then, the image feature sequence is used as the Query, and the first-stage enhanced hand feature sequence is used as the Key and Value, and input into the cross-attention module to obtain the first-stage enhanced image feature sequence. Then, the above cross-attention operation is repeated once to obtain the second-stage enhanced hand feature sequence and image feature sequence. Finally, the second-stage enhanced image feature sequence and hand feature sequence are added element-wise to obtain the final fused feature sequence. Subsequently, a gated cross-attention mechanism is introduced to deeply fuse the feature sequences of the two modalities. The cross-attention modules (such as the first and second cross-attention units mentioned above) employ a multi-head self-attention structure (4 heads, random dropout rate of 0.1) and integrate gating and residual connections to enhance modeling capabilities. The structure of the cross-attention module is as follows: Figure 10 As shown, the query tag (Query) and key / value tags (Key / Value) are input into the multihead attention module. The original query tag (Query) is simultaneously fed into a gate branch (Linear). The linearly transformed value is compressed to the (0,1) interval by the sigmoid activation function to generate a gate signal. The output of the multihead attention (AttnOutput1) is multiplied element-wise with the gate signal to obtain the gated output (AttnOutput2). The gated output (AttnOutput2) is then residually concatenated with the original query tag (Query). Finally, it is processed through a layer normalization layer to output the normalized feature (AttnOutput).

[0122] In practice, the first step is to use hand feature sequences. For query tags, image feature sequences Assuming key and value labels, the input graph is used to perform a cross-attention module on the hand to obtain the first-stage enhanced hand feature sequence. Next, using image feature sequences For query tags, Assuming key and value labels are input to the hand-image cross-attention module, the first-stage enhanced image feature sequence is obtained. The above process is further iterated as follows: and Interact again to obtain and This achieves secondary feature enhancement. The entire process can be formalized as follows: = (Q = , K,V = ) = (Q = , K,V = ) = (Q = , K,V = ) = (Q = , K,V = ) Finally, the enhanced image feature sequence and the hand feature sequence are added element-wise to obtain the final fused feature sequence. With a size of 16×1024, it achieves deep alignment and fusion of multimodal features: = + This module effectively enhances the network's ability to comprehensively model information from different modalities, providing a solid feature foundation for subsequent temporal modeling and gesture classification.

[0123] (iv) Temporal Modeling Module See Figure 11 After completing the bimodal feature alignment and fusion, the resulting fused feature sequence has a size of 16×1024. To further model the dynamic features of gestures changing over time, temporal modeling of this sequence is required. Specifically, firstly, a learnable 1024-dimensional positional embedding is added to each element in the fused feature sequence to explicitly inject temporal positional information. Subsequently, the fused feature sequence is input to a stacked 4-layer Transformer Encoder Layer, each layer employing 8 attention heads with a random dropout rate of 0.1 to fully capture the temporal dependencies in the sequence. Finally, the output is normalized by a LayerNorm layer to obtain a normalized temporally enhanced feature sequence. The output size remains 16×1024, where 16 is the sequence length and 1024 is the feature dimension at each time step.

[0124] (v) Classification Module See Figure 12 For the 16×1024 temporal augmentation feature sequence output by the temporal modeling module, a global temporal average pooling operation is first applied to compress the temporal dimension into a single 1×1024 feature vector. Subsequently, this feature vector is input into a fully connected classifier, namely the linear transformation layer, and finally outputs the gesture category prediction result data. Figures 8 to 12In the figure, 2048, 1024, 25, 336×336×3, 168×168×64, 84×84×256, 42×42×512, 21×21×1024, 11×11×2048, 16×1024, 1×1024 and 1×11 represent different dimensions.

[0125] In summary, the application example in this application, by fusing visual stream (image features) and hand keypoint stream information, fully retains the complementary advantages of both types of information. The image stream provides rich scene and gesture appearance information, while the hand keypoint stream provides accurate hand skeletal kinematic information. The combination of the two makes the model more robust to complex backgrounds, lighting changes, and different gesture shapes. Compared to the single-modal model (AttentionLSTM-ImgOnly) and the original Attention LSTM model based on the Paddle Video framework, the dual-modal fusion model in this application example significantly improves the accuracy and generalization performance of dynamic gesture recognition, and can better meet the high recognition rate requirements of practical application scenarios such as assistive robots. The model structure design of this application example is simple and efficient. It performs dual-modal fusion by simply adding frames one by one, avoiding the computational burden caused by complex fusion mechanisms, which is conducive to deployment on embedded or edge computing platforms such as assistive robots.

[0126] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the dynamic gesture recognition method and / or dynamic gesture recognition model training method based on dual-branch fusion mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0127] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0128] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the dynamic gesture recognition method and / or dynamic gesture recognition model training method based on dual-branch fusion in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the dynamic gesture recognition method and / or dynamic gesture recognition model training method based on dual-branch fusion in the above method embodiments.

[0129] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0130] The one or more modules are stored in the memory, and when executed by the processor, they execute the dynamic gesture recognition method and / or dynamic gesture recognition model training method based on dual-branch fusion in the embodiment.

[0131] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0132] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0133] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0134] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the aforementioned dynamic gesture recognition method and / or dynamic gesture recognition model training method based on dual-branch fusion. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0135] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned dynamic gesture recognition method and / or dynamic gesture recognition model training method based on dual-branch fusion.

[0136] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0137] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0138] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0139] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A dynamic gesture recognition method based on dual-branch fusion, characterized in that, include: Extract multiple target images containing hand targets from the target video data, and generate a first image and a second image corresponding to each of the target images to form a target image sequence, wherein the resolution of the first image is smaller than the resolution of the second image; The target image sequence is input into a dynamic gesture recognition model, so that the image feature extraction branch of the dynamic gesture recognition model extracts the image feature sequence corresponding to the target image sequence based on each of the first images, and the hand feature extraction branch of the dynamic gesture recognition model extracts the hand feature sequence corresponding to the target image sequence based on each of the second images; and the recognition head of the dynamic gesture recognition model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence. Output the gesture category prediction results data.

2. The dynamic gesture recognition method based on dual-branch fusion according to claim 1, characterized in that, The step of extracting multiple target images containing hands from the target video data and generating a target image sequence based on each of the target images includes: Hand target detection is performed on all image frames in the target video data. If a hand is detected, uniform sampling is performed within the range of the image frames in which the hand is detected to obtain multiple target images. Each of the target images is resolution adjusted to obtain a first image and a second image corresponding to each target image, thereby generating a target image sequence containing a first image sequence and a second image sequence; wherein, the first image sequence contains each of the first images; and the second image sequence contains each of the second images; Correspondingly, the image feature extraction branch is used to extract image features from each of the first images in the first image sequence to obtain an image feature sequence containing the image feature vectors corresponding to each of the first images. The hand feature extraction branch is used to extract hand features from each of the second images in the second image sequence, to obtain a hand feature sequence containing the hand feature vectors corresponding to each of the second images.

3. The dynamic gesture recognition method based on dual-branch fusion according to claim 2, characterized in that, The image feature extraction branch includes: The ResNet50 network is used to extract features from each of the first images in the first image sequence to obtain the feature map corresponding to each of the first images. A spatial global average pooling layer is used to process each of the feature maps by taking the average value of all pixels in each channel, so as to obtain the first feature vector corresponding to each of the first images. A linear transformation layer is used to reduce the dimensionality of each of the first feature vectors to obtain the image feature vectors corresponding to each of the first images, so as to form an image feature sequence containing each of the image feature vectors.

4. The dynamic gesture recognition method based on dual-branch fusion according to claim 2, characterized in that, The hand feature extraction branch includes: The YOLO11 model is used to perform hand target detection on each of the second images in the second image sequence, and obtain hand target detection result data corresponding to each of the second images; wherein, the hand target detection result data includes: the confidence score and position coordinates of the hand target; The result filtering module is used to sort the hand targets in the hand target detection result data according to their respective position coordinates in a top-down and left-to-right order if the number of hand targets in the hand target detection result data is greater than a number threshold, and select multiple hand targets from the sorted hand targets as filtered hand targets; if the number of hand targets in the hand target detection result data is less than or equal to the number threshold, all hand targets in the hand target detection result data are used as filtered hand targets. The stitching module is used to stitch together the confidence scores and position coordinates of each of the filtered hand targets corresponding to the hand target detection result data if the hand target detection result data corresponds to multiple filtered hand targets, so as to obtain the target feature vector corresponding to the hand target detection result data; if the hand target detection result data corresponds to one filtered hand target, then the confidence score and position coordinates of the filtered hand target are used as the target feature vector corresponding to the hand target detection result data. A multilayer perceptron is used to perform dimensionality upscaling on each of the target feature vectors to obtain hand feature vectors corresponding to each of the second images, so as to form a hand feature sequence containing each of the hand feature vectors.

5. The dynamic gesture recognition method based on dual-branch fusion according to claim 1, characterized in that, The identification head includes: A dual-modal fusion module is used to perform two cross-attention enhancement processes on the image feature sequence and the hand feature sequence to obtain a secondary enhanced image feature sequence corresponding to the image feature sequence and a secondary enhanced hand feature sequence corresponding to the hand feature sequence; and to fuse the secondary enhanced image feature sequence and the secondary enhanced hand feature sequence to output a corresponding fused feature sequence; wherein the image feature sequence and the hand feature sequence have the same dimension; The temporal modeling module is used to add preset position encoding data to each fusion feature vector in the fusion feature sequence to obtain the target fusion feature sequence, and input the target fusion feature sequence into a multi-layer stacked Transformer encoder so that the multi-layer stacked Transformer encoder outputs the temporal enhancement feature sequence corresponding to the target fusion feature sequence; The classification module is used to perform global temporal average pooling on the temporal enhancement feature sequence to obtain a compressed feature vector corresponding to the temporal enhancement feature sequence; and input the compressed feature vector into a fully connected classifier so that the fully connected classifier outputs the gesture category prediction result data corresponding to the target image sequence.

6. The dynamic gesture recognition method based on dual-branch fusion according to claim 5, characterized in that, The dual-modal fusion module includes: The first cross-attention unit is used to perform feature enhancement processing on the image feature sequence based on a preset gating mechanism, using the image feature sequence as a query marker and the hand feature sequence as a key marker and a value marker, to obtain a first-order enhanced image feature sequence corresponding to the image feature sequence; then, using the hand feature sequence as a query marker and the first-order enhanced image feature sequence as a key marker and a value marker, it performs feature enhancement processing on the hand feature sequence based on a preset gating mechanism to obtain a first-order enhanced hand feature sequence corresponding to the hand feature sequence; The second cross-attention unit is used to perform feature enhancement processing on the first-enhanced image feature sequence based on a preset gating mechanism, using the first-enhanced image feature sequence as a query marker and the first-enhanced hand feature sequence as a key marker and a value marker, to obtain the second-enhanced image feature sequence corresponding to the first-enhanced image feature sequence; and then, using the first-enhanced hand feature sequence as a query marker and the second-enhanced image feature sequence as a key marker and a value marker, perform feature enhancement processing on the first-enhanced hand feature sequence based on a preset gating mechanism to obtain the second-enhanced hand feature sequence corresponding to the first-enhanced hand feature sequence; The feature fusion unit is used to add the secondary enhanced image feature sequence and the secondary enhanced hand feature sequence element by element, and output the corresponding fused feature sequence.

7. A method for training a dynamic gesture recognition model, characterized in that, include: In the current iteration, the target image sequence is input into the deep learning model, so that the image feature extraction branch of the deep learning model extracts the image feature sequence corresponding to the target image sequence, and the hand feature extraction branch of the deep learning model extracts the hand feature sequence corresponding to the target image sequence; and the recognition head of the deep learning model recognizes the gesture category prediction result data corresponding to the target image sequence based on the image feature sequence and the hand feature sequence. Based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, the total loss of the deep learning model in the current iteration is determined, and the parameters of the deep learning model are updated based on the total loss. If the current deep learning model meets the preset training termination condition, the deep learning model with the updated parameters is used as a dynamic gesture recognition model for output, wherein the dynamic gesture recognition model is used to execute the dynamic gesture recognition method based on dual-branch fusion as described in any one of claims 1 to 6.

8. The dynamic gesture recognition model training method according to claim 7, characterized in that, The step of determining the total loss of the deep learning model in the current iteration based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, and updating the parameters of the deep learning model based on the total loss, includes: Based on the preset gesture category labels corresponding to the target image sequence and the gesture category prediction result data, the classification loss of the deep learning model in the current iteration is calculated; The center loss and supervised contrast loss corresponding to the target image sequence are calculated; Based on the classification loss, the center loss, the supervised comparison loss, the preset weight hyperparameters corresponding to the center loss and the preset weight hyperparameters corresponding to the supervised comparison loss, the total loss of the deep learning model in the current iteration is calculated. The model parameters in the deep learning model, excluding the frozen modules, are updated based on the total loss. The frozen modules include the YOLO11 model in the hand feature extraction branch and the other convolutional blocks in the ResNet50 network in the image feature extraction branch, excluding the last two convolutional blocks.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the dynamic gesture recognition method based on dual-branch fusion as described in any one of claims 1 to 6, and / or the dynamic gesture recognition model training method as described in claim 7 or 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the dynamic gesture recognition method based on dual-branch fusion as described in any one of claims 1 to 6, and / or the dynamic gesture recognition model training method as described in claim 7 or 8.

Images