Gesture instruction recognition method based on improved contactless data flow network

Abstract

This invention provides a gesture command recognition method based on an improved contactless data stream network. It utilizes the coordinates of key points in hand data extracted using an improved MediaPipe network algorithm framework based on artificial intelligence. The improvements to the MediaPipe network include increasing the recognition rate of the network model, enhancing the accuracy of key gesture features, and optimizing scene adaptability. The method achieves rapid gesture category determination by "precise hand region cropping and positioning—multi-scale key feature point fusion enhancement—gesture category static and dynamic classification adjustment." This invention addresses the recognition bias caused by excessively small gestures in long-distance photography and detailed gestures in close-up photography by adding a multi-scale feature enhancement module; it also reduces the misjudgment rate of gesture recognition by adding limb nodes and fusing the association between limbs such as the arm and torso; it enhances robustness in complex environments, reducing background interference and occlusion; and it strengthens the motion features between image frames, distinguishing between static and dynamic gestures and improving the dynamic gesture recognition rate.

Description

Technical Field

[0001] This invention relates to the field of image or video recognition or understanding technology, and specifically to a gesture command recognition method based on an improved contactless data stream network. Background Technology

[0002] Gestures are an important means of communication and command. In daily training, in environments with high noise levels and poor visibility, gestures can quickly and accurately convey instructions and information, providing a silent communication function. Using gestures to communicate allows for the rapid, accurate, and orderly deployment of various training tasks.

[0003] With the rapid development of unmanned systems, unmanned swarms based on various unmanned vehicles / drones and other equipment have gradually acquired the capabilities of swarm collaborative perception, collaborative navigation, and autonomous decision-making. However, when unmanned devices are integrated into daily training environments for collaborative tasks, gesture commands do not require any transmission medium, making it difficult for various devices to quickly intercept action information. Currently, most intelligent devices lack adaptive action recognition capabilities, making it difficult to recognize gestures during actions.

[0004] Therefore, a method for rapid self-recognition of action categories for various gesture commands is needed. Summary of the Invention

[0005] This invention aims to solve the problem of rapid and automatic recognition of gesture commands by providing a gesture command recognition method based on an improved contactless data stream network. It utilizes a lightweight MediaPipe network with an improved contactless design to achieve rapid recognition and processing of various gesture command action types, establishing a high-performance, high-speed, and high-precision gesture command rapid recognition technology with certain universality in the field of human posture and motion recognition technology.

[0006] This invention provides a gesture command recognition method based on an improved contactless data stream network, comprising the following steps:

[0007] S1. Establish a gesture command action database. Testers repeat N common gesture command actions and collect data to obtain gesture data. The gesture data includes at least two video data packets for each gesture command to be studied.

[0008] S2. Perform data preprocessing on the gesture data. Use the computer vision network OpenCV to convert the format of the input gesture data frame, and then input it into the modified BlazePalm detection network in chronological order to expand the positioning area from the hand to the key parts including the forearm area. Then perform adaptive cropping and standardization to obtain the preprocessed gesture.

[0009] S3. Build the gesture recognition network model architecture, configure the MediaPipe model network and OpenCV-python dependency package, and use the built-in tools of the MediaPipe model network to draw the skeletal line visualization results of the preprocessed gesture;

[0010] S4. Fine-tune the MediaPipe model network by replacing the fourth stage P4 of the MediaPipe model network with a multi-scale feature enhancement module, a spatial-channel dual attention filter, and a lightweight temporal coding module connected in sequence to perform multi-dimensional feature enhancement, suppress background interference, and dynamic semantic modeling.

[0011] The multi-scale feature enhancement module is used to adapt to changes in gesture scale during close-up and long-distance shooting. It focuses on finger joints and textures, captures hand posture and the angle of palm bending, and extracts global correlation of limbs to output enhanced feature maps.

[0012] Spatial-channel dual attention filtering preserves the feature channels that are strongly associated with gesture recognition in the enhanced feature map, removes redundant channels, and reduces the interference of background noise in complex training scenarios;

[0013] The lightweight temporal coding module captures the inter-frame variation patterns of key features and recognizes dynamic gestures to output static + dynamic fused features;

[0014] S5. Fine-tune the classification head of the MediaPipe model network, modify the single-branch classification head of the MediaPipe model network, and add a dual-branch classification head to optimize the classification of similar gestures to obtain an improved gesture recognition network model. The classification results of the dual-branch classification head are output through the final fully connected layer of the MediaPipe model network. The dual-branch classification head includes a main branch classification head and an auxiliary branch classification head.

[0015] S6. Use the preprocessed gestures to train and evaluate the improved gesture recognition network model. After passing the evaluation, the trained improved gesture recognition network model is obtained.

[0016] S7. Use the trained improved gesture recognition network model to recognize gesture commands.

[0017] The gesture command recognition method based on an improved contactless data stream network described in this invention, as a preferred embodiment, in step S1, N is 18, and the gesture data includes 5 video data packets for each gesture command to be studied;

[0018] In step S3, the processing method of the gesture recognition network model is as follows:

[0019] The input image is read, and the number of input frames for the operation is filtered using the FlowLimiter computation unit. After the previous frame image is calculated, the next frame image is sent for processing and converted into RGB format according to the MediaPipe model network.

[0020] Hand key points are extracted using BlazePalm for hand detection and HandLandmark for key point prediction and classification. The hand detector is initialized using the Hand model of the MediaPipe model network, and the preprocessed gesture is passed to the Hand model.

[0021] The key point results are displayed using OpenCV's visualization tools. The results are rendered on the output image using the DetectionToRenderData, RectToRenderData, and AnnotationOverlay calculation units in the dependency library to obtain the skeleton line visualization results.

[0022] In step S6, the model evaluation method is to calculate the mean square error (MSE) and average accuracy (mAP) between the predicted key point coordinates and the actual coordinates.

[0023] The trained improved gesture recognition network model can be applied to the recognition system of various unmanned swarm devices.

[0024] The gesture command recognition method based on an improved contactless data stream network described in this invention, as a preferred embodiment, involves converting the input gesture data frame into RGB format in step S2.

[0025] By modifying the detection head of the BlazePalm detection network and adding associated prediction boxes for the wrist and elbow, the key areas of the hand, arm, and torso are located in real time, resulting in the modified BlazePalm detection network. The modified BlazePalm detection network avoids misjudgment due to similar static and dynamic gestures.

[0026] Based on the key area output of the modified BlazePalm detection network, adaptive cropping is performed. When the location area includes the hand and forearm, the cropping size is fixed. When the hand is occluded within the location area and the occlusion range is less than 30%, the cropped area after occlusion is filled by edge padding to prevent the loss of key hand feature points.

[0027] Standardization is a process that limits pixel values ​​to between 0 and 1.

[0028] In step S4, the multi-scale feature enhancement module is located in the high-level feature processing stage of the MediaPipe model network, after stage P3.

[0029] The multi-scale feature enhancement module includes an input feature compression layer, a parallel multi-scale convolutional layer, and a dynamic attention weighted fusion layer;

[0030] The input feature compression layer unifies the number of input channels, compressing the features output by the third stage P3 of the MediaPipe model network to the same number of channels as the parallel multi-scale convolutional layer;

[0031] The parallel multi-scale convolutional layer includes three depthwise separable convolutions with padding. These three convolutional layers are adapted to gesture features at multiple scales, including finger joint features, texture features, hand pose features, palm bending angle features, arm information features, and wrist information features.

[0032] The dynamic attention weighted fusion layer performs dynamic process weighted fusion, including a global pooling layer, a fully connected layer, and a sigmoid activation function;

[0033] The dynamic attention weighted fusion layer takes the output feature maps of the three branches of the parallel multi-scale convolutional layer as input to this fusion layer, and compresses them into three pooled vector values ​​through global average pooling. The three pooled vector values ​​are then input into the fully connected layer to learn weights. The sigmoid activation function distributes the weights between 0 and 1, and outputs three weight values. The output feature maps of the three branches are weighted and summed according to the weight values ​​to output an enhanced feature map that fuses the three branches. Finally, the enhanced feature map is input into the two-branch classification head.

[0034] The gesture command recognition method based on an improved contactless data stream network described in this invention, as a preferred embodiment, has an input feature compression layer consisting of a 1×1 convolutional layer, which compresses the features output by the third stage P3 of the MediaPipe model network from 256 channels to 128 channels.

[0035] Parallel multi-scale convolutional layers include small-scale branch convolutional layers, medium-scale branch convolutional layers, and large-scale branch convolutional layers;

[0036] The small-scale branch convolutional layer is a 3×3 depthwise separable convolution with a stride of 1. It focuses on finger joint features and texture features, and outputs a 14×14 resolution, 128-channel feature map, retaining key information at the corresponding scale.

[0037] The mid-scale branch convolutional layer is a 5×5 depthwise separable convolution with a stride of 1. It focuses on capturing hand pose features and palm bending angle features, and outputs a 14×14 resolution, 128-channel feature map, retaining key information at the corresponding scale.

[0038] The large-scale branch convolutional layer is a 7×7 depthwise separable convolution with a stride of 1. It focuses on extracting global limb association features, which include arm information features and wrist information features. The output is a 14×14 resolution, 128-channel feature map, which retains key information at the corresponding scale.

[0039] The enhanced feature map is the result of fusing small-scale branch convolutional layers, medium-scale branch convolutional layers, and large-scale branch convolutional layers. The size of the enhanced feature map is 14×14×128.

[0040] The gesture command recognition method based on an improved contactless data stream network described in this invention, as a preferred embodiment, includes a spatial-channel dual attention filtering module comprising a spatial attention submodule and a channel attention submodule.

[0041] The spatial attention submodule suppresses features in the background region of the enhanced feature map, while enhancing features in the hand region, forearm region, and associated torso region. The background region includes equipment and terrain.

[0042] The spatial attention submodule's processing method includes: compressing the enhanced feature map, performing global average pooling to preserve spatial distribution information, compressing only the channel dimension, and outputting two pooling results; fusing and concatenating the two pooling results according to channels, and generating a spatial attention map using a 1×1 convolutional layer and a sigmoid activation function. The pixel values ​​of the spatial attention map are the probability values ​​of the gesture region at the location, with values ​​between 0 and 1; spatially weighting the input feature map and the spatial attention map, amplifying the feature values ​​of the hand position, and reducing the weight values ​​of the feature values ​​of the background region.

[0043] The channel attention submodule retains the feature channels that are strongly correlated with gesture recognition and removes redundant channels. The feature channels that are strongly correlated with gesture recognition include wrist and arm angles, while the redundant channels include background color features and clothing textures.

[0044] The channel attention submodule is processed as follows: global average pooling is used to compress the spatial dimension of the enhanced feature map, and the output feature vector with unchanged channel number and compressed spatial distribution information is output; channel attention weights are calculated, and the feature vector is input into two fully connected layers. The first layer is a ReLU activation function to compress the channels, and the second layer is a Sigmoid activation function to expand the channels. The output channel attention weights are the importance level of the channel, and the value is between 0 and 1.

[0045] The feature map output by the spatial attention submodule is multiplied with each channel of the channel attention weight. The feature values ​​of the feature channels that are strongly associated with gesture recognition are amplified, while the feature values ​​of redundant channels are reduced, so that the MediaPipe model network focuses on the essential features of gestures.

[0046] The present invention discloses a gesture command recognition method based on an improved contactless data stream network. In a preferred embodiment, the spatial attention submodule compresses and pools a 14×14×128 enhanced feature map to obtain a 7×7×128 feature map, then performs global average pooling to output two 7×7×1 pooling results. These two pooling results are then fused and concatenated according to channels to form a 7×7×2 pooling map. A 7×7×1 spatial attention map is generated using a 1×1 convolutional layer and a sigmoid activation function. Finally, the input feature map and the spatial attention map are spatially weighted, resulting in a 7×7×128×7×7×1→7×7×128 pooling map. The weight values ​​for hand position features are increased to between 0.7 and 0.9, while the weight values ​​for background region features are reduced to between 0.1 and 0.3.

[0047] In the channel attention submodule, the number of channels remains unchanged, and the feature vector size of the compressed spatial distribution information is 1×1×128. The first fully connected layer compresses the number of channels from 128 to 32, and the second activation layer expands the number of channels from 32 to 128, outputting 1×1×128 channel attention weights.

[0048] The present invention discloses a gesture command recognition method based on an improved contactless data stream network. In a preferred embodiment, the lightweight temporal coding module is located after the spatial-channel dual attention filtering module and before the dual-branch classification head. The lightweight temporal coding module can capture the inter-frame variation pattern of the output features of the spatial-channel dual attention filtering module and improve the accuracy of dynamic gesture recognition.

[0049] The spatial-channel dual attention filtering module outputs at least two consecutive feature maps, the corresponding hand coordinates, and the coordinates of limb skeletal key points to the lightweight temporal coding module; the limb skeletal key points include upper limb core nodes and trunk core nodes. The upper limb core nodes constitute the basic skeletal chain of the upper limb, covering the connection relationship of the wrist, elbow, and shoulder.

[0050] The lightweight temporal coding module separates static and dynamic features, calculates the mean of static features of feature maps of at least two consecutive frames of input, calculates inter-frame motion parameters of dynamic features based on hand coordinates and limb skeletal keypoint coordinates, and compresses the dynamic parameters into dynamic feature vectors through a one-dimensional convolutional layer to preserve the motion patterns of hand gestures.

[0051] Inter-frame motion parameters include displacement rate and acceleration. Displacement rate is the coordinate change of the same key node between adjacent frames, and acceleration is used to distinguish between deceleration and rapid motion. Acceleration is the rate of change of displacement rate.

[0052] Dynamic parameters are compressed into dynamic feature vectors through a one-dimensional convolutional layer, preserving the motion patterns of hand gestures;

[0053] Lightweight temporal modeling is then used to calculate inter-frame dependencies. The dynamic feature vector is input into the lightweight temporal modeling layer in chronological order. By updating the gate weights, the sigmoid function is used to activate and output the temporal feature vector with the complete motion trend.

[0054] The enhanced feature map is a static feature, and the temporal feature vector is a dynamic feature. The static feature is compressed by global average pooling and then concatenated with the dynamic feature to obtain the static + dynamic fusion feature. The static + dynamic fusion feature includes gesture spatial information, related limb associations, and dynamic gesture trends.

[0055] The gesture command recognition method based on an improved contactless data stream network described in this invention, as a preferred embodiment, uses a spatial-channel dual attention filtering module to process five consecutive frames of features. Figure 7 ×7×128, corresponding to 21 hand coordinates and 10 limb skeleton key point coordinates, are output to the lightweight timing coding module;

[0056] The 10 key skeletal points are the core nodes of the upper limbs and the core nodes of the trunk:

[0057] The core nodes of the upper limb include the left wrist, left elbow, left shoulder, right wrist, right elbow, and right shoulder;

[0058] The core nodes of the trunk include the neck, head, left hip, and right hip;

[0059] Calculate the static feature mean of the input 5 consecutive frame feature maps, and calculate the inter-frame motion parameters displacement rate and acceleration based on the dynamic features of the coordinate sequence of 31 key points;

[0060] The displacement rate is:

[0061]

[0062] The acceleration is:

[0063]

[0064] Where △x is the displacement of adjacent frames in the x-direction, △y is the displacement of adjacent frames in the y-direction, and △t is the time difference between adjacent frames;

[0065] The enhanced feature map is 7×7×128, the dynamic feature vector and the temporal feature vector are both 512-dimensional, the value of the temporal feature vector is between 0 and 1, and the static + dynamic fusion feature is 1024-dimensional.

[0066] The gesture command recognition method based on an improved contactless data stream network described in this invention, as a preferred embodiment, uses the following processing method with a dual-branch classification head:

[0067] First, the static and dynamic fused features are normalized using the LayerNorm layer and then output to the main branch classification head and the auxiliary branch classification head.

[0068] The main branch classification head performs global coarse classification of the input feature values ​​and normalizes them through the first, second, and third fully connected layers. The auxiliary branch classification head corrects the fine-grained differences in gestures and extracts key difference features from K highly similar gestures out of N gesture actions to correct the main branch results.

[0069] The first fully connected layer uses the ReLU activation function for computation, preserving core features for the first compression. The second fully connected layer uses the ReLU activation function for computation, preserving core features for the second compression. The third fully connected layer does not use an activation function and directly outputs the raw N-dimensional gesture logits. The raw N-dimensional gesture logits are then activated by Softmax to output the initial probability distribution value P of the main branch classification head. main Pmain consists of N values, which sum to 1, representing the confidence of N gesture categories;

[0070] The method for processing auxiliary branch classification heads is as follows:

[0071] Similarity category and difference feature extraction: K groups of similar gesture difference features are extracted from the input features through convolution and global pooling, and each group outputs L-dimensional difference features;

[0072] Difference Probability Calculation: The difference features are processed through a fully connected network to output the difference probabilities between the two classes. The K×L dimension is reduced to 1 / 2×K×L, and then further reduced to 2 dimensions, outputting the within-group difference probability values ​​between the two classes. Finally, the corrected weights w are obtained through a Sigmoid activation function. i ,w i ∈(0~1);

[0073] Main branch correction calculation: based on the correction weight w i For the initial probability distribution value P main The probability of similar categories is calculated, amplifying the probability difference between two gestures in the same group of similar gestures, while the probabilities of other dissimilar categories remain unchanged;

[0074] Classification: Output gesture category, and output the corrected gesture category probability distribution result P. final Select P final The category corresponding to the maximum value is the final identification result.

[0075] The gesture command recognition method based on an improved contactless data stream network described in this invention, as a preferred method, uses a dual-branch classification head to map 1024-dimensional feature vector values ​​to 18 types of gesture labels, serving as the final decision layer of the improved gesture recognition network model.

[0076] The first fully connected layer compresses the 1024-dimensional features to 512-dimensional features. The second fully connected layer compresses the 512-dimensional features to 256-dimensional features while retaining the core features. The third fully connected layer directly outputs the original 18-dimensional logits output value.

[0077] K is 7, and the highly similar hand gestures are: Stop - Pause action, Advance - Come over, Hear - See, I - You, Quick - Come over, Quiet - Crouch, Cover - Crouch;

[0078] With L = 20, each group outputs 20-dimensional difference features, totaling 7 * 20 = 140 dimensions.

[0079] When a set of similar gestures A and B are corrected by weight w i If the probability is greater than 0.5, then the probability of gesture A is greater than the probability of gesture B. Therefore, the probability of gesture A is increased to be greater than P. main The value of reduces the probability of gesture B to less than P. main The value of .

[0080] This invention primarily addresses the problem of extracting motion features and recognizing various gesture commands generated during action tasks in daily training environments.

[0081] The technical solution of this invention is as follows: For common commands in input gestures, the coordinate positions of key points of hand data are extracted using an improved version of the MediaPipe network algorithm framework based on artificial intelligence. The improvements of the MediaPipe network are to increase the recognition rate of the network model, improve the accuracy of key gesture features, and optimize scene adaptability. The gesture features are quickly determined by "precise cropping and positioning of hand region - multi-scale key feature point fusion enhancement - static and dynamic classification adjustment of gesture category".

[0082] The specific steps of this invention are as follows:

[0083] (1) Establish a gesture command action database: study common commands in 18 types of gestures, testers repeat the command actions to collect data, forming 5 video data packets for each of the 18 actions, for a total of 90 data packets;

[0084] (2) Gesture data preprocessing: To improve the recognition accuracy of the algorithm network, the input data frames are format-converted using the computer vision network OpenCV. After converting the training dataset, the improved lightweight backbone and channel-pruned BlazePalm detection network is used to locate key areas of the hand, arm, and torso in real time. The hand area, which excludes background interference, is quickly located. Invalid pixel background interference is reduced by cropping to avoid the loss of gesture features in long-distance scenes. Finally, the converted and cropped data frames are standardized, and their pixel values ​​are normalized to [0,1] to eliminate the influence of backlight and other lighting differences on subsequent gesture recognition.

[0085] (3) Build and train the gesture recognition network model: Configure the MediaPipe model network and OpenCV-python dependency package, use OpenCV to read the gesture action video stream or image data, convert it into RGB format and input it into the model, use the improved MediaPipe model to extract the key point coordinates of the training data frame, and use the built-in tools of the network model to draw the visualization results of the skeleton line.

[0086] (4) MediaPipe Network Model Fine-tuning: In order to achieve high accuracy, high speed and high robustness in gesture recognition, the MediaPipe network is deeply optimized. This invention adds three modules: multi-scale feature enhancement module, spatial-channel dual attention filtering and lightweight temporal coding. Through multi-dimensional feature enhancement, targeted background interference suppression and dynamic semantic modeling, it solves the shortcomings of the native MediaPipe network in recognizing gestures in complex scenes, multi-scale gesture localization and dynamic gesture commands.

[0087] ① Multi-scale feature enhancement module: Since there are scale differences in gestures during action, the proportion of gestures is larger when shooting at close range and smaller when shooting at long distance. This results in poor adaptability of the model network when extracting multi-scale features of the hand. Therefore, a multi-scale feature enhancement module is added to adapt to the changes in gesture scale.

[0088] ② Spatial-channel dual attention filtering: After feature fusion of the MediaPipe network, spatial-channel dual attention filtering is inserted to reduce background noise interference in complex training scenarios, such as when gestures are obscured by devices / people or when the gestures are captured in backlit environments.

[0089] ③ Lightweight Timing Coding Module: Because the original MediaPipe network recognizes static features, it cannot capture the temporal correlation of dynamic gestures, resulting in low accuracy for dynamic gestures. This invention improves the MediaPipe network by adding a "lightweight timing coding module," overcoming the limitations of the original MediaPipe's single-frame static features.

[0090] (5) Network model classification: In order to reduce the mixing efficiency of highly similar gestures, the single-branch classification head of the MediaPipe network architecture was modified and a "double-branch classification head" was added. The "main branch + auxiliary branch" structure was used to optimize similar gestures. The network model outputs 18 categories of gestures through the final fully connected layer (hidden dimension changed from 256→128→18).

[0091] (6) Model Algorithm Evaluation: The mean square error (MSE) and mean accuracy (mAP) between the predicted keypoint coordinates and the actual coordinates are calculated to evaluate the accuracy of keypoint localization in the MediaPipe network. This technology can be applied to the identification systems of various unmanned swarm devices.

[0092] The present invention has the following advantages:

[0093] (1) Rich recognition of gesture commands: It can support the recognition of 18 kinds of gesture commands;

[0094] (2) High accuracy in gesture recognition: The MediaPipe network uses deep learning algorithms to detect and track key hand points, accurately identifying hand position, shape, and movement posture, providing reliable gesture recognition results. By improving the network model and adding a "multi-scale feature enhancement module," the recognition biases of "too small gestures in long-distance shooting" and "detailed gestures in close-up shooting" are resolved, reducing the likelihood of lost hand details or semantic deviations and improving the accuracy of multi-scale gesture recognition. Compared to the native MediaPipe network, the overall gesture accuracy has increased from 70.1% to 89.5%.

[0095] (3) Improve the correlation between limb features and reduce the misjudgment rate of gesture recognition: Existing technologies mostly focus on key feature points of the human hand and cannot integrate the correlation between the human arm and torso. In the key node feature recognition, this invention adds 10 limb nodes such as the shoulder, elbow, and hip joints. By integrating the correlation of the "multi-scale feature fusion module", the misjudgment rate of similar gestures such as "forward" and "come over", "pause action" and "stop" is reduced by 20%. Compared with the native MediaPipe network, the accuracy of similar gestures is improved from 30.9% to 7.9%.

[0096] (4) Enhance robustness in complex environments and reduce background interference and occlusion: Existing technologies often directly process the full image of the training action after acquisition, which is easily affected by background interference such as equipment, environment, and other people in the training scene. In order to resist background interference and limb occlusion in the captured image, this invention performs "region cropping" on the input image data to locate the key areas of the hand, arm and torso. At the same time, it filters key features through "spatial-channel dual attention filtering", which can show good robustness under different lighting conditions (backlight, etc.), background complexity (occlusion and interference) and camera quality.

[0097] (5) Differentiating between static and dynamic gestures to improve dynamic gesture recognition rate: Because the original MediaPipe network recognizes static features, it cannot capture the temporal correlation of dynamic gestures, resulting in low accuracy for dynamic gestures. This invention improves the MediaPipe network by adding a "lightweight temporal coding module" to enhance the motion features (displacement rate and motion acceleration) between image frames, thereby increasing the accuracy of dynamic hand gestures to over 88% and reducing the misjudgment rate between static and dynamic actions. Compared to the native MediaPipe network, the accuracy of static gestures increases from 80.2% to 90%, and the accuracy of dynamic gestures increases from 62.5% to 88.5%.

[0098] (6) Fast real-time network processing speed: Existing technologies such as YOLOv8 have low network processing accuracy, while gesture recognition based on the Transformer network model with high processing accuracy has poor inference speed, which cannot meet the requirements of real-time recognition and recognition accuracy of gesture actions in task scenarios. In order to improve the network processing capabilities of both speed and accuracy, this invention improves the original MediaPipe network model by making backbone network lightweighting and channel pruning, etc., to achieve the function of fast recognition of gesture actions under video stream input with a higher frame rate. The improved real-time processing frame rate is increased from 31FPS to 35FPS.

[0099] (7) High category recognition capability under small sample gesture training dataset conditions: Since it is difficult to obtain training gestures in actual scenarios, in order to solve the problem of few samples, this invention adds a "dual-branch classification head" to calculate higher loss weights for small sample datasets, so that the recognition accuracy of small sample gestures is increased to more than 85% under the condition of few samples, ensuring that all 18 types of action commands are covered and recognized, and guaranteeing their recognition accuracy.

[0100] (8) Strong cross-platform compatibility of the algorithm: Existing technologies are mostly designed for specific scenarios and have poor portability. This system can run on multiple platforms, can be deployed on different hardware and operating systems, and can also be mounted on unmanned cluster devices to complete automatic detection and recognition of gesture commands. At the same time, this invention has scalability. The "multi-scale feature enhancement module" and "static / dynamic feature encoding" modules of the technology can be migrated to pose recognition scenarios such as typical human pose recognition networks such as Openpose and HRNet.

[0101] This invention achieves hand keypoint detection and tracking through an improved MediaPipe network, accurately identifying hand position, shape, and movement posture. It can be applied to recognition systems for various unmanned swarm devices, overcoming the technical shortcomings of the original network in dynamic gestures, similar gestures, and small sample categories. This invention features rich gesture recognition commands, high recognition accuracy, fast real-time processing speed, and good robustness. It is a high-performance, high-speed, and high-precision gesture command rapid recognition technology. Attached Figure Description

[0102] Figure 1 A flowchart of a gesture command recognition method based on an improved contactless data stream network;

[0103] Figure 2 This is a diagram of the original MediaPipe network architecture.

[0104] Figure 3 This is a hand detection model inference framework that uses the original MediaPipe model;

[0105] Figure 4 This is a diagram of the architecture of an improved gesture recognition network model for a gesture command recognition method based on an improved contactless data stream network.

[0106] Figure 5 This is a schematic diagram of the hand joint distribution for a gesture command recognition method based on an improved contactless data stream network.

[0107] Figure 6 A diagram illustrating the recognition effect of a "forward" gesture based on a gesture command recognition method using an improved contactless data stream network.

[0108] Figure 7 A diagram illustrating the "come here" gesture recognition effect of a gesture command recognition method based on an improved contactless data stream network;

[0109] Figure 8 A diagram illustrating the effect of "you" gesture recognition using a gesture command recognition method based on an improved contactless data stream network;

[0110] Figure 9A diagram illustrating the effect of "I" gesture recognition using a gesture command recognition method based on an improved contactless data stream network. Detailed Implementation

[0111] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0112] Example 1

[0113] like Figure 1 As shown, a gesture command recognition method based on an improved contactless data stream network includes the following steps:

[0114] S1. Dataset preparation: Collect video streams of action data of specified categories. In this embodiment, the action video streams include the following gesture commands: forward, come over, understand, question, fast, quiet, enter, alternating cover, cover, surround, please repeat, target personnel, protection team members, hear, see, reconnaissance, stop, crouch down, assemble, a total of 18*5=90 data packets, which are used as input data for the MediaPipe network.

[0115] S2. Data Preprocessing: Before inputting the data into the network model for training, this invention first performs data frame format conversion, gesture target region localization and cropping on the training dataset, and normalization processing on the cropped data frames.

[0116] ①BGR image to RGB data conversion: The acquired video is preprocessed. Since the MediaPipe network uses C++ as its core framework, the video data is processed after converting the BGR image to RGB format using OpenCV when compiling the network with Python.

[0117] ② Gesture and key area localization and cropping: Modify the BlazePalm detection network of the MediaPipe network model, and input the converted video data (640×480 resolution, 30FPS frame rate) into the modified BlazePalm detection network in chronological order.

[0118] a) Adaptive Region Localization: Modify the detection head of the BlazePalm detection network to add associated prediction boxes for the wrist and elbow, expanding the localization area from the hand to include the forearm region, which has a pixel size of 224×224. Output the boundingbox coordinates (x1, y1, x2, y2) of the hand to ensure the localization of key parts of various gestures. This invention avoids misjudgment due to similar static and dynamic gestures by introducing multiple key node features of the human body, such as the hand, arm, and wrist.

[0119] b) Adaptive Region Cropping: This invention performs adaptive and refined cropping based on the key region output by the modified BlazePalm detection network. When the location region includes the hand and forearm, the cropping size is a fixed 224×224. When the hand is occluded within the region and the occlusion range is <30%, the cropped region is filled by edge padding of the network to prevent the loss of key hand feature points.

[0120] ③ Image standardization after cropping: In order to eliminate the influence of background interference such as backlighting during shooting, and to improve the recognition accuracy of MediaPipe network, the image sequence in the video data is normalized, and the pixel value is limited to between 0 and 1. This improves the convergence speed of the network model, enhances numerical stability, and reduces background interference.

[0121] S3, such as Figure 2 , 3 As shown, the MediaPipe model environment is configured, and compilation from source code is set up for easier improvement of the network framework. The OpenCV-python dependency package is installed, the hand detection model is initialized, and the input video stream data is processed.

[0122] ① Input node: Reads frames from the camera. input_video and output_video are the input and output images. In the detection model, the FlowLimiter calculation unit is used to filter the number of input frames for calculation. In order to ensure that the video stream can be processed and calculated in real time, the next frame is sent for processing after the calculation of the previous frame is completed. It is converted into RGB format according to the MediaPipe model.

[0123] ② Processing nodes: Extract hand key points (hand detection BlazePalm, key point prediction and classification HandLandmark); Initialize the hand detector using the Hand model of the MediaPipe network, and pass the gesture image (video image captured by the camera or video after shooting) to the Hand model;

[0124] ③ Output Node: Display the key point results using OpenCV's visualization tools, and render the results onto the output image using the DetectionToRenderData, RectToRenderData, and AnnotationOverlay calculation units from the dependency library.

[0125] S4. Model fine-tuning: such as Figure 4As shown, this invention fine-tunes the native MediaPipe network by adding three modules: a multi-scale feature enhancement module, a spatial-channel dual attention filter, and a lightweight temporal coding module. This improves the MediaPipe network's deep optimization in gesture recognition, achieving high precision, high speed, and high robustness.

[0126] ① Multi-scale feature enhancement module: In the high-level feature processing stage of the native MediaPipe backbone network, a multi-scale feature enhancement module is added mainly between stage 3 (P3) and stage 4 (P4). This module mainly consists of an input feature compression layer, a parallel multi-scale convolutional layer, and a dynamic attention weighted fusion layer.

[0127] a) Input Feature Compression Layer: This layer unifies the input dimensions of subsequent modules, forming the basis for multi-scale feature extraction. This layer consists of 1×1 convolutional layers, compressing the features output from stage P3 of the native MediaPipe backbone network, reducing 256 channels to 128 channels.

[0128] b) Parallel multi-scale convolutional layer: This layer includes 3 depthwise separable convolutions with padding, and the 3 convolutional layers are adapted to gesture features at multiple scales;

[0129] Small-scale branched convolutional layer: 3×3 depthwise separable convolution with stride of 1, focusing on finger joints and texture, outputting a 14×14 resolution, 128-channel feature map, retaining key information at the corresponding scale;

[0130] Mid-scale branched convolutional layer: 5×5 depthwise separable convolution with a stride of 1, focusing on capturing hand pose and the angle of hand bending, outputting a 14×14 resolution, 128-channel feature map, retaining key information at the corresponding scale;

[0131] Large-scale branched convolutional layers: 7×7 depthwise separable convolutions with a stride of 1, focusing on extracting global associations of limbs, such as information about the arm and wrist, and outputting a 14×14 resolution, 128-channel feature map, retaining key information at the corresponding scale.

[0132] c) Dynamic attention weighted fusion layer (dynamic process weighted fusion): mainly composed of global pooling + fully connected layer + Sigmoid activation.

[0133] First, the output feature maps of the three branches of the above parallel multi-scale convolutional layer are used as the input of this fusion layer and compressed into vector values ​​through global average pooling;

[0134] Secondly, the three pooled compressed vector values ​​are input into the fully connected layer to learn the weights. Sigmoid activation ensures that the weights are between 0 and 1, and outputs three values ​​(corresponding to three branches).

[0135] Finally, the feature maps of the three branches are weighted and summed according to the final output weights. The summation process is as follows: Fusion feature = w1 × small-scale feature + w2 × medium-scale feature + w3 × large-scale feature.

[0136] The output is an enhanced feature map fused from the three branches, with a size of 14×14×128. The resulting 14×14×128 enhanced feature map is then input into stage P4 of the MediaPipe network.

[0137] ② Spatial-Channel Dual Attention Filtering Module: This module is located after the multi-scale feature enhancement module and before the temporal coding module, corresponding to stage P4 of the backbone MediaPipe network, and performs noise filtering. Since the feature map after multi-scale fusion already possesses complete gesture features, it can selectively preserve the core regions and key nodes of the gesture, improving the input data for subsequent temporal modeling and classification with a high signal-to-noise ratio. The spatial-channel dual attention filtering module mainly consists of a spatial attention submodule and a channel attention submodule.

[0138] a) Spatial Attention Submodule: Suppresses features of background areas (such as equipment and terrain) to enhance core gesture areas, such as the hand area, forearm area, and associated torso features. Specific operations are as follows:

[0139] First, feature compression is performed on the feature map output by the previous module: the feature map output by the multi-scale feature enhancement module, which has a size of 14×14×128, is compressed. After pooling, the input feature map becomes 7×7×128. Global average pooling is then performed on this feature map to output two 7×7×1 feature maps, preserving their spatial distribution information and compressing only their channel dimensions.

[0140] Secondly, the two pooling results are fused and stitched together according to the channels to form a 7×7×2. A 7×7×1 spatial attention map is generated using a 1×1 convolutional layer and a sigmoid activation function. The pixel value is the probability value of the gesture region at that location, with the value between 0 and 1.

[0141] Finally, the input feature map and the spatial attention map are spatially weighted (multiplied) to obtain 7×7×128×7×7×1→7×7×128. The feature values ​​of the hand position are amplified with a weight value between 0.7 and 0.9, while the weight values ​​of the feature values ​​of the background region are reduced to between 0.1 and 0.3.

[0142] b) Channel Attention Submodule: Retains feature channels strongly correlated with gesture recognition (such as wrist and arm angles), and removes redundant channels, such as background color and clothing texture. Specific operations are as follows:

[0143] First, the features output by the aforementioned spatial attention submodule Figure 7The spatial dimension is compressed by ×7×128 (by performing global average pooling), resulting in a 1×1×128 feature vector. The number of channels remains unchanged, while compressing the spatial distribution information.

[0144] Next, channel attention weights are calculated. A 1×1×128 feature vector is input into two fully connected layers. The first layer uses the ReLU activation function to change the number of channels from 128 to 32. The second layer uses the Sigmoid activation function to change the number of channels from 32 to 128. The output is a 1×1×128 channel attention weight, where each value represents the importance level of the channel and is between 0 and 1.

[0145] Finally, the features of the above spatial attention processing Figure 7 Multiply each channel by the 7×128 and channel attention weight 1×1×128:

[0146] 7×7×128×1×1×128→7×7×128

[0147] After multiplying the feature values, the key channel feature values, such as those for fingers and wrists, are amplified, while channels such as "background texture" are reduced, so that the network model focuses on the essential features of the gesture.

[0148] ③ Lightweight Temporal Coding Module: This module follows the spatial-channel dual attention filtering module and precedes the dual-branch classification head, belonging to the temporal enhancement stage of key gesture feature processing. After completing the spatial feature extraction operations (multi-scale feature enhancement and dual attention filtering), the feature map remains a static feature of a single frame, and the network model cannot distinguish dynamic gesture features. By enhancing spatial features and reducing spatial information redundancy, the lightweight temporal coding module can capture the inter-frame variation patterns of key features, improving the accuracy of dynamic gesture recognition.

[0149] First, the features of five consecutive frames output by the dual attention filtering module are processed. Figure 7 The coordinates of the ×7×128 and the corresponding 21 hand and 10 limb skeletal key points are input into this module. All hand and limb skeletal key points are shown in Table 1, and the hand joint distribution is as follows: Figure 5 As shown.

[0150] To construct a concise and robust Google topology, 10 limb key points were selected from the core joints of the upper limbs and torso:

[0151] a) Core nodes of the upper limb (6 nodes): left wrist, left elbow, left shoulder; right wrist, right elbow, right shoulder

[0152] To form the basic skeletal chain of the upper limbs, covering the connections between the wrist, elbow, and shoulder.

[0153] b) Core nodes of the trunk (4 nodes): neck, head, left hip, right hip.

[0154] Table 1

[0155]

[0156]

[0157] Secondly, static and dynamic features are separated. The mean value of static features in 5 frames is calculated. The inter-frame motion parameters displacement rate and acceleration are calculated for dynamic features based on the coordinate sequence of 31 key points.

[0158] Displacement rate: Calculates the coordinate changes of the same key nodes between adjacent frames.

[0159]

[0160] Acceleration: It can distinguish between deceleration and rapid motion, such as "coming over" and "rapidly", and calculate the rate of change of displacement.

[0161]

[0162] The dynamic parameters are compressed into a dynamic feature vector through a one-dimensional convolutional layer, preserving the motion patterns of the gestures. (31 nodes × 2 directions × 4 inter-frame differences = 248 dimensions, one-dimensional convolution results in a 512-dimensional dynamic feature vector)

[0163] Lightweight temporal modeling is then used to calculate inter-frame dependencies. The 512-dimensional dynamic feature vector output from the above steps is input into the lightweight temporal modeling layer in chronological order. By updating the gate weights, the input 512-dimensional feature vector is processed through a single fully connected layer to output 512-dimensional data. The Sigmoid function is activated, with a value between 0 and 1, to output a temporal feature vector with a complete motion trend.

[0164] Finally, the static and dynamic features are fused. The static features (7×7×128) output by the spatial-channel dual attention filtering module are compressed into a 512-dimensional vector through global average pooling. This vector is then concatenated with the temporal feature vector (512-dimensional vector) output by lightweight temporal modeling to obtain a 1024-dimensional static + dynamic fused feature, which is used as the output feature value of this module.

[0165] S5. Network Model Classification: A "dual-branch classification head" is added to the last layer of the native MediaPipe network to achieve accurate classification of 18 types of gestures. This layer is located after the lightweight temporal coding module, and its input data is a 1024-dimensional feature vector value that is statically and dynamically fused, containing complete information such as gesture spatial information, related limb associations, and dynamic gesture trends.

[0166] The dual-branch classification head maps 1024-dimensional feature vector values ​​to 18 gesture labels, serving as the final decision layer of this network. The specific operation is as follows:

[0167] ① Input feature processing: In order to reduce the difference in feature scale and avoid some category features from dominating the classification, the 1024-dimensional feature vector value output by the lightweight temporal coding module is normalized by the LayerNorm layer. The normalized 1024-dimensional feature vector value output is used as the input of the main branch and auxiliary branch of the dual-branch classification head.

[0168] ② The main branch classification head performs a coarse global feature classification on the input feature values. The features are then normalized using a 3-layer fully connected network.

[0169] a) Layer 1: ReLU activation function calculation, compressing the 1024-dimensional features to retain core features and then to 512-dimensional features (1024-dimensional features → 512-dimensional features);

[0170] b) Layer 2: ReLU activation function calculation, which compresses the 512-dimensional features into 256-dimensional features while retaining the core features (512-dimensional features → 256-dimensional features);

[0171] c) Layer 3: No activation function is calculated; the original logits of 18 types of gestures are directly output.

[0172] The original 18-dimensional logits output values ​​are activated by Softmax to output the initial probability distribution value Pmain of the main branch classification head. Pmain consists of 18 values, which sum to 1, representing the confidence of the 18 gesture categories.

[0173] After the main branch classifier outputs the initial confidence score, its branches misjudge the "stop" and "pause action" gestures. The static features are similar, and they cannot be distinguished under the global feature classification.

[0174] ③ Auxiliary Branch Classification Head: Addressing the issue of category confidence in the main branch classification head's output, this auxiliary branch classification head corrects for fine-grained differences in gestures. The auxiliary branch classification head primarily extracts key difference features from seven highly similar gesture groups (stop-pause, forward-come over, hear-see, I-you, quick-come over, quiet-crouch, cover-crouch) to correct the main branch results. The specific operation is as follows:

[0175] a) Similar category and difference feature extraction: 7 sets of similar gesture difference features are extracted from the input 1024-dimensional features through convolution and global pooling. Each set outputs 20-dimensional difference features, 7*20=140 dimensions.

[0176] b) Calculate the difference probability: The difference features are processed through a fully connected network to output the difference probabilities between the two classes. Dimensionality is reduced from 140 to 70, then from 70 to 2, outputting the within-group difference probability values ​​between the two classes. Finally, the corrected weights w are obtained through a Sigmoid activation function. i ∈(0~1).

[0177] c) Calculate the main branch correction and output the main branch probability P from the main branch classification head. main The probability of similar categories is calculated. An example calculation is as follows:

[0178] w i =0.7, if stopping > pausing action, then the probability of stopping increases to:

[0179] P 停止 =P main ×1.2

[0180] The probability of suspending operations has decreased to:

[0181] P 暂停行动 =P main ×0.8

[0182] The differences between the two are significantly amplified, while the probabilities of other dissimilar categories remain unchanged.

[0183] ④ Classification

[0184] a) Output gesture category: Output the corrected gesture category probability distribution result P final Select P final The category corresponding to the maximum value is the final identification result.

[0185] When P final <0.6, output not recognized.

[0186] The S6 and MediaPipe networks are used to evaluate the accuracy of key point localization, calculate the mean square error (MSE) and mean accuracy (mAP) between the predicted key point coordinates and the actual coordinates; through data processing, the relative positions and motion trajectories of gesture joints are analyzed, the gesture recognition results are fed back, the feedback results are compared with the actual actions, and the error rate of yes / no is calculated.

[0187] The 18 gestures and their corresponding category labels are shown in Table 2.

[0188] Table 2

[0189]

[0190]

[0191] Evaluation metrics for network models are divided into core metrics and real-time processing metrics.

[0192] ①Core Indicators:

[0193] a) Overall recognition accuracy, which is the average correct recognition rate of 18 types of gestures;

[0194] b) Static gesture accuracy rate: The average correct recognition rate of 11 types of static gestures is as follows: "I, you, target personnel, protection team members, commander, received, question, quiet, pause action, stop, squat down";

[0195] c) Accuracy of dynamic gestures: the average correct recognition rate of 7 types of dynamic gestures, namely "forward, come over, fast, cover, assemble, hear, see";

[0196] ② Real-time processing indicators:

[0197] To demonstrate the need for rapid recognition, a comparison is made using the recognition inference frame rate (FPS).

[0198]

[0199]

[0200] S7. Use the trained improved gesture recognition network model to recognize gesture commands. The recognition results of "forward", "come here", "you", and "I" in this embodiment are as follows: Figures 6-9 As shown, this embodiment demonstrates excellent recognition performance.

[0201] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A gesture command recognition method based on an improved contactless data stream network, characterized in that: Includes the following steps: S1. Establish a gesture command action database. Testers repeat N common gesture command actions and collect data to obtain gesture data. The gesture data includes at least two video data packets for each gesture command to be studied. S2. Perform data preprocessing on the gesture data. Use the computer vision network OpenCV to convert the format of the input gesture data frame, and then input it into the modified BlazePalm detection network in chronological order to expand the positioning area from the hand to the key part area including the forearm area. Then perform adaptive cropping and standardization to obtain the preprocessed gesture. S3. Build the gesture recognition network model architecture, configure the MediaPipe model network and OpenCV-python dependency package, and use the built-in tools of the MediaPipe model network to draw the skeletal visualization results of the preprocessed gesture; S4. Fine-tune the MediaPipe model network by replacing the fourth stage P4 of the MediaPipe model network with a multi-scale feature enhancement module, a spatial-channel dual attention filter, and a lightweight temporal coding module connected in sequence to perform multi-dimensional feature enhancement, suppress background interference, and dynamic semantic modeling. The multi-scale feature enhancement module is used to adapt to changes in gesture scale during close-up and long-distance shooting, focusing on finger joints and textures, capturing hand posture and the angle of palm bending, and extracting global correlation of limbs to output enhanced feature maps. The spatial-channel dual attention filter retains the feature channels that are strongly correlated with gesture recognition in the enhanced feature map, removes redundant channels, and reduces the interference of background noise in complex training scenarios. The lightweight temporal coding module captures the inter-frame variation patterns of key features and identifies dynamic gestures to output static + dynamic fused features; S5. Fine-tune the classification head of the MediaPipe model network, modify the single-branch classification head of the MediaPipe model network, and add a dual-branch classification head to optimize the classification of similar gestures to obtain an improved gesture recognition network model. The classification result of the dual-branch classification head is output through the final fully connected layer of the MediaPipe model network. The dual-branch classification head includes a main branch classification head and an auxiliary branch classification head. S6. Use the preprocessed gesture to train and evaluate the improved gesture recognition network model. After passing the evaluation, the trained improved gesture recognition network model is obtained. S7. Use the trained improved gesture recognition network model to recognize gesture commands.

2. The gesture command recognition method based on an improved contactless data stream network according to claim 1, characterized in that: In step S1, N is 18, and the gesture data includes 5 video data packets for each gesture command to be studied; In step S3, the processing method of the gesture recognition network model is as follows: The input image is read, and the number of input frames for the operation is filtered using the FlowLimiter computation unit. After the previous frame image is calculated, the next frame image is sent for processing and converted into RGB format according to the MediaPipe model network. Hand key points are extracted using BlazePalm for hand detection and HandLandmark for key point prediction and classification. The hand detector is initialized using the Hand model of the MediaPipe model network, and the preprocessed gesture is passed to the Hand model. The key point results are displayed using OpenCV's visualization tools. The results are rendered on the output image using the DetectionToRenderData, RectToRenderData, and AnnotationOverlay calculation units in the dependency library to obtain the skeleton line visualization results. In step S6, the model evaluation method is to calculate the mean square error (MSE) and average accuracy (mAP) between the predicted key point coordinates and the actual coordinates. The trained improved gesture recognition network model can be applied to the recognition system of various unmanned swarm devices.

3. The gesture command recognition method based on an improved contactless data stream network according to claim 1, characterized in that: In step S2, the input gesture data frame is converted into RGB format; By modifying the detection head of the BlazePalm detection network and adding associated prediction boxes for the wrist and elbow, the key areas of the hand, arm, and torso are located in real time, resulting in the modified BlazePalm detection network. The modified BlazePalm detection network avoids misjudgment due to similar static and dynamic gestures. Based on the key area output by the modified BlazePalm detection network, adaptive cropping is performed. When the positioning area includes the hand and forearm, the cropping size is fixed. When the hand is occluded within the positioning area and the occlusion range is less than 30%, the cropped area after occlusion is filled by edge filling to prevent the loss of key feature points of the hand. The standardization process is a normalization process that limits pixel values ​​to between 0 and 1; In step S4, the multi-scale feature enhancement module is located in the high-level feature processing stage of the MediaPipe model network, after stage P3. The multi-scale feature enhancement module includes an input feature compression layer, a parallel multi-scale convolutional layer, and a dynamic attention weighted fusion layer; The input feature compression layer unifies the number of input channels, compressing the features output by the third stage P3 of the MediaPipe model network to the same number of channels as the parallel multi-scale convolutional layer. The parallel multi-scale convolutional layer includes three depthwise separable convolutions with padding. The three convolutional layers are adapted to gesture features at multiple scales, including finger joint features, texture features, hand pose features, palm bending angle features, arm information features, and wrist information features. The dynamic attention weighted fusion layer performs dynamic process weighted fusion, including a global pooling layer, a fully connected layer, and a Sigmoid activation function; The dynamic attention weighted fusion layer takes the output feature maps of the three branches of the parallel multi-scale convolutional layer as input to this fusion layer, and compresses them into three pooled vector values ​​through global average pooling. Then, the three pooled vector values ​​are input to the fully connected layer to learn weights. The sigmoid activation function distributes the weights between 0 and 1, and outputs three weight values. The output feature maps of the three branches are weighted and summed according to the weight values ​​to output an enhanced feature map that fuses the three branches. Finally, the enhanced feature map is input to the dual-branch classification head.

4. The gesture command recognition method based on an improved contactless data stream network according to claim 3, characterized in that: The input feature compression layer consists of a 1×1 convolutional layer, which compresses the features output by the third stage P3 of the MediaPipe model network from 256 channels to 128 channels. The parallel multi-scale convolutional layer includes small-scale branch convolutional layers, medium-scale branch convolutional layers, and large-scale branch convolutional layers; The small-scale branch convolutional layer is a 3×3 depth separable convolution with a stride of 1. It focuses on finger joint features and texture features, and outputs a 14×14 resolution, 128-channel feature map, retaining key information at the corresponding scale. The mid-scale branch convolutional layer is a 5×5 depth separable convolution with a stride of 1. It focuses on capturing hand pose features and palm bending angle features, and outputs a 14×14 resolution, 128-channel feature map, retaining key information at the corresponding scale. The large-scale branch convolutional layer is a 7×7 depthwise separable convolution with a stride of 1. It focuses on extracting global limb association features, which include arm information features and wrist information features. It outputs a 14×14 resolution, 128-channel feature map, retaining key information at the corresponding scale. The enhanced feature map is the result of fusing the small-scale branch convolutional layer, the medium-scale branch convolutional layer, and the large-scale branch convolutional layer, and the size of the enhanced feature map is 14×14×128.

5. The gesture command recognition method based on an improved contactless data stream network according to claim 1, characterized in that: The spatial-channel dual attention filtering module includes a spatial attention submodule and a channel attention submodule; The spatial attention submodule suppresses features of the background region in the enhanced feature map and enhances features of the hand region, forearm region and associated torso region, the background region including equipment and terrain; The processing method of the spatial attention submodule includes: compressing the enhanced feature map, performing global average pooling, retaining spatial distribution information, compressing only the channel dimension, and outputting two pooling results; fusing and concatenating the two pooling results according to the channels, and generating a spatial attention map using a 1×1 convolutional layer and a sigmoid activation function, wherein the pixel value of the spatial attention map is the probability value of the gesture region at the location, and the value is between 0 and 1; spatially weighting the input feature map and the spatial attention map, amplifying the feature value of the hand position, and reducing the weight value of the feature value of the background region; The channel attention submodule retains the feature channels that are strongly associated with gesture recognition and removes redundant channels. The feature channels that are strongly associated with gesture recognition include wrist and arm angles, and the redundant channels include background color features and clothing textures. The processing method of the channel attention submodule is as follows: global average pooling is used to compress the spatial dimension of the enhanced feature map, and a feature vector with unchanged channel number and compressed spatial distribution information is output; channel attention weights are calculated, and the feature vector is input into two fully connected layers. The first layer is a ReLU activation function to compress the channels, and the second layer is a Sigmoid activation function to expand the channels. The channel attention weights are output, and the attention weight of each channel is the importance level of the channel, with a value between 0 and 1. The feature map output by the spatial attention submodule is multiplied with each channel of the channel attention weight, thereby amplifying the feature values ​​of the feature channels strongly associated with gesture recognition and reducing the feature values ​​of the redundant channels, so that the MediaPipe model network focuses on the essential features of gestures.

6. The gesture command recognition method based on an improved contactless data stream network according to claim 5, characterized in that: The spatial attention submodule compresses and pools the 14×14×128 enhanced feature map to obtain a 7×7×128 feature map, then performs global average pooling to output two 7×7×1 pooling results; the two pooling results are then fused and concatenated according to channels to obtain a 7×7×2 pooling result, and a 7×7×1 spatial attention map is generated using a 1×1 convolutional layer and a sigmoid activation function; finally, the input feature map and the spatial attention map are spatially weighted, resulting in a 7×7×128×7×7×1→7×7×128, with the weight values ​​of the hand position feature values ​​increased to between 0.7 and 0.9, and the weight values ​​of the background region feature values ​​decreased to between 0.1 and 0.

3. In the channel attention submodule, the number of channels remains unchanged, the feature vector size of the compressed spatial distribution information is 1×1×128, the first fully connected layer compresses the number of channels from 128 to 32, the second activation layer expands the number of channels from 32 to 128, and outputs 1×1×128 channel attention weights.

7. The gesture command recognition method based on an improved contactless data stream network according to claim 1, characterized in that: The lightweight temporal coding module is located after the spatial-channel dual attention filtering module and before the dual-branch classification head. The lightweight temporal coding module can capture the inter-frame variation pattern of the output features of the spatial-channel dual attention filtering module and improve the accuracy of dynamic gesture recognition. The spatial-channel dual attention filtering module outputs at least two consecutive feature maps, corresponding hand coordinates, and limb skeletal key point coordinates to the lightweight temporal coding module; the limb skeletal key points include upper limb core nodes and trunk core nodes, the upper limb core nodes constitute the basic skeletal chain of the upper limb, covering the connection relationship of the wrist, elbow, and shoulder; The lightweight temporal coding module separates static and dynamic features, calculates the mean of static features of at least two consecutive frames of input feature maps, calculates inter-frame motion parameters of dynamic features based on the hand coordinates and the coordinates of the key points of the limb bones, and compresses the dynamic parameters into a dynamic feature vector through a one-dimensional convolutional layer, preserving the motion pattern of the gesture. The inter-frame motion parameters include displacement rate and acceleration. Displacement rate is the coordinate change of the same key node between adjacent frames. Acceleration is used to distinguish between deceleration and rapid motion. Acceleration is the rate of change of displacement rate. Dynamic parameters are compressed into dynamic feature vectors through a one-dimensional convolutional layer, preserving the motion patterns of hand gestures; Lightweight temporal modeling is then used to calculate inter-frame dependencies. The dynamic feature vectors are input into the lightweight temporal modeling layer in chronological order. By updating the gate weights, the sigmoid function is used to activate and output the temporal feature vectors with complete motion trends. The enhanced feature map is a static feature, and the temporal feature vector is a dynamic feature. The static feature is compressed by global average pooling and then concatenated with the dynamic feature to obtain the static + dynamic fusion feature. The static + dynamic fusion feature includes gesture spatial information, related limb associations, and dynamic gesture trends.

8. The gesture command recognition method based on an improved contactless data stream network according to claim 7, characterized in that: The spatial-channel dual attention filtering module outputs five consecutive frames of feature maps (7×7×128), along with the corresponding 21 hand coordinates and 10 limb skeleton keypoint coordinates, to the lightweight temporal coding module. The 10 key skeletal points are the core nodes of the upper limbs and the core nodes of the trunk: The core nodes of the upper limb include the left wrist, left elbow, left shoulder, right wrist, right elbow, and right shoulder; The core nodes of the torso include the neck, head, left hip, and right hip; Calculate the static feature mean of the input 5 consecutive frame feature maps, and calculate the inter-frame motion parameters displacement rate and acceleration based on the dynamic features of the coordinate sequence of 31 key points; The displacement rate is: The acceleration is: Where △x is the displacement of adjacent frames in the x-direction, △y is the displacement of adjacent frames in the y-direction, and △t is the time difference between adjacent frames; The enhanced feature map is 7×7×128, the dynamic feature vector and the temporal feature vector are both 512-dimensional, the value of the temporal feature vector is between 0 and 1, and the static + dynamic fusion feature is 1024-dimensional.

9. The gesture command recognition method based on an improved contactless data stream network according to claim 1, characterized in that: The processing method for the dual-branch classification head is as follows: First, the static and dynamic fused features are normalized using the LayerNorm layer and then output to the main branch classification head and the auxiliary branch classification head. The main branch classification head performs global coarse classification of the input feature values ​​and normalizes them through the first fully connected layer, the second fully connected layer, and the third fully connected layer. The auxiliary branch classification head corrects the fine-grained differences in gestures and extracts key difference features from K groups of highly similar gestures in N groups of gesture actions to correct the main branch results. The first fully connected layer uses the ReLU activation function for computation, preserving core features for the first compression. The second fully connected layer uses the ReLU activation function for computation, preserving core features for the second compression. The third fully connected layer does not use an activation function and directly outputs the original N-dimensional gesture logits. The original N-dimensional gesture logits are then activated by Softmax to output the initial probability distribution value P of the main branch classification head. main Pmain consists of N values, which sum to 1, representing the confidence of N gesture categories; The method for processing the auxiliary branch classification head is as follows: Similarity category and difference feature extraction: K groups of similar gesture difference features are extracted from the input features through convolution and global pooling, and each group outputs L-dimensional difference features; Difference Probability Calculation: The difference features are processed through a fully connected network to output the difference probabilities of the two classes. The K×L dimension is reduced to 1 / 2×K×L dimension, and the 1 / 2×K×L dimension is further reduced to 2 dimensions. The difference probability values ​​of the two classes within the group are output. Finally, the corrected weights w are obtained through the Sigmoid activation function. i ,w i ∈(0~1); Main branch correction calculation: based on the correction weight w i For the initial probability distribution value P main The probability of similar categories is calculated, amplifying the probability difference between two gestures in the same group of similar gestures, while the probabilities of other dissimilar categories remain unchanged; Classification: Output gesture category, and output the corrected gesture category probability distribution result P. final Select P final The category corresponding to the maximum value is the final identification result.

10. A gesture command recognition method based on an improved contactless data stream network according to claim 9, characterized in that: The dual-branch classification head maps 1024-dimensional feature vector values ​​to 18 types of gesture labels, serving as the final decision layer of the improved gesture recognition network model. The first fully connected layer compresses the 1024-dimensional features to 512-dimensional features. The second fully connected layer compresses the 512-dimensional features to 256-dimensional features while retaining the core features. The third fully connected layer directly outputs the original 18-dimensional logits output value. K is 7, and the highly similar gestures are: Stop - Pause Action, Forward - Come Over, Hear - See, I - You, Quick - Come Over, Quiet - Crouch, Cover - Crouch; L is 20, and each group outputs 20-dimensional difference features, so 7*20=140 dimensions; When a set of similar gestures A and B are corrected by weight w i If the probability is greater than 0.5, then the probability of gesture A is greater than the probability of gesture B. Therefore, the probability of gesture A is increased to be greater than P. main The value of reduces the probability of gesture B to less than P. main The value of .

Images