Hand action recognition method and system based on lightweight cross-modal space-time network

Abstract

The invention relates to the technical field of action recognition, and provides a hand action recognition method and system based on a lightweight cross-modal space-time network, and the method comprises the following steps: carrying out the preprocessing of a hand-washing video stream, and obtaining an RGB image sequence and a hand key point coordinate sequence; performing separation convolution operation of multi-layer space convolution and time convolution on the RGB image sequence, extracting motion-related features in combination with a channel attention mechanism, and generating a visual feature mark sequence; performing semantic extraction and time sequence compression on the key point coordinate sequence to obtain a structural feature mark sequence aligned with the X; and dynamically calculating a gating coefficient based on context information, fusing X and H to obtain fusion features, and outputting classification probability distribution of the hand washing steps. According to the method, a lightweight visual feature extraction structure is adopted, and a dynamic gating mechanism is introduced to realize adaptive collaborative fusion of visual and structural modes, so that high-precision and low-delay hand action recognition is realized on edge equipment.

Description

Technical Field

[0001] This invention relates to the field of motion recognition technology, specifically to a hand motion recognition method and system based on a lightweight cross-modal spatiotemporal network. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In high-risk environments such as clinical medicine, food processing, and public health, standardized hand hygiene procedures are the most direct and economical means of preventing the spread of pathogens. With the accelerating pace of intelligent technology development, automated and real-time handwashing compliance monitoring equipment has become a necessity. Intelligent systems capable of autonomously recognizing handwashing steps are crucial for ensuring public safety. However, traditional methods relying on manual inspections or post-incident checks are inefficient, prone to omissions, and unable to meet the demands of high-frequency, high-density applications. Therefore, developing a lightweight identification solution suitable for edge deployment that can automatically identify the entire handwashing process has significant practical urgency and social value.

[0004] Current handwashing action recognition technologies still face multiple challenges in practical applications. On the one hand, handwashing involves several fine movements, such as rubbing fingertips, rubbing palms together, and crossing fingers. These movements involve rapid changes in detail and small amplitude, and are often accompanied by strong visual interference such as water flow and foam splashing. Methods that rely solely on RGB images for recognition are unstable in complex backgrounds, lighting changes, and water flow obstruction, easily leading to false positives or false negatives. On the other hand, while traditional skeleton point recognition methods show strong robustness under low light and occlusion conditions, they lack detailed information such as texture and color, making it difficult to independently support high-precision recognition. Furthermore, although 3D convolutional neural networks (such as I3D and SlowFast) with strong spatiotemporal modeling capabilities can improve recognition accuracy, they consume enormous computational resources, failing to meet the deployment requirements of edge devices for low power consumption and low latency. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a hand action recognition method and system based on a lightweight cross-modal spatiotemporal network. It integrates a lightweight visual feature extraction structure that combines separable convolution and channel attention, and introduces a dynamic gating mechanism to achieve adaptive collaborative fusion of visual and structural modalities, thereby enabling high-precision, low-latency hand action recognition on edge devices.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: One or more embodiments provide a hand action recognition method based on a lightweight cross-modal spatiotemporal network, including the following steps: The acquired handwashing video stream is preprocessed to obtain an RGB image sequence and a sequence of hand key point coordinates; For each image in the RGB image sequence, perform multi-level spatial convolution and temporal convolution operations separately. After each separate convolution operation, enhance the motion-related channel response through channel attention, and obtain the visual feature label sequence X after pooling. Semantic extraction and temporal compression are performed on the input hand keypoint coordinate sequence, and the output is a structural feature sequence H aligned with the visual feature label sequence X; The visual feature label sequence X and the structural feature label sequence H are dynamically calculated using joint contextual information, and then fused through a gating mechanism to obtain fused features, thereby generating the classification probability distribution of handwashing steps.

[0007] One or more embodiments provide a hand motion recognition system based on a lightweight cross-modal spatiotemporal network, including: Video data acquisition device and processor; Video data acquisition devices are used to acquire video streams; The processor is configured to perform the steps in the hand motion recognition method based on a lightweight cross-modal spatiotemporal network described above.

[0008] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the aforementioned hand motion recognition method based on a lightweight cross-modal spatiotemporal network.

[0009] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the aforementioned hand motion recognition method based on a lightweight cross-modal spatiotemporal network.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: This embodiment integrates the visual detail representation capabilities of RGB images with the structural stability of skeletal key points, effectively improving the model's robustness in real-world scenarios such as complex backgrounds, varying lighting, and occlusion. By introducing a separate convolutional structure of spatial and temporal convolutions, key spatiotemporal dynamic features are preserved while reducing computational complexity, meeting the dual requirements of lightweight design and real-time performance for edge devices. The channel attention mechanism further enhances the model's perception of motion-related regions, resulting in higher accuracy in recognizing refined handwashing actions such as rubbing the palms, alternating hand movements, and cleaning between the fingers. Structural feature extraction, through compression and semantic modeling of temporal information of hand key points, improves the model's understanding of the action evolution process and ensures precise temporal alignment with visual modal features. The dynamic gating fusion mechanism adaptively adjusts the fusion weights based on multimodal context features, achieving efficient collaboration between visual and structural information, further improving overall recognition accuracy and stability, and ensuring high reliability and practical value of the system even under edge deployment conditions.

[0011] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0012] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 This is a schematic diagram of the lightweight cross-modal spatiotemporal network of Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the structure of the enhanced 3D convolutional visual extraction module of Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the structure-aware key point encoding module of Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the cross-modal gating fusion module performing fusion according to Embodiment 1 of the present invention; Figure 5 This is a flowchart of the hand motion recognition method of Embodiment 1 of the present invention; Detailed Implementation The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0014] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0015] It should be noted that the terminology used herein is for describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0016] Example 1 In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 5 As shown, the hand action recognition method based on a lightweight cross-modal spatiotemporal network includes the following steps: Step 1: Preprocess the acquired handwashing video stream to obtain an RGB image sequence and hand key point coordinate vectors; Step 2: Perform multi-level spatial convolution and temporal convolution operations on each image in the RGB image sequence. After each separable convolution operation, enhance the motion-related channel response through channel attention, and obtain the visual feature label sequence X after pooling. Step 3: Perform semantic extraction and temporal compression on the input hand key point coordinate sequence [B,T,84], and output the structural feature sequence H[B,Tcnn,128] aligned with the visual feature label sequence X; Step 4: Combine the visual feature label sequence X and the structural feature label sequence H, dynamically calculate the gating coefficients using joint context information, fuse them through a gating mechanism to obtain fused features, and then generate the classification probability distribution of handwashing steps.

[0017] In this embodiment, a lightweight cross-modal neural network model is used to integrate visual images and hand skeleton data to improve the accuracy and robustness of handwashing action recognition. First, preprocessing operations convert the raw handwashing video data into RGB image sequences and corresponding hand keypoint coordinate sequences, preparing for subsequent modal feature extraction. In the visual modality, a separate spatial and temporal convolution strategy is employed to achieve effective spatiotemporal feature decoupling, thereby accurately capturing action changes. Simultaneously, a channel attention mechanism is used to enhance channel responses related to hand movements, making the model focus more on feature regions with semantic information. Next, pooling operations are used to compress the feature dimensions and extract the visual feature sequence X. In the structural modality, the input hand keypoint coordinate sequence has dimensions [B, T, 84], where B is the batch size, T is the number of time frames, and 84 is the total dimension of the keypoints. This sequence is processed by an embedding layer to extract spatial semantics and then compressed by a temporal compression module into a structural feature sequence H aligned with X, with dimensions [B, Tcnn, 128]. Finally, a context-based dynamic gating mechanism is used to adaptively fuse visual and structural features. The gating coefficients are calculated jointly by X and H to adjust the proportion of information from different modalities, thereby achieving cross-modal information synergistic enhancement and outputting the classification probability distribution of handwashing action steps.

[0018] This embodiment integrates the visual detail representation capabilities of RGB images with the structural stability of skeletal key points, effectively improving the model's robustness in real-world scenarios such as complex backgrounds, varying lighting, and occlusion. By introducing a separate convolutional structure of spatial and temporal convolutions, key spatiotemporal dynamic features are preserved while reducing computational complexity, meeting the dual requirements of lightweight design and real-time performance for edge devices. The channel attention mechanism further enhances the model's perception of motion-related regions, resulting in higher accuracy in recognizing refined handwashing actions such as rubbing the palms, alternating hand movements, and cleaning between the fingers. Structural feature extraction, through compression and semantic modeling of temporal information of hand key points, improves the model's understanding of the action evolution process and ensures precise temporal alignment with visual modal features. The dynamic gating fusion mechanism adaptively adjusts the fusion weights based on multimodal context features, achieving efficient collaboration between visual and structural information, further improving overall recognition accuracy and stability, and ensuring high reliability and practical value of the system even under edge deployment conditions.

[0019] Steps 1 to 4 above are achieved by constructing a lightweight cross-modal spatiotemporal network. The lightweight cross-modal spatiotemporal network includes a preprocessing module, an enhanced 3D convolutional visual extraction module, and a structure-aware keypoint encoding module and a cross-modal gating fusion module, which are arranged in parallel in sequence. Specifically, in step 1, the RGB image sequence is extracted through a preprocessing module, including the following steps: Step 11: Extract a sequence of RGB images with a duration of T frames from the continuous handwashing video stream; Optionally, the number of frames T can be set to cover a complete or partial handwashing process to ensure the continuity of the action sequence and contextual information; it can be set to a fixed value, such as 16 frames. Step 12: Adjust each frame of RGB image to the set resolution to obtain an RGB image sequence; Optionally, the resolution can be set to 112×112, 224×224, etc.; uniform adjustment of image resolution can ensure that the image input is consistent with the input specifications of the subsequent neural network model and eliminate resolution differences caused by different acquisition devices.

[0020] Specifically, the preprocessing module extracts the coordinate vectors of key hand points, including the following steps: Step 101: Call a hand key point detection algorithm (such as MediaPipe Hands) on each frame of RGB image to automatically identify the key points of the hands in the image and form the hand structure through skeleton connection.

[0021] This embodiment identifies 21 key points, each containing two-dimensional or three-dimensional coordinate information, such as three-dimensional coordinates, and forms a stable hand structure through a skeleton connection method; Step 102, Key Point Coordinate Normalization: The key point coordinates extracted from each frame of RGB image are normalized according to the image size to obtain the hand key point coordinate vector, and then the hand key point coordinate sequence is constructed. Specifically, normalization to the range of [0, 1] can eliminate the effects of different shooting distances or hand position shifts, making it easier for the subsequent lightweight cross-modal spatiotemporal network to learn spatial variation patterns.

[0022] Further technical solutions include temporal alignment and interpolation of key points: when key points are missing or incompletely detected, linear interpolation can be used to complete the key point sequence, or a sliding window method can be used to ensure that each segment of input data maintains the temporal consistency and stability of key points.

[0023] The method in step 2 can be implemented by constructing an enhanced 3D convolutional visual extraction module, such as... Figure 2 As shown, the enhanced 3D convolutional visual extraction module includes multi-layered cascaded spatiotemporal convolutional extraction units. Each spatiotemporal convolutional extraction unit includes a spatial convolutional layer, a temporal convolutional layer, a channel attention layer, and a pooling layer connected in sequence. The pooling layer of the last spatiotemporal convolutional extraction unit adopts spatial pyramid pooling, and the extracted features are output through the stitching and flattening units as visual feature label sequence X. The enhanced 3D CNN branch in this embodiment, serving as the backbone of the visual modality, employs a decompositional design, breaking down the computationally intensive 3D convolution into spatial and temporal convolutions. This module does not use standard 3D convolution kernels. Instead, it is decomposed into spatial convolution ( ) and temporal convolution ( For example, first through Convolution extracts intra-frame texture, then... Convolutional modeling of inter-frame motion significantly reduces the number of parameters; at the same time, it integrates a 3D channel attention mechanism to dynamically suppress background noise, focus on the hand motion region, and generate a visual feature label sequence X with a high compression ratio.

[0024] Step 2: Perform multi-level spatial convolution and temporal convolution operations on each image in the RGB image sequence. After each separable convolution operation, enhance the motion-related channel response through channel attention. After pooling, obtain the visual feature label sequence X, including the following steps: Step 21: Spatial convolution processing to model intra-frame features: Perform spatial convolution on the input RGB image sequence to extract local spatial features from each frame as the spatial convolution result; local spatial features include texture information, edge contours, etc. In this implementation, the spatial convolution operation uses a convolution kernel size of [size missing]. It only operates on the spatial dimension, avoiding the high computational cost of a one-time 3D convolution.

[0025] Step 22: Temporal convolution processing to model inter-frame motion: After obtaining the spatial convolution result, temporal convolution processing is performed to model the motion in the time dimension, capture the action changes and gesture motion trajectories across frames, and obtain the feature map U after spatiotemporal convolution. Step 23: For the feature map U after each spatiotemporal convolution, embed a 3D channel attention mechanism to enhance the response of the feature channels related to hand movements, and obtain the enhanced feature map. The specific steps are as follows: Step 231: Perform global average pooling on the spatiotemporal feature map U to obtain the global descriptor for each channel. ; Step 232: Input the channel descriptor into two layers of multilayer perceptron (MLP) and sigmoid activation function to generate channel weights g; Step 233: Perform channel-weighted calibration on the current feature map to enhance the motion-related channel responses, resulting in a feature map with enhanced responses. ; Step 24: After channel attention, spatial pooling is used to compress the feature map size, reducing spatial redundancy and enhancing the translation invariance of the model. Step 25: Repeat steps 21 to 24 to complete multiple spatiotemporal convolution extractions. After spatial pyramid pooling, feature splicing and flattening are performed to output the visual feature label sequence X. Specifically, in step 25, steps 21 to 24 above are combined into a spatiotemporal convolution extraction stage, which can be performed as follows: Figure 2 The model employs three or more stacked stages to extract higher-level spatiotemporal features, with the number of channels increasing sequentially to 32, 64, and 128 channels. Following the final (128-channel) spatiotemporal convolutional unit, a Spatial Pyramid Pooling (SPP) module is introduced. This module uses pooling windows of different sizes to perform multi-scale abstraction of the feature maps, extracting global and local spatial contextual information, effectively enhancing the model's adaptability to actions at different scales.

[0026] Step 3 is achieved through the constructed structure-aware keypoint encoding branch. The structure-aware keypoint encoding branch is used to extract semantic dependencies and temporal dynamics in the anatomical structure of the hand, inject structural consistency features, and improve cross-modal feature alignment capabilities.

[0027] Furthermore, such as Figure 3 As shown, the structure-aware keypoint encoding module includes a semantic feature extraction unit and a spatiotemporal alignment unit; The semantic feature extraction unit is used to map the original key point coordinates to a high-dimensional latent space through a multilayer perceptron and combine a channel attention mechanism to highlight hand joint features that are strongly related to the current gesture. Specifically, the semantic feature extraction unit includes a first linear layer and a channel attention layer connected in sequence. The linear layer projects the coordinates of the input key points and maps them into a latent representation to enhance semantic expressiveness. Specifically, the dimensionality transformation is from 84 dimensions to 64 dimensions. Optionally, the first linear layer adopts a multilayer perceptron. The channel attention layer uses a channel attention mechanism to enhance the channel response of key points that are recognizable by the current gesture (such as fingertips, palms, and wrists). The spatiotemporal alignment unit is used to perform adaptive average pooling in the temporal dimension on the semantic features after key point feature extraction, compressing the temporal length to be consistent with the temporal resolution of the visual feature label sequence X, and expanding the spatial dimension to generate a structural feature label sequence that is geometrically compatible with the visual features. Specifically, the spatiotemporal alignment unit includes a second linear layer, a first transposed layer, an adaptive pooling layer, and a second transposed layer connected in sequence; The second linear layer uses a multilayer perceptron to expand the dimensions of the semantic features output by the semantic feature extraction unit. For example, the 64-dimensional features are expanded to 128 dimensions through the linear layer to construct a multimodal consistent representation space. The first transpose layer is used to transpose the dimensions, transforming the expanded feature dimensions from [B, T, 128] to [B, 128, T], so as to facilitate subsequent pooling along the time dimension; The adaptive pooling layer applies adaptive average pooling to the time dimension, compressing the time length to the same dimension Tcnn as the visual feature label sequence X, thus obtaining the structural feature sequence. The second transpose layer transposes and restores the structural feature sequence to obtain the structural feature label sequence H. The structure-aware keypoint encoding module constructed in this embodiment mines the dependencies between joints through multilayer perceptron and channel attention, such as changes in fingertip distance and wrist rotation angle, and solves the alignment problem between high frame rate keypoints and low frame rate video features through adaptive temporal pooling.

[0028] Step 3, which involves semantic extraction and temporal compression of the input hand keypoint coordinate sequence [B,T,84] to output a structural feature sequence [B,Tcnn,128] aligned with the visual feature label sequence X, includes the following steps: Step 31: Extract semantic features from the key point sequence: Use a multilayer perceptron to map the coordinates of key points in each frame to the semantic feature space, and introduce a channel attention mechanism to enhance the response of joints (such as fingertips, wrists, etc.) in the key points, model the interdependence of joints in the anatomical structure of the hand, and obtain the extracted semantic features. In this embodiment, a channel attention mechanism is introduced to precisely highlight and enhance key joint information most relevant to the current gesture. Through this mechanism, the system can intelligently identify and differentiate the contributions of different parts of the body, such as effectively distinguishing the high-frequency subtle movements of the fingertips from the relative stability of the palm region, thereby significantly improving the accuracy and robustness of gesture recognition. This approach not only enhances the model's sensitivity to key features but also effectively suppresses interference from background or irrelevant joints, making the overall gesture analysis more focused and efficient.

[0029] Step 32: Compress and align the extracted semantic features in the time dimension. Use an adaptive pooling method to compress the feature sequence along the time dimension and align the key point features to the frame rate length Tcnn of the visual feature label sequence X to ensure the alignment consistency of the two modal features in the time dimension, and obtain the structural feature label sequence H.

[0030] Finally, a sequence of structural feature labels consistent with the visual modality dimension is generated, which serves as one of the inputs for subsequent multimodal fusion, thereby improving the model's ability to model structural information in dynamic gestures.

[0031] Step 4: Combine the visual feature label sequence X and the structural feature label sequence H, dynamically calculate the gating coefficients using joint context information, fuse them through a gating mechanism to obtain fused features, and then generate the classification probability distribution of handwashing steps; Step 4, the method for generating fused features, includes the following steps: Step 41, Cross-attention operation: Use the visual feature label sequence X as the query vector and the structural feature label sequence H as the key vector and value vector, calculate the attention weights between modalities and generate the refined features after structural guidance, namely the attention-optimized features H1; Step 42, Joint Context Awareness: Concatenate the visual feature label sequence X with the refined feature H1 along the channel dimension to construct a joint context tensor. ; Step 43, Dynamic Gating Generation: Generate the joint context tensor Discriminative cues are extracted using a multilayer perceptron (MLP) containing dimensionality reduction and dimensionality increase layers, and then dynamic gating coefficients G are generated by activation functions. Optionally, the activation function can be the Sigmoid activation function, and the generated dynamic gating coefficient G value is between 0 and 1; Step 44, Complementary Residual Fusion: Based on the obtained dynamic gating coefficients G, the visual feature label sequence X and the refined features are weighted and fused to obtain the final cross-modal fused features.

[0032] Step 4 is achieved by constructing a cross-modal gated fusion module, which is used to organically combine multimodal information. The cross-modal gated fusion module includes a cross attention unit, a joint context-aware unit, a dynamic gated generation unit, and a complementary residual fusion unit, which are configured to execute steps 41 to 44, respectively. The cross-modal gating fusion module first uses a cross-attention mechanism to inject structural features into the visual feature space; then it introduces an adaptive gating fusion mechanism to dynamically calculate gating coefficients based on joint context information, and adaptively adjusts the weights of the original visual information and the prior structural information at the pixel level to achieve soft-switching feature fusion.

[0033] Cross-modal gating fusion module such as Figure 4 As shown, this paper addresses the problems of modality alignment and dynamic complementarity. First, a cross-attention mechanism is used to label sequences with visual features. For the query, the structural feature label sequence H is used as the key and value. Attention weights are calculated and refined features are generated. However, relying solely on Subtle visual textures may be lost. Therefore, this embodiment introduces an adaptive gating fusion mechanism; By combining context-aware units, the original visual features With refining characteristics Concatenate along the channel dimension to form a joint context. ; To accurately assess the reliability of each modality at the current time step, this embodiment constructs a learnable gating network. The joint context tensor [H,X] (e.g., with a dimension of 2C) is input into a multilayer perceptron (MLP) to perform channel compression and recovery, generating gating coefficients.

[0034] The multilayer perceptron (MLP) in this embodiment includes two fully connected layers: The first layer compresses the feature channels and extracts nonlinear discriminative features using the ReLU activation function; Optionally, the compression ratio can be set to r=4; The second layer restores the number of channels to C. Finally, the output is mapped to the (0,1) interval using the Sigmoid activation function to generate the dynamic gating coefficient G, the specific calculation formula of which is as follows: ; in, and Here, δ represents the ReLU activation function, σ represents the Sigmoid activation function, and [.;.] represents the concatenation operation along the channel dimension. The physical meaning of G lies in adaptively measuring the confidence of the original visual feature label sequence X: when the visual information is clear and unobstructed, G approaches 1; when there is bubble occlusion or motion blur, G approaches 0.

[0035] Complementary Residual Fusion: Based on the generated gating coefficients, a complementary residual structure is used to weight and fuse features. Unlike traditional feature addition, this embodiment achieves pixel-level feature selection through a "soft-switching" mechanism. The final fused features... The calculation formula is: ; Here, ⊙ represents element-wise multiplication, and 1 represents a matrix of all ones. This mechanism ensures that the model can automatically switch and rely on structural features H under visual interference, while retaining rich visual textures X under normal scenes, thus achieving dynamic complementarity of cross-modal information.

[0036] Step 4 can be achieved by constructing an action classification module, including an enhanced spatiotemporal Transformer unit, a global spatiotemporal pooling unit, and an MLP classifier; Enhanced spatiotemporal Transformer unit for capturing long-range global spatiotemporal dependencies in fused features; The global spatiotemporal pooling unit is used to compress feature tensors into segment-level global semantic vectors; An MLP classifier is used to map semantic vectors to probability scores for handwashing action categories through a lightweight fully connected layer.

[0037] The action classification module receives fused features. After the enhanced spatiotemporal Transformer processes long-range dependencies, global spatiotemporal average pooling is performed, and the probability of the handwashing action category is output through an MLP classifier.

[0038] During the training process of the aforementioned lightweight cross-modal spatiotemporal network model, a combined loss function of cross-entropy loss and Focal Loss is used to address the sample imbalance problem.

[0039] This embodiment proposes an ultra-lightweight network architecture (GestureWashNet) with only 0.37M parameters and a computational cost of only 3.61 GFLOPs, achieving a balance between performance and efficiency through modular collaborative design. Addressing the foam occlusion problem unique to handwashing scenarios, this invention utilizes key hand points to provide stable geometric priors and automatically increases structural feature weights when visual obstruction occurs through an adaptive gating mechanism, significantly improving robustness. Experiments on a self-built handwashing dataset and the publicly available Jester dataset demonstrate that the system maintains recognition accuracy exceeding that of mainstream heavyweight networks (such as I3D and ResNet101) while significantly reducing computational power requirements, making it particularly suitable for embedded intelligent hygiene monitoring terminals.

[0040] To illustrate the effectiveness of the method in this embodiment, a simulation experiment was conducted, the details of which are as follows: The network model constructed in this embodiment and the existing model were tested on a self-made handwashing dataset and a portion of the publicly available Jester dataset. The experimental results are shown in Table 1 and Table 2, respectively. Table 1. Performance comparison results on the self-made hand cleaning dataset; Model Data modal Top-1 Acc (%) Macro-F1 2D CNN + LSTM RGB 82.3 0.78 I3D RGB 88.6 0.83 X3D-S RGB 87.2 0.82 GestureWashNet (the model in this embodiment) RGB + Keypoints 98.5 0.98 In Table 1, 2D CNN+LSTM means: a two-dimensional convolutional neural network + a long short-term memory network, which is a hybrid model that combines a two-dimensional convolutional neural network and a long short-term memory network.

[0041] I3D: refers to Inflated 3D CNN, which is a model obtained by inflating the convolutional kernels of 2D CNN to 3D (adding a temporal dimension), and capturing the spatial and temporal features of the video simultaneously through 3D convolution.

[0042] X3D-S: is a small version of the X3D model (X3D includes variants of different sizes, such as X3D-S, X3D-M, etc.). It is a lightweight 3D CNN model that achieves efficient spatiotemporal feature extraction by balancing parameters such as network depth, width, and time dimension.

[0043] As shown in Table 1, using a multimodal input of RGB (Red / Green / Blue, three-channel color video) fusion keypoints, this embodiment achieves a Top-1 Accuracy of 98.5% on a self-built handwashing dataset. Top-1 accuracy is the proportion of the predicted first class that matches the true class. Simultaneously, the Macro-F1 score (a macro-average F1 score used to measure overall multi-class balance performance) reaches 0.98. Compared to comparative methods using only RGB (such as 2D CNN+LSTM, I3D, X3D-S, etc.), this embodiment demonstrates superior performance in both accuracy and class balance, proving the significant gain of fusing keypoint information for fine-grained handwashing action discrimination.

[0044] Table 2 shows the performance comparison results on the general dataset Jester dataset; Model Data modal Params(M) GFLOPs Top-1 Acc (%) MFNet RGB 8.0 1.2 96.6 STM RGB 25.6 8.3 96.7 DirecFormer RGB 63.2 10.8 98.1 X3D MobileNet-V3 RGB 3.1 0.45 95.5 PAN ResNet101 RGB 44.5 7.6 97.4 GestureWashNet (the model in this embodiment) RGB + Keypoints 0.37 3.61 94.4 MFNet stands for Multi-Flow Network, a video understanding model with a multi-branch structure. It extracts features from different flows (such as spatial and temporal flows) and then fuses them. It is often used for action recognition tasks.

[0045] STM stands for Spatio-Temporal Memory network, a model that combines spatio-temporal context with memory mechanisms, and is good at capturing long-term dependencies in video sequences.

[0046] DirecFormer is an action recognition model based on the Transformer architecture. It improves the efficiency of spatiotemporal feature modeling by optimizing the computation of the self-attention mechanism.

[0047] X3D MobileNet-V3: This is a combination of the X3D (Extended 3D) model and MobileNet-V3 (Lightweight Convolutional Network). It is a lightweight 3D CNN model that balances spatiotemporal feature extraction capabilities with deployment efficiency.

[0048] PAN ResNet101: This model combines ResNet101 as the basic feature extraction backbone network with the feature aggregation structure of PAN, forming a model that balances deep spatial feature extraction and multi-scale feature fusion capabilities.

[0049] As shown in Table 2, this embodiment achieves a Top-1 accuracy of 94.4% on Jester (a public dataset for general gesture recognition), while requiring only 3.61 GFLOPs (Giga Floating-point Operations, used to measure inference computational overhead). Compared to heavy-duty contrastive networks such as DirecFormer and STM, this embodiment maintains high recognition accuracy while significantly reducing computational power consumption and model size. Therefore, this embodiment is more suitable for real-time deployment in computing-constrained scenarios such as mobile devices and edge computing, demonstrating a balance between lightweight design and high accuracy.

[0050] Experimental results show that the method in this embodiment achieves a Top-1 accuracy of 98.5% on the self-built handwashing dataset and 94.4% accuracy on the Jester general gesture dataset, with a computational cost of only 3.61 GFLOPs. This is superior to existing heavy networks such as DirecFormer and STM, achieving a balance between lightweight design and high precision.

[0051] Example 2 Based on Embodiment 1, this embodiment provides a hand motion recognition system based on a lightweight cross-modal spatiotemporal network, including: Video data acquisition device and processor; Video data acquisition devices are used to acquire video streams; The processor is configured to perform the steps in the hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in Example 1.

[0052] Optionally, the video data acquisition device can use an RGB camera or a depth camera to adapt to different application scenarios and needs.

[0053] RGB cameras can capture rich color information and are suitable for scenes with good lighting conditions and no need for depth perception; while depth cameras, by acquiring three-dimensional spatial information, can provide more accurate hand motion data in complex backgrounds or low-light conditions.

[0054] The video data acquisition device can also integrate infrared sensors to enhance the ability to recognize hand contours, especially in the presence of occlusion or interference.

[0055] The processor is responsible for real-time processing and analysis of the acquired video stream. It integrates a lightweight cross-modal spatiotemporal network model, enabling efficient completion of the entire process from data preprocessing to feature extraction, fusion, and classification. To further improve system response speed and stability, the processor employs a multi-threaded parallel computing architecture and combines hardware acceleration technologies (such as GPUs or TPUs) to optimize the execution efficiency of key algorithms. Simultaneously, the processor supports dynamically adjusting computing resource allocation to adapt to the performance requirements of different devices and the complexity of tasks.

[0056] In practical deployments, this system can be flexibly applied to various terminal devices. By combining hand gesture recognition technology with specific application scenarios, the system can not only achieve high-precision gesture classification but also provide users with intuitive feedback and guidance. For example, in a smart handwashing scenario, the system can provide real-time prompts based on the recognition results to indicate whether the user has completed the correct handwashing steps, thereby effectively improving the standardization and efficiency of hygiene management.

[0057] Example 3 Based on Embodiment 1, this embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps in the hand motion recognition method based on a lightweight cross-modal spatiotemporal network described in Embodiment 1.

[0058] Example 4 Based on Embodiment 1, this embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they complete the steps in the hand motion recognition method based on a lightweight cross-modal spatiotemporal network described in Embodiment 1.

[0059] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0060] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A hand motion recognition method based on a lightweight cross-modal spatiotemporal network, characterized in that, Includes the following steps: The acquired handwashing video stream is preprocessed to obtain an RGB image sequence and a sequence of hand key point coordinates; For each image in the RGB image sequence, perform multi-level spatial convolution and temporal convolution operations separately. After each separate convolution operation, enhance the motion-related channel response through channel attention, and obtain the visual feature label sequence X after pooling. Semantic extraction and temporal compression are performed on the input hand keypoint coordinate sequence, and the output is a structural feature sequence H aligned with the visual feature label sequence X; The visual feature label sequence X and the structural feature label sequence H are dynamically calculated using joint contextual information, and then fused through a gating mechanism to obtain fused features, thereby generating the classification probability distribution of handwashing steps.

2. The hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in claim 1, characterized in that: The method for extracting RGB image sequences includes the following steps: Extract a sequence of RGB images with a duration of T frames from the handwashing video stream; Adjust each frame of the RGB image to the set resolution; The method for extracting the coordinate sequence of key points on the hand includes the following steps: The hand key point detection algorithm is called on each frame of RGB image to automatically identify the key points of the hands in the image and form the hand structure through skeleton connection. The key point coordinates extracted from each frame of RGB image are normalized according to the image size to obtain the hand key point coordinate vector, and then the hand key point coordinate sequence is constructed.

3. The hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in claim 1, characterized in that: For each image in the RGB image sequence, perform multi-level spatial convolution and temporal convolution operations separately. After each separate convolution operation, enhance the motion-related channel response through channel attention. After pooling, obtain the visual feature label sequence X, including the following steps: Step 21: Perform spatial convolution on the input RGB image sequence to extract local spatial features from each frame as the spatial convolution result; Step 22: After obtaining the spatial convolution result, perform temporal convolution processing to model in the time dimension, capture cross-frame action changes and gesture motion trajectories, and obtain the feature map after spatiotemporal convolution. Step 23: For the feature map after each spatiotemporal convolution, embed a 3D channel attention mechanism to enhance the response of the feature channels related to hand movements, and obtain the enhanced feature map; Step 24: After channel attention, spatial pooling is used to compress the feature map size, reducing spatial redundancy and enhancing the translation invariance of the model. Step 25: Repeat steps 21 to 24 to complete multiple spatiotemporal convolution extractions. After spatial pyramid pooling, feature splicing and flattening are performed to output the visual feature label sequence X.

4. The hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in claim 3, characterized in that: For each feature map resulting from spatiotemporal convolution, a 3D channel attention mechanism is embedded to enhance the response of feature channels related to hand movements, resulting in an enhanced feature map. The specific steps are as follows: Global average pooling is performed on the spatiotemporal feature map to obtain the global descriptor for each channel. ; The global descriptor of the channel is input into two layers of multilayer perceptron and a sigmoid activation function to generate channel weights; Based on channel weights, the current feature map is weighted and calibrated by channel to enhance the motion-related channel responses, resulting in a feature map with enhanced responses.

5. The hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in claim 1, characterized in that: A method for semantic extraction and temporal compression of the input hand keypoint coordinate sequence [B,T,84] to output a structural feature sequence [B,Tcnn,128] aligned with the visual feature label sequence X includes the following steps: Semantic feature extraction of key point sequences: Multilayer perceptron is used to map the coordinates of key points in each frame to the semantic feature space, and channel attention mechanism is introduced to enhance the response of joints in key points, model the interdependence of joints in the anatomical structure of the hand, and obtain the extracted semantic features. The extracted semantic features are compressed and aligned in the time dimension. An adaptive pooling method is used to compress the feature sequence along the time dimension and align the key point features to the frame rate length Tcnn of the visual feature label sequence X to ensure the alignment consistency of the two modal features in the time dimension, thus obtaining the structural feature label sequence H.

6. The hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in claim 1, characterized in that: The method for generating fused features includes the following steps: Using the visual feature label sequence X as the query vector and the structural feature label sequence H as the key vector and value vector, we calculate the attention weights between modalities and generate the refined features after structural guidance, namely the attention optimization feature H1. The visual feature label sequence X and the refined feature H1 are concatenated along the channel dimension to construct a joint context tensor. ; Joint context tensor Discriminative cues are extracted using a multilayer perceptron containing dimensionality reduction and dimensionality increase layers, and then dynamic gating coefficients G are generated by activation functions. Based on the obtained dynamic gating coefficient G, the visual feature label sequence X and the refined features are weighted and fused to obtain the final cross-modal fusion feature.

7. The hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in claim 1, characterized in that: It also includes the construction of a lightweight cross-modal spatiotemporal network, which includes a preprocessing module, an enhanced 3D convolutional visual extraction module, and a structure-aware keypoint encoding module and a cross-modal gating fusion module, arranged in parallel. The enhanced 3D convolutional visual extraction module includes multi-layered cascaded spatiotemporal convolutional extraction units. Each spatiotemporal convolutional extraction unit includes a spatial convolutional layer, a temporal convolutional layer, a channel attention layer, and a pooling layer connected in sequence. The pooling layer of the last spatiotemporal convolutional extraction unit adopts spatial pyramid pooling, and the extracted features are output through the stitching and flattening units as visual feature label sequence X. The structure-aware keypoint encoding module includes a semantic feature extraction unit and a spatiotemporal alignment unit; The semantic feature extraction unit consists of a first linear layer and a channel attention layer connected in sequence. The linear layer projects the coordinates of the input key points and maps them into a latent representation. The spatiotemporal alignment unit comprises a second linear layer, a first transposed layer, an adaptive pooling layer, and a second transposed layer connected in sequence. The second linear layer is used to expand the dimensions of the semantic features output by the semantic feature extraction unit and construct a multimodal consistent representation space. The first transpose layer is used to transpose the dimensions, transforming the expanded feature dimensions from [B, T, 128] to [B, 128, T], so as to facilitate subsequent pooling along the time dimension; The adaptive pooling layer applies adaptive average pooling to the time dimension, compressing the time length to the same dimension Tcnn as the visual feature label sequence X, thus obtaining the structural feature sequence. The second transpose layer transposes and restores the structural feature sequence to obtain the structural feature label sequence H.

8. A hand motion recognition system based on a lightweight cross-modal spatiotemporal network, characterized in that, include: Video data acquisition device and processor; Video data acquisition devices are used to acquire video streams; The processor is configured to perform the steps of the hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in any one of claims 1-7.

9. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps in the hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps in the hand motion recognition method based on a lightweight cross-modal spatiotemporal network as described in any one of claims 1-7.

Images