Lightweight pseudo-multi-mode gesture recognition method and system based on segmented motion energy

Abstract

The invention discloses a lightweight pseudo-multi-mode gesture recognition method and system based on segmented motion energy, and relates to the technical field of image recognition, and the method comprises the steps: employing a monocular camera to collect continuous video clips containing complete gesture actions, and carrying out the preprocessing of the continuous video clips; based on a preset pseudo multi-mode branch, respectively extracting pseudo multi-mode data representing static appearance, dynamic motion and time sequence significance in the RGB video sequence; constructing an expert committee structure consisting of a multi-modal exclusive expert network and a dynamic gating network by taking the pseudo multi-modal data as input; a pseudo-multi-mode branch and an expert committee structure are used as a teacher model, and a student model corresponding to training is constructed. According to the method, the information of the time dimension is encoded to the channel dimension, so that a lightweight two-dimensional convolutional network can directly sense the evolution sequence and the dominant direction of the gesture action, and explicit modeling of the dynamic time sequence characteristics is realized with extremely low calculation cost.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a lightweight pseudo-multimodal gesture recognition method and system based on segmented motion energy. Background Technology

[0002] With the development of gesture recognition technology, it has been widely used in application scenarios with high requirements for real-time performance, reliability, and concealment, such as public safety law enforcement collaboration, emergency response, special operations, and intelligent terminal control. For example, when law enforcement officers are carrying out on-site handling or arrest coordination tasks, they often need to quickly convey instructions to colleagues or systems through standardized gestures in low-noise, low-visibility, or inconvenient voice communication environments, which places higher demands on the response speed, stability, and resistance to environmental interference of gesture recognition.

[0003] In the above application scenarios, since gestures are usually characterized by short duration, large range of change, and uneven rhythm, the static hand shape information and dynamic motion information contained in a single RGB video are prone to perceptual conflict on the time scale. Furthermore, complex lighting changes and background interference will further amplify this conflict, making it difficult for existing gesture recognition methods based on single appearance features or simple temporal modeling to stably capture key motion information while ensuring real-time performance.

[0004] Currently, gesture recognition technology can be broadly categorized into two types: 1) Gesture recognition methods based on real multimodal sensors. These methods typically acquire RGB images, depth images, infrared information, or inertial sensor data simultaneously. They extract features from each modality using multi-branch convolutional neural networks (CNNs) with RGB and depth branches, and then perform feature fusion and classification in later stages. The basic principle relies on complementary information from different sensors, such as using depth information to separate foreground and background or using infrared information to adapt to low-light scenes, thereby improving recognition accuracy and robustness in complex environments. 2) Gesture recognition methods based on single RGB video. These methods use ordinary RGB cameras to acquire video sequences, directly inputting consecutive frames into a convolutional neural network, or combining simple temporal modules such as average pooling and simple recurrent neural networks for end-to-end classification. These methods are relatively simple in structure and easy to deploy, but most only explicitly utilize the static appearance features of the image, lacking dedicated modeling of the dynamic temporal information of the gesture and the spatial salience reflecting the hand's position relative to a complex background.

[0005] However, these methods face certain shortcomings in practical applications: 1) High cost and insufficient flexibility: Many effective multimodal fusion methods rely on dedicated sensors such as depth cameras, which not only increases the hardware cost of the system but also limits its use on a large number of existing devices equipped only with ordinary RGB cameras. True multimodal fusion models, such as dual-branch convolutional neural networks, have complex structures, a large number of parameters, and high computational overhead, making it difficult to achieve real-time operation on terminal devices with limited computing power, power consumption, and memory, such as mobile phones, embedded devices, and XR headsets, thus limiting the popularization and application of the technology. 2) Difficulty in data acquisition and alignment: Methods based on real multimodal data require the simultaneous acquisition of data from multiple modalities. This process is not only cumbersome but also faces the technical challenge of strictly aligning data from different sensors in time and space. Any alignment deviation may introduce noise, which will reduce the fusion effect, and the cost of building a large-scale, accurately aligned multimodal dataset is extremely high. 3) Model robustness faces the risk of single point of failure. Although multimodal models aim to improve robustness, when data of a certain modality, such as depth information, is completely lost or its quality drops sharply due to sensor failure or environmental interference such as strong light, the branch that depends on the real input of that modality will fail, which may lead to the collapse of the entire system performance. There is a lack of effective degradation response mechanisms. Summary of the Invention

[0006] Therefore, it is necessary to provide a lightweight pseudo-multimodal gesture recognition method and system based on segmented motion energy to address the aforementioned technical problems.

[0007] In a first aspect, the present invention provides a lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy, comprising:

[0008] S1. Use a monocular camera to capture continuous video clips containing complete hand gestures, and preprocess the continuous video clips to obtain an RGB video sequence; the preprocessing includes video enhancement, multi-frame temporal alignment, and motion compensation.

[0009] S2. Based on the preset pseudo-multimodal branches, extract pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency from the RGB video sequence; the pseudo-multimodal data includes RGB enhanced images, segmented motion energy maps, and spatial attention maps;

[0010] S3. Using pseudo-multimodal data as input, construct an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network, and introduce dynamic weights to achieve adaptive fusion of the expert network outputs in order to complete gesture recognition in different environments.

[0011] S4. Using pseudo-multimodal branches and expert committee structure as teacher models, construct and train corresponding student models; and deploy student models to edge devices, triggering incremental learning with differential privacy protection when the triggering conditions are met.

[0012] Furthermore, a monocular camera is used to capture continuous video clips containing complete hand gestures, and these continuous video clips are preprocessed to obtain an RGB video sequence including:

[0013] S11. Use a monocular camera to acquire a continuous video stream, and use a sliding window mechanism to extract a continuous video segment containing complete hand gestures from the video stream.

[0014] S12. Extract image frames from continuous video segments, perform contrast-adaptive histogram equalization on the brightness channel of each image frame to enhance local contrast, and use multi-scale illumination correction to enhance the video quality of continuous video segments under different illumination conditions.

[0015] S13. The sharpness of the enhanced image frame is evaluated. When the calculated Laplacian variance is lower than the preset threshold, the corresponding image frame is determined to be a blurred frame, and the blurred frame is corrected by a gradient-based non-blind deblurring method.

[0016] S14. Use the sparse optical flow method to track the key points of the hand in adjacent image frames, calculate the corresponding affine transformation relationship based on the motion relationship of the key points between image frames, and perform sub-pixel alignment on consecutive image frames based on the affine transformation relationship.

[0017] S15. For consecutive image frames after alignment, calculate the pixel difference between adjacent image frames in the hand region. If the pixel difference exceeds the dynamic threshold, mark it as an abnormal frame and use bidirectional optical flow interpolation between the preceding and following image frames for reconstruction and replacement.

[0018] S16. After preprocessing, the image frames are scaled to a uniform size and then standardized according to the preset mean and standard deviation to obtain an RGB video sequence with enhanced quality and consistent timing.

[0019] Furthermore, based on the preset pseudo-multimodal branches, pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency are extracted from the RGB video sequence, including:

[0020] S21. Extract the RGB image of each frame in the RGB video sequence, input it into a two-dimensional convolutional neural network for encoding, obtain the RGB enhanced image through spatial domain and frequency domain transformation, and stack all the processed RGB enhanced images in the original time order to form an appearance pseudomodal data sequence.

[0021] S22. A dynamic modeling method based on time-series frequency domain analysis and decoupling of motion components extracts motion information from RGB images and constructs a main motion energy channel and a detail motion direction channel. By stitching them together, a segmented motion energy map is generated as motion pseudomodal data.

[0022] S23. An attention generation method based on motion-invariant subspace learning and spectral clustering separation processes RGB images to generate spatial attention maps, which serve as spatial attention pseudomodal data.

[0023] Furthermore, RGB images of each frame in the RGB video sequence are extracted and input into a two-dimensional convolutional neural network for encoding. RGB enhanced images are obtained through spatial and frequency domain transformations. All processed RGB enhanced images are then stacked in their original temporal order to form an appearance pseudomodal data sequence, including:

[0024] S211. Input each frame of RGB image in the RGB video sequence into a two-dimensional convolutional neural network, perform shallow feature extraction through depthwise separable convolutional layers, and output a 64-channel feature map.

[0025] S212. In a two-dimensional convolutional neural network, a frequency domain weight matrix is ​​introduced. A block-based two-dimensional discrete cosine transform is performed on each channel of the feature map to convert the spatial domain information to the frequency domain representation. The weighted frequency domain coefficients are then reconstructed back into the spatial domain through an inverse discrete cosine transform to obtain a frequency domain enhanced feature map.

[0026] S213. The frequency domain enhanced feature map is fused with the original RGB image after dimensionality reduction by a convolutional layer through a residual connection. The fusion result is then passed through another convolutional layer to adjust the number of channels to three, and an activation function is used to constrain the pixel values ​​to a preset range to obtain an RGB enhanced image of a preset size.

[0027] Furthermore, based on a dynamic modeling method that decouples time-series frequency domain analysis from motion components, motion information is extracted from RGB images, and main motion energy channels and detail motion direction channels are constructed. These are then stitched together to generate segmented motion energy maps, which serve as pseudo-modal motion data, including:

[0028] S221. Convert each frame of the RGB video sequence into a grayscale image to obtain a grayscale sequence, and calculate the absolute difference map sequence between adjacent frames.

[0029] S222. Extract the intensity value of each spatial location in the difference map sequence as a function of time to form a one-dimensional signal, and perform discrete Fourier transform to obtain the amplitude spectrum; add up the frequency components corresponding to the amplitude spectra of all spatial locations and calculate the average to obtain the global average motion energy spectrum.

[0030] S223. Identify peak frequencies with amplitudes higher than the average level in the global average motion energy spectrum, and find segmentation points in the time domain according to the period corresponding to the peak frequencies to divide the original RGB video sequence into several continuous sub-stages.

[0031] S224. Stack the difference graphs contained in each sub-stage into a three-dimensional spatiotemporal tensor. Separate the low-rank component tensor and sparse component tensor from the three-dimensional spatiotemporal tensor using the Tucker-2 decomposition method, which respectively represent the main motion information and detailed motion information within the sub-stage.

[0032] S225. By weighted averaging of each frame of RGB image along the time dimension of the low-rank component, the main motion energy channel is generated; by analyzing the time vector of each spatial location in the sparse component, the detail motion direction channel is generated and divided into two matrices: horizontal and vertical components.

[0033] S226. The main motion energy channel and two detailed motion direction channels generated in each sub-stage are stitched together along the channel dimension to form a three-channel image; and the three-channel images obtained from all sub-stages are stitched together along the channel dimension to generate a composite two-dimensional image with a total number of channels equal to three times the number of sub-stages, which serves as the segmented motion energy map.

[0034] Furthermore, an attention generation method based on motion-invariant subspace learning and spectral clustering separation processes RGB images to generate spatial attention maps, which serve as spatial attention pseudomodal data, including:

[0035] S231. Convert each frame of the RGB video sequence into a grayscale image to obtain a grayscale sequence, and calculate the dense optical flow field of adjacent frames to obtain the motion velocity sequence of each pixel in the horizontal and vertical directions, forming the original motion features.

[0036] S232. Divide the original motion features of each frame into local blocks, divide the grayscale image into multiple overlapping local blocks, and collect the motion velocity sequence of all pixels in each local block in consecutive frames to form a motion observation set.

[0037] S233. Perform motion-invariant subspace learning on the motion observation set of each local block, construct a graph structure based on motion subspace similarity, take the local block as a node in the graph structure, calculate the motion subspace similarity between any two nodes, and form a similarity matrix.

[0038] S234. Perform symmetric normalization on the similarity matrix, calculate the corresponding Laplacian matrix, and perform eigenvalue decomposition on the Laplacian matrix to calculate a preset number of non-zero eigenvalues ​​and their corresponding eigenvectors; then arrange the eigenvectors by column to form a new feature matrix.

[0039] S235. Cluster the new feature matrix, divide all local blocks into different clusters, and after identifying the main cluster belonging to the hand, map all local blocks in the main cluster back to the pixel space to generate the initial binary mask.

[0040] S236. Perform spatiotemporal consistency filtering on the binary mask, perform a logical OR operation on the binary mask after processing all frames along the time dimension to cover all hand areas reached by the gesture, and normalize to obtain the final spatial attention map.

[0041] Furthermore, using pseudo-multimodal data as input, an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network is constructed. Dynamic weights are introduced to achieve adaptive fusion of the expert network outputs, enabling gesture recognition in different environments, including:

[0042] S31. Input the RGB enhanced image into the appearance expert network, extract spatial features frame by frame and pool them into a frame-level feature vector sequence, perform convolution operation through a one-dimensional temporal convolution module, and perform global average pooling in the time dimension to output the appearance feature vector.

[0043] S32. Input the segmented motion energy map into the motion expert network, extract spatial pattern features through the adapted backbone network, and output the motion feature vector through global pooling.

[0044] S33. Input the spatial attention map into the spatial attention expert network, and output the attention feature vector after global average pooling;

[0045] S34. Using appearance feature vector, motion feature vector and attention feature vector as input, a dynamic gating network is introduced as the chairman network. Global average pooling is used to capture the global context, and two fully connected layers are used for processing to generate a set of dynamic fusion weights.

[0046] S35. Use fusion weights to sum the feature vectors to generate fusion features and input them into a fully connected classifier. After activation by the Softmax function, output the probability distribution of the gesture category.

[0047] Furthermore, using pseudo-multimodal branches and an expert committee structure as the teacher model, a corresponding student model is constructed and trained; and the student model is deployed to edge devices. When triggering conditions are met, incremental learning with differential privacy protection is triggered, including:

[0048] S41. The teacher model is trained end-to-end using the complete pseudo-multimodal branch, and the classification cross-entropy loss is optimized until the teacher model converges. Feature maps of multiple levels inside the teacher model network are extracted to construct a feature pyramid covering different semantic levels.

[0049] S42. Using aggregated pseudo-multimodal data as input, a lightweight network is used to construct a student model, and several feature adaptation modules are embedded in the intermediate layer. Under the guidance of the teacher model, a progressive knowledge transfer process is carried out to achieve end-to-end joint optimization of all parameters of the student model.

[0050] S43. Deploy the trained student model to the edge device, introduce an edge-side overhead adaptive mechanism, and form a personal dataset by locally caching the recognition of gestures and classification results with confidence exceeding a preset threshold within a preset period.

[0051] S44. Automatically monitor trigger conditions during idle periods of edge devices. When the trigger conditions are met, automatically trigger incremental learning and integrate local differential privacy protection functions during the incremental learning process to maintain a balance between personalized adaptation, privacy protection and real-time performance of edge devices.

[0052] Furthermore, during idle periods of the edge device, trigger conditions are automatically monitored. When the trigger conditions are met, incremental learning is automatically initiated, and local differential privacy protection functions are integrated during the incremental learning process, including:

[0053] S441. Based on the number of samples that meet the preset quality standards and the category distribution entropy value in the local cache of the edge device, the triggering condition is set. When the number of samples reaches the preset number threshold and the category distribution entropy value is lower than the preset entropy value threshold, incremental learning is automatically triggered.

[0054] S442. When incremental learning is triggered, the gradient of the model parameters is calculated in each incremental learning iteration, the gradient vector is clipped to limit the magnitude to a preset range, and random noise of the corresponding scale is generated according to the preset privacy protection strength parameter. The random noise is superimposed with the clipped gradient vector to obtain a noisy gradient and injected into the critical path of the student model update.

[0055] S443. During the incremental learning process, continuously track and calculate the consumption of the privacy budget, limit the accumulated privacy leakage risk to the preset security limit, and adopt an adaptive optimization strategy to dynamically optimize the gradient clipping threshold and adjust the clipping range of the gradient vector.

[0056] S444. Continuously track the performance metrics of the student model within the edge device. If the performance metrics deteriorate due to adaptive updates and exceed the preset performance threshold for ensuring real-time interaction, the rollback mechanism will be automatically triggered to restore the model parameters to the stable version before the incremental update.

[0057] Secondly, the present invention also provides a lightweight pseudo-multimodal gesture recognition system based on segmented motion energy, the system comprising:

[0058] The data acquisition and processing module is used to acquire continuous video clips containing complete hand gestures using a monocular camera, and to preprocess the continuous video clips to obtain RGB video sequences; the preprocessing includes video enhancement, multi-frame temporal alignment, and motion compensation.

[0059] The pseudo-multimodal generation module is used to extract pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency from RGB video sequences based on preset pseudo-multimodal branches; the pseudo-multimodal data includes RGB enhanced images, segmented motion energy maps, and spatial attention maps;

[0060] The feature extraction and inference module is used to construct an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network, using pseudo-multimodal data as input. Dynamic weights are introduced to achieve adaptive fusion of the expert network outputs in order to complete gesture recognition in different environments.

[0061] The edge deployment update module is used to build and train the corresponding student model using pseudo-multimodal branches and expert committee structure as teacher models; and to deploy the student model to edge devices, triggering incremental learning with differential privacy protection when the triggering conditions are met.

[0062] The beneficial effects of this invention are as follows:

[0063] 1. This invention abandons computationally complex methods like dense optical flow or 3D convolution. Its core lies in dividing a gesture video segment into multiple consecutive sub-stages along the temporal dimension, and simultaneously accumulating motion intensity and directional gradients within each stage. By stitching the three-channel energy maps of each stage along the channel dimension, a composite motion energy map is finally generated. This process essentially encodes information from the temporal dimension to the channel dimension, enabling a lightweight 2D convolutional network to directly perceive the evolution sequence and dominant direction of gesture movements, thus achieving explicit modeling of dynamic temporal features with extremely low computational cost.

[0064] 2. This invention derives a spatial attention pseudo-modality from the original RGB video sequence. Through a lightweight spatiotemporal saliency detection module, it integrates appearance and inter-frame motion cues to generate a spatial attention map that highlights the hand region. This map serves as an independent pseudo-modality input, guiding the corresponding expert network to focus its computation on the hand. This proactively suppresses background noise interference at the feature extraction level, enhancing the model's robustness in cluttered environments.

[0065] 3. This invention employs an expert committee architecture, where each expert network specializes in processing a specific pseudomodality and performing targeted feature extraction. A lightweight chairperson network serves as the decision-making core, analyzing the scene context in real time based on the features output by the three experts and dynamically generating the optimal fusion weights. This mechanism simulates the attention allocation during human comprehensive judgment, achieving adaptive content fusion at the feature level and ensuring stable decision-making performance even when information from any single modality decays.

[0066] 4. This invention employs a knowledge distillation framework. By constructing a lightweight student model with aggregated pseudomodal images as input, and using feature pyramids and output soft labels for progressive distillation, the student model inherits the multimodal decision-making capabilities of the teacher model. After deployment, the system continuously learns from users' personalized high-confidence data, performs lightweight incremental updates to the model's key parameters, and combines differential privacy and performance monitoring rollback mechanisms to achieve a balance between personalized adaptation, privacy protection, and real-time performance assurance. Attached Figure Description

[0067] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0068] Figure 1 This is a flowchart of the training phase in a lightweight pseudo-multimodal gesture recognition method based on segmented motion energy according to an embodiment of the present invention.

[0069] Figure 2 This is a schematic diagram of the lightweight deployment stage in the lightweight pseudo-multimodal gesture recognition method based on segmented motion energy according to an embodiment of the present invention;

[0070] Figure 3 This is a diagram illustrating the differentiated design of the expert network in a lightweight pseudo-multimodal gesture recognition method based on segmented motion energy, according to an embodiment of the present invention.

[0071] Figure 4 This is a schematic diagram of the dynamic gating fusion mechanism in the lightweight pseudo-multimodal gesture recognition method based on segmented motion energy according to an embodiment of the present invention;

[0072] Figure 5 This is a system principle block diagram of a lightweight pseudo-multimodal gesture recognition system based on segmented motion energy according to an embodiment of the present invention.

[0073] The reference numerals are as follows: 1. Data acquisition and processing module; 2. Pseudo-multimodal generation module; 3. Feature extraction and inference module; 4. Edge deployment and update module. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0075] Please see Figure 1 A lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy includes:

[0076] S1. Use a monocular camera to capture continuous video clips containing complete hand gestures, and preprocess the continuous video clips to obtain an RGB video sequence; the preprocessing includes video enhancement, multi-frame temporal alignment, and motion compensation.

[0077] In the description of this invention, a continuous video segment containing a complete hand gesture is acquired using a monocular camera, and the continuous video segment is preprocessed to obtain an RGB video sequence, including:

[0078] S11. Use a monocular camera to acquire a continuous video stream, and use a sliding window mechanism to extract a continuous video segment containing complete hand gestures from the video stream.

[0079] Specifically, this invention uses a standard monocular RGB camera to capture continuous video streams as input data. To ensure the continuity and integrity of rapid hand gestures in the temporal dimension, the camera frame rate is preferably set to 60 frames per second, and the video resolution is no less than 640×480 pixels. During video acquisition, a sliding window mechanism is used to extract continuous video segments containing complete hand gestures from the video stream. The time window length T is set to 30 frames, corresponding to a time span of approximately 0.5 seconds, and the window sliding step size is set to 4 frames to capture the complete dynamic process of rapid hand gestures within a short time range, while ensuring a high time response frequency in the recognition process.

[0080] S12. Extract image frames from continuous video segments, perform contrast-adaptive histogram equalization on the brightness channel of each image frame to enhance local contrast, and use multi-scale illumination correction to enhance the video quality of continuous video segments under different illumination conditions.

[0081] Specifically, after obtaining continuous video clips, the video quality is enhanced to adapt to complex lighting conditions. For each captured image frame, contrast-limited adaptive histogram equalization is first performed on its luminance channel to enhance local contrast and reduce detail loss in shadow or highlight areas caused by uneven lighting. Based on this, a multi-scale illumination correction method is further employed to process the image. By separating the illumination and reflection components in the image frame, the effects of non-uniform lighting, background glare, and low-light environments on image quality are effectively suppressed, ensuring that the hand area maintains relatively stable visual characteristics under different lighting conditions.

[0082] S13. The sharpness of the enhanced image frame is evaluated. When the calculated Laplacian variance is lower than the preset threshold, the corresponding image frame is determined to be a blurred frame, and the blurred frame is corrected by a gradient-based non-blind deblurring method.

[0083] Specifically, considering that motion blur is easily generated during rapid hand gestures, a sharpness assessment is performed on each image frame during image enhancement. The presence of significant motion blur is determined by calculating the Laplacian variance of the image frame. When the variance is lower than a preset threshold, the corresponding image frame is identified as a blurred frame, and a gradient-based non-blind deblurring method is used to quickly correct the blurred frame, thereby improving the overall sharpness and detail discernibility of the image.

[0084] S14. Use the sparse optical flow method to track the key points of the hand in adjacent image frames, calculate the corresponding affine transformation relationship based on the motion relationship of the key points between image frames, and perform sub-pixel alignment on consecutive image frames based on the affine transformation relationship.

[0085] Specifically, after performing quality enhancement at the single-frame level, multi-frame temporal alignment and motion compensation processing are performed on the frame sequence of continuous video segments. This invention tracks hand key points in adjacent video frames based on the sparse optical flow method, calculates the corresponding affine transformation relationship based on the motion relationship of the key points between frames, and performs sub-pixel alignment on continuous frames accordingly to compensate for global displacement and local deformation caused by camera shake or target movement.

[0086] S15. For consecutive image frames after alignment, calculate the pixel difference between adjacent image frames in the hand region. If the pixel difference exceeds the dynamic threshold, mark it as an abnormal frame and use bidirectional optical flow interpolation between the preceding and following image frames for reconstruction and replacement.

[0087] Specifically, for the aligned consecutive frame sequence, the pixel difference between adjacent frames is further calculated within the hand region to evaluate temporal consistency. When the difference between a frame and its preceding and following frames exceeds a dynamic threshold, the frame is marked as an anomalous frame. This anomalous situation may be caused by instantaneous occlusion or sudden changes in illumination. For anomalous frames, bidirectional optical flow interpolation between its preceding and following frames can be used to reconstruct and replace them, or the weight of the frame can be significantly reduced in subsequent feature extraction processes, thereby ensuring the consistency and reliability of the input sequence in the temporal dimension.

[0088] S16. After preprocessing, the image frames are scaled to a uniform size and then standardized according to the preset mean and standard deviation to obtain an RGB video sequence with enhanced quality and consistent timing.

[0089] Specifically, after completing the above enhancement, alignment, and consistency verification processes, each image frame is uniformly scaled to 224×224 pixels and standardized according to preset mean and standard deviation to finally obtain a stable and time-consistent RGB video sequence, which serves as the input for subsequent pseudo-multimodal feature construction and gesture recognition processes.

[0090] The aforementioned complete enhancement and alignment process is primarily used during the model training phase to improve data quality and model robustness. In practical application deployments, to meet real-time requirements, only critical size scaling and standardization operations can be performed based on the device's computing power, with some enhancement steps serving as optional modules.

[0091] S2. Based on the preset pseudo-multimodal branches, extract pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency from the RGB video sequence respectively; the pseudo-multimodal data includes RGB enhanced images, segmented motion energy maps, and spatial attention maps.

[0092] After obtaining a stable RGB video sequence processed in stage S1, this step aims to construct three pseudo-multimodal features from the same source data, focusing on static appearance, dynamic motion, and temporal saliency, respectively. These features are explicitly separated at the perceptual level, providing a multi-angle description of the target gesture and laying the foundation for subsequent differential processing and collaborative decision-making by the expert network.

[0093] In the description of this invention, based on preset pseudo-multimodal branches, the extraction of pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency from RGB video sequences includes:

[0094] S21. Extract each frame of RGB image from the RGB video sequence, input it into a two-dimensional convolutional neural network for encoding, obtain RGB enhanced images through spatial and frequency domain transformation, and stack all the processed RGB enhanced images in the original time order to form an appearance pseudomodal data sequence.

[0095] In the description of this invention, each frame of RGB image is extracted from an RGB video sequence, input into a two-dimensional convolutional neural network for encoding, and RGB enhanced images are obtained through spatial and frequency domain transformation. All processed RGB enhanced images are then stacked in their original chronological order to form an appearance pseudomodal data sequence, including:

[0096] S211. Input each frame of RGB image in the RGB video sequence into a two-dimensional convolutional neural network, perform shallow feature extraction through depthwise separable convolutional layers, and output a 64-channel feature map.

[0097] Specifically, to extract static hand shape, contour, and texture information crucial for gesture category discrimination, each temporally aligned RGB image frame is independently input into a lightweight 2D convolutional neural network for encoding. This network employs a selective feature sharpening lightweight network structure, with input being a single-frame RGB image of size 3×224×224. The network first performs shallow feature extraction through a lightweight depthwise separable convolutional layer using a 3×3 kernel, stride 1, and padding 1, outputting a 64-channel feature map to preserve rich spatial details.

[0098] S212. Introduce a frequency domain weight matrix into a two-dimensional convolutional neural network, perform a block-based two-dimensional discrete cosine transform on each channel of the feature map, convert the spatial domain information to the frequency domain representation, and reconstruct the weighted frequency domain coefficients back to the spatial domain through an inverse discrete cosine transform to obtain a frequency domain enhanced feature map.

[0099] Specifically, a block-based two-dimensional discrete cosine transform is performed on each channel of the obtained feature map to convert the spatial domain information to a frequency domain representation. A learnable frequency domain weight matrix is ​​introduced into the two-dimensional convolutional neural network, the size of which corresponds to the block spectrum of the discrete cosine transform. The parameters of this matrix are initialized in the early stage of training as a distribution with higher weights on mid-to-high frequency components to a priori emphasize edge and texture information. The matrix is ​​adaptively optimized through the network training process to learn the optimal frequency domain filtering characteristics for the hand appearance discrimination task.

[0100] S213. The frequency domain enhanced feature map is fused with the original RGB image after dimensionality reduction by a convolutional layer through a residual connection. The fusion result is then passed through another convolutional layer to adjust the number of channels to three, and an activation function is used to constrain the pixel values ​​to a preset range to obtain an RGB enhanced image of a preset size.

[0101] Specifically, the weighted frequency domain coefficients are reconstructed back into the spatial domain using inverse discrete cosine transform to obtain a frequency domain enhanced feature map. This enhanced feature map is then fused with the original input image, which has undergone dimensionality reduction via a 1×1 convolutional layer, through a residual connection. The fused result is then passed through another 1×1 convolutional layer to adjust the number of channels back to 3, and a sigmoid activation function is used to constrain the pixel values ​​to a reasonable range. Finally, an enhanced RGB image with a size maintained at 3×224×224 is output.

[0102] This process enables the output RGB enhanced image to maintain a natural appearance while specifically enhancing the contour sharpness and texture details of the hand area, and effectively suppressing low-frequency gradient components and random high-frequency noise in the background area. Each frame in a continuous video clip undergoes the above processing independently, and all processed frames are stacked in their original chronological order to form an appearance pseudomodal data sequence. This sequence serves as the direct input to the appearance expert network in step S3.

[0103] S22. A dynamic modeling method based on time-series frequency domain analysis and decoupling of motion components extracts motion information from RGB images and constructs a main motion energy channel and a detail motion direction channel. By stitching them together, a segmented motion energy map is generated as motion pseudomodal data.

[0104] In the description of this invention, a dynamic modeling method based on time-series frequency domain analysis and decoupling of motion components is used to extract motion information from RGB images and construct a main motion energy channel and a detail motion direction channel. These are then stitched together to generate a segmented motion energy map, which serves as motion pseudomodal data.

[0105] S221. Convert each frame of the RGB video sequence into a grayscale image to obtain a grayscale sequence, and calculate the absolute difference map sequence between adjacent frames.

[0106] Specifically, to highlight the motion information in rapid gestures, this invention constructs a direction-aware segmented motion energy map, converts each frame of the RGB video sequence into a grayscale image to obtain a grayscale sequence, then calculates the absolute difference map sequence between adjacent frames to obtain three-dimensional data representing the motion intensity between frames; then adaptive temporal segmentation is performed, and the difference map sequence is regarded as a three-dimensional spatiotemporal signal.

[0107] S222. Extract the intensity value of each spatial location in the difference map sequence as a function of time to form a one-dimensional signal, and perform discrete Fourier transform to obtain the amplitude spectrum; add up the frequency components corresponding to the amplitude spectrum of all spatial locations and calculate the average to obtain the global average motion energy spectrum.

[0108] Specifically, for each spatial location in the difference map sequence, its intensity value changing over time is extracted to form a one-dimensional signal, and a discrete Fourier transform is performed to obtain the amplitude spectrum. The frequency components corresponding to the amplitude spectra of all spatial locations are added together and averaged to obtain the global average motion energy spectrum.

[0109] S223. Identify peak frequencies with amplitudes higher than the average level in the global average motion energy spectrum, and find segmentation points in the time domain according to the period corresponding to the peak frequencies to divide the original RGB video sequence into several continuous sub-stages.

[0110] Specifically, peak frequencies with amplitudes significantly higher than the average level are identified in the global average motion energy spectrum. Based on the periods corresponding to these peak frequencies, segmentation points are determined in the time domain, thereby dividing the original sequence into K consecutive sub-stages.

[0111] S224. Stack the difference graphs contained in each sub-stage into a three-dimensional spatiotemporal tensor. Use the Tucker-2 decomposition method to separate the low-rank component tensor and sparse component tensor from the three-dimensional spatiotemporal tensor, which respectively represent the main motion information and detailed motion information within the sub-stage.

[0112] Specifically, for each sub-stage, its contained difference graphs are stacked to form a three-dimensional spatiotemporal tensor. To separate the main motion and detailed motion from this tensor, the Tucker-2 decomposition method is used. This decomposition aims to approximate the original tensor as the product of two spatial factor matrices and a core tensor.

[0113] In practice, two low-rank dimensions smaller than the original spatial size are first preset to define the size of the factor matrices and constrain the complexity of the model. Then, these factor matrices and the core tensor are initialized, and iterative optimization is performed using alternating least squares. In each iteration, other variables are fixed sequentially, and one of the factor matrices or the core tensor is updated by minimizing the reconstruction error. This update process is repeated until the solution converges or a preset iteration stopping condition is met. After convergence, the final factor matrix is ​​multiplied by the core tensor to reconstruct the low-rank component tensor, which smoothly represents the coherent, large-scale main motion pattern within the sub-stage. Simultaneously, the residual between the original tensor and the low-rank component is calculated to obtain the sparse component tensor, which concentrates the local, rapidly changing detailed motion information in the original data.

[0114] S225. By weighted averaging of the low-rank components along the time dimension of each frame of RGB image, a main motion energy channel is generated. By analyzing the time vector of each spatial location in the sparse components, a detail motion direction channel is generated and divided into two matrices: a horizontal matrix and a vertical matrix.

[0115] Specifically, the generation of the main motion energy channel is achieved by weighted averaging of the low-rank components along the time dimension for each frame. The weighting coefficients are generated by a Gaussian function, emphasizing motion information in the middle of the stage. The generation of the detailed motion direction channel is accomplished by analyzing the time vector of each spatial location in the sparse components. The discrete gradient of this time vector is calculated to obtain the horizontal and vertical gradient sequences. Then, the covariance matrix of these two gradient sequences is calculated, and eigenvalues ​​and corresponding eigenvectors are obtained from the eigenvalues.

[0116] To distinguish between the true direction of motion and random directions caused by noise, a significance threshold is set for judgment. This threshold is automatically determined based on the statistical characteristics of the current covariance matrix; for example, it can be set as a scaling factor of the mean of the diagonal elements of the covariance matrix (i.e., the variances of the two directional gradients themselves). The largest eigenvalue and its corresponding unit eigenvector are selected. If the largest eigenvalue is greater than the significance threshold, a dominant direction of motion is considered to exist at that location, and the two components of this eigenvector are used as the values ​​for the horizontal and vertical directions, respectively. Otherwise, the location is determined to have no significant directionality, and the direction components are set to zero. After traversing all spatial locations, two matrices representing the horizontal and vertical components of the dominant direction of detailed motion are obtained. These matrices are then concatenated to form the "horizontal direction channel" and "vertical direction channel" of the motion pseudomodal data.

[0117] S226. The main motion energy channel and two detailed motion direction channels generated in each sub-stage are stitched together along the channel dimension to form a three-channel image. The three-channel images obtained from all sub-stages are then stitched together along the channel dimension to generate a composite two-dimensional image with a total number of channels equal to three times the number of sub-stages, which serves as the segmented motion energy map.

[0118] Specifically, the main motion energy channel and two detailed motion direction channels generated in each sub-stage are stitched together along the channel dimension. During stitching, these three matrices are directly stacked to form a three-channel image with three channels and a spatial dimension of H×W. Its data can be represented as a three-dimensional array. The channel order is usually conventional; for example, the first channel stores the main motion energy matrix, the second channel stores the horizontal direction matrix, and the third channel stores the vertical direction matrix.

[0119] The three-channel images I1, I2, ..., I obtained from all K sub-stages K The data is then stitched together again along the channel dimension, with the three channels of I2 appended to the three channels of I1, followed by the three channels of I3, and so on, ultimately generating a composite two-dimensional image with a total of 3×K channels. This image is then used as the motion pseudomodal data input to the subsequent motion expert network. This method achieves adaptive stage division through temporal frequency domain analysis and decouples motion components through tensor decomposition, constructing a more informative motion representation.

[0120] S23. An attention generation method based on motion-invariant subspace learning and spectral clustering separation processes RGB images to generate spatial attention maps, which serve as spatial attention pseudomodal data.

[0121] In order to guide the subsequent feature extraction network to focus on the hand region and suppress interference from complex backgrounds, a spatial attention map needs to be generated from the RGB video sequence.

[0122] Specifically, the time-aligned RGB video sequence is processed using a lightweight attention generation module based on motion-invariant subspace learning and spectral clustering separation. The spatial attention map generated in step S23 will serve as the sole input data for the spatial attention expert network.

[0123] In the description of this invention, the attention generation method based on motion-invariant subspace learning and spectral clustering separation processes RGB images to generate spatial attention maps, which serve as spatial attention pseudomodal data, including:

[0124] S231. Convert each frame of the RGB video sequence into a grayscale image to obtain a grayscale sequence, and calculate the dense optical flow field of adjacent frames to obtain the motion velocity sequence of each pixel in the horizontal and vertical directions, forming the original motion features.

[0125] Specifically, the RGB video sequence is converted to a grayscale sequence, and the dense optical flow field of adjacent frames is calculated to obtain the motion velocity sequence of each pixel in the horizontal and vertical directions, forming the original motion features. To capture the inherent structure of the motion pattern, the motion features of each frame are divided into local blocks, dividing the image into multiple overlapping local blocks. For each local block, the motion velocity sequences of all pixels within it in consecutive frames are collected to form a motion observation set.

[0126] S232. Divide the original motion features of each frame into local blocks, divide the grayscale image into multiple overlapping local blocks, and collect the motion velocity sequence of all pixels in each local block in consecutive frames to form a motion observation set.

[0127] Specifically, motion-invariant subspace learning is performed on the motion observation set of each local block. Specifically, it is assumed that hand movements are consistent within the local spatiotemporal range, and their motion patterns can be approximated by a low-dimensional linear subspace.

[0128] Principal component analysis (PCA) is performed on all motion observations within the block to extract the top k principal component vectors. The subspace spanned by these principal components is the local motion subspace of the block. Simultaneously, the reconstruction error of this subspace to the observation data is calculated. The smaller the reconstruction error, the more the motion of the block conforms to the low-dimensional subspace assumption, and the more likely it is to be a coherent hand movement.

[0129] S233. Perform motion-invariant subspace learning on the motion observation set of each local block, construct a graph structure based on motion subspace similarity, take the local block as a node in the graph structure, calculate the motion subspace similarity between any two nodes, and form a similarity matrix.

[0130] Specifically, this invention constructs a graph structure based on motion subspace similarity, treating each local block as a node in the graph and calculating the motion subspace similarity between any two nodes. The similarity metric uses the cosine similarity between the principal components of the subspace, combined with their spatial proximity. Based on this similarity matrix, spectral clustering is performed to separate motion patterns.

[0131] S234. Perform symmetric normalization on the similarity matrix, calculate the corresponding Laplacian matrix, and perform eigenvalue decomposition on the Laplacian matrix to calculate a preset number of non-zero eigenvalues ​​and their corresponding eigenvectors; then arrange the eigenvectors by column to form a new feature matrix.

[0132] Specifically, the similarity matrix is ​​symmetrically normalized, and its corresponding Laplacian matrix is ​​calculated. Eigenvalue decomposition is then performed on this Laplacian matrix, calculating its first m smallest non-zero eigenvalues ​​and their corresponding eigenvectors. These eigenvectors are arranged column-wise to form a new feature matrix, where each row represents the coordinates of an original local block in the reduced feature space. The row vectors of this feature matrix are then clustered using the K-means clustering algorithm, dividing all blocks into m distinct categories. The number of clusters, m, can be set to a small, fixed value, such as 2 to 4, based on the eigenvalue distribution or through preliminary experiments.

[0133] S235. Cluster the new feature matrix, divide all local blocks into different clusters, and after identifying the main cluster belonging to the hand, map all local blocks in the main cluster back to the pixel space to generate the initial binary mask.

[0134] Specifically, the core objective is to group similar features in the motion subspace into the same category through a clustering process. Because hand movements have inherent spatiotemporal consistency, their corresponding blocks cluster tightly in the feature space and are therefore assigned to the same primary, and usually largest, cluster. Inconsistent or heterogeneous motion patterns in the background are separated into other subcategories. Based on this, the algorithm identifies the cluster containing the most blocks as the "main hand cluster."

[0135] S236. Perform spatiotemporal consistency filtering on the binary mask, perform a logical OR operation on the binary mask after processing all frames along the time dimension to cover all hand areas reached by the gesture, and normalize to obtain the final spatial attention map.

[0136] Specifically, after identifying the main cluster belonging to the hand, all blocks in the main cluster are mapped back to pixel space to generate an initial binary mask. This mask undergoes spatiotemporal consistency filtering: morphological operations are used in the spatial dimension to fill holes and smooth boundaries, while median filtering is applied to the masks of consecutive frames in the temporal dimension to eliminate jitter. The processed masks of all frames are then logically ORed along the temporal dimension to cover all areas reached by the hand during the entire gesture, and normalized to obtain the final spatial attention map, which serves as the input to the spatial attention pseudomodality.

[0137] S3. Using pseudo-multimodal data as input, an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network is constructed, and dynamic weights are introduced to achieve adaptive fusion of the expert network outputs in order to complete gesture recognition in different environments.

[0138] To address the differences in information density and discrimination criteria among various pseudomodalities and avoid feature conflicts and insufficient robustness, this step constructs an expert committee structure consisting of multiple modality-specific expert networks and a dynamic gating network. This structure uses asymmetric expert design to model various features specifically and leverages dynamic weights to achieve adaptive fusion, thereby achieving robust gesture recognition in complex environments. All three expert networks are built upon the lightweight MobileNetV3-Small backbone, which is primarily composed of stacked inverse residual structures and channel attention modules.

[0139] In the description of this invention, pseudo-multimodal data is used as input to construct an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network. Dynamic weights are introduced to achieve adaptive fusion of the expert network outputs, thereby completing gesture recognition in different environments, including:

[0140] S31. Input the RGB enhanced image into the appearance expert network, extract spatial features frame by frame and pool them into a sequence of frame-level feature vectors, perform convolution operation through a one-dimensional temporal convolution module, and perform global average pooling in the time dimension to output the appearance feature vector.

[0141] Specifically, the input to the appearance expert network is a preprocessed RGB enhanced image with dimensions [T, 3, 224, 224]. The network first extracts spatial features frame by frame and pools them into a sequence of frame-level feature vectors. This sequence then passes through a dedicated one-dimensional temporal convolutional module. This module contains two one-dimensional convolutional layers with kernel sizes of 3 and 5, respectively, and employs ReLU activation and layer normalization to explicitly model the temporal dependency of gestures. Global average pooling is then performed in the temporal dimension to output the appearance feature vector.

[0142] S32. Input the segmented motion energy map into the motion expert network, extract spatial pattern features through the adapted backbone network, and output the motion feature vector through global pooling.

[0143] Specifically, the input to the motion expert network is the segmented motion energy map generated in step S2, with dimensions [1, 9, 224, 224]. To adapt the input channels, the number of input channels in the first convolutional layer of the backbone network is changed from 3 to 9, and the newly added weights are randomly initialized. Since the motion energy map has already encoded temporal information into the channel dimension, the temporal convolution module is removed from the network. Spatial pattern features are directly extracted through the adapted backbone network, and then the motion feature vector is output through global average pooling.

[0144] S33. Input the spatial attention map into the spatial attention expert network, and output the attention feature vector after global average pooling.

[0145] Specifically, the input to the spatial attention expert network is the spatial attention map generated in step S2, with dimensions [1, 1, 224, 224]. Similarly, the number of input channels for the first convolutional layer of the backbone network is modified to 1, and the temporal convolutional module is removed. Considering the relatively concentrated information in the attention map, a shallower backbone network can be used to reduce computational overhead, and the attention feature vector is output after global average pooling.

[0146] S34. Using appearance feature vector, motion feature vector and attention feature vector as input, a dynamic gating network is introduced as the chairman network. Global average pooling is used to capture the global context, and two fully connected layers are used for processing to generate a set of dynamic fusion weights.

[0147] Specifically, to dynamically fuse the feature vectors output by the three expert networks, a lightweight dynamic gating network is introduced as the chairman network. This network takes the concatenated result of the three feature vectors as input, first capturing the global context through global average pooling, and then processing it through a two-layer fully connected layer. The first fully connected layer maps the feature dimension to 256 dimensions and uses ReLU activation, while the second fully connected layer outputs the raw three-dimensional scores, which are then normalized using the Softmax function to obtain a set of dynamically fused weights. These weights are related to the current input content and can adaptively adjust according to scene changes such as lighting, motion speed, and background complexity. For example, in low-light environments, the weights of motion features are automatically increased; in cluttered backgrounds, the weights of spatial attention features are enhanced.

[0148] S35. Use fusion weights to sum the feature vectors to generate fusion features and input them into a fully connected classifier. After activation by the Softmax function, output the probability distribution of the gesture category.

[0149] Specifically, the fused features are obtained by weighting and summing the feature vectors of each expert using dynamic fusion weights, and then input into a fully connected classifier. After Softmax activation, the output is the probability distribution of the gesture category. For example... Figures 3-4 As shown, the expert committee structure, through differentiated design and dynamic adaptive integration, significantly improves the robustness and accuracy of the gesture recognition system in complex environments.

[0150] In a more specific parameterized implementation, the output feature vectors of each expert network have a dimension of 1024, resulting in a joint feature dimension of 3072 after concatenation. The parameters of the two fully connected layers in the chairman network are Linear(3072, 256) and Linear(256, 3). In the temporal convolutional module of the appearance expert network, both convolutional layers have a stride of 1, and padding is set to maintain the same length. After modifying the weights of the first convolutional layer in the motion expert network, its kernel shape is [9, 16, 3, 3], where 16 is the number of output channels. The final classifier is a Linear(1024, N) layer, where N is the total number of gesture categories. The weighted fusion formula is:

[0151] F final =α×F RGB +β×F Motion +γ×F attention ;

[0152] In the above formula, F final For the final fused feature vector, F RGB F is the feature vector output by the appearance expert network. Motion F is the feature vector output by the motion expert network. Attentionα represents the feature vector output by the attention expert network, and β and γ represent the normalized weights output by the dynamic gating network.

[0153] S4. Using pseudo-multimodal branches and an expert committee structure as the teacher model, construct and train the corresponding student model. Deploy the student model to an edge device, and when the triggering conditions are met, trigger incremental learning with differential privacy protection.

[0154] In the description of this invention, as Figure 2 As shown, a pseudo-multimodal branch and expert committee structure are used as the teacher model to construct and train the corresponding student model. The student model is then deployed to an edge device, and when triggering conditions are met, incremental learning with differential privacy protection is initiated, including:

[0155] S41. The teacher model is trained end-to-end using complete pseudo-multimodal branches, and the classification cross-entropy loss is optimized until the teacher model converges. Feature maps of multiple levels inside the teacher model network are extracted to construct a feature pyramid covering different semantic levels.

[0156] Specifically, the pseudo-multimodal fusion model, constructed based on steps S1 to S3 and incorporating multiple expert committees, is used as the teacher model. To transfer its capabilities to a lightweight model suitable for edge deployment, a student model needs to be constructed and trained. The teacher model is trained as described in S3, using three complete pseudo-multimodal branches for end-to-end training. The classification cross-entropy loss is optimized until the model converges, resulting in a baseline model with strong representational capabilities. After the teacher model is trained, feature maps from multiple levels within its network are further extracted to construct a feature pyramid covering high, medium, and low semantic levels. These features originate from the expert committee network that collaboratively models multimodal information. Their multi-scale representations contain complementary cross-modal cues, providing a more robust and rich source of structured knowledge for the student model compared to the hierarchical features of a single-modal network.

[0157] S42. Using aggregated pseudo-multimodal data as input, a student model is constructed using a lightweight network, and several feature adaptation modules are embedded in the intermediate layer. Under the guidance of the teacher model, a progressive knowledge transfer process is carried out to achieve end-to-end joint optimization of all parameters of the student model.

[0158] Specifically, the student model employs a simplified, lightweight network with a completely different structure from the teacher network. Its backbone is MobileNetV3-Small, and several lightweight feature adaptation modules are embedded in its key intermediate layers. Training the student model is a progressive knowledge transfer process guided by the teacher model. First, the parameters of the pre-trained teacher model are fixed, serving as a static knowledge source. The input to the student model is aggregated pseudo-multimodal data (pseudo-modal representation images). Representative pseudo-modal data generated in step S2, such as temporal average frames extracted from RGB sequences, motion energy maps, and spatial attention maps, are concatenated along the channel dimension to form a multi-channel fused image as input. This invention allows the lightweight student model to directly receive and learn the multi-source information relied upon by the teacher model without requiring a complex multi-branch structure.

[0159] The student model training process sequentially performs output layer knowledge distillation, intermediate layer feature alignment, and final joint fine-tuning. Output layer distillation is achieved by minimizing the KL divergence loss between the softened outputs of the student and the teacher. Feature alignment is achieved through an adaptation module, which guides the intermediate feature maps of the student model to align with the feature maps of the corresponding levels in the feature pyramid of the teacher model in the vector space.

[0160] Finally, the classification cross-entropy loss, output distillation loss, and multi-level feature imitation loss are weighted and combined to jointly optimize all parameters of the student model end-to-end, thereby completing deep knowledge transfer and obtaining a lightweight, high-performance model.

[0161] S43. Deploy the trained student model to the edge device, introduce an edge-side overhead adaptive mechanism, and form a personal dataset by locally caching gestures and classification results whose confidence exceeds a preset threshold within a preset period.

[0162] Specifically, the fully trained lightweight student model will be deployed to resource-constrained edge devices. To enhance the model's personalized adaptability in real-world end-user scenarios, this method introduces a low-overhead adaptive mechanism at the edge during deployment. During operation, the device locally caches recently identified user gesture data with high confidence levels and their results, forming a miniature personal dataset.

[0163] S44. Automatically monitor trigger conditions during idle periods of edge devices. When the trigger conditions are met, automatically trigger incremental learning and integrate local differential privacy protection functions during the incremental learning process to maintain a balance between personalized adaptation, privacy protection and real-time performance of edge devices.

[0164] In the description of this invention, triggering conditions are automatically monitored during idle periods of the edge device. When the triggering conditions are met, incremental learning is automatically triggered, and local differential privacy protection functions are integrated during the incremental learning process, including:

[0165] S441. Based on the number of samples that meet the preset quality standards and the category distribution entropy value in the local cache of the edge device, the triggering condition is set. When the number of samples reaches the preset number threshold and the category distribution entropy value is lower than the preset entropy value threshold, incremental learning is automatically triggered.

[0166] Specifically, the system automatically monitors triggering conditions during device idle periods and employs a multimodal confidence joint evaluation strategy. This strategy not only statistically analyzes the confidence of the recognition results but also analyzes the motion coherence and appearance clarity indices of the gesture data. Motion coherence is evaluated by calculating the temporal variance of the motion energy map in the gesture segment, while appearance clarity is evaluated by calculating the local contrast of the hand region image. Only when the recognition confidence of a sample is higher than a threshold, and both motion coherence and appearance clarity meet preset standards, is the sample considered a high-quality sample and included in the cache. The triggering conditions are based on a comprehensive judgment of the number of high-quality samples in the cache and their category distribution entropy values: when the number of high-quality samples reaches a preset threshold, such as 50, and the category distribution entropy value is lower than the threshold, indicating that user habits are stabilizing, the system automatically triggers incremental learning.

[0167] Once the triggering conditions are met, the system initiates a rapid incremental learning process. During the initial model training phase, the importance of each parameter is estimated by calculating the second derivative of the loss function with respect to the model parameters, such as using a diagonal Fisher information matrix. During incremental learning, the system only selects a small number of parameters with high importance, such as the top 10%, for updates. These parameters are typically concentrated in the later layers of the network and in dynamically gated networks. Simultaneously, the update process introduces elastic weights to reinforce regularization terms, preventing catastrophic forgetting of old knowledge.

[0168] Specifically, a term is added to the loss function to penalize changes in important parameters, allowing the model to adapt to new user habits while retaining as much of the original general knowledge as possible. This update process uses a cached high-quality dataset and employs a momentum-driven stochastic gradient descent algorithm, with cross-entropy loss as the primary objective, performing iterative optimization in a limited number of rounds to achieve accurate and stable personalized adaptation with extremely low computational overhead.

[0169] S442. When incremental learning is triggered, the gradient of the model parameters is calculated in each incremental learning iteration, the gradient vector is clipped to limit the amplitude to a preset range, and random noise of the corresponding scale is generated according to the preset privacy protection strength parameter. The random noise is superimposed with the clipped gradient vector to obtain a noisy gradient and injected into the critical path of the student model update.

[0170] Specifically, to protect user privacy, the incremental learning process integrates local differential privacy technology. This technology ensures that no original user data can be deduced from the updated parameters by injecting carefully controlled noise into the critical path of model updates. In practice, after calculating the gradient of the model parameters in each incremental learning iteration, the gradient vector is pruned to limit its magnitude to a preset range. This aims to constrain the maximum impact that any single training sample can have on the update, providing a deterministic boundary for subsequent noise additions.

[0171] Based on preset privacy protection strength parameters, random noise of a corresponding scale is generated. This noise is sampled from a specific probability distribution, and its generation scale is directly related to the aforementioned gradient clipping range, ensuring the mathematical rigor of privacy protection. Subsequently, the generated noise vector is superimposed with the clipped gradient vector to obtain the noisy gradient.

[0172] S443. During incremental learning, continuously track and calculate the consumption of the privacy budget, limit the accumulated privacy leakage risk to the preset security limit, and adopt an adaptive optimization strategy to dynamically optimize the gradient clipping threshold and adjust the clipping range of the gradient vector.

[0173] Specifically, throughout the entire incremental learning lifecycle, the system continuously tracks and calculates the consumption of the privacy budget. Each noisy gradient update consumes a portion of the pre-set overall privacy budget, which is managed by the system through a rigorous privacy ledger to ensure that the accumulated risk of privacy breaches is always kept within the pre-committed security limits.

[0174] To maintain the effectiveness of model updates while ensuring robust privacy protection, this method also employs an adaptive optimization strategy. For example, before each gradient pruning operation, the system queries the statistics module to obtain the latest gradient norm statistical features. The pruning threshold is not fixed but dynamically calculated based on these statistical features. Specifically, the new pruning threshold is set to the current gradient norm mean plus an offset proportional to its standard deviation. This design allows the threshold to adapt to the current model optimization stage and the characteristics of the user data: when gradient fluctuations are large, the threshold is widened accordingly to reduce over-pruning of the effective learning signal. When the gradient is relatively stable, the threshold is automatically tightened, allowing for the injection of less noise while maintaining the same level of privacy protection.

[0175] The dynamically adjusted structure is seamlessly embedded in the incremental learning update process. The gradient statistics module rapidly updates its statistics online after each update, while the recomputation overhead of the pruning threshold is extremely low. Through this closed-loop feedback design, the system can intelligently retain the most important model update information within a strict mathematical framework that protects privacy, significantly improving the efficiency of privacy budget utilization and the ultimate effectiveness of personalized learning.

[0176] S444. Continuously track the performance metrics of the student model within the edge device. If the performance metrics deteriorate due to adaptive updates and exceed the preset performance threshold for ensuring real-time interaction, the rollback mechanism will be automatically triggered to restore the model parameters to the stable version before the incremental update.

[0177] Specifically, the deployment system also includes a lightweight runtime monitoring module that continuously tracks key performance indicators such as model inference latency and memory usage. By continuously monitoring single inference latency, if this key performance indicator deteriorates due to adaptive updates and exceeds a preset threshold for ensuring real-time interaction, a rollback mechanism is automatically triggered to restore the model parameters to the previous stable version. This ensures that the core real-time responsiveness of the system is always reliably guaranteed during the dynamic process of pursuing personalized performance optimization.

[0178] Please see Figure 5 The present invention also provides a lightweight pseudo-multimodal gesture recognition system based on segmented motion energy, the system comprising:

[0179] Data acquisition and processing module 1 is used to acquire continuous video clips containing complete hand gestures using a monocular camera, and to preprocess the continuous video clips to obtain RGB video sequences. Preprocessing includes video enhancement, multi-frame temporal alignment, and motion compensation.

[0180] The pseudo-multimodal generation module 2 is used to extract pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency from RGB video sequences based on preset pseudo-multimodal branches. The pseudo-multimodal data includes RGB enhanced images, segmented motion energy maps, and spatial attention maps.

[0181] Feature extraction and inference module 3 is used to construct an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network, using pseudo multimodal data as input, and introduces dynamic weights to achieve adaptive fusion of the expert network outputs to complete gesture recognition in different environments.

[0182] Edge deployment update module 4 is used to construct and train the corresponding student model using pseudo-multimodal branches and an expert committee structure as the teacher model. The student model is then deployed to edge devices, triggering incremental learning with differential privacy protection when certain conditions are met.

[0183] Example: In the experimental environment and data preparation section, a dedicated hardware and software platform was built, using the internationally recognized dynamic gesture video dataset 20BN-Jester V1 as the basic data source. For hardware configuration, model training was performed on a general-purpose deep learning workstation equipped with a dedicated graphics card. To verify the algorithm's deployment performance on general-purpose mobile terminals, model inference tests were performed on devices without dedicated graphics card acceleration. For the software environment, the system was based on the Windows 10 / 11 operating system and developed using the PyTorch deep learning framework in conjunction with the OpenCV image processing library. The experiment selected 10 typical dynamic gestures from the 20BN-Jester dataset: "Swiping Left," "Swiping Right," "Stop Sign," "Thumb Up," "Thumb Down," "Sliding Two Fingers Up," "Sliding Two Fingers Down," "Zooming In With Full Hand," "Zooming Out With Full Hand," and "No Gesture." A total of 2000 valid video samples were selected and randomly divided into training and test sets in an 8:2 ratio. In the preprocessing stage, a sliding window mechanism was used to uniformly set the time window length of all video samples to T=30 frames. The image size was scaled to 224×224 pixels and standardized to ensure the consistency of the input data.

[0184] Regarding the specific implementation parameters for model training, the teacher model is constructed based on the aforementioned technical solution, with its three expert branches all adapted to a lightweight MobileNetV3-Small network as the backbone. The student model is a simplified single-branch network, also using MobileNetV3-Small as its backbone, with a feature adaptation module embedded in the intermediate layers. The training process uses the AdamW optimizer, with a batch size of 32, an initial learning rate of 0.001, and cosine annealing for adjustment. The teacher model is first trained end-to-end on complete three-branch pseudo-multimodal data, iterating for 100 epochs until convergence. Subsequently, in the knowledge distillation stage, the teacher model parameters are fixed to guide the student model's training. During distillation, the temperature coefficient T is set to 4 to soften the teacher model's output distribution; the total loss function is a weighted sum of cross-entropy loss and KL divergence loss, with the weight coefficients λ1 and λ2 both set to 0.5 to ensure that the student model, while learning real labels, fully mimics the probability distribution rich in multimodal information from the teacher model. The student model was trained iteratively for 150 epochs in this setting.

[0185] In terms of accuracy, the student model of this invention achieves an accuracy of 93.8% on the test set, outperforming the baseline of 82.5% and the traditional two-stream network of 91.2%. Especially when distinguishing between highly time-dependent gestures such as "left shift" and "right shift," this invention demonstrates higher robustness. Regarding inference speed and resource consumption, on a general-purpose processor (CPU only) device without a dedicated graphics card, the student model of this invention achieves a single inference time of 25ms, fully meeting the requirements of real-time interaction. In contrast, the traditional two-stream network, due to the need to calculate dense optical flow and run two large networks, takes over 145ms. The model of this invention has only 2.9M parameters, reducing storage space by more than 80% compared to the traditional two-stream network.

[0186] Compared to the baseline using only RGB, this invention improves average accuracy by 11.3% while maintaining the same order of magnitude of parameters. In a low-light test subset simulated by reducing gamma values, the baseline's accuracy plummeted to 68.3% due to texture loss, while this invention, utilizing motion energy map input, maintained accuracy at 89.4%, demonstrating the complementary advantages of pseudo-multimodal approaches. While traditional two-stream networks offer good accuracy, they require complex optical flow calculations and running two large networks, consuming up to 145ms, making real-time operation impossible. The student model of this invention has an inference time of 25ms, meeting the 40fps real-time requirement and validating the advantages of knowledge distillation strategies in edge deployment.

[0187] In summary, by utilizing the above-mentioned technical solution of the present invention, this application does not introduce additional sensors or real multimodal data, but only constructs multiple pseudomodal features in the time and space dimensions to characterize the stable appearance of gestures, short-term motion changes, and significant regional distribution based on video sequences captured by a common monocular RGB camera, so as to explicitly separate the influence of different perceptual factors on the recognition results.

[0188] Based on this, an expert committee is constructed, consisting of multiple lightweight expert networks designed for different perceptual emphases. The participation level of each expert is dynamically adjusted by the chair network according to the temporal characteristics of the input gesture sequence, thereby alleviating the conflict between different pseudomodalities and improving the stability and accuracy of fast gesture recognition in complex environments.

[0189] This invention completely eliminates the reliance on depth cameras, infrared sensors, or wearable devices, requiring only a regular monocular camera to operate. This overcomes the drawbacks of existing high-performance gesture recognition technologies, such as expensive hardware and difficulty in widespread adoption on existing devices. All pseudomodalities originate from the mathematical transformation of the same RGB video stream, ensuring time synchronization and spatial alignment from a physical perspective. This eliminates the accuracy degradation caused by sensor calibration errors or timestamp asynchronization in traditional multimodal technologies. The "pseudo-multimodal" design replaces hardware costs with algorithmic overhead, enriching feature representation. An expert committee and dynamic fusion mechanism enhance decision robustness. Ultimately, through knowledge distillation, the capabilities of the high-performance "teacher" are compressed into a lightweight, real-time-running "student" model, maintaining high recognition accuracy while significantly reducing the number of parameters. The dynamic fusion weight mechanism enables the system to adapt to environmental factors such as lighting changes and background complexity. Post-deployment edge-side incremental learning allows the model to be quickly fine-tuned using limited user data, adapting to users' personalized gesture habits and improving the user experience.

[0190] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

Claims

1. A lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy, characterized in that, include: S1. Use a monocular camera to capture continuous video clips containing complete hand gestures, and preprocess the continuous video clips to obtain an RGB video sequence; The preprocessing includes video enhancement, multi-frame temporal alignment, and motion compensation. S2. Based on preset pseudo-multimodal branches, extract pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency from the RGB video sequence; the pseudo-multimodal data includes RGB enhanced images, segmented motion energy maps, and spatial attention maps; S3. Using pseudo-multimodal data as input, construct an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network, and introduce dynamic weights to achieve adaptive fusion of the expert network outputs in order to complete gesture recognition in different environments. S4. Using pseudo-multimodal branches and expert committee structure as teacher models, construct and train corresponding student models; and deploy student models to edge devices, triggering incremental learning with differential privacy protection when the triggering conditions are met.

2. The lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy according to claim 1, characterized in that, The process of acquiring continuous video clips containing complete hand gestures using a monocular camera and preprocessing these clips to obtain an RGB video sequence includes: S11. Use a monocular camera to acquire a continuous video stream, and use a sliding window mechanism to extract a continuous video segment containing complete hand gestures from the video stream. S12. Extract image frames from continuous video segments, perform contrast-adaptive histogram equalization on the brightness channel of each image frame to enhance local contrast, and use multi-scale illumination correction to enhance the video quality of continuous video segments under different illumination conditions. S13. The sharpness of the enhanced image frame is evaluated. When the calculated Laplacian variance is lower than the preset threshold, the corresponding image frame is determined to be a blurred frame, and the blurred frame is corrected by a gradient-based non-blind deblurring method. S14. Use the sparse optical flow method to track the key points of the hand in adjacent image frames, calculate the corresponding affine transformation relationship based on the motion relationship of the key points between image frames, and perform sub-pixel alignment on consecutive image frames based on the affine transformation relationship. S15. For consecutive image frames after alignment, calculate the pixel difference between adjacent image frames in the hand region. If the pixel difference exceeds the dynamic threshold, mark it as an abnormal frame and use bidirectional optical flow interpolation between the preceding and following image frames for reconstruction and replacement. S16. After preprocessing, the image frames are scaled to a uniform size and then standardized according to the preset mean and standard deviation to obtain an RGB video sequence with enhanced quality and consistent timing.

3. The lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy according to claim 1, characterized in that, The pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency in the RGB video sequence, based on the preset pseudo-multimodal branches, includes: S21. Extract the RGB image of each frame in the RGB video sequence, input it into a two-dimensional convolutional neural network for encoding, obtain the RGB enhanced image through spatial and frequency domain transformation, and stack all the processed RGB enhanced images in the original time order to form an appearance pseudomodal data sequence. S22. A dynamic modeling method based on time-series frequency domain analysis and decoupling of motion components extracts motion information from RGB images and constructs a main motion energy channel and a detail motion direction channel. By stitching them together, a segmented motion energy map is generated as motion pseudomodal data. S23. An attention generation method based on motion-invariant subspace learning and spectral clustering separation processes RGB images to generate spatial attention maps, which serve as spatial attention pseudomodal data.

4. The lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy according to claim 3, characterized in that, The process of extracting each frame of RGB image from the RGB video sequence, inputting it into a two-dimensional convolutional neural network for encoding, obtaining RGB enhanced images through spatial and frequency domain transformation, and stacking all the processed RGB enhanced images in their original chronological order to form an appearance pseudomodal data sequence includes: S211. Input each frame of RGB image in the RGB video sequence into a two-dimensional convolutional neural network, perform shallow feature extraction through depthwise separable convolutional layers, and output a 64-channel feature map. S212. In a two-dimensional convolutional neural network, a frequency domain weight matrix is ​​introduced. A block-based two-dimensional discrete cosine transform is performed on each channel of the feature map to convert the spatial domain information to the frequency domain representation. The weighted frequency domain coefficients are then reconstructed back into the spatial domain through an inverse discrete cosine transform to obtain a frequency domain enhanced feature map. S213. The frequency domain enhanced feature map is fused with the original RGB image after dimensionality reduction by a convolutional layer through a residual connection. The fusion result is then passed through another convolutional layer to adjust the number of channels to three, and an activation function is used to constrain the pixel values ​​to a preset range to obtain an RGB enhanced image of a preset size.

5. The lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy according to claim 3, characterized in that, The dynamic modeling method based on time-series frequency domain analysis and motion component decoupling extracts motion information from RGB images and constructs a main motion energy channel and a detail motion direction channel. By stitching these together, a segmented motion energy map is generated, which serves as motion pseudomodal data, including: S221. Convert each frame of the RGB video sequence into a grayscale image to obtain a grayscale sequence, and calculate the absolute difference map sequence between adjacent frames. S222. Extract the intensity value of each spatial location in the difference map sequence as a function of time to form a one-dimensional signal, and perform discrete Fourier transform to obtain the amplitude spectrum; add up the frequency components corresponding to the amplitude spectra of all spatial locations and calculate the average to obtain the global average motion energy spectrum. S223. Identify peak frequencies with amplitudes higher than the average level in the global average motion energy spectrum, and find segmentation points in the time domain according to the period corresponding to the peak frequencies to divide the original RGB video sequence into several continuous sub-stages. S224. Stack the difference graphs contained in each sub-stage into a three-dimensional spatiotemporal tensor. Separate the low-rank component tensor and sparse component tensor from the three-dimensional spatiotemporal tensor using the Tucker-2 decomposition method, which respectively represent the main motion information and detailed motion information within the sub-stage. S225. By weighted averaging of each frame of RGB image along the time dimension of the low-rank component, the main motion energy channel is generated; by analyzing the time vector of each spatial location in the sparse component, the detail motion direction channel is generated and divided into two matrices: horizontal and vertical components. S226. The main motion energy channel and two detailed motion direction channels generated in each sub-stage are stitched together along the channel dimension to form a three-channel image; and the three-channel images obtained from all sub-stages are stitched together along the channel dimension to generate a composite two-dimensional image with a total number of channels equal to three times the number of sub-stages, which serves as the segmented motion energy map.

6. The lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy according to claim 3, characterized in that, The attention generation method based on motion-invariant subspace learning and spectral clustering separation processes RGB images to generate spatial attention maps, which serve as spatial attention pseudomodal data. S231. Convert each frame of the RGB video sequence into a grayscale image to obtain a grayscale sequence, and calculate the dense optical flow field of adjacent frames to obtain the motion velocity sequence of each pixel in the horizontal and vertical directions, forming the original motion features. S232. Divide the original motion features of each frame into local blocks, divide the grayscale image into multiple overlapping local blocks, and collect the motion velocity sequence of all pixels in each local block in consecutive frames to form a motion observation set. S233. Perform motion-invariant subspace learning on the motion observation set of each local block, construct a graph structure based on motion subspace similarity, take the local block as a node in the graph structure, calculate the motion subspace similarity between any two nodes, and form a similarity matrix. S234. Perform symmetric normalization on the similarity matrix, calculate the corresponding Laplacian matrix, and perform eigenvalue decomposition on the Laplacian matrix to calculate a preset number of non-zero eigenvalues ​​and their corresponding eigenvectors; then arrange the eigenvectors by column to form a new feature matrix. S235. Cluster the new feature matrix, divide all local blocks into different clusters, and after identifying the main cluster belonging to the hand, map all local blocks in the main cluster back to the pixel space to generate the initial binary mask. S236. Perform spatiotemporal consistency filtering on the binary mask, perform a logical OR operation on the binary mask after processing all frames along the time dimension to cover all hand areas reached by the gesture, and normalize to obtain the final spatial attention map.

7. The lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy according to claim 1, characterized in that, The process involves using pseudo-multimodal data as input to construct an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network, and introducing dynamic weights to achieve adaptive fusion of the expert network outputs to complete gesture recognition in different environments. S31. Input the RGB enhanced image into the appearance expert network, extract spatial features frame by frame and pool them into a frame-level feature vector sequence, perform convolution operation through a one-dimensional temporal convolution module, and perform global average pooling in the time dimension to output the appearance feature vector. S32. Input the segmented motion energy map into the motion expert network, extract spatial pattern features through the adapted backbone network, and output the motion feature vector through global pooling. S33. Input the spatial attention map into the spatial attention expert network, and output the attention feature vector after global average pooling; S34. Using appearance feature vector, motion feature vector and attention feature vector as input, a dynamic gating network is introduced as the chairman network. Global average pooling is used to capture the global context, and two fully connected layers are used for processing to generate a set of dynamic fusion weights. S35. Use fusion weights to sum the feature vectors to generate fusion features and input them into a fully connected classifier. After activation by the Softmax function, output the probability distribution of the gesture category.

8. The lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy according to claim 1, characterized in that, The pseudo-multimodal branch and expert committee structure are used as the teacher model to construct and train the corresponding student model. The student model is then deployed to edge devices, and when certain triggering conditions are met, incremental learning with differential privacy protection is initiated, including: S41. The teacher model is trained end-to-end using the complete pseudo-multimodal branch, and the classification cross-entropy loss is optimized until the teacher model converges. Feature maps of multiple levels inside the teacher model network are extracted to construct a feature pyramid covering different semantic levels. S42. Using aggregated pseudo-multimodal data as input, a lightweight network is used to construct a student model, and several feature adaptation modules are embedded in the intermediate layer. Under the guidance of the teacher model, a progressive knowledge transfer process is carried out to achieve end-to-end joint optimization of all parameters of the student model. S43. Deploy the trained student model to the edge device, introduce an edge-side overhead adaptive mechanism, and form a personal dataset by locally caching the recognition of gestures and classification results with confidence exceeding a preset threshold within a preset period. S44. Automatically monitor trigger conditions during idle periods of edge devices. When the trigger conditions are met, automatically trigger incremental learning and integrate local differential privacy protection functions during the incremental learning process to maintain a balance between personalized adaptation, privacy protection and real-time performance of edge devices.

9. The lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy according to claim 8, characterized in that, The automatic monitoring of triggering conditions during idle periods of the edge device, and the automatic triggering of incremental learning when the triggering conditions are met, along with the integration of local differential privacy protection functions during the incremental learning process, includes: S441. Based on the number of samples that meet the preset quality standards and the category distribution entropy value in the local cache of the edge device, the triggering condition is set. When the number of samples reaches the preset number threshold and the category distribution entropy value is lower than the preset entropy value threshold, incremental learning is automatically triggered. S442. When incremental learning is triggered, the gradient of the model parameters is calculated in each incremental learning iteration, the gradient vector is clipped to limit the magnitude to a preset range, and random noise of the corresponding scale is generated according to the preset privacy protection strength parameter. The random noise is superimposed with the clipped gradient vector to obtain a noisy gradient and injected into the critical path of the student model update. S443. During the incremental learning process, continuously track and calculate the consumption of the privacy budget, limit the accumulated privacy leakage risk to the preset security limit, and adopt an adaptive optimization strategy to dynamically optimize the gradient clipping threshold and adjust the clipping range of the gradient vector. S444. Continuously track the performance metrics of the student model within the edge device. If the performance metrics deteriorate due to adaptive updates and exceed the preset performance threshold for ensuring real-time interaction, the rollback mechanism will be automatically triggered to restore the model parameters to the stable version before the incremental update.

10. A lightweight pseudo-multimodal gesture recognition system based on piecewise motion energy, used to implement the lightweight pseudo-multimodal gesture recognition method based on piecewise motion energy as described in any one of claims 1-9, characterized in that, The system includes: The data acquisition and processing module is used to acquire continuous video segments containing complete hand gestures using a monocular camera, and to preprocess the continuous video segments to obtain an RGB video sequence; the preprocessing includes video enhancement, multi-frame temporal alignment, and motion compensation. The pseudo-multimodal generation module is used to extract pseudo-multimodal data representing static appearance, dynamic motion, and temporal saliency from RGB video sequences based on preset pseudo-multimodal branches; the pseudo-multimodal data includes RGB enhanced images, segmented motion energy maps, and spatial attention maps; The feature extraction and inference module is used to construct an expert committee structure consisting of a multimodal dedicated expert network and a dynamic gating network, using pseudo-multimodal data as input. Dynamic weights are introduced to achieve adaptive fusion of the expert network outputs to complete gesture recognition in different environments. The edge deployment update module is used to build and train the corresponding student model using pseudo-multimodal branches and expert committee structure as teacher models; and deploy the student model to edge devices, triggering incremental learning with differential privacy protection when the triggering conditions are met.

Images