Dynamic gesture recognition lightweight method based on staged pipeline strategy

Abstract

The invention provides a dynamic gesture recognition lightweight method based on a staged assembly line strategy, and belongs to the technical field of computer vision and human-computer interaction. According to the method, a video stream collected by an RGB camera is taken as input, a detection-tracking-feature extraction-classification staged pipeline architecture is adopted, accurate positioning of a multi-gesture area is realized through a YOLOv5 model, real-time stable tracking of hand key points is completed through a MediaPipe framework, geometric features of the key points are coded into deep features through a lightweight CNN network, and the deep features are extracted through a multi-gesture image. And finally, the classification decision is optimized by using a support vector machine. According to the method, a full-process collaborative optimization strategy of pipeline parallelization and dynamic resource scheduling is introduced, and the identification accuracy of 94.1%, the real-time processing speed of 32FPS and the power consumption control within 5W are realized. The method adapts to the resource constraint of the smart home terminal equipment, and provides a light-weight technical scheme capable of being actually deployed for the natural interaction of the smart home.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and human-computer interaction technology, specifically relating to a lightweight method for dynamic gesture recognition based on a phased pipeline strategy. Background Technology

[0002] Currently, the application of dynamic gesture recognition technology in smart home scenarios mainly involves two types of technical solutions: end-to-end deep learning solutions and traditional single-algorithm and hybrid architecture solutions.

[0003] Mainstream research employs end-to-end deep sequence modeling methods, based on 3D-CNN, LSTM / GRU, or Transformer architectures, to directly extract spatiotemporal features from raw video data and output recognition results. For example, some solutions use an improved YOLOv5-Conv model, optimizing the model structure through an integrated "detection-analysis" architecture, achieving an mAP@0.996 on the HaGRID public dataset. Other solutions are based on 3D convolutional neural networks or Transformer+LSTM combined models, pursuing high accuracy metrics on general datasets and relying on high-performance GPUs to support model inference computation.

[0004] Traditional single-algorithm and hybrid architecture solutions employ a phased process of "hand detection - manual feature extraction - temporal classification". For example, methods based on skin color segmentation and Hu moment features are used for lighting control scenarios, while solutions combining optical flow and dynamic time warping (DTW) are used for curtain and sound control. Some hybrid architecture solutions utilize traditional methods (such as skin color detection) to quickly locate the hand region, and then combine them with lightweight neural networks (such as MobileNet) for feature extraction and classification, attempting to balance computational efficiency and recognition robustness.

[0005] The common technical approach of the two types of solutions is to use RGB cameras or depth cameras (such as Kinect and RealSense) as sensing devices, process gesture data through a single model or a simple combination of models, and achieve gesture command recognition in home scenarios to support the control of smart home devices.

[0006] The limitations of end-to-end solutions: End-to-end deep learning models have a large number of parameters (usually millions to tens of millions of parameters), high computational complexity, and long single-frame inference time (e.g., the improved YOLOv5-Conv model has a latency of up to 120ms on a Raspberry Pi 4B). They also rely on high-performance GPUs, making it difficult to achieve real-time and continuous operation on smart home terminal devices with limited computing power, memory, and power consumption. At the same time, the models are sensitive to the quality of input images, with accuracy dropping by up to 10.5% in low-light (200Lux) scenarios, indicating insufficient robustness.

[0007] Performance limitations of traditional and hybrid solutions: Traditional single-algorithm solutions rely on handcrafted features (such as HOG features), which cannot model deep spatiotemporal relationships of gestures and have poor adaptability to changes in lighting (accuracy fluctuates by 12.5% ​​at 200-800 Lux) and complex backgrounds (accuracy drops sharply to 57.9% when the texture coefficient is 0.8); Although hybrid architecture solutions simplify computation, they lack collaborative optimization between modules, resulting in low dynamic gesture tracking completeness (mostly below 85%) and a recall rate of only 76.5%-88.2% for continuous dynamic gestures (such as "brighten → medium brightness"), which is difficult to meet the natural interaction needs of smart homes.

[0008] Imbalance between resources and performance: Existing solutions generally suffer from the contradiction of "high precision - high resource consumption" or "low consumption - low robustness" - end-to-end solutions have high power consumption (the improved YOLOv5-Conv consumes 5.8W on Raspberry Pi 4B), and traditional solutions have low accuracy (KNN / SVM model accuracy is only 73.0%). Neither can meet the actual needs of smart home terminals for "low power consumption (≤5W), high real-time performance (≥30FPS), and high robustness (accuracy ≥90% in complex scenarios)". Summary of the Invention

[0009] This invention addresses the problems of existing technologies by providing a lightweight dynamic gesture recognition system based on a phased pipeline strategy. It aims to solve the issues of high computational complexity, high power consumption, and poor real-time performance inherent in existing end-to-end deep learning models deployed on resource-constrained smart home devices. This invention can be specifically applied to smart home scenarios, particularly for resource-constrained smart home terminal devices such as smart control screens, embedded speakers, and IoT gateways, achieving high recognition accuracy, real-time processing performance, and low power consumption in dynamic gesture recognition.

[0010] To address the above technical problems, this invention provides the following technical solution: a lightweight method for dynamic gesture recognition based on a phased pipeline strategy, comprising the following steps:

[0011] S1. Using the video stream as input and the hand bounding box as output, construct a gesture detection module to achieve hand region localization based on video frames.

[0012] S2. Using the cropped hand region image as input and the temporal coordinates of hand key points as output, construct a hand tracking and key point localization module;

[0013] S3. Using the temporal coordinates of key hand points as input and high-dimensional feature vectors as output, a feature extraction module is constructed to extract discriminative features from the key point sequence and build a spatiotemporal feature set.

[0014] S4. Using high-dimensional feature vectors as input and corresponding gesture category labels as output, construct a gesture classification module to achieve gesture category classification.

[0015] S5. A dynamic gesture recognition neural network is constructed based on the gesture detection module, hand tracking and key point localization module, feature extraction module, and gesture classification module. The video stream is used as input and the gesture category label is used as output. The neural network is trained using a dual mechanism of dynamic scheduling and temporal fusion to obtain a dynamic gesture recognition model. The video stream is input into the dynamic gesture recognition model to obtain the gesture recognition result.

[0016] Furthermore, in the aforementioned step S1, a gesture detection module is constructed using the YOLOv5 object detection model, and the input image resolution is set to 640×640 pixels.

[0017] Furthermore, in the aforementioned step S2, the MediaPipeHands machine learning framework is used, with the input being a cropped image of the hand region from the gesture detection module, to estimate the three-dimensional coordinates of 21 key points of the hand skeleton in real time, covering key positions such as fingertips, knuckles, and wrists.

[0018] Furthermore, in the aforementioned step S3, the feature extraction module is configured to perform the following actions:

[0019] S3.1 After extracting the key nodes of the hand in frame t using MediaPipe, calculate the vector angle between adjacent bones and introduce a temporal smoothing factor to suppress noise interference in fast motion.

[0020] S3.2 To address the semantic importance differences between different keypoint pairs, keypoint weights and temporal deviation penalties are introduced to calculate the weighted distance feature.

[0021] S3.3. The dynamic angle features and weighted distance features are concatenated into an 18-dimensional geometric feature vector, which is then input into the lightweight CNN feature extraction module to output the temporal coordinates of the key points of the hand.

[0022] Furthermore, in step S3.1 above, the dynamic joint angle is calculated as follows: ,

[0023] In the formula: These are the bone vectors corresponding to joint points i and j in frame t, respectively. The angle of the same joint pair in the previous frame For time series smoothing factor, , The base of the natural logarithm. The variance of the trajectory of key points in the current frame; the more intense the motion, the greater the variance. The larger the value, the more likely the information of the current frame will be retained.

[0024] Furthermore, in step S3.2 above, the weighted distance feature is calculated as follows:

[0025] ,

[0026] In the formula: For key point pairs The importance weights are learned through the feature contribution of pre-trained samples. Let m be the x-coordinate of the key point m in frame t; Let n be the x-coordinate of the joint point n in frame t; Let m be the ordinate of the joint point m in frame t; Let n be the ordinate of the joint point n in frame t; The weighted distance feature value of the keypoint pair (m,n) in the (t-1)th frame; This is the historical average distance between the two key points; This is a time-series deviation penalty coefficient to avoid the impact of abrupt distance changes on feature stability.

[0027] Furthermore, in step S3.3 above, the lightweight CNN feature extraction module adopts a dual optimization strategy of dynamic channel pruning and quantization compensation, specifically implemented as follows:

[0028] The dynamic channel selection mechanism, specifically, is based on the idea of ​​deep hybridization (MoD), introducing a dual factor of feature discriminative power and frame complexity to dynamically adjust the number of channels retained in each convolutional layer. As shown in the following formula:

[0029] ,

[0030] In the formula: This represents the original number of channels in this layer. Minimum number of reserved channels; The space complexity of the current frame feature map; This represents the average batch complexity. The class discriminative power of the features at this layer; The maximum discriminative power value;

[0031] Quantization error compensation specifically involves designing a dynamic compensation formula based on channel energy to address the feature precision attenuation caused by TensorRT INT8 quantization.

[0032] ,

[0033] In the formula: These are the eigenvalues ​​after quantization compensation; Original features before quantization; This is the quantization error coefficient for this channel; Let be the characteristic energy norm of channel c.

[0034] Furthermore, the aforementioned gesture classification module uses a support vector machine (SVM) as the classifier, inputs a 512-dimensional deep feature vector, and optimizes the classification decision boundary through a multinomial kernel function and a soft margin mechanism.

[0035] Furthermore, in step S5 above, the neural network is trained using a dual mechanism of dynamic scheduling and temporal fusion. Specifically, the dynamic scheduling is a load-aware pipeline scheduling mechanism: a multi-dimensional scheduling priority formula is designed to address the resource requirements differences among the four modules—preprocessing, feature extraction, fusion, and recognition—to achieve dynamic allocation of thread resources.

[0036] ,

[0037] In the formula: Let k be the scheduling priority of the k-th module at time t. This represents the average task time for this module; This represents the CPU utilization rate of the module at the previous moment. The length of the module's task queue; , , This is a weighting factor. When the system load exceeds 80%, the preprocessing module's priority is automatically increased to accelerate frame filtering and improve pipeline throughput.

[0038] Furthermore, in step S5 mentioned above, the temporal fusion dual mechanism specifically refers to temporal enhancement feature fusion: to address the temporal dependency of dynamic gestures, a temporal decay factor is added to the fusion of spatial features, i.e., CNN output and motion features.

[0039] ,

[0040] In the formula: Let t be the spatial feature vector of the t-th frame; The feature vector of the motion trajectory; , For dynamic weights, These are motion feature weights, taken from the feature extraction module; This is the length of the timing window; The attenuation coefficient is... This is a time-series step index.

[0041] Compared with the closest existing technologies (end-to-end improved YOLOv5-Conv model, traditional KNN / SVM+HOG feature model), this application has the following significant technical advantages:

[0042] (a) Comparison with the end-to-end improved YOLOv5-Conv model

[0043] Lower resource consumption and stronger deployment adaptability: The improved YOLOv5-Conv consumes 5.8W of power, has a latency of 120ms, and a processing speed of 25FPS on Raspberry Pi 4B, while this application consumes 4.7W of power (reduced by 19%), has a latency of 31.2ms (reduced by 74.2%), and a processing speed of 32FPS (increased by 28%). It is fully adapted to the hardware constraints of "low power consumption and high real-time performance" of smart home terminals and can run stably without relying on GPU.

[0044] Greater robustness in complex scenarios: The improved YOLOv5-Conv achieved a 10.5% accuracy reduction in low-light (200 Lux) scenarios, an 89.8% accuracy in complex backgrounds (texture coefficient 0.8), and an 88.2% recall rate for fast dynamic gestures. The corresponding indicators of this application are a 3.5% reduction, an accuracy of 93.5%, and a recall rate of 92.3%. Through multi-module collaboration and feature optimization, the system's adaptability to complex home environments is significantly improved.

[0045] Superior ability to distinguish similar gestures: The improved YOLOv5-Conv relies on a single CNN classifier, with a misclassification rate of 9.7% for similar gestures such as "four" and "five". This application optimizes the classification boundary through SVM kernel function, reducing the misclassification rate of similar gestures to 4.8%, achieving better classification accuracy and avoiding accidental triggering of smart home control commands.

[0046] (II) Comparison with traditional KNN / SVM+HOG feature models

[0047] Significantly improved recognition accuracy: The accuracy of traditional KNN / SVM models is only 73.0%, while the accuracy of this application is 94.1% (an improvement of 21.1 percentage points). Through deep feature extraction and multi-module collaboration, it solves the problem that manual features cannot model deep spatiotemporal correlations, thus meeting the precise control needs of smart homes.

[0048] Significantly enhanced environmental adaptability: Traditional models show an accuracy fluctuation of 12.5% ​​under varying illumination (200-800 Lux) and an accuracy of 57.9% against complex backgrounds (texture coefficient 0.8); this application shows a fluctuation of ≤3.5% and an accuracy of 93.5%, reducing the impact of environmental interference on recognition results through YOLOv5 background filtering and MediaPipe key point tracking.

[0049] Superior dynamic gesture processing capability: Traditional models lack temporal modeling capabilities, with a continuous dynamic gesture accuracy of 65.7%; this application achieves a continuous dynamic gesture accuracy of 91.2% through key point temporal features and pipeline parallel processing, and can support multi-step control commands such as "brighten → medium brightness → turn off the light", meeting the needs of complex interactive scenarios in smart homes.

[0050] In summary, this application, through a phased pipeline architecture, multi-module collaborative optimization, and system-level resource scheduling, comprehensively surpasses existing technologies in four dimensions: recognition accuracy, real-time performance, low power consumption, and robustness, providing a practically deployable dynamic gesture recognition solution for resource-constrained smart home terminals. Attached Figure Description

[0051] Figure 1 This is a flowchart of the overall process framework of the present invention. Detailed Implementation

[0052] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0053] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0054] This invention proposes a lightweight dynamic gesture recognition method (DTFC-DGRM) based on a staged pipeline strategy of "Detection-Tracking-Feature Extraction-Classification" (DTFC). Through modular design and end-to-end collaborative optimization, it achieves a balance between accuracy, real-time performance, and power consumption on resource-constrained terminals. This method takes video streams captured by a monocular RGB camera as input and adopts a serial pipeline structure, sequentially passing through four core modules: gesture detection, hand tracking and keypoint localization, feature extraction, and gesture classification, outputting gesture category labels (e.g., "turn on the light (one), brighten (two)"). Simultaneously, a system-level end-to-end collaborative optimization module is introduced, maximizing the utilization of embedded platform hardware through pipeline parallelization and dynamic resource scheduling. The overall architecture of DTFC-DGRM is as follows: Figure 1 As shown. The specific steps are as follows:

[0055] S1. Using the video stream as input and the hand bounding box as output, construct a gesture detection module to achieve hand region localization based on video frames;

[0056] S2. Using the cropped hand region image as input and the temporal coordinates of hand key points as output, construct a hand tracking and key point localization module;

[0057] S3. Using the temporal coordinates of key hand points as input and high-dimensional feature vectors as output, a feature extraction module is constructed to extract discriminative features from the key point sequence and construct a spatiotemporal feature set.

[0058] S4. Using high-dimensional feature vectors as input and corresponding gesture category labels as output, construct a gesture classification module to achieve gesture category classification.

[0059] S5. A dynamic gesture recognition neural network is constructed based on the gesture detection module, hand tracking and key point localization module, feature extraction module, and gesture classification module. The video stream is used as input and the gesture category label is used as output. The neural network is trained using a dual mechanism of dynamic scheduling and temporal fusion to obtain a dynamic gesture recognition model. The video stream is input into the dynamic gesture recognition model to obtain the gesture recognition result.

[0060] In a preferred embodiment of the present invention, in step S1, the YOLOv5 target detection model is selected, and the model size and computational load are optimized for embedded devices. The input image resolution is set to 640×640 pixels (balancing accuracy and computational complexity). After fine-tuning on a custom smart home gesture dataset, the mAP@0.5 reaches 83.6%, which can accurately locate single or multiple hand regions in video frames and output the coordinates of the hand bounding box, thus limiting the processing range for subsequent modules and avoiding background interference.

[0061] As a preferred embodiment of the present invention, in step S2, the MediaPipeHands machine learning framework is used. The input is a cropped image of the hand region (instead of the entire image) from the gesture detection module. The three-dimensional coordinates of 21 key points of the hand skeleton (covering key positions such as fingertips, knuckles, and wrists) are estimated in real time, and the trajectory integrity rate reaches 93.2%. The hand movement trajectory is recorded by the temporal coordinates of the key points, providing core data support for dynamic gestures for feature extraction, while reducing the computational overhead of full image processing.

[0062] In a preferred embodiment of the present invention, in step S3, the feature extraction module is configured to perform the following actions:

[0063] S3.1 After extracting 19 key nodes of the hand in frame t using MediaPipe, calculate the vector angle between adjacent bones (such as finger root-fingertip, wrist-finger root), and introduce a temporal smoothing factor to suppress noise interference in fast motion.

[0064] ,

[0065] In the formula: These are the bone vectors corresponding to joint points i and j in frame t, respectively. The angle of the same joint pair in the previous frame For time series smoothing factor, , The base of the natural logarithm. The variance of the trajectory of key points in the current frame; the more intense the motion, the greater the variance. The larger the value, the more likely the information of the current frame will be retained.

[0066] S3.2. To address the semantic importance differences between different keypoint pairs (e.g., the thumb-index finger distance is more critical for the "pinch" gesture), keypoint weights and temporal deviation penalties are introduced to calculate the weighted distance feature.

[0067] ,

[0068] In the formula: For key point pairs The importance weights are learned through the feature contribution of pre-trained samples, and their values ​​range from 0.8 to 1.2. Let m be the x-coordinate of the key point m in frame t; Let n be the x-coordinate of the joint point n in frame t; Let m be the ordinate of the joint point m in frame t; Let n be the ordinate of the joint point n in frame t; The weighted distance feature value of the keypoint pair (m,n) in the (t-1)th frame; This is the historical average distance between the two key points; The time-series deviation penalty coefficient is used to avoid the impact of abrupt distance changes on feature stability. In this example, it is selected as... .

[0069] S3.3. Concatenate the dynamic angle features (10-dimensional) and weighted distance features (8-dimensional) into an 18-dimensional geometric feature vector. The feature map is then input into a lightweight CNN feature extraction module, which outputs the temporal coordinates of key hand points. This preserves the interpretability of handcrafted features while improving robustness to dynamic scenes through temporal constraints.

[0070] As a preferred embodiment, the lightweight CNN feature extraction module adopts a dual optimization strategy of dynamic channel pruning and quantization compensation, as specifically implemented as follows:

[0071] The dynamic channel selection mechanism, specifically, is based on the idea of ​​deep hybridization (MoD), introducing a dual factor of feature discriminative power and frame complexity to dynamically adjust the number of channels retained in each convolutional layer. As shown in the following formula:

[0072] ,

[0073] In the formula: This represents the original number of channels in this layer. To minimize the number of channels to retain and avoid feature loss; The space complexity (gradient norm) of the current frame feature map; This represents the average batch complexity. The class discriminative power of the features at this layer (calculated by inter-class variance); The maximum discriminative power value is given. Experiments show that this mechanism can reduce the computational cost of convolution by 40%-60% with an accuracy loss of less than 2%.

[0074] Quantization error compensation specifically involves designing a dynamic compensation formula based on channel energy to address the feature precision attenuation caused by TensorRT INT8 quantization.

[0075] ,

[0076] In the formula: These are the eigenvalues ​​after quantization compensation; Original features before quantization; This is the quantization error coefficient for this channel (obtained offline from the calibration set, with a value of 0.01-0.05). Let be the feature energy norm of channel c. After compensation, the feature reconstruction error of the quantized model decreased from 12.3% to 4.7%. After double optimization, the number of CNN parameters decreased from 8.2M to 3.8M, and the inference speed increased by 2.1 times, meeting the lightweight requirements of smart home terminals.

[0077] In a preferred embodiment of the present invention, the gesture classification module uses a support vector machine (SVM) as the classifier. It takes a 512-dimensional deep feature vector as input and optimizes the classification decision boundary through a multinomial kernel function (parameters set to degree=2, coef0=0.2) and a soft margin mechanism (parameter C=100.0). SVM has strong generalization ability and fast inference speed in small sample scenarios (inference time of only 3.2ms per frame). It can effectively reduce the overlapping area of ​​similar gestures in the feature space (from 11.2% of the improved YOLOv5-Conv to 4.5%), and improve classification accuracy.

[0078] In a preferred embodiment of the present invention, in step S5, a dual mechanism of dynamic scheduling and temporal fusion is used to train the neural network. Specifically, the dynamic scheduling is a load-aware pipeline scheduling mechanism: a multi-dimensional scheduling priority formula is designed to address the resource requirements differences among the four modules—preprocessing, feature extraction, fusion, and recognition—to achieve dynamic allocation of thread resources.

[0079] ,

[0080] In the formula: Let k be the scheduling priority of the k-th module at time t. This represents the average task time for this module; This represents the CPU utilization rate of the module at the previous moment. The length of the module's task queue; , , These are weighting coefficients, which are set to 0.4, 0.3, and 0.3 respectively in the embodiments. When the system load exceeds 80%, the priority of the preprocessing module is automatically increased (accelerating frame filtering), thereby increasing the pipeline throughput by more than 30%.

[0081] The temporal fusion dual mechanism specifically refers to temporal enhancement feature fusion: Addressing the temporal dependence of dynamic gestures, a temporal decay factor is added to the fusion of spatial features (CNN output) and motion features (trajectory variance).

[0082] ,

[0083] In the formula: Let t be the spatial feature vector of the t-th frame; The feature vector of the motion trajectory; , For dynamic weights, These are motion feature weights, taken from the feature extraction module; This is the length of the timing window, with a value of 5. For time-series step-size index, As the attenuation coefficient (with higher weight for recent frame features), this fusion method reduces the misjudgment rate of similar gestures such as "waving" and "arm swinging" from 15.6% to 6.2%.

[0084] The system training and testing are based on the "Light-HaGRID public dataset + custom smart home gesture library": Light-HaGRID contains 18 types of static gestures, providing diverse background data; the custom gesture library covers 5 types of high-frequency control gestures: "turn on the light (one), brighten (two), medium brightness (three), dim (four), turn off the light (five)", collected from five perspectives: front, top, left 45°, right 45°, and diagonally upward 45°. The background complexity is controlled within the texture coefficient range of 0.3-0.8. The dataset is divided into training, validation, and test sets in a ratio of 7:1.5:1.5, and data augmentation strategies are adopted, including random rotation (±20°), brightness and contrast adjustment (±30%), Gaussian noise injection (σ=0.01-0.05), and region occlusion (maximum 32×32 pixels).

[0085] Experimental results on the Raspberry Pi 4B hardware platform (smart central control screen test platform) show that the system recognition accuracy reaches 94.1% (comprehensive test of Light-HaGRID and custom library), the processing speed reaches 32FPS, the average power consumption is 4.7W (≤5W), and the end-to-end latency is 31.2ms. The accuracy is 90.6% in low light (200Lux) scenes, 93.5% in complex backgrounds (texture coefficient 0.8), and 92.3% in fast dynamic gestures (speed >0.5m / s), meeting the practical application needs of smart homes.

[0086] The key points and areas to be protected in this application are as follows:

[0087] (I) Key Points of Core Technologies

[0088] The "detection-tracking-feature extraction-classification" phased pipeline architecture decomposes the complex recognition task into four independently optimizable sub-tasks by splitting the modules. Each module uses a lightweight model adapted to the embedded platform (YOLOv5, MediaPipe, lightweight CNN, SVM) to avoid the high resource consumption of a single complex model.

[0089] Multi-module collaborative optimization mechanism: Gesture detection and tracking module collaboration (YOLOv5 precise positioning + MediaPipe stable tracking) solves the problem of dynamic gesture trajectory loss; feature extraction and classification module collaboration (geometric features + 512-dimensional deep features + SVM kernel function optimization) improves the ability to distinguish similar gestures and reduces cross-category misclassification rate.

[0090] System-level resource scheduling strategy: Combining pipeline parallelization with dynamic resource scheduling, it enables parallel processing of multi-frame data and load-aware resource allocation, achieving 32FPS real-time processing and 4.7W low power consumption on embedded platforms such as Raspberry Pi 4B, breaking through the deployment bottleneck of resource-constrained terminals.

[0091] (ii) The technical solution to be protected

[0092] A dynamic gesture recognition system based on a phased pipeline is characterized by comprising a YOLOv5 gesture detection module, a MediaPipe hand tracking and key point localization module, a lightweight CNN feature extraction module, an SVM gesture classification module, and a system-level optimization module for realizing pipeline parallelization and dynamic resource scheduling, which are connected in series.

[0093] According to the above system, the features are as follows: the gesture detection module inputs a 640×640 resolution RGB image, with an mAP@0.5 of 83.6%; the hand tracking module outputs the temporal coordinates of 21 hand key points, with a trajectory integrity rate of 93.2%; the feature extraction module outputs a 512-dimensional deep feature vector; and the classification module uses a multinomial kernel function SVM, with a similar gesture misclassification rate of ≤4.8%.

[0094] According to the above system, the feature is that the system-level optimization module processes different frame data in parallel through multi-threading, dynamically adjusts the CPU core allocation and calculation strategy, and achieves a processing speed of 32FPS, power consumption of ≤5W, end-to-end latency of ≤31.2ms on Raspberry Pi 4B, with an accuracy of ≥90% in complex scenes (low light, complex background, fast dynamics).

[0095] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A dynamic gesture recognition lightweight method based on a staged pipeline strategy, characterized in that, The method comprises the following steps: S1, taking a video stream as input and a hand boundary box as output, constructing a gesture detection module to realize hand region positioning based on a video frame; S2, taking a cropped hand region image as input and a time sequence coordinate of a hand key point as output, constructing a hand tracking and key point positioning module; S3, taking a time sequence coordinate of a hand key point as input and a high-dimensional feature vector as output, constructing a feature extraction module to extract discriminative features from a key point sequence and construct a space-time feature set; S4, taking a high-dimensional feature vector as input and a corresponding gesture category label as output, constructing a gesture classification module to realize gesture category classification; S5, based on the gesture detection module, the hand tracking and key point positioning module, the feature extraction module, and the gesture classification module, constructing a dynamic gesture recognition neural network, taking a video stream as input and a gesture category label as output, training the neural network by using a dynamic scheduling + time sequence fusion dual mechanism to obtain a dynamic gesture recognition model, and inputting the video stream into the dynamic gesture recognition model to obtain a gesture recognition result.

2. The dynamic gesture recognition light-weighting method based on a phased pipeline strategy according to claim 1, characterized in that, In step S1, a YOLOv5 target detection model is used to construct the gesture detection module, and the input image resolution is set to 640x640 pixels.

3. The dynamic gesture recognition light-weighting method based on a phased pipeline strategy according to claim 1, characterized in that, In step S2, a MediaPipeHands machine learning framework is used, the input is a hand region image cropped by the gesture detection module, and three-dimensional coordinates of 21 hand skeleton key points are estimated in real time, covering finger tips, finger joints, and wrist key positions.

4. The dynamic gesture recognition light-weighting method based on a phased pipeline strategy according to claim 1, characterized in that, In step S3, the feature extraction module is configured to perform the following actions: S3.1, after extracting the hand key node of the tth frame by MediaPipe, the vector angle of adjacent bones is calculated, and a time sequence smoothing factor is introduced to suppress noise interference in rapid motion; S3.2, according to the semantic importance difference of different joint node pairs, a joint node weight and a time sequence deviation penalty are introduced to calculate a weight distance feature; S3.3, the dynamic angle feature and the weighted distance feature are spliced into an 18-dimensional geometric feature vector to input a lightweight CNN feature extraction module, and the time sequence coordinates of the hand key point are output.

5. The dynamic gesture recognition lightweight method based on a staged pipeline strategy according to claim 4, characterized in that, In step S3.1, the dynamic joint angle is calculated as follows: , In the formula: are the skeleton vectors corresponding to the joint pair i, j in the t-th frame, respectively; is the angle of the same joint pair in the previous frame is a time smoothing factor, , denotes the base of the natural logarithm, is the key point trajectory variance of the current frame, the more intense the motion, the greater, and the current frame information is preferentially retained.

6. The dynamic gesture recognition light-weighting method based on a phased pipeline strategy according to claim 4, characterized in that, In step S3.2, the weight distance feature is calculated as follows: ， In the formula: is the importance weight of the joint node pair is learned by the feature contribution degree of the pre-training sample, is the horizontal coordinate of the joint node m in the t-th frame; is the horizontal coordinate of the joint node n in the t-th frame; is the vertical coordinate of the joint node m in the t-th frame; is the vertical coordinate of the joint node n in the t-th frame; is the weighted distance feature value of the joint node pair (m, n) in the t-1-th frame; is the historical average distance of the joint node pair; is the time sequence deviation penalty coefficient, which avoids the influence of distance mutation on the stability of the feature.

7. The dynamic gesture recognition lightweight method based on phased pipeline strategy according to claim 4, characterized in that, In step S3.3, the lightweight CNN feature extraction module adopts a dynamic channel pruning + quantization compensation double optimization strategy, and the specific implementation is as follows: The dynamic channel selection mechanism specifically comprises: referring to a deep mixing MoD idea, introducing double factors of feature discrimination and frame complexity, and dynamically adjusting the reserved channel number of each layer of convolution as follows: ， In the formula: is the original channel number of the layer; is the minimum reserved channel number; is the spatial complexity of the current frame feature map; is the batch average complexity; is the class discriminability of the feature of the layer; is the maximum discriminability value; Quantization error compensation, specifically: for the feature precision decay caused by TensorRT INT8 quantization, a dynamic compensation formula based on channel energy is designed: ， In the formula: is the quantized feature value after compensation; is the original feature before quantization; is the quantization error coefficient of the channel; is the feature energy norm of the channel c.

8. The dynamic gesture recognition lightweight method based on a staged pipeline strategy according to claim 1, characterized in that, The gesture classification module adopts a support vector machine SVM as a classifier, inputs a 512-dimensional deep feature vector, and optimizes a classification decision boundary through a polynomial kernel function and a soft interval mechanism.

9. The dynamic gesture recognition lightweight method based on a staged pipeline strategy according to claim 1, characterized in that, In step S5, the neural network is trained by using a dynamic scheduling + time sequence fusion dual mechanism, wherein the dynamic scheduling is specifically a load-aware pipeline scheduling: in view of the resource requirement difference of four modules of preprocessing, feature extraction, fusion and recognition, a multi-dimensional scheduling priority formula is designed to realize dynamic allocation of thread resources: ， In the formula: is the scheduling priority of the kth module at the tth time; is the average task duration of the module; is the CPU occupancy rate of the module at the previous time; is the task queue length of the module; , , is the weight coefficient. When the system load exceeds 80%, the priority of the preprocessing module is automatically raised to speed up frame filtering and improve pipeline throughput.

10. The dynamic gesture recognition lightweight method based on phased pipeline strategy according to claim 1, characterized in that, In step S5, the time sequence fusion dual mechanism is specifically time sequence enhanced feature fusion: in view of the time sequence dependency of a dynamic gesture, a time sequence attenuation factor is added in the fusion of spatial features, that is, CNN output and motion features: ， In the formula: is the t-th frame spatial feature vector; is the motion trajectory feature vector; , is the dynamic weight, is the motion feature weight, taken from the feature extraction module; is the time window length; is the decay coefficient, is the time step index.

Images