Dynamic gesture recognition method and system based on static detection

Abstract

The invention discloses a dynamic gesture recognition method and system based on static detection. The method comprises the following steps: constructing a dynamic gesture standard library; constructing a data set training static detection model; inputting the gesture image into a trained static detection model to obtain a static time sequence detection result; performing rule matching based on the static time sequence detection result and the dynamic gesture standard library, and confirming a dynamic gesture result; according to the method, dynamic actions are identified through short-term frame caching and lightweight rules, a video model does not need to be trained, each frame is classified independently, the data requirement is simplified, and the reasoning performance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control, and more particularly to a dynamic gesture recognition method and system based on static detection. BACKGROUND

[0002] The Jester, ChaLearnLAP, IPNHand and other data sets are mainly video-level annotations, that is, each video has only one action label (static or dynamic), and there is no hand position (Bounding Box) or key point; the MediaPipeHands focuses on key point detection and can give 21 points of hand skeleton, but there is no classification label (such as "like", "fist" and other gesture categories); the NVGesture and MotionFusedFrames propose some methods of fusing space-time information, but they are all for whole video or single-frame classification, and do not provide a detection level data structure.

[0003] The Jester and the NVGesture are similar, and mainly cover sliding, dialing and other control gestures;

[0004] The ChaLearnLAP is also mainly imperative actions such as "turn on the light" and "close the door"; the IPNHand also focuses on a small number of static touch gestures such as tapping and sliding, but lacks social / emotional gestures; the MediaPipe only provides skeleton points and cannot support gesture semantic expression. SUMMARY

[0005] Therefore, the present application provides a dynamic gesture recognition method and system based on static detection, which simplifies the data requirement and improves the inference performance.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0007] A dynamic gesture recognition method based on static detection comprises the following steps:

[0008] Constructing a dynamic gesture standard library;

[0009] Constructing a data set to train a static detection model;

[0010] Inputting a gesture image into the trained static detection model to obtain a static time sequence detection result;

[0011] Based on the static time sequence detection result and the dynamic gesture standard library, rule matching is performed to confirm a dynamic gesture result.

[0012] Preferably, the step of constructing a data set comprises:

[0013] Obtaining an original training data set and training a hand detector;

[0014] Automatic hand extraction is performed on the original training data based on the hand detector, and a final data set is obtained through manual labeling.

[0015] Preferably, the step of manual labeling comprises:

[0016] A single-hand gesture image is obtained through the automatic hand extraction, and another hand not performing a gesture is labeled based on the single-hand gesture image.

[0017] Preferably, the step of manual labeling comprises:

[0018] A double-hand gesture image is obtained through the automatic hand extraction, and corresponding two detection boxes are stretched and fused based on the double-hand gesture image.

[0019] Preferably, the step of rule matching comprises:

[0020] A first sequence length and a sequence element of a current standard gesture sequence are obtained.

[0021] A corresponding length is cut from a sequence tail in the static timing detection result according to the first sequence length, as a feature sequence.

[0022] Element-by-element comparison is performed based on the feature sequence and the current standard gesture sequence, and a label of the current standard gesture sequence is output as the dynamic gesture result when each element corresponds to each other.

[0023] Preferably, the step of rule matching comprises:

[0024] An independent FSM instance is configured for each dynamic gesture.

[0025] Each new label in the static timing detection result is listened to and broadcast to each FSM instance.

[0026] Each FSM judges whether a transition condition is met according to its current state and the new frame information, and is updated to a next state.

[0027] When any FSM instance enters its final state, it is declared that the matching is successful.

[0028] Preferably, the step of rule matching comprises:

[0029] Sequence reduction is performed on the static timing detection result, and the same static labels appearing in continuous frames are merged.

[0030] Based on the merged detection result, fuzzy matching is performed with the standard dynamic labels in the dynamic gesture standard library, and when the matching is successful, the corresponding dynamic label is output as the dynamic gesture result.

[0031] A dynamic gesture recognition system based on static detection includes a dynamic gesture standard library, a data acquisition module, a static gesture recognition module, and a dynamic gesture recognition module.

[0032] The dynamic gesture standard library is used to store standard dynamic sequence labels of dynamic gestures; the data acquisition module is used to acquire gesture images; the static gesture recognition module is used to perform frame-by-frame recognition according to the gesture images and correspondingly generate frame-by-frame static gesture recognition results to obtain a static timing detection result; and the dynamic gesture recognition module is used to match the static timing detection result with each dynamic sequence label in the dynamic gesture standard library and output a dynamic gesture result that meets the matching relationship.

[0033] A gesture instruction input system includes a user terminal, a gesture recognition module, and an instruction generation module; the gesture recognition module is used to receive image frames acquired by the user terminal and obtain corresponding dynamic gesture recognition results by using the above dynamic gesture recognition method; and the instruction generation module generates an instruction corresponding to the current dynamic gesture recognition result according to a preset mapping relationship.

[0034] Compared with the prior art, the technical solution provides a dynamic gesture recognition method and system based on static detection, which classifies each frame independently, simplifies data requirements, and improves inference performance by short-term frame caching + lightweight rule-based dynamic action (such as sliding, clicking, and zooming) recognition without training a video model; each image is labeled with a hand bounding box, a gesture category label, and optional key points (estimated by MediaPipe), and supports integrated detection + classification training, taking into account detection, classification, and key point regression three downstream tasks, and improves data structure and labeling methods. BRIEF DESCRIPTION OF DRAWINGS

[0035] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0036] Figure 1 A dynamic gesture recognition method based on static detection is provided. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] Embodiment 1

[0039] The embodiment of the present application discloses a dynamic gesture recognition method based on static detection, comprising the following steps:

[0040] S1: constructing a dynamic gesture standard library;

[0041] S2: constructing a data set to train a static detection model;

[0042] S3: inputting a gesture image into the trained static detection model to obtain a static time sequence detection result;

[0043] S4: performing rule matching based on the static time sequence detection result and the dynamic gesture standard library to confirm a dynamic gesture result.

[0044] In the embodiment, the static detection model obtains a static label corresponding to each frame gesture through frame-by-frame recognition of the gesture image, forms a short-term frame cache, and on this basis, the present application introduces a lightweight rule recognition to recognize the matching relationship in the short-term frame cache and the dynamic gesture standard, recognizes a dynamic action, does not need to train a video model, each frame is independently classified, simplifies data requirements, and improves inference performance.

[0045] In an embodiment, the step of constructing a data set comprises: obtaining an original training data set, and training a hand detector; performing automatic hand extraction on the original training data based on the hand detector, and obtaining a final data set through manual labeling.

[0046] The embodiment needs to perform a large amount of labeling work, and then provides an automatic labeling mode combined with a hand detector. On the basis of hand detection, labeling is realized through simple adjustment.

[0047] In addition, in order to make the training sample have sufficient coverage, in addition to gestures with specific needs, a large number of natural hand postures are also configured, such as hand close to face, hand holding cup, natural waving, etc. Through labeling, they are defined as a "no gesture" category.

[0048] Specifically, in the data collection phase, volunteers complete the specified gestures in different indoor environments and lighting conditions; in the verification link, the uploaded photos are audited, and only the gesture pictures that meet the requirements are retained; in the filtering link, sensitive information (such as minors) and other non-compliant samples are removed. In addition, a total of 28,394 different subjects participated in data collection, each subject was in a different scene, ensuring the diversity of the dataset in terms of age, gender, race, lighting, and scene.

[0049] In the annotation phase, to improve efficiency, HRDv2 adopts an annotation process of automated hand detection combined with manual correction. By first detecting the hand position and preliminarily judging the gesture through algorithm, the workload of manual annotation is greatly reduced, and only a small amount of manual review and correction is needed to complete the annotation, improving the annotation speed and consistency. First, a YOLOv10x lightweight hand detector is trained using HRD original data (all gesture categories are unified as "hand" category), which detects hands in newly collected images. For single-hand gesture images, if the other hand appears in the picture, the hand below the gesture hand is labeled; for double-hand gesture images, first detect two boundary boxes for left and right hands, then merge the left and right hand boundaries into a joint boundary box containing both hands. For example, for the "X" gesture, a single square boundary box is used to annotate the entire double-hand region, and the distance between the two hands is covered by stretching the vertical line. This processing ensures that when another hand appears in the image without performing a gesture, it is still labeled, avoiding missing labels affecting model training; at the same time, double-hand gestures are labeled with a joint boundary box, so that the subsequent model recognizes both hands as a whole, improving the recognition accuracy of double-hand composite gestures.

[0050] In many real scenarios, when a user's hand is performing a gesture, the other hand may unintentionally appear in the camera (e.g., on the table or at the side). If only the hand performing the gesture is labeled in the training data, the model may incorrectly learn a negative correlation, i.e., "valid single-hand gestures are always accompanied by only one hand in the picture." This can lead to false negatives or misjudgments during inference, as the model may not conform to the "pattern" it has learned.

[0051] By labeling this non-performing hand and marking it as background or ignored area in subsequent training, the model can be explicitly taught to ignore the presence of non-gesture hands, significantly improving the robustness and accuracy of the model in real and complex scenarios.

[0052] For double-hand gesture images, this embodiment labels them with a unified joint boundary box, guiding the model to recognize both hands as a whole, improving the recognition accuracy of such complex composite gestures.

[0053] In one embodiment, the application implements dynamic gesture recognition by using strict sequence matching. In the dynamic gesture standard library, each standard dynamic gesture has a fixed-length, strictly ordered static label list. For example, the rule library = {"single click": ["single finger pointing", "fist", "single finger pointing"], "OK": ["palm open", "OK gesture"]}.

[0054] In this embodiment, a matching check is triggered every time a new static label L_t enters the frame buffer queue. When the check is performed, the subsequence at the end of the queue that is equal in length to the rule is intercepted and compared with each rule in the rule library in an exact, element-by-element manner. If the subsequence is identical to a certain rule, the matching is successful, and the corresponding dynamic gesture is recognized.

[0055] In another embodiment, finite state machines (FSMs) are used for matching, which greatly improves the flexibility and robustness of matching and can adapt to the speed changes and uncertainties of user actions in real-world scenarios.

[0056] In this embodiment, each dynamic gesture in the dynamic gesture standard library is defined as a finite state machine (FSM). The state machine contains a set of states, transition conditions between states, and one or more terminal states. Each frame of new labels in the static timing detection result is listened to and broadcast to each FSM instance; each FSM updates to the next state according to its current state and the new frame information, and judges whether the transition condition is met; when any FSM instance enters its final state, it is declared that the matching is successful.

[0057] Further, this embodiment can easily handle the duration changes of static gestures through state self-loop, prevent infinite waiting by introducing a timeout mechanism, and determine the direction by combining position information, which is very close to the complexity of the real world.

[0058] In another embodiment, the steps of rule matching include: sequence reduction on the static timing detection result, merging the same static labels that appear in consecutive frames; based on the merged detection result, fuzzy matching with the standard dynamic labels in the dynamic gesture standard library, and outputting the corresponding dynamic label as the dynamic gesture result when the matching is successful.

[0059] This embodiment balances the simplicity of implementation and a certain degree of flexibility, and can effectively solve the problem of strict matching failure caused by different action durations.

[0060] To further implement the above technical solutions, for static gesture recognition (classification and detection), two methods of full-frame classifier and target detector are respectively adopted. In the full-frame classification task, the commonly used convolutional neural network (such as ResNet18 / 152, MobileNetV3, ConvNeXt, etc.) is used to classify the whole image into the corresponding gesture label; in the gesture detection task, a lightweight target detection model (such as YOLOv10 and SSDLite) is used to predict the boundary box and gesture category of each hand.

[0061] During training, the corresponding optimizer, learning rate scheduling strategy, weight decay and other hyperparameters are configured for different network architectures, and during evaluation, F1-score is used for classification results and mean average precision mAP is used for detection results to measure performance. The experimental results show that the model trained on HRDv2 achieves excellent results on static gesture tasks: for example, ResNet-18 can achieve 98.3% F1-score on full-frame classification, ConvNeXt-Base can achieve 96.4%, and the lightweight YOLOv10n detection model can achieve more than 88% mAP in gesture detection tasks. These high-precision indicators illustrate the effectiveness of the HRDv2 dataset in improving model robustness and generalization.

[0062] The main process of the application includes:

[0063] Each frame of the input video is processed independently (i.e. without requiring pre-segmentation of complete gesture sequences), a lightweight detector based on the RFB module is used to detect all hand regions in each frame, and then a small classification network containing a single residual block is used to classify the cropped hand images. In the continuous frame recognition process, we maintain a recognition queue with a length of T (T=30 in the experiment), record the recognition results in the last 30 frames, and track the duration and position of the same target gesture. Only when the same static gesture is recognized in a series of consecutive frames and the duration exceeds the set threshold, a dynamic gesture is determined to be completed (for example, if the "zoom in" gesture is detected in consecutive frames and the duration meets the requirement, it is considered that the user has completed a "zoom in" action).

[0064] In the real-time inference phase, the static gesture recognition module and the dynamic gesture recognition module of the system work together: the static module independently performs gesture classification on each frame to output a gesture label, and the dynamic module continuously monitors the change of the frame label over time to identify complex gesture patterns. For example, when a certain static gesture is repeatedly detected in multiple consecutive frames and the duration exceeds a preset threshold, the dynamic recognition module determines that a specific dynamic gesture pattern has been formed, thereby outputting the corresponding dynamic gesture instruction. In this way, the static and dynamic recognition units cooperate with each other: the static module provides the instantaneous gesture recognition result of each frame, and the dynamic module performs pattern analysis based on the result sequence, to jointly achieve accurate recognition of complex dynamic gestures.

[0065] Embodiment 2

[0066] Based on the same inventive concept, the present application discloses a dynamic gesture recognition system based on static detection, which comprises a dynamic gesture standard library, a data acquisition module, a static gesture recognition module and a dynamic gesture recognition module.

[0067] The dynamic gesture standard library is used for storing standard dynamic sequence labels of dynamic gestures.

[0068] The data acquisition module is used for acquiring gesture images.

[0069] The static gesture recognition module is used for frame-by-frame recognition according to the gesture images, and generates corresponding static gesture recognition results of each frame to obtain static timing detection results.

[0070] The dynamic gesture recognition module is used for matching the static timing detection results with each dynamic sequence label in the dynamic gesture standard library, and outputting a dynamic gesture result that meets the matching relationship.

[0071] Embodiment 3

[0072] Based on the same inventive concept, the present application discloses a gesture instruction input system, which comprises a user terminal, a gesture recognition module and an instruction generation module; wherein the gesture recognition module is used for receiving image frames acquired by the user terminal, and obtaining corresponding dynamic gesture recognition results by using the dynamic gesture recognition method of embodiment 1; and the instruction generation module generates an instruction corresponding to the current dynamic gesture recognition result according to a preset mapping relationship.

[0073] The technical effects of the present application are as follows:

[0074] In the data set construction, the present application labels each image with a hand bounding box, a gesture category label, and optional key points (estimated by MediaPipe), and supports integrated detection + classification training, taking into account three downstream tasks of detection, classification and key point regression, improving the data structure and labeling method, and introducing a large number of non-gesture categories to reduce the false detection rate.

[0075] For the static gesture recognition task, we selected multiple classic network architectures for model training and evaluation. For example, in the full-frame gesture classification, we used ResNet18, ConvNeXt-B, and other convolutional neural network models, and in the gesture detection task, we used lightweight object detection architectures such as YOLOv10-nano, SSDLite, etc. We adjusted the hyperparameters such as the optimizer and learning rate schedule for each model to achieve the best performance. The classification task evaluated the model accuracy using F1-score, and the detection task measured the recognition effect using the average precision mAP. The experimental results show that the models trained on the HRDv2 dataset achieved excellent performance: for example, ResNet18 achieved an F1-score of 98.3% in the static gesture classification task, and ConvNeXt-B achieved an F1-score of 96.4%. YOLOv10-nano achieved an mAP of more than 88% in the gesture detection task. These high-performance indicators show that the dataset improves the robustness and generalization ability of the model, fully demonstrating the effectiveness of the scheme in static gesture recognition.

[0076] The static and dynamic hierarchical decoupling and collaborative design adopted by the scheme makes the system very efficient: the detector part only needs about 270,000 parameters (106 MFLOPs), and the classifier only needs about 100,000 parameters (6.9 MFLOPs), so the overall model can run in real time on a normal CPU. The reason why the parameter amount and computational demand can be greatly reduced is that we avoid training a single large model and split the task into two lightweight modules to complete collaboratively. The cooperation of static detection and dynamic recognition produces additional efficiency advantages - this collaborative design effectively reduces model redundancy and improves resource utilization. The performance improvement brought by this is not simply achieved by stacking existing technologies. This further demonstrates the ingenuity and innovation of the scheme in architecture design.

[0077] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0078] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dynamic gesture recognition method based on static detection, characterized in that, Includes the following steps: Build a standard library of dynamic gestures; Build a dataset to train a static detection model; A gesture image is acquired and input into a trained static detection model to obtain a static temporal detection result. The dynamic gesture result is confirmed by matching the static timing detection result with the dynamic gesture standard library.

2. The dynamic gesture recognition method based on static detection according to claim 1, characterized in that, The steps for constructing the dataset include: Obtain the original training dataset and train the hand detector; The hand detector is used to automatically extract hands from the original training data, and the final dataset is obtained through manual annotation.

3. The dynamic gesture recognition method based on static detection according to claim 2, characterized in that, The manual annotation steps include: The automatic hand extraction process yields a single-hand gesture image, and the other hand, which did not perform a gesture, is then labeled based on this single-hand gesture image.

4. A dynamic gesture recognition method based on static detection according to claim 2 or 3, characterized in that, The manual annotation steps include: The automatic hand extraction process yields images of two hand gestures, which are then stretched and fused into two corresponding detection boxes.

5. The dynamic gesture recognition method based on static detection according to claim 1, characterized in that, The rule matching steps include: Get the length and elements of the first sequence of the current standard gesture sequence; Based on the first sequence length, a corresponding length is truncated from the end of the sequence in the static time-series detection result and used as a feature sequence. The feature sequence and the current standard gesture sequence are compared element by element, and when all elements correspond to the same value, the label of the current standard gesture sequence is output as the dynamic gesture result.

6. The dynamic gesture recognition method based on static detection according to claim 1, characterized in that, The rule matching steps include: Configure a separate FSM instance for each dynamic gesture; Listen for new tags in each frame of the static time-series detection results and broadcast them to each of the FSM instances; Each FSM determines whether the transition conditions are met based on its current state and the information of the new frame, and updates itself to the next state. A successful match is declared when any FSM instance enters its final state.

7. The dynamic gesture recognition method based on static detection according to claim 1, characterized in that, The rule matching steps include: The static temporal detection results are sequenced, and the same static tags appearing in consecutive frames are merged. Based on the merged detection results, a fuzzy match is performed with the standard dynamic labels in the dynamic gesture standard library, and the corresponding dynamic label is output as the dynamic gesture result when the match is successful.

8. A dynamic gesture recognition system based on static detection, characterized in that, It includes a dynamic gesture standard library, a data acquisition module, a static gesture recognition module, and a dynamic gesture recognition module; The dynamic gesture standard library is used to store standard dynamic sequence tags for dynamic gestures; The data acquisition module is used to acquire gesture images; The static gesture recognition module is used to perform frame-by-frame recognition based on the gesture image and generate static gesture recognition results for each frame accordingly, thereby obtaining static temporal detection results. The dynamic gesture recognition module is used to match the static temporal detection results with each dynamic sequence label in the dynamic gesture standard library, and output dynamic gesture results that meet the matching relationship.

9. A gesture command input system, characterized in that, It includes a user terminal, a gesture recognition module, and an instruction generation module; wherein, the gesture recognition module is used to receive image frames acquired by the user terminal and obtain corresponding dynamic gesture recognition results using any one of the dynamic gesture recognition methods described in claims 1-7; The instruction generation module generates instructions corresponding to the current dynamic gesture recognition result based on a preset mapping relationship.

Images