General robot gesture control method and system based on time sequence large model

Abstract

The invention discloses a general robot gesture control method based on a time sequence large model. A continuous gesture image sequence of a user is collected and preprocessed; performing unified modeling on the evolution process of the gesture action in the time dimension based on a time sequence large model, and extracting time sequence feature representation of the gesture; outputting corresponding gesture semantics or control instructions according to the time sequence characteristics; and the control instruction is sent to the robot to drive the robot to execute a corresponding action. The invention further provides a general robot gesture control system based on the time sequence large model. According to the method, continuous gesture actions are modeled by introducing the time sequence large model, the ability of understanding dynamic gesture semantics is improved, efficient cooperation between gesture recognition and robot control is achieved, and the method is suitable for various robot platforms and application scenes and has good real-time performance, universality and engineering deployability.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot control and human-computer interaction technology, specifically a general robot gesture control method and system based on a time-series large model. Background Technology

[0002] With the continuous evolution of deep learning technology, large-scale models have achieved breakthroughs in fields such as computer vision, speech understanding, and natural language processing. By pre-training on large-scale datasets, large-scale models can learn feature representations with high generalization capabilities, demonstrating superior performance compared to traditional models in complex perception and understanding tasks. In recent years, with the increasing demand for dynamic information understanding in application scenarios, large-scale models with time-series modeling capabilities have gradually become a research hotspot. By uniformly modeling time-series data, they provide new technical pathways for tasks such as action recognition, behavior understanding, and continuous decision-making.

[0003] In the fields of gesture recognition and human-computer interaction, gestures are essentially dynamic behaviors that change continuously over time. Their semantic information is reflected not only in their spatial form at a single moment but also in their temporal structure during the evolution of the action. Therefore, methods that rely solely on static images or short-term features for modeling are insufficient to fully characterize the dynamic features of complex gestures. To address this, researchers have begun to introduce deep models with temporal modeling capabilities to jointly model information from consecutive frames, thereby improving the ability to understand dynamic gestures.

[0004] As model size continues to increase, large-scale temporal models have demonstrated significant advantages in feature representation capabilities, simultaneously capturing local dynamic changes and long-term dependencies, achieving good results in complex action understanding tasks. However, existing large-scale temporal models still face many pressing technical challenges in practical applications. First, large-scale temporal models typically rely on large-scale parameters and complex computational structures, requiring significant computing and storage resources during training and inference, making them difficult to deploy directly in resource-constrained real-world systems. Especially in interactive application scenarios requiring real-time responses, model inference latency and computational overhead become crucial factors limiting system availability.

[0005] Secondly, existing large-scale time series models often focus on general sequence modeling capabilities in their structural design, lacking targeted optimization for the dynamic characteristics of specific tasks. When processing time series data with different time scales and action rhythms, the models struggle to achieve a balance between performance and efficiency. On the one hand, while overly complex time series modeling structures can improve recognition accuracy, they significantly increase the computational burden; on the other hand, simplifying the model structure may lead to insufficient modeling capabilities for fine-grained dynamic information, thus affecting the overall recognition performance.

[0006] Furthermore, the training and optimization costs of large models in engineering applications cannot be ignored. To adapt to specific tasks or scenarios, existing methods typically require full or near-full parameter updates for large models. This not only significantly increases training time and memory usage but also raises the cost of system development and maintenance. In practical applications, especially in scenarios requiring frequent iterations and rapid deployment, such as robot control, this high-cost model update method is difficult to meet engineering needs.

[0007] The aforementioned problems are particularly prominent in robot gesture control applications. Robot systems typically run on embedded or edge computing platforms with limited computing resources, while simultaneously demanding high real-time performance and stability of control commands. Existing gesture control solutions often simply connect the perception model and the control system in series, lacking an overall design that considers the relationship between timing modeling capabilities and control requirements. This leads to fluctuations in recognition results in dynamic scenarios, thereby affecting the accuracy and continuity of robot motion execution.

[0008] Therefore, how to fully leverage the powerful expressive capabilities of large-scale temporal models while reducing resource consumption during training and inference phases, and better serving the specific application scenario of robot gesture control, has become a key technical problem urgently needing to be solved in this field. There is a pressing need for a gesture interaction method for general robot control tasks, capable of effectively temporally modeling continuous gestures, achieving efficient coordination between perception results and robot control commands while ensuring system real-time performance and stability, and providing robots with more natural, reliable, and scalable human-machine interaction control capabilities. Summary of the Invention

[0009] To address the shortcomings of existing robot gesture control methods, such as insufficient dynamic gesture modeling capabilities, poor system versatility, and high computational overhead and insufficient real-time performance of large models in engineering deployment, this invention proposes a general robot gesture control method and system based on a time-series large model. The general robot is preferably a wheeled robot, but is not limited to this.

[0010] This invention aims to fully utilize the advantages of temporal large models in continuous action understanding and long-term dependency modeling to uniformly model the temporal evolution process of user gestures. While ensuring the expressive power of the model, it also takes into account the requirements of robot control systems for real-time performance, stability, and engineering deployability, thereby realizing a universal gesture control scheme applicable to various robot platforms and application scenarios.

[0011] To achieve the above objectives, the present invention provides the following technical solution: A general robot gesture control method based on a large temporal model includes the following steps: A, as an auxiliary input optimization method, acquires the user's gesture visual data without affecting the main line of temporal modeling. The gesture visual data includes a sequence of gesture images continuously acquired by an image acquisition device, which is used to characterize the change process of gesture actions in the time dimension. B, as an auxiliary input optimization method, preprocesses the gesture visual data without affecting the main line of temporal modeling. The preprocessing includes image size standardization, background interference suppression, key region extraction and temporal alignment operation to reduce the impact of environmental factors on subsequent temporal modeling. C. Construct a gesture recognition model based on a temporal large model, wherein the temporal large model is used to uniformly model continuous gesture image sequences and extract temporal representations that reflect the dynamic evolution characteristics of gestures; D, input the timing representation into the gesture classification and decision module, and output the corresponding gesture category or control semantics; E, according to a preset mapping relationship, convert the gesture category or control semantics into robot control instructions; F, the control commands are sent to the robot control terminal via the communication module; G. After receiving the control command, the robot control terminal drives the actuator to perform the corresponding action, thereby realizing real-time control of the robot.

[0012] In this invention, the gesture recognition model uses a large-scale temporal model to uniformly model continuous gesture actions, effectively depicting the dynamic evolution of gestures over time. This enhances the understanding of complex gesture semantics and reduces control instability caused by single-frame recognition errors. Simultaneously, by establishing a flexible control command mapping relationship, users can customize gesture and control command configurations according to different robot platforms and task requirements, enhancing the system's versatility and scalability. Furthermore, control commands are transmitted between the host computer and the robot via a standardized communication protocol, ensuring the stability, reliability, and real-time performance of the command interaction process.

[0013] Furthermore, in C, the temporal large model includes at least one backbone coding network and multiple lightweight adaptation modules, wherein the adaptation modules perform task-related adjustments to the model without updating the backbone parameters.

[0014] Preferably, the adaptation module includes a temporal adaptation module, which is used to perform temporal feature modeling on continuously acquired gesture image sequences to enhance the model's ability to recognize dynamic gesture changes.

[0015] More preferably, the timing adaptation module includes a recurrent neural network structure for feature modeling of short-term or long-term gesture sequences.

[0016] More preferably, the adaptation module further includes a feature adaptation module, which is used to perform nonlinear transformation and fine-grained adjustment on high-level semantic features to improve the expressive power of multimodal features.

[0017] Furthermore, in C, during model training or fine-tuning, an optimization strategy of parameter block update is adopted, which only performs gradient updates on some adaptation module parameters and freezes the parameters that are not updated, so as to reduce the GPU memory usage and improve training efficiency during model training.

[0018] Preferably, the parameter block update optimization strategy updates different parameter blocks at different training stages to ensure the overall convergence of the model and training stability.

[0019] Furthermore, in A, the gesture image acquisition process includes sampling under various ambient lighting conditions, different backgrounds, different angles, and different individual users to enhance the diversity of the dataset and the generalization ability of the model.

[0020] In B, the background interference suppression includes one or more of the following: skin color region extraction, edge detection, background blurring, or foreground segmentation.

[0021] A general robot gesture control system based on a time-series large model, the system comprising: The image acquisition module is used to acquire user gesture images or image sequences in real time; The gesture recognition module is used to recognize the captured gesture images; The control command mapping module is used to convert gesture recognition results into general robot control commands; A communication module is used to send the control commands to the general-purpose robot; The execution module is used to parse control commands and drive the general-purpose robot to perform corresponding actions.

[0022] The beneficial technical effects of this invention include: Based on a large temporal model, this invention models continuous hand gestures, effectively improving the understanding of dynamic gesture semantics. It is suitable for applications with high real-time and stability requirements, such as military, security, inspection, and service robots. Through overall design oriented towards robot control tasks, it achieves efficient coordination between gesture recognition results and robot control commands, improving the reliability and versatility of the system. Simultaneously, through a parameter-efficient model adaptation method, it reduces model training and deployment costs while ensuring recognition performance, making this method more suitable for application in robot control systems in embedded or edge computing environments, demonstrating promising engineering application prospects and promotional value. Attached Figure Description

[0023] Figure 1This is a flowchart of a general robot gesture control method based on a large temporal model.

[0024] Figure 2 This is a block diagram of a general robot gesture control system based on a large-scale temporal model. Detailed Implementation

[0025] The technical solutions of the present invention will be further described below through specific embodiments, which are intended to help those skilled in the art to better understand and implement the present invention. However, it should be understood that these embodiments do not limit the scope of protection of the present invention.

[0026] Example 1 Reference Figure 1 A general robot gesture control method based on a large temporal model includes the following steps: A, as an auxiliary input optimization method, acquires the user's gesture visual data without affecting the main line of temporal modeling. The gesture visual data includes a sequence of gesture images continuously acquired by an image acquisition device, which is used to characterize the change process of gesture actions in the time dimension. In this embodiment, the gesture image acquisition process includes sampling under various ambient lighting conditions, different backgrounds, different angles, and different individual users to enhance the diversity of the dataset and the generalization ability of the model.

[0027] B, as an auxiliary input optimization method, preprocesses the gesture visual data without affecting the main line of temporal modeling. The preprocessing includes image size standardization, background interference suppression, key region extraction and temporal alignment operation to reduce the impact of environmental factors on subsequent temporal modeling. The background interference suppression includes one or more of the following: skin color region extraction, edge detection, background blurring, or foreground segmentation.

[0028] C. Construct a gesture recognition model based on a temporal large model, wherein the temporal large model is used to uniformly model continuous gesture image sequences and extract temporal representations that reflect the dynamic evolution characteristics of gestures; In C, the time-series large model includes at least one backbone coding network and multiple lightweight adaptation modules. The adaptation modules make task-related adjustments to the model without updating the backbone parameters.

[0029] The adaptation module also includes a feature adaptation module, which is used to perform nonlinear transformations and fine-grained adjustments on high-level semantic features to improve the expressive power of multimodal features.

[0030] During model training or fine-tuning, a parameter block update optimization strategy is adopted, which only updates the gradient of some adaptation module parameters and freezes the parameters that are not updated, in order to reduce the GPU memory usage and improve training efficiency during model training.

[0031] The optimization strategy of parameter block update updates different parameter blocks at different training stages to ensure the overall convergence of the model and training stability.

[0032] The adaptation module includes a timing adaptation module. The timing adaptation module is used to perform timing feature modeling on continuously acquired gesture image sequences to enhance the model's ability to recognize dynamic gesture changes.

[0033] The timing adaptation module includes a recurrent neural network structure and a gating network, used for feature modeling of short-term or long-term gesture sequences. The recurrent neural network is used to model short-term dynamic changes. ( ), and recurrent neural networks for modeling long-term dependencies. ( This allows the system to take into account the dynamic characteristics of gestures at different time scales, improving the accuracy of recognizing complex, continuous gestures. A gating network is a simple fully connected layer that receives the video length... As input, and output a fusion weight. in, It is the Sigmoid function. and These are the learnable parameters of the gating network. The output of the timing adaptation module is a weighted sum of the outputs of the two timing modules: in, It is the output of the timing adaptation module. Given the input features, ( )and ( ) represents linear dimensionality reduction and linear dimensionality increase mappings, used to capture temporal feature combinations along the dominant direction.

[0034] D, input the timing representation into the gesture classification and decision module, and output the corresponding gesture category or control semantics; E, according to a preset mapping relationship, convert the gesture category or control semantics into robot control instructions; F, the control commands are sent to the robot control terminal via the communication module; G. After receiving the control command, the robot control terminal drives the actuator to perform the corresponding action, thereby realizing real-time control of the robot.

[0035] In this invention, the gesture recognition model uses a large-scale temporal model to uniformly model continuous gesture actions, effectively depicting the dynamic evolution of gestures over time. This enhances the understanding of complex gesture semantics and reduces control instability caused by single-frame recognition errors. Simultaneously, by establishing a flexible control command mapping relationship, users can customize gesture and control command configurations according to different robot platforms and task requirements, enhancing the system's versatility and scalability. Furthermore, control commands are transmitted between the host computer and the robot via a standardized communication protocol, ensuring the stability, reliability, and real-time performance of the command interaction process.

[0036] The general robot gesture control method based on a large temporal model in this embodiment includes the following steps: In this embodiment, a gesture recognition dataset for training is first constructed. This dataset contains a series of consecutive gesture images for multiple predetermined gesture categories (such as forward, stop, turn left, turn right, etc.). Each sequence is continuously acquired by an image acquisition device (such as a camera) to characterize the complete dynamic process of the gesture.

[0037] B. Data Preprocessing: The acquired image samples are converted to different image formats (e.g., from JPEG to PNG), their size is standardized (e.g., to 224×224), and background is suppressed (e.g., skin color detection or edge filtering techniques are used to remove irrelevant backgrounds). Data augmentation techniques, such as cropping, rotation, scaling, and adding noise, are also employed to improve the stability of model training.

[0038] C. Gesture Recognition and Inference Based on a Temporal Large-Scale Model: A preprocessed sequence of gesture images is input into the temporal large-scale model. The model jointly models the information from consecutive frames, extracting a temporal representation reflecting the dynamic evolution of the gesture. Based on this temporal representation, the gesture classification and decision module outputs the gesture category or control semantics corresponding to the current gesture. Compared to methods based solely on single-frame images, this approach effectively reduces transient motion interference and improves the stability of recognizing complex, continuous gestures.

[0039] D. Online Recognition and Reasoning: During the system deployment phase, user gesture images are captured in real time via a camera and fed into a pre-trained model for inference output. The model returns the gesture category number corresponding to the current image.

[0040] E. Control Command Mapping: The system has a built-in mapping table between gesture categories and control actions, typically: gesture 1 → forward, gesture 2 → backward, gesture 3 → turn left, gesture 4 → turn right, gesture 5 → stop. The recognition results will be converted into standard control command data packets.

[0041] F. Command Communication: Control commands are sent from the host computer to the robot receiving module via serial port, Bluetooth, or Wi-Fi. The communication process uses a custom protocol, including frame headers, category fields, and checksum fields, to ensure stable command transmission and parsing.

[0042] G. Action Execution: The robot control unit parses the received instructions, drives the motors to perform corresponding actions, and completes operations such as path movement, turning, or stopping.

[0043] Example 2 Based on Example 1, in order to further improve the stability of the system in dynamic environments, this example adopts a continuous time-series fusion strategy.

[0044] During the inference phase, the system performs statistical and weighted fusion on the gesture recognition results within multiple consecutive time windows, and outputs the final gesture category, thereby reducing recognition fluctuations caused by single-frame misjudgments or short-term interference and improving the overall stability of the system.

[0045] Example 3 Based on Example 1, in order to reduce the training and update costs of large time series models in actual deployment, this example adopts a parameter-efficient model update strategy during the model training or adaptation phase.

[0046] Without altering the core structure of the time-series model, only parameters relevant to the current robot control task are updated, while the remaining parameters are frozen. This approach ensures gesture recognition performance while reducing computational and storage overhead during model training. Example 4 This embodiment further explains the model structure and training method of the gesture recognition module based on embodiments 1 to 3.

[0047] In this embodiment, the gesture recognition model uses a pre-trained large model with temporal modeling capabilities as the basic network structure. This large model has already learned general feature representation capabilities through large-scale visual data during the pre-training stage. When training for robot control scenarios, the backbone parameters of the large model are frozen, and only a lightweight adaptation module is introduced internally for task-related adjustments.

[0048] The adaptation module includes a temporal adaptation module for modeling the temporal relationships of continuous gesture actions, and a feature adaptation module for adjusting the distribution of high-level semantic features. Through these adaptation modules, the model can effectively improve its ability to understand dynamic gesture changes without significantly increasing the parameter scale.

[0049] During model training or fine-tuning, a parameter block update optimization strategy is adopted. Gradient updates are performed only on the parameters of the currently participating adaptation modules, while parameters not participating in the update are frozen, thereby reducing GPU memory usage during training and improving model training efficiency.

[0050] Through the above method, this embodiment can quickly learn new gesture categories under a small number of samples and stably map the recognition results into robot control commands, thereby achieving efficient and reliable human-computer interaction control.

[0051] Example 5 Reference Figure 2 A general robot gesture control system based on a time-series large model includes: The image acquisition module is responsible for acquiring user gesture visual data in real time and continuously. This module typically consists of an RGB camera, a depth camera, or a combination of both, and is deployed on the robot itself or in the interactive environment. Its processing includes: 1) Data capture: Continuously acquiring video streams containing user gestures at a frame rate of at least 30 frames per second. To adapt to different interaction distances and angles, wide-angle lenses or gimbal adjustment mechanisms can be used. 2) Initial transmission: The acquired raw image sequence is transmitted in real time via USB, MIPI, Ethernet, or wireless network interfaces to a host computer or edge computing device (such as an embedded Jetson platform, industrial PC, etc.) responsible for computation, providing a data source for subsequent processing.

[0052] The gesture recognition module first standardizes the received raw image sequence to unify image sizes. Then, it employs background suppression techniques to highlight the hand region and reduce environmental interference. To further enhance model robustness, online data augmentation operations such as random cropping, rotation, and brightness adjustment can be applied. The preprocessed continuous image frame sequence is then input into a pre-built temporal model. This model preferably uses a Visual Transformer (ViT) architecture and incorporates a hybrid adapter to enhance temporal modeling capabilities, simultaneously capturing both the long-range dependencies and short-term dynamics of gesture actions.

[0053] The instruction mapping module is responsible for converting gesture semantics into standardized control instructions that the robot can understand. This module maintains a flexibly configurable gesture-instruction mapping table. The processing steps are: 1) Semantic mapping: Receive the category number output by the gesture recognition module, query the mapping table, and find the corresponding robot atomic operation instruction. 2) Instruction encapsulation: Encapsulate the mapped operation instruction, along with necessary timestamps, priority, checksums, and other information, into a standard instruction data packet with a preset format for subsequent transmission and parsing. Users can dynamically modify or extend this mapping relationship according to different robot platforms or task scenarios, reflecting the system's versatility.

[0054] The communication module is responsible for establishing a stable, real-time data link between the host computer and the robot control terminal. Its processing includes: 1) Protocol transmission: Selecting an appropriate communication method based on the robot's hardware interface, such as serial port (UART), Bluetooth, Wi-Fi, or ROS (Robot Operating System) topics / services. The data packets generated by the instruction mapping module are sent to the robot via the selected protocol. 2) Reliability assurance: The communication protocol design includes fields such as frame header identification, data segments, and cyclic redundancy check (CRC) to ensure the integrity and accuracy of instructions. The module can implement a simple acknowledgment mechanism or packet loss retransmission logic to adapt to unstable wireless network environments.

[0055] The robot execution module is the final execution unit of the control system, located in the robot control terminal (such as an STM32, Raspberry Pi, or robot main controller). Its processing includes: 1) Instruction parsing and verification: Receiving instruction data packets from the communication module, unpacking, parsing the format, and verifying their validity. 2) Action execution: Based on the instruction content, calling the underlying driver or control library (such as controlling motors, servos, joints, etc.) to execute the corresponding action, such as moving straight, turning, stopping, or robotic gripping. The execution process may include speed planning, interpolation calculations, and abnormal state monitoring to ensure smooth and safe movements.

[0056] The modules work together through logical connections to achieve natural and efficient control of the robot.

[0057] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A general robot gesture control method based on a large temporal model, characterized in that, The method includes the following steps: A. Acquire the user's gesture visual data, which includes a sequence of gesture images continuously acquired by an image acquisition device, used to characterize the change process of gesture actions in the time dimension. B. Preprocess the gesture visual data, including image size normalization, background interference suppression, key region extraction, and temporal alignment operations, to reduce the impact of environmental factors on subsequent temporal modeling. C. Construct a gesture recognition model based on a temporal large model, wherein the temporal large model is used to uniformly model continuous gesture image sequences and extract temporal representations that reflect the dynamic evolution characteristics of gestures; D, input the timing representation into the gesture classification and decision module, and output the corresponding gesture category or control semantics; E, according to a preset mapping relationship, convert the gesture category or control semantics into robot control instructions; F, the control commands are sent to the robot control terminal via the communication module; G. After receiving the control command, the robot control terminal drives the actuator to perform the corresponding action, thereby realizing real-time control of the robot.

2. The general robot gesture control method based on a large temporal model as described in claim 1, characterized in that, In C, the time-series large model includes at least one backbone coding network and multiple lightweight adaptation modules. The adaptation modules make task-related adjustments to the model without updating the backbone parameters.

3. The general robot gesture control method based on a large temporal model as described in claim 2, characterized in that, The adaptation module includes a temporal adaptation module, which is used to perform temporal feature modeling on continuously acquired gesture image sequences to enhance the model's ability to recognize dynamic gesture changes.

4. The general robot gesture control method based on a large temporal model as described in claim 3, characterized in that, The timing adaptation module includes a recurrent neural network structure for feature modeling of short-term or long-term gesture sequences.

5. The general robot gesture control method based on a large temporal model as described in any one of claims 2 to 4, characterized in that, The adaptation module also includes a feature adaptation module, which is used to perform nonlinear transformations and fine-grained adjustments on high-level semantic features to improve the expressive power of multimodal features.

6. The general robot gesture control method based on a large temporal model as described in claim 1, characterized in that, In C, during model training or fine-tuning, an optimization strategy of parameter block update is adopted, which only performs gradient updates on some adaptation module parameters and freezes the parameters that are not updated, so as to reduce the GPU memory usage and improve training efficiency during model training.

7. The general robot gesture control method based on a large temporal model as described in claim 6, characterized in that, The optimization strategy of parameter block update updates different parameter blocks at different training stages to ensure the overall convergence of the model and training stability.

8. The general robot gesture control method based on a large temporal model as described in claim 1, characterized in that, In A, the gesture image acquisition process includes sampling under various ambient lighting conditions, different backgrounds, different angles, and different individual users to enhance the diversity of the dataset and the generalization ability of the model.

9. The general robot gesture control method based on a large temporal model as described in claim 1, characterized in that, In B, the background interference suppression includes one or more of the following: skin color region extraction, edge detection, background blurring, or foreground segmentation.

10. A system for implementing the general robot gesture control method based on a large temporal model as described in claim 1, characterized in that, The system includes: The image acquisition module is used to acquire user gesture images or image sequences in real time; The gesture recognition module is used to recognize the captured gesture images; The control command mapping module is used to convert gesture recognition results into general robot control commands; A communication module is used to send the control commands to the general-purpose robot; The execution module is used to parse control commands and drive the general-purpose robot to perform corresponding actions.

Images