A robot motion generation method and system combining general and special models

Abstract

The application relates to a robot action generation method and system combining a general model and a special model, wherein the method comprises the following steps: constructing a general model and a special model, pre-training the general model, fine-tuning the parameters of the pre-trained general model, and training the special model based on the fine-tuned general model; acquiring a task instruction and real-time visual information, inputting the task instruction and the real-time visual information into the fine-tuned general model, and outputting an action sequence and a task latent feature; acquiring real-time point cloud perception data, and inputting the real-time point cloud perception data and the action sequence and the task latent feature into the trained special model, and outputting a robot continuous action. Compared with the prior art, the application improves the speed of robot action generation and enhances the generalization of robot action generation.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method and system for generating robot motions that combines general and special models. Background Technology

[0002] Developing robots with multi-tasking and adaptive capabilities has always been a core goal in the field of robotics. Traditional robot learning methods typically learn policies by specifying a robot and a dataset for a particular task; these operational policies are called specialized models. Specialized models exhibit high accuracy in specific scenarios and tasks, but often have limited generalization ability. As robots are increasingly used in open and multi-tasking scenarios, the demand for multi-tasking robots is surging.

[0003] To address the needs of multi-tasking robots, general-purpose robot strategies, such as RT-2 and OpenVLA, have begun to emerge. These strategies attempt to apply internet knowledge to robot control, leveraging extensive heterogeneous datasets to enhance the robot's cross-domain versatility. General-purpose strategies integrate large amounts of cross-ontology data with pre-trained large language models to achieve functions such as commonsense reasoning and instruction tracking in robot policy learning. While general-purpose strategies excel at knowledge transfer and generalization across different scenarios, they still have some limitations: 1) Without fine-tuning, they cannot be directly deployed to new applications or environments. Compared to dedicated strategies, the fine-tuning process requires more data and training. 2) Although general-purpose strategies excel at decision-making, their large model size leads to extremely high inference latency. This critical bottleneck makes them unsuitable for fine-grained control in dynamic environments. Currently, lightweight models are used to address these shortcomings, but this results in a significant decrease in model performance.

[0004] Therefore, providing a robot motion generation method that can satisfy both the generalization of general models and the high performance of special models is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a robot motion generation method and system that combines general and special models. By effectively combining the general model and the special model, the general model is used to process complex multimodal data, the point cloud data is used as the input of the special model, and the output of the general model is used as the diffusion denoising condition. This preserves both the generalizability of the general model and the accuracy of the special model.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a method for generating robot motions combining general and special models is provided, characterized in that the method comprises:

[0008] Construct a general model and a special model, pre-train the general model, fine-tune the parameters of the pre-trained general model, and train the special model based on the fine-tuned general model.

[0009] Acquire task instructions and real-time visual information, input the task instructions and real-time visual information into a finely tuned general model, and output action sequences and potential task features.

[0010] Acquire real-time point cloud perception data, and input the real-time point cloud perception data, along with the action sequence and potential task features, into a trained dedicated model to output continuous robot actions.

[0011] As a preferred technical solution, the general model is constructed based on a large visual language model.

[0012] As a preferred technical solution, the dedicated model is built based on a lightweight and scalable diffusion model.

[0013] As a preferred technical solution, the pre-training of the general model includes:

[0014] Acquire robot operation data, which includes language commands, image data, and robot motion data;

[0015] Based on the aforementioned language instructions and image data, language features and visual features are extracted using the aforementioned general model, and the language features and visual features are aligned.

[0016] Iterate through the following steps until pre-training is complete:

[0017] The aligned linguistic and visual features are input into the general model, the first discretization action is output, and the first discretization action is decoded.

[0018] Based on the decoded first discretized action and robot action data, the first loss is calculated and backpropagation is performed. The first gradient is calculated, and the parameters of the general model are updated based on the first gradient.

[0019] As a preferred technical solution, the fine-tuning method of the general model includes:

[0020] Obtain robot operation data for a specific task, and use the parameters of the pre-trained general model as initial parameters;

[0021] Determine the fine-tuning range of the parameters involved in visual feature extraction, language feature extraction, and action decoding in the initial parameters, and freeze the other parameters in the initial parameters;

[0022] Input the robot operation data under the specific task into the general model and output the second discretized action;

[0023] Based on the robot motion data in the second discretized action and the robot operation data under the specific task, the second loss is calculated, and backpropagation is performed to calculate the second gradient. The parameters of the general model are updated based on the second gradient.

[0024] As a preferred technical solution, the training of the dedicated model includes:

[0025] Acquire point cloud datasets and corresponding robot operation data at different times. The point cloud datasets include point cloud data, robot state ontology perception data, and robot motion data.

[0026] Based on the robot operation data at the corresponding time, the fine-tuned general model is used to output the action sequence and potential task features;

[0027] A dedicated model is trained based on the aforementioned point cloud dataset, action sequences, and potential task features. The loss during the training process is calculated, backpropagation is performed, the gradient is calculated, and the parameters of the dedicated model are updated based on the gradient.

[0028] As a preferred technical solution, the general model and the special model cooperate asynchronously, and the method of asynchronous cooperation is as follows:

[0029] Obtain the task instructions and real-time visual information at the current time t1, and use a general model to output the action sequence and potential task features at time t1;

[0030] Get t1 to t n Real-time point cloud sensing data at time t1, and based on the action sequence and potential task features at time t1, a dedicated model is used to output data from time t1 to time t2. n Continuous actions at any given moment.

[0031] As a preferred technical solution, the method for outputting the action sequence and potential task features at time t1 is as follows:

[0032] Based on the visual information at time t1, visual features are extracted using a general model; based on the task instructions at time t1, language features are extracted using a general model.

[0033] The aforementioned visual and linguistic features are projected into a unified latent space to generate the task latent features at time t1.

[0034] Based on the aforementioned visual and linguistic features, a discretized action sequence is generated according to time steps using a general model, the expression of which is:

[0035] a t1 =g φ (a＜t1 c),

[0036] Among them, a t1 Let a represent the action sequence at time t1. ＜t1 Let c represent the action sequence before time t1, and g represent visual and linguistic features. φ (·) indicates general model processing.

[0037] As a preferred technical solution, the outputs t1 to t n The method for continuous actions at any given time is as follows:

[0038] Based on the aforementioned t1 to t n Real-time point cloud sensing data is used to extract real-time point cloud sensing features.

[0039] The real-time point cloud sensing features at each time step are aligned and fused with the action sequence and task latent features at time step t1, and the corresponding action at each time step is generated using a diffusion denoising mechanism. The expression is as follows:

[0040]

[0041] in, Indicates t i The action at time i ∈ [1, n], where ∈ represents Gaussian noise. Indicates t i The alignment and fusion results of real-time point cloud perception features at time t1 with action sequences and potential task features at time t1.

[0042] According to a second aspect of the present invention, a robot motion generation system combining general and special models is provided, the system being used to implement the above-described method.

[0043] Compared with existing technologies, this invention utilizes a general model to process multimodal data composed of vision and language, and uses the output of the general model as the diffusion denoising condition for the special model, combined with point cloud data that can provide better spatial information, to output the robot's continuous actions, while retaining the generalization of the general model and the accuracy of the special model. Furthermore, this invention maintains asynchronous cooperation between the general model and the special model, that is, the general model infers once at the current moment, and the special model infers actions including the current moment and multiple future moments based on the inference result of the general model at the current moment. This reduces the resource consumption of the general model and can also ensure the accuracy of the generated actions. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention;

[0045] Figure 2 This is a hardware architecture diagram of the system of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0048] This embodiment combines the generalization of a multimodal general model with the high efficiency of a dedicated model, improving the generalization and inference speed of the robot motion generation system. Furthermore, the dedicated model uses point cloud data as input, which provides better spatial information compared to RGB image data, thus helping to enhance the robot's operational capabilities.

[0049] Specifically, the process of this method is as follows: Figure 1 As shown, it includes:

[0050] S1. Model Building:

[0051] S11. Based on a large visual language model such as OpenVLA, construct an autoregressive visual language action model with approximately 7B parameters as a general model. Specifically, this general model includes: a visual encoder, a large model (LLaMA-2) module, and an action generation module, wherein the action generation module includes an action decoder.

[0052] S12, Pre-trained general model:

[0053] S121. Acquire robot operation data, including language commands, RGB images, and robot motion data;

[0054] S122. Visual features are extracted from RGB images using visual encoders (DINOv2 and ViT), language features are extracted from language instructions using a large language model, and visual features are mapped to a latent space consistent with language features through a multilayer perceptron for feature alignment.

[0055] S123. Input the aligned language features and visual features into the action generation module, output the first discretized action, and decode the first discretized action. The action range is mapped to a uniform distribution of [-1,1] to ensure that the general model has clear physical constraints when generating actions.

[0056] S124. Based on the decoded first discretized action and robot action data, calculate the first loss and perform backpropagation, calculate the first gradient, and update the parameters of the general model using the AdamW optimizer based on the first gradient.

[0057] The formula for calculating the first loss is:

[0058]

[0059] Where p represents the input language prompt, a ＜i For the action sequence generated before step i, N a The length of the action sequence.

[0060] Iterate through steps S123 to S124 until the pre-training is complete.

[0061] S13, Fine-tuning of the general model:

[0062] S131. Obtain robot operation data under a specific task, such as robot operation data generated when folding clothes, and obtain the parameters of the pre-trained general model as initial parameters to retain the general knowledge learned during the pre-training process.

[0063] S132. Determine the fine-tuning range of the parameters involved in visual feature extraction, language feature extraction, and action decoding in the initial parameters, and freeze the other parameters in the initial parameters.

[0064] S133. Input the robot operation data under a specific task into the general model and output the second discretized action.

[0065] S134. Based on the second discretized action and the robot action data in the robot operation data under the specific task, calculate the second loss and its calculation expression is consistent with that in step S124. Perform backpropagation, calculate the second gradient, update the parameters of the general model based on the second gradient, and dynamically adjust the learning rate to avoid overfitting.

[0066] The formula for calculating the second loss is as follows:

[0067]

[0068] Where p represents the input language prompt, a ＜i For the action sequence generated before step i, N a The length of the action sequence.

[0069] S14. Based on the DiT diffusion model, construct a lightweight and scalable dedicated model, which is a multimodal action denoising model. Specifically, the dedicated model includes a lightweight point cloud visual encoder (PointNet++) and an action generator. The action generator includes: a causal self-attention layer for processing time-series actions, a cross-modal attention layer for fusing point cloud features and the output features of a general model, a feedforward network for nonlinear feature transformation, and a position encoder that maintains the sequential consistency of the time series.

[0070] S15, Specialized Model Training:

[0071] S151. Obtain the point cloud dataset and the robot operation data at the corresponding time. The point cloud dataset includes point cloud data, robot state ontology perception data and robot motion data. Then, use a lightweight point cloud visual encoder to extract point cloud features based on the point cloud dataset.

[0072] S152. Based on the robot operation data at the corresponding time, the fine-tuned general model is used to output the action sequence and potential task features.

[0073] S153. Train a dedicated model based on point cloud features, action sequences, and latent task features. Calculate the loss during training, perform backpropagation, calculate the gradient, and update the parameters of the dedicated model based on the gradient. The expression for calculating the loss is:

[0074]

[0075] Where, π θis a dedicated model; a0 is the real action; ∈ is Gaussian noise; t is the time step; c is the conditional input, including the action sequence and task latent features from the fine-tuned general model, as well as point cloud features.

[0076] S2, Output action sequence and potential task features:

[0077] S21. Adjust the size of the RGB image at time t1 to a preset size, such as 224×224, and after standardization, extract visual features using a visual encoder; based on the task instructions at time t1, i.e. natural language task instructions, extract language features using a large model, i.e. semantic embedding.

[0078] S22. Project the visual and linguistic features into a unified latent space to generate the task latent features at time t1.

[0079] S23. Based on visual and linguistic features, a discretized action sequence is generated using a general model and an autoregressive mechanism according to time steps. Its expression is:

[0080]

[0081] in, This represents the sequence of actions at time t1. Let c represent the action sequence before time t1, and g represent visual and linguistic features. φ (·) indicates general model processing.

[0082] S3, Output continuous robot motion:

[0083] S31, Based on t1 to t n Real-time point cloud perception data is obtained at all times, and a lightweight point cloud visual encoder is used to extract real-time point cloud perception features.

[0084] S32. Input the real-time point cloud perception features at each time step and the action sequence and task latent features at time step t1 into the shared latent space for alignment and fusion, and use the diffusion denoising mechanism to generate the action at the corresponding time step, the expression of which is:

[0085]

[0086] in, Indicates t i The action at time i ∈ [1, n], where ∈ represents Gaussian noise. Indicates t i The alignment and fusion results of real-time point cloud perception features at time t1 with action sequences and potential task features at time t1.

[0087] Furthermore, steps S2 and S3 employ a fixed window mechanism for asynchronous collaboration, specifically as follows:

[0088] A1. Obtain the task instructions and real-time visual information at the current time t1, and use a general model to output the action sequence and potential task features at time t1.

[0089] A2. Acquire real-time point cloud perception data from time t1 to t8, and based on the action sequence and potential task features at time t1, output continuous actions from time t1 to t8 using a dedicated model.

[0090] A3. When the general model updates the output action sequence and task latent features, the special model uses the updated output action sequence and task latent features to perform step A2.

[0091] A4. Repeat steps A1 to A3 until a complete robot motion is generated.

[0092] This embodiment also provides a robot motion generation system that combines general and special models to implement the above method, and its hardware framework is as follows: Figure 2 As shown, it includes a controller, a teleoperated device, a depth camera, and a robotic arm. Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes described herein can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0093] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating robot motions that combines general and specialized models, characterized in that, The method includes: Construct a general model and a special model, pre-train the general model, fine-tune the parameters of the pre-trained general model, and train the special model based on the fine-tuned general model. The general model and the special model collaborate asynchronously, and the specific process is as follows: The task instructions and real-time visual information at the current time t1 are obtained. These instructions and information are then input into a finely tuned general model, which outputs the action sequence and potential task features at time t1. The specific implementation is as follows: Based on the visual information at time t1, visual features are extracted using a general model; based on the task instructions at time t1, language features are extracted using a general model. The aforementioned visual and linguistic features are projected into a unified latent space to generate the task latent features at time t1. Based on the aforementioned visual and linguistic features, a discretized action sequence is generated according to time steps using a general model, the expression of which is: ， in, express The sequence of actions at any given moment express The sequence of actions prior to time, where c represents visual features and linguistic features. Indicates general model processing; Get t1 to t n The robot obtains real-time point cloud perception data at time t1, and inputs this real-time point cloud perception data, along with the action sequence and potential task features at time t1, into a trained dedicated model to output the robot's data from time t1 to time t2. n The continuous actions at time points are specifically implemented as follows, based on the aforementioned t1 to t n Real-time point cloud sensing data is used to extract real-time point cloud sensing features. The real-time point cloud sensing features at each time step are aligned and fused with the action sequence and task latent features at time step t1, and the corresponding action at each time step is generated using a diffusion denoising mechanism. The expression is as follows: ， in, express The sequence of actions at time and , Indicates Gaussian noise. express Real-time point cloud sensing features and The result of aligning and fusing the action sequence at any given time with the latent features of the task.

2. The robot motion generation method combining general and special models according to claim 1, characterized in that, The general model described is built upon a large visual language model.

3. The robot motion generation method combining general and special models according to claim 1, characterized in that, The specialized model is built upon a lightweight and scalable diffusion model.

4. The robot motion generation method combining general and special models according to claim 1, characterized in that, The pre-training of the general model includes: Acquire robot operation data, which includes language commands, image data, and robot motion data; Based on the aforementioned language instructions and image data, language features and visual features are extracted using the aforementioned general model, and the language features and visual features are aligned. Iterate through the following steps until pre-training is complete: The aligned linguistic and visual features are input into the general model, the first discretization action is output, and the first discretization action is decoded. Based on the decoded first discretized action and robot action data, the first loss is calculated and backpropagation is performed. The first gradient is calculated, and the parameters of the general model are updated based on the first gradient.

5. The robot motion generation method combining general and special models according to claim 4, characterized in that, The fine-tuning methods for the general model include: Obtain robot operation data for a specific task, and use the parameters of the pre-trained general model as initial parameters; Determine the fine-tuning range of the parameters involved in visual feature extraction, language feature extraction, and action decoding in the initial parameters, and freeze the other parameters in the initial parameters; Input the robot operation data under the specific task into the general model and output the second discretized action; Based on the robot motion data in the second discretized action and the robot operation data under the specific task, the second loss is calculated, and backpropagation is performed to calculate the second gradient. The parameters of the general model are updated based on the second gradient.

6. The robot motion generation method combining general and special models according to claim 5, characterized in that, The training of the specialized model includes: Acquire point cloud datasets and corresponding robot operation data at different times. The point cloud datasets include point cloud data and robot state ontology perception data. Based on the robot operation data at the corresponding time, the fine-tuned general model is used to output the action sequence and potential task features; A dedicated model is trained based on the aforementioned point cloud dataset, action sequences, and potential task features. The loss during the training process is calculated, backpropagation is performed, the gradient is calculated, and the parameters of the dedicated model are updated based on the gradient.

7. A robot motion generation system combining general and special models, characterized in that, The system is used to implement the method as described in any one of claims 1 to 6.

Images