Double-arm robot control method based on single-arm track prior guidance

Abstract

The invention discloses a double-arm robot control method based on single-arm trajectory prior guidance. The method comprises the following steps: acquiring a double-arm robot data set; skill primitives of different action types are obtained based on a single-arm action prediction model, the skill primitives of different action types are selected for a left arm and a right arm, and next action intentions of the left arm and the right arm are predicted respectively; constructing a double-arm diffusion generation model for establishing a mapping relation from initial action distribution to real double-arm coordination action distribution; training a single-arm action prediction model and a double-arm diffusion generation model based on the double-arm robot training data set; and acquiring motion prediction priori of the left arm and the right arm, constructing Gaussian distribution taking the single-arm priori as a mean value and a preset variance as a radius, sampling from the Gaussian distribution to obtain an initial motion vector, inputting the initial motion vector into a trained double-arm diffusion generation model, and de-noising the initial motion vector to generate double-arm motion. According to the invention, the independence of single-arm motion and the collaboration of double-arm control can be considered, and the double-arm motion control effect is optimized.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and specifically to a method for controlling a dual-arm robot based on prior guidance of a single-arm trajectory. Background Technology

[0002] Existing dual-arm robotic arm control methods mainly include rule-based planning methods, optimization-based motion planning methods, and learning-based methods. Rule-based or optimization-based methods typically rely on accurate environmental modeling and dynamic constraints, jointly planning the motion trajectories of the two arms to avoid collisions and complete the task. These methods can achieve high success rates under known environments and explicit task constraints, but they require high accuracy in environmental perception and model accuracy. Once the environment changes or task conditions change, it is often necessary to remodel and adjust parameters, lacking flexibility and adaptability. In addition, jointly planning the trajectory of the two arms usually leads to a significant increase in the dimensionality of the search space, resulting in high computational complexity and difficulty in meeting real-time requirements.

[0003] In recent years, with the development of deep learning, data-driven learning methods have been gradually introduced into the control of dual-arm robotic arms. Among these, imitation learning and reinforcement learning are widely used to learn dual-arm operation strategies from demonstration or interactive data. These methods model the movements of both arms as a whole, enabling the robotic arm to autonomously learn complex operational behaviors to a certain extent, reducing the reliance on precise modeling and manual rule design. However, existing dual-arm learning methods still have the following shortcomings, limiting their application effectiveness in real-world scenarios:

[0004] (1) High dependence on bi-arm demonstration data and high data acquisition cost. Existing bi-arm learning methods usually require a large amount of high-quality bi-arm collaborative demonstration data as the training basis. Compared with single-arm operation, bi-arm demonstration is not only more difficult to collect, but also has higher requirements for action synchronization, spatial coordination and safety, resulting in a significant increase in data acquisition cost. In addition, the bi-arm action space has high dimension and complex combination, making it difficult for a limited number of demonstration data to cover all possible collaborative modes, thereby reducing data utilization efficiency and limiting the generalization ability of the algorithm.

[0005] (2) Existing control architectures struggle to balance the rationality of single-arm motion with the coordination of two arms. In dual-arm operation tasks, spatial constraints and potential conflicts inevitably exist between robotic arms and between robotic arms and the environment. However, existing technologies often fall into a dilemma when dealing with this problem: First, end-to-end direct coordination. Some methods attempt to directly generate complete joint motion trajectories in the high-dimensional dual-arm motion space, strongly coupling the generation of single-arm motion with the coordination adjustment of two arms in the same learning process. This modeling method forces the model to simultaneously handle the rationality of single-arm motion (such as smoothness and reaching the target) and the geometric coordination relationship between the two arms in a single prediction, resulting in an excessive learning burden on the model, difficulty in convergence, and a tendency to overlook one aspect while focusing on the other. Second, hierarchical separation control. Design a dedicated coordination module and use this module to schedule or correct independent single-arm strategies, forcibly separating control and coordination. Although this method reduces the dimensionality, it ignores the inherent deep connection between the low-level motion characteristics of single arms and the high-level coordination logic, resulting in the coordination module often failing to accurately understand the intention of single arms, poor actual execution effect, and a tendency to experience timing misalignment or jamming. Summary of the Invention

[0006] To overcome the shortcomings and deficiencies of existing technologies, and addressing the scarcity of dual-arm data and the difficulty in balancing direct collaboration and hierarchical separation in existing architectures, this invention provides a dual-arm robot control method based on single-arm trajectory prior guidance. This method does not directly output dual-arm movements, nor is it a simple superposition of single-arm actions. Instead, it uses single-arm action prediction as prior information to guide dual-arm action generation. Specifically, it uses a single-arm strategy to predict the intended action of a single arm, and injects this prediction result as a prior distribution into the initial stage of the dual-arm action generation model. In this way, the dual-arm action model focuses on coordinating and correcting the predicted single-arm actions during the generation process, thereby reducing the burden on the model to learn motion trajectories from scratch. This not only reduces dependence on paired dual-arm demonstration data while preserving the effectiveness of single-arm skills, but also more effectively solves the problems of temporal and spatial coordination and stability of dual-arm actions.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention provides a method for controlling a dual-arm robot based on prior guidance of a single-arm trajectory, comprising the following steps:

[0009] Obtain the dataset for the dual-arm robot;

[0010] Based on the single-arm motion prediction model, skill primitives of different motion types are obtained. According to the instructions of both arms, skill primitives of different motion types are selected for the left and right arms, and the next motion intention of the left and right arms is predicted respectively.

[0011] A dual-arm diffusion generation model is constructed to establish a mapping relationship from the initial motion distribution to the actual dual-arm coordinated motion distribution;

[0012] Training a single-arm motion prediction model based on a dual-arm robot training dataset;

[0013] The dual-arm diffusion generation model was trained based on the dual-arm robot training dataset;

[0014] Prior predictions for the left and right arms are obtained based on the trained single-arm motion prediction model.

[0015] A Gaussian distribution with the single-arm action prediction prior as the mean and the preset variance as the radius is constructed. An initial action vector is sampled from the Gaussian distribution and input into the trained dual-arm diffusion generation model. The initial action vector is then denoised to generate dual-arm actions.

[0016] As a preferred technical solution, the dual-arm robot dataset includes serialized multimodal information. and natural language commands, among which, This represents environmental observation data, including images from multiple perspectives. This indicates a combined movement of both arms, including the end positions of the left and right arms.

[0017] As a preferred technical solution, skill primitives of different movement types are selected for the left and right arms according to the instructions of both arms, and the next movement intention of the left and right arms is predicted respectively, specifically including:

[0018] A visual alignment module is constructed. The input of the visual alignment module is the three-dimensional spatial voxel features obtained by image encoding and projection of the current observation image into three-dimensional space. The output is the observation features of the left arm and the observation features of the right arm.

[0019] Construct an action selection module that receives spatial voxel feature vectors from the visual alignment module and task instruction features from the language encoder, and outputs left arm language features and right arm language features.

[0020] Action priors for the left and right arms are generated based on a single-arm policy network.

[0021] As a preferred technical solution, in the visual alignment module, a left arm mask and a right arm mask are generated by a mask generator. The left arm mask and the right arm mask are then multiplied element-wise with the original input voxel feature vector, and the masked features are added to the original input voxel feature vector to obtain the left arm observation features and the right arm observation features, specifically represented as follows:

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] in, This represents the currently observed image as input. Indicates a mask generator. ViT represents the training parameters, and ViT represents the image encoder. This represents the dot product operation. This indicates a splicing operation. Indicates the left arm mask. This indicates the features observed in the left arm after masking. This indicates the observed features of the spliced ​​left arm. Indicates the right arm mask. This indicates the features observed in the left arm after masking. This indicates the observational features of the right arm after splicing.

[0028] The calculation process of the loss function is as follows:

[0029] ;

[0030] in, This represents the KL divergence.

[0031] As a preferred technical solution, in the action selection module, a voxel feature sequence is obtained through linear projection and pooling. The voxel feature sequence and the task instruction feature sequence are then concatenated in parallel in time to obtain a fused feature sequence.

[0032] The fused feature sequence is input into the Transformer model, and the output of the Transformer model is mapped to the primitive category space through a linear layer. The dimension of the primitive category space corresponds to the total number of action categories.

[0033] A skill primitive library is constructed to store action semantic embedding vectors corresponding to action categories. The vectors in the pre-constructed skill primitive library are then weighted and linearly combined to obtain the linguistic features of the left and right arms, specifically represented as follows:

[0034] ;

[0035] ;

[0036] ;

[0037] in, Indicates the observational features of the left arm. Indicates the characteristics of the task instruction. Represents a weighted predictor. Represents training parameters, Indicates the weight of the left arm. This indicates task-oriented compensation for the left arm. Primitives representing different types of actions, This represents the combined language features of the left arm. Indicates the observational features of the right arm. Indicates the weight of the right arm. This indicates task-oriented compensation for the right arm. This indicates the combined linguistic features of the right arm;

[0038] The calculation process of the loss function is as follows:

[0039] ;

[0040] In the formula, Describing the L1 norm, Represents the L21 norm. It is an adjustable balance parameter.

[0041] As a preferred technical solution, the action priors for the left and right arms are generated separately based on a single-arm policy network, specifically including:

[0042] The left arm observation features and left arm language features are used as input pairs, the right arm observation features and right arm language features are used as input pairs, and the current robot arm body perception state is also input.

[0043] Semantic association and spatial alignment are performed through a cross-attention layer. The pose and gripper state of the end effector are predicted by probability distribution, and the discretized action prediction is output to obtain the action priors of the left and right arms.

[0044] As a preferred technical solution, a dual-arm diffusion generation model is constructed to establish a mapping relationship from the initial motion distribution to the actual dual-arm coordinated motion distribution, specifically including:

[0045] Input the current proprioception state of the robotic arm into the state MLP to obtain the proprioception Token;

[0046] Input the single-arm action priors of the left and right arms into the noisy trajectory MLP to obtain the action prior token;

[0047] The images from all perspectives of the dual-arm robot are independently encoded using a pre-trained visual encoder, 2D feature maps are extracted, and each pixel on the 2D feature map is back-projected into the robot's 3D world coordinate system.

[0048] Input natural language instructions into a pre-trained text encoder to extract language instruction tokens;

[0049] A 3D flow matching Transformer model is constructed. Based on the cross-attention module, the noise trajectory token is used as the query. The 3D scene visual tokens O and language instruction features are used as keys and values ​​respectively. The interaction between the noise trajectory token and other internal features is processed based on the self-attention module. Finally, the arm movements are obtained based on the feedforward neural network.

[0050] As a preferred technical solution, the single-arm motion prediction model is trained based on the dual-arm robot training dataset, and the loss function is expressed as:

[0051] ;

[0052] ;

[0053] Where BCE represents the cross-entropy loss of the behavioral clone, This represents the difference between the actual actions of the expert and the predicted actions. and It is an adjustable parameter. and These are the loss functions for the visual alignment module and the action selection module, respectively.

[0054] As a preferred technical solution, the dual-arm diffusion generation model is trained based on a dual-arm robot training dataset, specifically including:

[0055] Environmental observation data and language commands are input into the single-arm motion prediction model with frozen parameters, and the predicted motion trajectories of the left and right arms are output respectively.

[0056] The predicted motion trajectories of the left and right arms are superimposed with random noise sampled from a Gaussian distribution;

[0057] The predicted motion trajectories of the left and right arms are randomly replaced with all-zero vectors with a set probability.

[0058] As a preferred technical solution, the conditional flow matching algorithm is used to train the two-arm diffusion generation model.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] (1) The present invention utilizes single-arm priors to reduce the search space of double-arm movements. It does not require learning basic movements from scratch, but only needs to use a small amount of double-arm data to learn movement details and correct them, thereby reducing the dependence on a large amount of double-arm demonstration data, reducing the learning burden of double-arm data, and improving the efficiency of double-arm movement generation.

[0061] (2) This invention connects single and double arm strategies through prior guidance, retains the rationality of single arm prediction, and automatically corrects the spatial conflict and temporal connection between the two arms through the generation process, realizing the organic balance between single arm movement and double arm coordination. It can effectively take into account the independence of single arm movement and the coordination of double arm control, and optimize the double arm movement control effect. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the dual-arm robot control method based on single-arm trajectory prior guidance according to the present invention.

[0063] Figure 2 This is a schematic diagram of the overall implementation framework of the dual-arm robot control method based on single-arm trajectory prior guidance of the present invention;

[0064] Figure 3 This is a schematic diagram of the process for predicting the prior movements of a single arm in this invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0066] Example

[0067] like Figure 1 , Figure 2 As shown, this embodiment provides a dual-arm robot control method based on single-arm trajectory prior guidance, including the following steps:

[0068] S1: Obtain the dual-arm robot dataset, which includes environmental observation images, task text descriptions, and sequences of joint dual-arm movements;

[0069] In this embodiment, the dual-arm robot dataset is selected from expert demonstration data collected in the RLBench2 simulation environment. Each demonstration data point contains serialized multimodal information. , and natural language instructions L, in which Representing environmental observations, this includes RGB-D images from five perspectives: frontal, left wrist, right wrist, left shoulder, and right shoulder. The resolution is uniformly adjusted to 256×256. Represents a combined bi-arm movement, including the end-effector poses of the left and right arms, including the position of the end effector. and and rotation and And the opening and closing states of the grippers on the left and right arms. and ;

[0070] S2: Utilizing existing single-arm motion prediction models and skill primitives of different motion types obtained from the language encoder in the single-arm motion prediction model, a primitive selection module is constructed. Based on dual-arm commands, different motion types of skill primitives are selected for the left and right arms. Through pre-training, the next motion intentions of the left and right arms are predicted separately, thus enabling the direct learning and output of independent single-arm motion priors from dual-arm data. Specifically, this includes:

[0071] S21: Construct a visual alignment module, such as Figure 3 As shown, the visual alignment module addresses the visual distribution shift problem in a two-arm environment using a single-arm strategy. Specifically, the input to the visual alignment module consists of 3D spatial voxel features obtained by encoding the currently observed image from the camera and projecting it into 3D space. The core structure of this module, the mask generator (Aligner), is mainly composed of a one-dimensional convolutional neural network (Conv1D). First, the input features pass through a shared feature extraction block, which consists of a concatenated one-dimensional convolutional layer and an activation function layer (ReLU) to extract global context features. Subsequently, the network is divided into two independent parallel branches, corresponding to the left arm mask generation branch and the right arm mask generation branch, respectively. Each branch also contains additional one-dimensional convolutional layers and activation function layers to further refine the visual distribution. The global feature mapping is a spatial soft mask for a specific robotic arm. The numerical range of these masks is limited to 0 to 1, representing the importance weight of different regions in the visual feature space for the current robotic arm. Finally, the generated left arm mask and right arm mask are element-wise multiplied with the original input voxel feature vector, and residual connections are introduced, that is, the masked features are added to the original input features, so as to obtain the final left arm-specific voxel features and right arm-specific voxel features. This ensures that while preserving the original environmental information, the feature saliency of the specific robotic arm's attention area is enhanced through the attention mechanism. In this embodiment, a pre-trained VisionTransformer (ViT) is used as the image encoder, and the parameters are frozen during training.

[0072] In this embodiment, the specific calculation process is represented as follows:

[0073] ;

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] in, This represents the currently observed image as input. Indicates a mask generator. ViT represents the training parameters, and ViT represents the image encoder. This represents the dot product operation. This indicates a splicing operation. Indicates the left arm mask. This indicates the features observed in the left arm after masking. This represents the complete left-arm observation features after splicing, which includes both the features required by the left-arm single-arm policy network and the global context. Similarly, right represents the right-arm related features.

[0079] In this embodiment, the calculation process of the loss function is as follows:

[0080] ;

[0081] in, This represents the KL divergence, and the complete loss function is the JS divergence. Its function is to maximize the difference between the two masks so that the complete visual features can be decomposed into pre-trained single-arm visual features.

[0082] S22: Construct the action selection module, such as Figure 3 As shown, the action selection module is used to dynamically schedule predefined skill primitives according to the current task instruction. Specifically, the module receives spatial voxel feature vectors from the visual alignment module and task instruction features from the language encoder. First, it uses a linear projection layer to align the data of the two modalities to a unified hidden dimension (e.g., 128-dimensional). For the projected voxel feature sequence, a one-dimensional average pooling operation is used for downsampling. The pooled voxel feature sequence and the language instruction feature sequence are then concatenated in parallel in time to form a unified fusion feature sequence that includes global task intent and local spatial geometry. Finally, the concatenated fusion feature sequence is processed by layer normalization.

[0083] In this embodiment, a pre-trained GPT-2 Transformer model is used as the core predictor. This model is configured with a 128-dimensional hidden layer and four attention heads. During training, the parameters of the GPT-2 model are frozen to preserve its sequence modeling capabilities learned on large-scale data. After the fused features are input into the Transformer model, its output is mapped to the primitive category space through a linear layer. The primitive category space is the action semantic dimension space of the model's output layer, and its dimension setting represents the total number of action categories covered by the system. The pre-built skill primitive library stores action semantic embedding vectors that correspond one-to-one with the above categories. These vectors are encoded by the language encoder (in this embodiment, a pre-trained CLIP model) for a specific task description, essentially constituting the semantic core representation of different action types. This layer contains the model's trainable parameters. The probability distribution is calculated using the Softmax function, and this probability distribution serves as the combination weight. It is used to perform a weighted linear combination of vectors from a pre-built skill primitive library, thereby generating specific skill representation vectors for the left and right arms at the current moment. This skill primitive library... The different task language instructions in the single-arm policy network are processed by the language encoder and represent different action categories, such as push, pull, lift, etc.

[0084] To accurately capture task execution details and environmental deviations, this embodiment constructs a dual-head parallel prediction architecture. The model is divided into two prediction branches in parallel at the Transformer output layer: the first branch calculates the probability distribution through a linear layer and the Softmax function, which serves as the weighted combination coefficient of the basic primitives; the second branch directly maps and outputs a task-oriented compensation with the same semantic dimension through an independent linear regression head.

[0085] In this embodiment, the text encoder is a pre-trained CLIP model, and its parameters are frozen during training.

[0086] In this embodiment, the specific calculation process is represented as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] In the formula, This indicates the complete observational features of the left arm after splicing. Indicates the characteristics of the task instruction. Represents a weighted predictor. Represents training parameters, Indicates the weight of the left arm. This indicates task-oriented compensation because, although the actions are similar, there are still subtle differences. Primitives representing different types of actions, This represents the language features resulting from the left arm combination. Similarly, "right" represents the relevant features of the right arm.

[0091] The calculation process of the loss function is as follows:

[0092] ;

[0093] In the formula, Describing the L1 norm, Represents the L21 norm. It is an adjustable balancing parameter. By minimizing this loss function, the weight predictor can select as few primitives as possible to reconstruct language features.

[0094] S23: Generate single-arm movement priors: such as Figure 3 As shown, after obtaining the specific visual features for the left and right arms through step S21 and the language features for the left and right arms through step S22, the pre-trained single-arm policy network is used to generate the action priors for the left and right arms respectively.

[0095] This embodiment selects the PerAct (Perceiver-Actor) network as the basic architecture of the single-arm policy network. PerAct is a multi-task manipulation network capable of processing high-dimensional 3D observation data. The specific implementation process of this step is as follows:

[0096] For the left arm, the left arm-specific visual features generated in step S21 and the left arm skill features generated in step S22 are used as input pairs. For the right arm, the same input pairs of right arm-specific visual features and right arm skill features are constructed. At the same time, the current proprioceptive state of the robotic arm is input, and then a set of learnable latent query vectors are introduced as feature detectors. The visual features and skill combination features are semantically associated and spatially aligned through a cross-attention layer. The mapped features are projected onto a discretized action bucket space. The pose of the end effector and the gripper state are predicted through probability distribution. Finally, a discretized action prediction is output, including the translation, rotation, and gripper opening and closing states of the end effector. In this embodiment, the deterministic action sequence output by PerAct is directly used as the single-arm action prior.

[0097] S3: Construct a dual-arm diffusion generation model to establish a mapping relationship from the initial motion distribution to the actual dual-arm cooperative motion distribution, specifically including:

[0098] S31: Multimodal feature embedding and tokenization: such as Figure 2As shown, in order to input data of different modalities into the Transformer network, a set of encoders maps the raw data into feature vectors, including:

[0099] State encoding: Input the current proprioception state of the robotic arm into the state MLP to obtain the proprioception token;

[0100] Prior trajectory encoding: The single-arm action priors for the left and right arms generated in step S23 are input into the noisy trajectory MLP to obtain action prior tokens. To enhance the model's robustness to prior errors, during the training phase, random noise injection and random dropping strategies are applied to the input prior trajectories, forcing the model not to rely entirely on the priors, but to learn how to correct the priors.

[0101] 3D Scene Visual Feature Encoding: A pre-trained visual encoder (CLIP) is used to independently encode the RGB images of the dual-arm robot from all perspectives, extracting semantically rich 2D feature maps. Combined with the corresponding depth maps and the intrinsic and extrinsic parameter matrices of each camera, each pixel on the 2D feature map is back-projected into the robot's 3D world coordinate system. At this point, each point contains both spatial coordinates (x, y, z) and a high-dimensional visual semantic vector, thus fusing the multi-view 2D feature maps into a unified 3D scene Visual Tokens O.

[0102] Language encoding: Input natural language instructions into a pre-trained text encoder (CLIP) to extract language instruction tokens.

[0103] S32: Construct a 3D flow matching Transformer model: such as Figure 2 As shown, this network is the core of action generation, and it contains multiple layers of attention modules, including:

[0104] Cross-attention module: The noise trajectory token is used as the query, and the 3D scene visual tokens O and language instruction features are used as the key and value respectively. The model can actively query scene and instruction features.

[0105] Self-attention module: used to handle the interaction between the noisy trajectory token and other internal features, capturing the geometric relationship between actions and the environment;

[0106] Feedforward neural network: performs further nonlinear transformation on the features to obtain the final arm movements;

[0107] This embodiment employs a trajectory correction strategy based on flow matching. The model learns a residual correction field, applying the single-arm prior to the distribution of cooperative movements of both arms. The boundary conditions for flow matching are defined as follows: initial distribution... Defined as prior to single-arm movement The noisy distribution centered on the center, i.e. This means the initial state is a roughly correct trajectory, but it contains noise and errors. Target distribution Defined as the distribution of coordinated arm movements demonstrated by a real expert. The linear interpolation process for flow matching is defined as:

[0108] ;

[0109] in, For time steps;

[0110] The loss function is:

[0111] ;

[0112] S4: Perform the first stage of training: fine-tune the single-arm motion prediction model for a specific task, using a small amount of dual-arm robot training dataset to train the single-arm motion prediction model. In this way, the model learns the implicit single-arm movement habits and cooperative temporal features in the dual-arm demonstration data without changing the data structure. After training, the model parameters are frozen.

[0113] This embodiment utilizes existing dual-arm robot training datasets to jointly train the composite model constructed in step S2, which includes a visual alignment module, a skill primitive selection module, and a single-arm policy network (PerAct). By introducing specific loss function constraints, the left and right arm observation features and combined linguistic features obtained from the visual alignment module and the skill primitive selection module are input into the single-arm model for action prediction. This training enables the model to decouple the implicit single-arm movement habits and dual-arm collaborative temporal features from the dual-arm demonstration data without changing the single-arm network architecture. After training, the model parameters are frozen and used as a reference generator for the second-stage dual-arm diffusion model. The hybrid loss function for this part is:

[0114] ;

[0115] ;

[0116] Where BCE represents the cross-entropy loss of the behavioral clone, This represents the difference between the actual actions of the expert and the predicted actions. and It is an adjustable parameter. and These are the loss functions for the visual alignment module and the action selection module, respectively.

[0117] S5: Perform the second stage of training, using a dual-arm diffusion generation model based on single-arm prior conditions. Train the dual-arm diffusion generation model using the same dual-arm robot training dataset. During training, the actions output by the frozen single-arm motion prediction model are introduced into the diffusion model as prior conditions. The model learns how to use this single-arm prior as a reference to construct an action distribution that is treated as a noisy action distribution, and then denoise it to obtain dual-arm actions. During training, a random mask or noise interference is applied to the input prior conditions.

[0118] The specific implementation process is as follows: For each training batch, environmental observation data and language commands are first input into the single-arm motion prediction model with frozen parameters. Based on the visual alignment in step S21 and the skill selection in step S22, the model outputs the predicted motion trajectory for the left arm. And the predicted motion trajectory for the right arm These two trajectories represent action suggestions that the single-arm expert considers reasonable, but they may lack precise spatiotemporal coordination between the two arms or contain local conflicts. To prevent the two-arm diffusion model from over-relying on the single-arm prior, this embodiment applies two key perturbations to the prior before inputting it into the main network:

[0119] Random noise interference: Random noise sampled from a Gaussian distribution is superimposed on the single-arm prior trajectory to obtain a noisy prior. The right arm is similar. This forces the dual-arm model to learn to identify and correct errors in the prior, transforming the training task from replicating the prior to denoising and refining based on the prior.

[0120] Prior random mask: Set a probability threshold ,like Figure 2 As shown, during the training process, with probability The input prior trajectory token is randomly replaced with an all-zero vector. This mechanism simulates the situation where the single-arm prior fails or is missing, ensuring that the dual-arm model can still independently generate reasonable actions based on its own 3D scene understanding ability even without prior guidance, thus guaranteeing the robustness of the system.

[0121] During the flow matching training process, this embodiment uses the conditional flow matching algorithm to train the two-arm policy network, with an initial distribution... To use a single-arm prior The noisy distribution centered on the center, i.e. .

[0122] According to the formula Calculate intermediate states ,Will Representing the actual action at time t, Time step t, 3D scene Visual Tokens O, original prior conditions The language instructions are input into the Transformer network for training.

[0123] The loss function is:

[0124] ;

[0125] In the formula, This represents the initial noise action selected from the distribution. This represents the actual movement; 'i' represents the denoised step. The same applies to the right arm. This represents the difference between the actual gripper switch state and the predicted gripper switch state, reflecting the actual actions of the expert.

[0126] S6, Collaborative Reasoning and Execution: In the reasoning phase, given the language instruction and the current observed image, the single-arm action prediction model is first input to obtain the action prediction priors for the left and right arms. A Gaussian distribution with the mean of the single-arm prior and the radius of the preset variance is constructed. The initial action vector is sampled from this initial distribution and input into the dual-arm diffusion generation model. Under the joint guidance of the observed image and the single-arm prior, the model gradually denoises the initial action vector to generate dual-arm actions.

[0127] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for controlling a dual-arm robot based on prior guidance of a single-arm trajectory, characterized in that, Includes the following steps: Obtain the dataset for the dual-arm robot; Based on the single-arm motion prediction model, skill primitives of different motion types are obtained. According to the instructions of both arms, skill primitives of different motion types are selected for the left and right arms, and the next motion intention of the left and right arms is predicted respectively. A dual-arm diffusion generation model is constructed to establish a mapping relationship from the initial motion distribution to the actual dual-arm coordinated motion distribution; Training a single-arm motion prediction model based on a dual-arm robot training dataset; The dual-arm diffusion generation model was trained based on the dual-arm robot training dataset; Prior predictions for the left and right arms are obtained based on the trained single-arm motion prediction model. A Gaussian distribution with the single-arm action prediction prior as the mean and the preset variance as the radius is constructed. An initial action vector is sampled from the Gaussian distribution and input into the trained dual-arm diffusion generation model. The initial action vector is then denoised to generate dual-arm actions.

2. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 1, characterized in that, The dual-arm robot dataset includes serialized multimodal information. and natural language commands, among which, This represents environmental observation data, including images from multiple perspectives. This indicates a combined movement of both arms, including the end positions of the left and right arms.

3. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 1, characterized in that, Based on the instructions given to both arms, different action types of skill primitives are selected for the left and right arms, and the next action intention of the left and right arms is predicted respectively, specifically including: A visual alignment module is constructed. The input of the visual alignment module is the three-dimensional spatial voxel features obtained by image encoding and projection of the current observation image into three-dimensional space. The output is the observation features of the left arm and the observation features of the right arm. Construct an action selection module that receives spatial voxel feature vectors from the visual alignment module and task instruction features from the language encoder, and outputs left arm language features and right arm language features. Action priors for the left and right arms are generated based on a single-arm policy network.

4. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 3, characterized in that, In the visual alignment module, a mask generator generates a left arm mask and a right arm mask. The left arm mask and the right arm mask are then element-wise multiplied with the original input voxel feature vector. Finally, the masked features are added back to the original input voxel feature vector to obtain the left arm observation features and the right arm observation features, specifically as follows: ； ； ； ； ； in, This represents the currently observed image as input. Indicates a mask generator. ViT represents the training parameters, and ViT represents the image encoder. This represents the dot product operation. This indicates a splicing operation. Indicates the left arm mask. This indicates the features observed in the left arm after masking. This indicates the observed features of the spliced ​​left arm. Indicates the right arm mask. This indicates the features observed in the left arm after masking. This indicates the observational features of the right arm after splicing. The calculation process of the loss function is as follows: ； in, This represents the KL divergence.

5. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 3, characterized in that, In the action selection module, the voxel feature sequence is obtained through linear projection and pooling. The voxel feature sequence is then concatenated with the task instruction feature sequence in parallel in time to obtain the fused feature sequence. The fused feature sequence is input into the Transformer model, and the output of the Transformer model is mapped to the primitive category space through a linear layer. The dimension of the primitive category space corresponds to the total number of action categories. A skill primitive library is constructed to store action semantic embedding vectors corresponding to action categories. The vectors in the pre-constructed skill primitive library are then weighted and linearly combined to obtain the linguistic features of the left and right arms, specifically represented as follows: ； ； ； in, Indicates the observational features of the left arm. Indicates the characteristics of task instructions. Represents a weighted predictor. Represents training parameters, Indicates the weight of the left arm. This indicates task-oriented compensation for the left arm. Primitives representing different types of actions, This represents the combined language features of the left arm. Indicates the observational features of the right arm. Indicates the weight of the right arm. This indicates task-oriented compensation for the right arm. This indicates the combined linguistic features of the right arm; The calculation process of the loss function is as follows: ； In the formula, Describing the L1 norm, Represents the L21 norm. It is an adjustable balance parameter.

6. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 3, characterized in that, Based on a single-arm policy network, action priors for the left and right arms are generated separately, specifically including: The left arm observation features and left arm language features are used as input pairs, the right arm observation features and right arm language features are used as input pairs, and the current robot arm body perception state is also input. Semantic association and spatial alignment are performed through a cross-attention layer. The pose and gripper state of the end effector are predicted by probability distribution, and the discretized motion prediction is output to obtain the motion priors of the left and right arms.

7. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 1, characterized in that, A dual-arm diffusion generation model is constructed to establish a mapping relationship from the initial motion distribution to the actual dual-arm coordinated motion distribution, specifically including: Input the current proprioception state of the robotic arm into the state MLP to obtain the proprioception Token; Input the single-arm action priors of the left and right arms into the noisy trajectory MLP to obtain the action prior token; The images from all perspectives of the dual-arm robot are independently encoded using a pre-trained visual encoder, 2D feature maps are extracted, and each pixel on the 2D feature map is back-projected into the robot's 3D world coordinate system. Input natural language instructions into a pre-trained text encoder to extract language instruction tokens; A 3D flow matching Transformer model is constructed. Based on the cross-attention module, the noise trajectory token is used as the query. The 3D scene visual tokens O and language instruction features are used as keys and values ​​respectively. The interaction between the noise trajectory token and other internal features is processed based on the self-attention module. Finally, the arm movements are obtained based on the feedforward neural network.

8. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 3, characterized in that, The single-arm motion prediction model is trained based on the dual-arm robot training dataset, and the loss function is expressed as: ； ； Where BCE represents the cross-entropy loss of the behavioral clone, This represents the difference between the actual actions of the expert and the predicted actions. and It is an adjustable parameter. and These are the loss functions for the visual alignment module and the action selection module, respectively.

9. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 1, characterized in that, The dual-arm diffusion generation model was trained based on a dual-arm robot training dataset, specifically including: Environmental observation data and language commands are input into the single-arm motion prediction model with frozen parameters, and the predicted motion trajectories of the left and right arms are output respectively. The predicted motion trajectories of the left and right arms are superimposed with random noise sampled from a Gaussian distribution; The predicted motion trajectories of the left and right arms are randomly replaced with all-zero vectors with a set probability.

10. The dual-arm robot control method based on single-arm trajectory prior guidance according to claim 1, characterized in that, A conditional flow matching algorithm was used to train a two-arm diffusion generation model.

Images