Motion control method and device based on vision and motion generation model

Abstract

The invention relates to the technical field of robot control, and discloses a motion control method and device based on vision and a motion generation model, and the method comprises the steps: collecting image information of a to-be-operated object corresponding to a two-arm robot, carrying out the scene perception analysis of the image information, obtaining a multi-modal scene representation vector, and obtaining a multi-modal scene representation vector; physical parameters of the double-arm robot are obtained, the basic motion generation model is dynamically modulated according to the physical parameters, a motion generation model is obtained, a task instruction and a double-arm state are obtained, the multi-modal scene representation vector, the task instruction and the double-arm state are input into the motion generation model, and a double-arm joint angle control sequence is generated; and controlling the double-arm robot to execute an operation task according to the double-arm joint angle control sequence. Therefore, by implementing the method, the three-dimensional scene can be accurately perceived, cross-platform multiplexing of the basic model is realized, the adaptation cost and period of the model are reduced, end-to-end mapping from visual perception to motion control is realized, and the automation and intelligence level of motion control is improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a motion control method and apparatus based on vision and motion generation models. Background Technology

[0002] Humanoid robot bi-arm motion generation technology is a core research area in the fields of artificial intelligence and robot imitation learning. Its goal is to enable robots to autonomously generate precise, coordinated, and safe bi-arm movements by observing human demonstrations, enabling them to complete complex tasks such as grasping and placing. Current technical solutions face several bottlenecks in practical applications: First, at the perception level, pure visual imitation learning relies heavily on monocular vision, lacking depth information support and failing to accurately construct 3D scenes. Furthermore, the lack of joint modeling of the coupling constraints between bi-arm coordination and whole-body balance leads to ambiguity in the robot's understanding of the operational space, affecting the execution accuracy and obstacle avoidance performance of the bi-arm terminals. Second, at the motion generation level, traditional motion planning methods rely on precise models and environmental parameters, making it difficult to generate natural, coordinated human-like movements. While the mainstream Transformer model can extract spatiotemporal features, it lacks sufficient characterization of the dual-domain coupling relationship between joint topology and temporal motion, easily resulting in long-sequence coordinated drift and failing to effectively handle the coupling relationship and self-collision problems between the bi-arms.

[0003] To address the aforementioned issues, existing technologies have employed various improvement methods. For instance, some solutions enhance visual perception accuracy by wearing markers on the limbs, but the additional sensing devices increase system complexity and operating costs. Other solutions attempt to combine Transformers with traditional graph convolutional networks to optimize feature extraction, but lack task-adaptive network structures, making it impossible to dynamically model the collaborative relationship between the two arms. Therefore, proposing a technical solution capable of accurately perceiving 3D scenes and improving the naturalness, coordination, and accuracy of robot arm movements is of paramount importance. Summary of the Invention This invention provides a motion control method and device based on vision and motion generation models, which can accurately perceive three-dimensional scenes and improve the naturalness, coordination and accuracy of robot dual-arm movements.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a motion control method based on a vision and motion generation model, the method comprising: Image information of the object to be operated corresponding to the dual-arm robot is collected, and scene perception analysis is performed on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated. The physical parameters of the dual-arm robot are obtained, and the preset basic motion generation model is dynamically modulated according to the physical parameters to obtain a motion generation model that matches the dual-arm robot. The physical parameters include at least one of the following: arm length parameter, mass distribution parameter, joint limit parameter, and joint transmission ratio parameter. Obtain the task instructions for the object to be operated and the state of the two arms of the dual-arm robot, and input the multimodal scene representation vector, the task instructions and the state of the two arms into the motion generation model to generate a control sequence for the joint angles of the two arms; The dual-arm robot is controlled to perform the operation task corresponding to the task instruction according to the dual-arm joint angle control sequence.

[0005] As an optional implementation, in a first aspect of the present invention, the motion generation model includes a feature fusion layer, a hybrid codec, and an output layer, wherein the hybrid codec includes an encoder, a motion primitive library, and a decoder. The step of inputting the multimodal scene representation vector, the task instructions, and the dual-arm states into the motion generation model to generate dual-arm joint angle control sequences includes: The feature fusion layer concatenates the features of the multimodal scene representation vector, the task instructions, and the dual-arm state input to obtain initial context features. The spatiotemporal features of the dual-arm robot's dual-arm collaboration are extracted from the initial context features by a multi-layer spatiotemporal convolutional network in the encoder. The spatiotemporal features include the spatial relationship between the joints of the dual-arm robot's two arms, the temporal evolution law, and semantic information related to the operation task corresponding to the task instruction. Based on the motion primitives stored in the motion primitive library and the spatiotemporal characteristics, generate motion instructions that match the operation task. The decoder generates an initial joint angle sequence based on the spatiotemporal features and the motion commands. The initial joint angle sequence includes joint angle control commands for multiple future time steps. The initial joint angle sequence is smoothed and filtered by the output layer to obtain the bi-arm joint angle control sequence.

[0006] As an optional implementation, in the first aspect of the present invention, the multilayer spatiotemporal convolutional network includes a visual network and a spatiotemporal graph convolutional network; The step of extracting the spatiotemporal features of the dual-arm robot's dual-arm collaboration from the initial context features through a multi-layer spatiotemporal convolutional network in the encoder includes: The global semantic information of the image in the initial context features is captured by the visual network to obtain global visual features. The historical joint trajectory information corresponding to the state of the two arms is input into the spatiotemporal graph convolutional network to obtain spatiotemporal joint features. The global visual features and the spatiotemporal joint features are then concatenated to obtain the initial fusion features. Cross-modal spatiotemporal attention calculation is performed on the initial fusion features to obtain a cross-modal attention weight set, and the initial fusion features are weighted and optimized according to the cross-modal attention weight set to obtain cross-modal fusion features. The cross-modal attention weight set includes the spatial attention weight matrix, the temporal attention weight matrix, and the correlation weight between the global visual features and the spatiotemporal joint features corresponding to the initial fusion features. Based on the task instructions, at least one functional partition is determined for the two arms of the dual-arm robot, and based on each functional partition and the cross-modal fusion feature, the local spatial relationship features of the joints within each functional partition are determined; Each of the local spatial relationship features is input into a preset multi-layer cascaded structure for multi-layer spatiotemporal convolution operations to obtain the spatiotemporal features of the dual-arm robot's dual-arm collaboration. The multi-layer cascaded structure is composed of multiple multi-configuration partition adaptive graph convolutional network layers and time one-dimensional convolutional layers cascaded alternately.

[0007] As an optional implementation, in the first aspect of the present invention, the method further includes: Construct a basic motion generation model network architecture and a training dataset, and input the training dataset into the basic motion generation model network architecture to obtain the model prediction motion of the basic motion generation model network architecture. The mean squared error loss between the model's predicted motion and the human demonstration motion in the training dataset is calculated based on the behavior cloning algorithm. The model's predicted motion and the human demonstration motion are judged based on the discriminator to obtain the judgment result. A joint optimization loss function is constructed based on the mean squared error loss and the judgment result. The basic motion generation model network architecture is pre-trained by imitation learning based on the joint optimization loss function to obtain the pre-trained model. In a preset simulation environment, the pre-trained model is used as the initial motion policy, a weighted reward function is constructed, and the initial motion policy is iteratively updated based on the proximal policy optimization algorithm and the weighted reward function to obtain a fine-tuned model. The fine-tuned model is validated using a preset validation dataset to obtain validation results. When the validation results indicate that the fine-tuned model meets preset model requirements, the fine-tuned model is determined to be a basic motion generation model. The model requirements include task execution accuracy requirements, motion coordination requirements, and physical constraint requirements.

[0008] As an optional implementation, in the first aspect of the present invention, the image information includes binocular synchronized RGB image information; The step of performing scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated on includes: The binocular synchronized RGB image information is segmented to obtain a 2D mask of the object corresponding to the dual-arm robot. Stereo matching is performed on the binocular synchronized RGB image information to obtain a dense depth map. The dense depth map is then filled with depth based on the 2D mask to obtain a three-dimensional scene point cloud. The object includes the two arms of the dual-arm robot, the object to be operated, and obstacles. Based on the 3D scene point cloud, the object information of the object is identified, the scene graph nodes are determined based on the object information, and an undirected scene graph is constructed based on the scene graph nodes and the spatial distance between each scene graph node. Based on the undirected scene graph, the task level and the cooperation relationship between the two arms are determined. RGB appearance features are extracted from the binocular synchronized RGB image information, and a first weight coefficient of the RGB appearance features and a second weight coefficient of the three-dimensional scene point cloud are calculated according to the task level and the dual-arm cooperation relationship. The RGB appearance features and the 3D scene point cloud are fused according to the first weight coefficient and the second weight coefficient to obtain a fused feature vector. The fused feature vector is then normalized to obtain a multimodal scene representation vector corresponding to the object to be operated. The multimodal scene representation vector includes the pose of the object to be operated, obstacle information, scene context, the task level, and the dual-arm collaboration relationship.

[0009] As an optional implementation, in the first aspect of the present invention, the step of dynamically modulating a preset basic motion generation model according to the physical parameters to obtain a motion generation model matching the dual-arm robot includes: The physical parameters are encoded to generate a condition vector; Determine the standard spatiotemporal features corresponding to the basic motion generation model, and generate the scaling factor and offset factor of the standard spatiotemporal features based on the condition vector; A layer normalization operation is performed on the standard spatiotemporal features to obtain the normalization operation result. Based on the normalization operation result, the scaling factor, and the offset factor, the standard spatiotemporal features are dynamically modulated to obtain modulated spatiotemporal features. The modulated spatiotemporal features are residually concatenated with the standard spatiotemporal features to obtain fused spatiotemporal features. Based on the model agnostic meta-learning framework, the basic motion generation model corresponding to the fused spatiotemporal features is fine-tuned by gradient update to obtain a motion generation model that matches the dual-arm robot.

[0010] As an optional implementation, in the first aspect of the present invention, the motion primitive library includes a plurality of basic motion primitives; The step of generating motion instructions matching the operation task based on the motion primitives stored in the motion primitive library and the spatiotemporal features includes: Based on the semantic information related to the operation task corresponding to the task instruction, the similarity between the spatiotemporal features and each basic motion primitive is calculated, and the basic motion primitives with similarity greater than a preset similarity threshold are selected as control motion primitives to obtain a set of control motion primitives. Based on the spatial relationship between the two arm joints and the temporal evolution law, the temporal constraints corresponding to the control motion primitive set are constructed, and the control motion primitive set is temporally sorted according to the temporal constraints to obtain the motion primitive sequence. The acceleration fluctuation value of the two arm joint motion corresponding to the motion primitive sequence is minimized. The motion primitive sequence is mapped to joint angle changes, and motion commands matching the operation task are generated based on the joint angle changes and the constraint parameters corresponding to the dual-arm robot. The constraint parameters include joint limit parameters and motion speed parameters, and the motion commands include dual-arm collaborative timing logic, joint motion amplitude, and action execution duration.

[0011] A second aspect of the present invention discloses a motion control device based on a vision and motion generation model, the device comprising: The acquisition module is used to acquire image information of the object to be operated by the dual-arm robot, and to perform scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated. An acquisition module is used to acquire the physical parameters of the dual-arm robot; The modulation module is used to dynamically modulate the preset basic motion generation model according to the physical parameters to obtain a motion generation model that matches the dual-arm robot. The physical parameters include at least one of arm length parameters, mass distribution parameters, joint limit parameters, and joint transmission ratio parameters. The acquisition module is also used to acquire the task instructions for the object to be operated and the state of the two arms of the dual-arm robot. The generation module is used to input the multimodal scene representation vector, the task instructions, and the state of the two arms into the motion generation model to generate a control sequence for the joint angles of the two arms; The control module is used to control the dual-arm robot to execute the operation task corresponding to the task instruction according to the dual-arm joint angle control sequence.

[0012] As an optional implementation, in a second aspect of the invention, the motion generation model includes a feature fusion layer, a hybrid codec, and an output layer, wherein the hybrid codec includes an encoder, a motion primitive library, and a decoder. The generation module inputs the multimodal scene representation vector, the task instruction, and the dual-arm state into the motion generation model, and the specific method for generating the dual-arm joint angle control sequence includes: The feature fusion layer concatenates the features of the multimodal scene representation vector, the task instructions, and the dual-arm state input to obtain initial context features. The spatiotemporal features of the dual-arm robot's dual-arm collaboration are extracted from the initial context features by a multi-layer spatiotemporal convolutional network in the encoder. The spatiotemporal features include the spatial relationship between the joints of the dual-arm robot's two arms, the temporal evolution law, and semantic information related to the operation task corresponding to the task instruction. Based on the motion primitives stored in the motion primitive library and the spatiotemporal characteristics, generate motion instructions that match the operation task. The decoder generates an initial joint angle sequence based on the spatiotemporal features and the motion commands. The initial joint angle sequence includes joint angle control commands for multiple future time steps. The initial joint angle sequence is smoothed and filtered by the output layer to obtain the bi-arm joint angle control sequence.

[0013] As an optional implementation, in a second aspect of the present invention, the multilayer spatiotemporal convolutional network includes a visual network and a spatiotemporal graph convolutional network; The generation module extracts the spatiotemporal features of the dual-arm robot's dual-arm collaboration from the initial context features through a multi-layer spatiotemporal convolutional network in the encoder, specifically including the following methods: The global semantic information of the image in the initial context features is captured by the visual network to obtain global visual features. The historical joint trajectory information corresponding to the state of the two arms is input into the spatiotemporal graph convolutional network to obtain spatiotemporal joint features. The global visual features and the spatiotemporal joint features are then concatenated to obtain the initial fusion features. Cross-modal spatiotemporal attention calculation is performed on the initial fusion features to obtain a cross-modal attention weight set, and the initial fusion features are weighted and optimized according to the cross-modal attention weight set to obtain cross-modal fusion features. The cross-modal attention weight set includes the spatial attention weight matrix, the temporal attention weight matrix, and the correlation weight between the global visual features and the spatiotemporal joint features corresponding to the initial fusion features. Based on the task instructions, at least one functional partition is determined for the two arms of the dual-arm robot, and based on each functional partition and the cross-modal fusion feature, the local spatial relationship features of the joints within each functional partition are determined; Each of the local spatial relationship features is input into a preset multi-layer cascaded structure for multi-layer spatiotemporal convolution operations to obtain the spatiotemporal features of the dual-arm robot's dual-arm collaboration. The multi-layer cascaded structure is composed of multiple multi-configuration partition adaptive graph convolutional network layers and time one-dimensional convolutional layers cascaded alternately.

[0014] As an optional implementation, in a second aspect of the invention, the apparatus further includes: A construction module is used to construct a basic motion generation model network architecture and a training dataset, and to obtain the model prediction motion of the basic motion generation model network architecture based on the training dataset input. The computation module is used to calculate the mean squared error loss between the model's predicted motion and the human demonstration motion in the training dataset based on the behavior cloning algorithm, judge the model's predicted motion and the human demonstration motion based on the discriminator, obtain the judgment result, construct a joint optimization loss function based on the mean squared error loss and the judgment result, and perform imitation learning pre-training on the basic motion generation model network architecture based on the joint optimization loss function to obtain a pre-trained model. The update module is used to construct a weighted reward function by using the pre-trained model as the initial motion policy in a preset simulation environment, and to iteratively update the initial motion policy based on the proximal policy optimization algorithm and the weighted reward function to obtain a fine-tuned model. The verification module is used to verify the fine-tuned model using a preset verification dataset and obtain verification results. When the verification results indicate that the fine-tuned model meets the preset model requirements, the fine-tuned model is determined to be a basic motion generation model. The model requirements include task execution accuracy requirements, motion coordination requirements, and physical constraint requirements.

[0015] As an optional implementation, in a second aspect of the present invention, the image information includes binocular synchronized RGB image information; The acquisition module performs scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated on. Specifically, this includes the following methods: The binocular synchronized RGB image information is segmented to obtain a 2D mask of the object corresponding to the dual-arm robot. Stereo matching is performed on the binocular synchronized RGB image information to obtain a dense depth map. The dense depth map is then filled with depth based on the 2D mask to obtain a three-dimensional scene point cloud. The object includes the two arms of the dual-arm robot, the object to be operated, and obstacles. Based on the 3D scene point cloud, the object information of the object is identified, the scene graph nodes are determined based on the object information, and an undirected scene graph is constructed based on the scene graph nodes and the spatial distance between each scene graph node. Based on the undirected scene graph, the task level and the cooperation relationship between the two arms are determined. RGB appearance features are extracted from the binocular synchronized RGB image information, and a first weight coefficient of the RGB appearance features and a second weight coefficient of the three-dimensional scene point cloud are calculated according to the task level and the dual-arm cooperation relationship. The RGB appearance features and the 3D scene point cloud are fused according to the first weight coefficient and the second weight coefficient to obtain a fused feature vector. The fused feature vector is then normalized to obtain a multimodal scene representation vector corresponding to the object to be operated. The multimodal scene representation vector includes the pose of the object to be operated, obstacle information, scene context, the task level, and the dual-arm collaboration relationship.

[0016] As an optional implementation, in a second aspect of the present invention, the modulation module dynamically modulates a preset basic motion generation model according to the physical parameters to obtain a motion generation model matching the dual-arm robot, specifically including: The physical parameters are encoded to generate a condition vector; Determine the standard spatiotemporal features corresponding to the basic motion generation model, and generate the scaling factor and offset factor of the standard spatiotemporal features based on the condition vector; A layer normalization operation is performed on the standard spatiotemporal features to obtain the normalization operation result. Based on the normalization operation result, the scaling factor, and the offset factor, the standard spatiotemporal features are dynamically modulated to obtain modulated spatiotemporal features. The modulated spatiotemporal features are residually concatenated with the standard spatiotemporal features to obtain fused spatiotemporal features. Based on the model agnostic meta-learning framework, the basic motion generation model corresponding to the fused spatiotemporal features is fine-tuned by gradient update to obtain a motion generation model that matches the dual-arm robot.

[0017] As an optional implementation, in a second aspect of the present invention, the motion primitive library includes a plurality of basic motion primitives; The generation module generates motion instructions that match the operation task based on the motion primitives stored in the motion primitive library and the spatiotemporal features, specifically including the following methods: Based on the semantic information related to the operation task corresponding to the task instruction, the similarity between the spatiotemporal features and each basic motion primitive is calculated, and the basic motion primitives with similarity greater than a preset similarity threshold are selected as control motion primitives to obtain a set of control motion primitives. Based on the spatial relationship between the two arm joints and the temporal evolution law, the temporal constraints corresponding to the control motion primitive set are constructed, and the control motion primitive set is temporally sorted according to the temporal constraints to obtain the motion primitive sequence. The acceleration fluctuation value of the two arm joint motion corresponding to the motion primitive sequence is minimized. The motion primitive sequence is mapped to joint angle changes, and motion commands matching the operation task are generated based on the joint angle changes and the constraint parameters corresponding to the dual-arm robot. The constraint parameters include joint limit parameters and motion speed parameters, and the motion commands include dual-arm collaborative timing logic, joint motion amplitude, and action execution duration.

[0018] A third aspect of the present invention discloses another motion control device based on a vision and motion generation model, the device comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the motion control method based on vision and motion generation model according to any of the first aspects of the present invention.

[0019] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the motion control method based on vision and motion generation model as described in any of the first aspects of the present invention.

[0020] Compared with the prior art, the present invention has the following beneficial effects: In this embodiment of the invention, image information of the object to be operated on corresponding to the dual-arm robot is acquired, and scene perception analysis is performed on the image information to obtain a multimodal scene representation vector corresponding to the object to be operated. The physical parameters of the dual-arm robot are obtained, and a preset basic motion generation model is dynamically modulated based on the physical parameters to obtain a motion generation model matching the dual-arm robot. The task instructions for the object to be operated and the states of the dual arms of the robot are obtained. The multimodal scene representation vector, task instructions, and dual-arm states are input into the motion generation model to generate a dual-arm joint angle control sequence. The dual-arm robot is controlled to execute the operation task corresponding to the task instructions based on the dual-arm joint angle control sequence. It is evident that implementing this invention can accurately perceive the 3D scene, providing comprehensive and reliable visual input for motion generation. Dynamic modulation of the basic motion generation model based on the robot's physical parameters enables cross-platform reuse of the basic model, significantly reducing the adaptation cost and cycle of the model. It achieves end-to-end mapping from visual perception to motion control, improving the automation and intelligence level of motion control, enhancing the naturalness, coordination, and accuracy of the robot's dual-arm movements, and increasing the success rate and stability of the robot's operation tasks. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a motion control method based on a vision and motion generation model disclosed in an embodiment of the present invention; Figure 2 This is a flowchart illustrating another motion control method based on a vision and motion generation model disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a motion control device based on a vision and motion generation model disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of another motion control device based on a vision and motion generation model disclosed in an embodiment of the present invention; Figure 5 This is a schematic diagram of another motion control device based on vision and motion generation model disclosed in an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] This invention discloses a motion control method and device based on vision and motion generation models. It can accurately perceive 3D scenes, providing comprehensive and reliable visual input for motion generation. Based on robot physical parameters, it dynamically modulates the basic motion generation model, enabling cross-platform reuse of the basic model. This significantly reduces the adaptation cost and cycle time, achieving end-to-end mapping from visual perception to motion control, improving the automation and intelligence level of motion control, enhancing the naturalness, coordination, and accuracy of robot dual-arm movements, and increasing the success rate and stability of robot task execution. Detailed descriptions follow.

[0027] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a motion control method based on a vision and motion generation model disclosed in an embodiment of the present invention. Figure 1The described motion control method based on vision and motion generation models can be applied to a dual-arm robot. This dual-arm robot may include a motion control device based on vision and motion generation models. This motion control device may include an intelligent server or intelligent platform for generating joint angle control sequences for the two arms of the dual-arm robot. The intelligent server may be a local server or a cloud server; this embodiment of the invention does not limit the scope. Figure 1 As shown, the motion control method based on vision and motion generation models can include the following operations: 101. Collect image information of the object to be operated corresponding to the dual-arm robot, and perform scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated.

[0028] In this embodiment of the invention, optionally, the head of the dual-arm robot can be equipped with a binocular vision acquisition device, which can acquire image information of the object to be operated. This image information can include binocular synchronous RGB image information, specifically including visual data of the object to be operated, the robot's own two arms, and surrounding obstacles. Scene perception analysis is performed on the image information. Specifically, a three-dimensional scene point cloud can be constructed through instance segmentation, stereo matching, and depth completion. Then, the spatial and temporal relationships of each object in the scene can be mined through scene graph construction and relation reasoning. Finally, a normalized multimodal scene representation vector is obtained through dynamic feature fusion. This vector eliminates redundant background information and retains only effective features related to the operation task. The multimodal scene representation vector is a high-dimensional feature vector that integrates RGB appearance features, three-dimensional scene geometric structure features, and task semantic features. It contains core information such as the pose of the object to be operated, obstacle information, scene context, task level, and the collaborative relationship between the two arms. This invention does not limit this.

[0029] 102. Obtain the physical parameters of the dual-arm robot, and dynamically modulate the preset basic motion generation model according to the physical parameters to obtain a motion generation model that matches the dual-arm robot.

[0030] In this embodiment of the invention, optionally, the physical parameters of the robot can be parsed from the URDF file of the dual-arm robot. The physical parameters may include at least one of the following: arm length parameters, mass distribution parameters, joint limit parameters, and joint transmission ratio parameters. The basic motion generation model is a general-purpose vision-motion mapping model trained based on human demonstration data. It can realize the mapping from multimodal scene representation vectors to dual-arm joint angle sequences. It is not bound to a specific robot platform and has the foundation for cross-platform adaptation. The preset basic motion generation model can be dynamically modulated according to the physical parameters to obtain a motion generation model that matches the dual-arm robot. The motion generation model is a dedicated model that completely matches the physical parameters of the target dual-arm robot and can generate a coordinated motion sequence that conforms to the joint limit and torque constraints of the robot.

[0031] 103. Obtain the task instructions for the object to be operated and the state of the two arms of the dual-arm robot. Input the multimodal scene representation vector, task instructions and the state of the two arms into the motion generation model to generate the joint angle control sequence of the two arms.

[0032] In this embodiment of the invention, optionally, the task instruction represents the type of operation task instruction for the object to be operated, such as "grab", "place", "carry", etc., and uses one-hot encoding as the input of the motion generation model. The dual-arm robot's dual-arm state represents the current joint motion state of the dual-arm robot, including the real-time joint angle, joint motion speed, joint acceleration, and pose information of the end effector of each joint. The end effector may include a simulated manipulator, a gripper, a clamp, etc. The multimodal scene representation vector, task instruction, and dual-arm state are concatenated and input into the adapted motion generation model. The model goes through a process of feature fusion, spatiotemporal feature extraction, motion primitive selection and combination, joint angle sequence generation, smoothing filtering, etc., and outputs a continuous and smooth dual-arm joint angle control sequence. The time step of the sequence can be adjusted according to the robot's control frequency.

[0033] 104. Control the dual-arm robot to execute the operation task corresponding to the task instruction according to the dual-arm joint angle control sequence.

[0034] In this embodiment of the invention, optionally, the joint angle control sequence of the two arms output by the motion generation model can be sent to the underlying motion controller of the dual-arm robot. The controller drives the servo motors of each joint to complete the corresponding rotation according to the joint angle control sequence, so as to realize the coordinated movement of the robot's two arms and thus perform corresponding operation tasks such as grasping, placing, and carrying. At the same time, the robot's binocular vision device collects scene feedback information in real time. If a task execution deviation is detected, an online replanning mechanism will be triggered to regenerate the joint angle control sequence to correct the motion trajectory and ensure that the task is completed smoothly.

[0035] It is evident that implementation Figure 1The described motion control method based on vision and motion generation models can acquire image information of the object to be manipulated by a dual-arm robot, perform scene perception analysis on the image information to obtain a multimodal scene representation vector corresponding to the object to be manipulated, obtain the physical parameters of the dual-arm robot, and dynamically modulate a preset basic motion generation model according to the physical parameters to obtain a motion generation model matching the dual-arm robot. It can also acquire task instructions for the object to be manipulated and the states of the two arms of the dual-arm robot, input the multimodal scene representation vector, task instructions, and arm states into the motion generation model to generate a joint angle control sequence for the two arms. Based on the joint angle control sequence, the method controls the dual-arm robot to execute the operation task corresponding to the task instruction. This method can accurately perceive the 3D scene, providing comprehensive and reliable visual input for motion generation. Dynamic modulation of the basic motion generation model based on the robot's physical parameters enables cross-platform reuse of the basic model, significantly reducing the adaptation cost and cycle time. It achieves end-to-end mapping from visual perception to motion control, improving the automation and intelligence level of motion control, enhancing the naturalness, coordination, and accuracy of the robot's dual-arm movements, and increasing the success rate and stability of the robot's operation tasks.

[0036] In an optional embodiment, the motion control method based on vision and motion generation models may further include the following operations: Construct a basic motion generation model network architecture and a training dataset, and input the training dataset into the basic motion generation model network architecture to obtain the model's predicted motion. The mean squared error loss of the model's predicted motion and the human demonstration motion in the training dataset is calculated based on the behavior cloning algorithm. The discriminator judges the model's predicted motion and the human demonstration motion to obtain the judgment result. A joint optimization loss function is constructed based on the mean squared error loss and the discrimination result. The basic motion generation model network architecture is pre-trained by imitation learning based on the joint optimization loss function to obtain the pre-trained model. In a pre-set simulation environment, the pre-trained model is used as the initial motion policy, a weighted reward function is constructed, and the initial motion policy is iteratively updated based on the proximal policy optimization algorithm and the weighted reward function to obtain a fine-tuned model. The fine-tuned model is validated using a pre-set validation dataset to obtain validation results. When the validation results indicate that the fine-tuned model meets the pre-set model requirements, the fine-tuned model is determined to be the basic motion generation model. The model requirements include task execution accuracy requirements, motion coordination requirements, and physical constraint requirements.

[0037] In this optional embodiment, the basic motion generation model network architecture may include a feature fusion layer, a hybrid encoder-decoder, and an output layer. The hybrid encoder-decoder includes an encoder, a motion primitive library, and a decoder. The training dataset contains multiple demonstration data of human bi-arm operations, such as 100,000 data points. The data covers typical operation tasks such as grasping, placing, carrying, and twisting. Each data point includes a binocular RGB image, operation task instructions, human bi-arm joint trajectories, and task execution results. All data has been labeled and standardized. The multimodal scene representation vectors, task instructions, and bi-arm states in the training dataset can be input into the untrained network architecture to obtain the model's predicted motion randomly output by the network.

[0038] In this optional embodiment, the behavior cloning algorithm is a supervised imitation learning algorithm. By minimizing the mean squared error between the model's predicted motion and the human-demonstrated motion, the model directly learns human motion strategies, resulting in fast convergence. The human-demonstrated motion represents the sequence of arm joint trajectories labeled in the training dataset when a human performs an operation task. The mean squared error loss between the model's predicted motion and the human-demonstrated motion in the training dataset can be calculated based on the behavior cloning algorithm. Specifically, the formula for calculating the mean squared error loss is as follows:

[0039] in, This represents the mean squared error loss, where N represents the number of training samples. This indicates that the model predicts motion. This represents a human demonstration of movement; optionally, the basic motion generation model network architecture can be pre-trained based on behavioral cloning, with training parameters set as follows: learning rate of 3e-4, batch size of 64, number of training epochs of 200, optimizer of Adam, and data augmentation using depth jitter and affine noise, minimized through gradient descent. The initial pre-training of the network is completed, but this embodiment does not impose any limitations.

[0040] In this optional embodiment, the discriminator is optionally a binary classification network built based on a multilayer perceptron (MLP). The input is a motion sequence, and the output is the probability that the sequence belongs to a human-demonstrated motion. This is used to distinguish between model-predicted motion and human-demonstrated motion. The model-predicted motion and human-demonstrated motion can be input into the discriminator to train it to distinguish between the two. The loss function is a binary cross-entropy loss. A reward function can be constructed based on the discriminator's judgment result, with the following formula:

[0041] in, This represents the discriminator's judgment of action a in the current state s, i.e., the probability that the action belongs to a human demonstration movement. Then, the PPO algorithm is used to update the model's policy network with this reward function as the objective, i.e., the network is trained using Generative Adversarial Imitation Learning (GAIL). Specifically: Calculate time-series cumulative reward based on single-step reward. The formula includes:

[0042] Where T represents the total time steps of the task. Indicates the discount factor; Based on time-series cumulative rewards Calculate the advantage function The formula includes:

[0043] in, Value network Predicted state at time step t Expected reward; Then based on the advantage function Construct PPO clipping loss The formula includes:

[0044]

[0045] in, Represents the learnable parameters of the motion generation model network. This represents the ratio of the probabilities of the old and new policies, reflecting the updated policy. Compared to the previous strategy In state Next action The probability ratio, Indicates the PPO cutting factor; A joint optimization loss function can be constructed based on the mean squared error loss and the discrimination result. Specifically:

[0046] in, This represents the weight of the behavioral cloning loss; the initial value can be set to 0.9. This represents the weight of the reinforcement learning loss, and its initial value can be set to 0.1; The basic motion generation model network architecture can be pre-trained using imitation learning based on a joint optimization loss function to obtain a pre-trained model. Specifically: first, behavior cloning can be used to initialize and pre-train the model, reducing the cold start exploration cost; then, the model is trained using a hybrid approach of behavior cloning and generative adversarial imitation learning (PPO); finally, convergence and generalization are achieved through independent PPO training. Throughout the training process, the pre-training phase... Much larger As training progresses, the cloning loss weights are gradually reduced, and the network architecture is iteratively trained based on this joint optimization loss function. When the model's success rate on the training dataset stabilizes above 90%, training is stopped, and a pre-trained model is obtained. This embodiment does not impose any limitations.

[0047] In this optional embodiment, the simulation environment may include a dual-arm robot, a target object, and obstacles. A pre-trained model can be used as the initial motion strategy within the simulation environment. A hybrid planner is constructed by combining Monte Carlo Tree Search (MCS) and Model Predictive Control (MPC). The MCS uses the UCB1 algorithm to traverse nodes, with an exploration coefficient of 1.414 and 50 parallel rollouts (K_rollouts). MPC has a time domain length H of 10, and each control step is optimized through 10 iterations. A proximal policy optimization algorithm is used as the optimizer, with a pruning coefficient of 0.2 and a learning rate of 1e-4. A weighted reward function is set, incorporating task success rate, motion smoothness, energy efficiency, and dynamic equilibrium constraints. The basic motion generation model network architecture is pre-trained using imitation learning based on the joint optimization loss function to obtain the pre-trained model. The multimodal scene representation vectors from the validation dataset are then used... The task instructions and the states of both arms are input into the fine-tuning model. The task execution results of the model are recorded, and indicators such as task success rate, joint angle prediction error, and joint angle acceleration fluctuation are calculated. The motion sequence generated by the model is verified to meet the physical constraints of the robot, i.e., the model requirements. The model requirements include task execution accuracy requirements, motion coordination requirements, and physical constraint requirements. It is checked for joint limit violations, self-collisions of both arms, etc. If the verification results meet the preset model requirements, the fine-tuning model is determined to be the basic motion generation model, and the network weights and structural parameters of the model are saved. If they do not meet the requirements, the model returns to the reinforcement learning fine-tuning stage, the reward function weights are adjusted, and training continues until the model meets the requirements. This embodiment does not limit the scope of the model.

[0048] As can be seen, implementing this optional embodiment can construct a basic motion generation model network architecture and training dataset to obtain the model's predicted motion. By combining the mean squared error loss of the behavior cloning algorithm and the judgment results of the discriminator to construct a joint optimization loss function, a pre-trained model is obtained through imitation learning pre-training. In a simulation environment, the pre-trained model is used as the initial motion strategy. A weighted reward function is constructed and iteratively updated based on the proximal policy optimization algorithm to obtain a fine-tuned model. After verification with a validation dataset, the basic motion generation model that meets the requirements is determined. It can quickly enable the model to learn basic vision-motion mapping capabilities through imitation learning. The joint optimization loss function makes the motion generated by the model more in line with human demonstration patterns, improving the model's generalization ability. The reinforcement learning fine-tuning combined with the multi-dimensional weighted reward function allows the model to take into account multiple indicators such as task execution and motion coordination. The validation process ensures that the performance of the basic model meets the standards. The final basic motion generation model has good generalization and robustness, providing high-quality and highly universal basic support for subsequent model dynamic modulation, and significantly reducing the overall training cost and deployment difficulty of the model.

[0049] In another optional embodiment, the image information includes binocular synchronized RGB image information; Scene-aware analysis of image information to obtain a multimodal scene representation vector corresponding to the object to be manipulated can include the following operations: Instance segmentation is performed on the binocular synchronized RGB image information to obtain the 2D mask of the object corresponding to the dual-arm robot. Stereo matching is performed on the binocular synchronized RGB image information to obtain a dense depth map. Depth completion is performed on the dense depth map based on the 2D mask to obtain a 3D scene point cloud. The objects include the two arms of the dual-arm robot, the object to be operated, and obstacles. Based on the 3D scene point cloud, object information is identified, scene graph nodes are determined based on the object information, and an undirected scene graph is constructed based on the scene graph nodes and the spatial distance between each scene graph node. Based on the undirected scene graph, the task level and the cooperation relationship between the two arms are determined. RGB appearance features are extracted from binocular synchronized RGB image information, and the first weight coefficient of RGB appearance features and the second weight coefficient of 3D scene point cloud are calculated according to task level and binocular cooperation relationship. The RGB appearance features and the 3D scene point cloud are fused according to the first and second weight coefficients to obtain a fused feature vector. The fused feature vector is then normalized to obtain the multimodal scene representation vector corresponding to the object to be operated. The multimodal scene representation vector includes the pose of the object to be operated, obstacle information, scene context, task level, and the two-arm collaboration relationship.

[0050] In this optional embodiment, scene perception analysis of image information can be performed based on the multimodal vision perception module of the dual-arm robot. After the robot's binocular camera acquires binocular synchronous RGB image information, it transmits the image information to the multimodal vision perception module. The image information includes binocular synchronous RGB image information. The multimodal vision perception module includes a bottom layer, a middle layer, and a high layer. The bottom layer is used to perform instance segmentation of the binocular synchronous RGB image information using the SOLO-v2 lightweight instance segmentation algorithm, accurately segmenting the dual-arm robot's arms, the object to be operated, and obstacles in the image, and outputting a 2D mask for each object. The bottom layer is also used to perform stereo matching of the binocular synchronous RGB image information using a combination of the SGBM algorithm and a neural thinning network. Specifically, firstly, the initial disparity map of the binocular image is calculated using the SGBM algorithm, and then the SGBM is set. The algorithm uses a window size of 11 and a disparity range of 0-128. A neural refinement network is then used to refine the disparity map, eliminating holes and noise and converting it into a dense depth map with the same resolution as the original RGB image. Then, guided by a 2D mask, depth completion is performed on low-texture regions (such as the surface of a solid-color object) in the dense depth map, and interpolation algorithms are used to fill in the holes. The completed dense depth map is then registered with the RGB image to generate a 3D scene point cloud. Objects include the arms of the dual-arm robot, the object to be manipulated, and obstacles; this embodiment is not limited to these.

[0051] In this optional embodiment, the object information may include object type, object pose, object size, object center coordinates, etc. Scene graph nodes represent abstract graph nodes representing key objects and / or key parts of objects in the scene. Specifically, these may include the joints of the dual-arm robot, the center point of the object to be manipulated, the vertices of obstacles, etc. An undirected scene graph can be constructed at a mid-level based on the scene graph nodes and the spatial distance between each scene graph node. That is, the dual-arm joints, the center of the object to be manipulated, and the vertices of obstacles are determined as scene graph nodes. The Euclidean spatial distance between any two nodes is calculated, and the reciprocal of the spatial distance is used as the edge weight to construct the undirected scene graph. The task level and the dual-arm collaboration relationship are determined based on the undirected scene graph. The task level is... The structured hierarchical description of the tasks to be performed by the dual-arm robot is used to define the execution logic of the task, the priority of sub-tasks, and the division of labor mode of dual-arm cooperation. Taking "grasping and placing the object to be operated" as an example, the task hierarchy can be divided into: High-level task objective: complete the entire process of "grasping-carrying-placing"; Mid-level sub-task decomposition: move both arms to the vicinity of the object to be operated; grasp the object with both hands in coordination; maintain the stability of the object's posture and carry it to the target position; release the object to complete the placement; Low-level motion instructions: details of the joint motion of the two arms corresponding to each sub-task (such as the wrist joint angle adjustment in the grasping stage, and the synchronous movement amplitude of the two arms in the carrying stage); The dual-arm cooperation relationship can include the sequential and / or synchronous movement order of the two arms, which is not limited in this embodiment.

[0052] In this optional embodiment, the RGB appearance features may include appearance features such as image texture, color, and contour. The higher layer may use a differentiable weight controller to calculate the first weight coefficient of the RGB appearance features and the second weight coefficient of the 3D scene point cloud according to the task level and the two-arm collaboration relationship. Then, the RGB appearance features and the 3D scene point cloud are fused according to the first weight coefficient and the second weight coefficient to obtain a fused feature vector. The fused feature vector is then normalized to obtain the multimodal scene representation vector corresponding to the object to be operated. The multimodal scene representation vector includes the pose of the object to be operated, obstacle information, scene context, task level, and two-arm collaboration relationship. This embodiment does not limit this.

[0053] As can be seen, implementing this optional embodiment can segment the image information to obtain a 2D mask, perform stereo matching to obtain a dense depth map, and complete the depth to obtain a 3D scene point cloud. Based on the 3D scene point cloud, an undirected scene graph is constructed, and the task level and the two-arm collaboration relationship are determined. RGB appearance features are extracted, and dual weight coefficients are calculated according to the task level and the two-arm collaboration relationship. After feature fusion and normalization, a multimodal scene representation vector is obtained. It can overcome the perception limitations of monocular vision by utilizing the disparity information of binocular synchronized RGB images, accurately restore the 3D geometric structure of the operation scene, and mine the task level and the two-arm collaboration relationship through scene graph reasoning. This allows scene perception to have both geometric information and semantic guidance capabilities. The dynamic weight coefficients realize the task-oriented fusion of features, and the normalization process improves the robustness of features. This further improves the accuracy and comprehensiveness of the multimodal scene representation vector, providing a visual input basis that is more in line with the actual operation task requirements for the motion generation model, and greatly improving the support effect of scene perception for subsequent motion generation.

[0054] In another optional embodiment, dynamically modulating a preset basic motion generation model based on physical parameters to obtain a motion generation model matching the dual-arm robot may include the following operations: Encode the physical parameters to generate condition vectors; Determine the standard spatiotemporal features corresponding to the basic motion generation model, and generate the scaling factor and offset factor of the standard spatiotemporal features based on the condition vector; A layer normalization operation is performed on the standard spatiotemporal features to obtain the normalization operation result. Based on the normalization operation result, scaling factor, and offset factor, the standard spatiotemporal features are dynamically modulated to obtain the modulated spatiotemporal features. The modulated spatiotemporal features are residually connected with the standard spatiotemporal features to obtain fused spatiotemporal features. Based on the model agnostic meta-learning framework, the basic motion generation model corresponding to the fused spatiotemporal features is fine-tuned by gradient update to obtain a motion generation model that matches the dual-arm robot.

[0055] In this optional embodiment, the physical parameters may include at least one of the following: arm length parameter, mass distribution parameter, joint limit parameter, and joint transmission ratio parameter. All physical parameters may be normalized using a minimum-maximum normalization formula to map the parameter values ​​to the [0,1] interval, eliminating the dimensional differences between different physical parameters. Then, the normalized physical parameters are encoded to generate a condition vector. This embodiment does not impose any limitations on this.

[0056] In this optional embodiment, the encoder output of the basic motion generation model can be retrieved as standard spatiotemporal features, and scaling and offset factors of the standard spatiotemporal features can be generated based on the conditional vector. Specifically, the conditional vector can be input into a two-layer lightweight MLP network, which is a modulation parameter generator, and outputs scaling and offset factors consistent with the number of channels of the standard spatiotemporal features. Layer normalization is performed on the standard spatiotemporal features h, that is, mean and variance normalization is performed on each channel of the feature to eliminate the dimensional differences between channels. Specifically:

[0057] in, This represents the result of the normalization operation. The channel mean of the standard spatiotemporal feature h is represented. The channel variance representing the standard spatiotemporal characteristic h, To represent the minimum value, we can take 1e-8 to avoid a denominator of 0; then, based on the normalization result, scaling factor, and offset factor, we dynamically modulate the standard spatiotemporal features to obtain the modulated spatiotemporal features, specifically:

[0058] in, Indicates the spatiotemporal characteristics of modulation. Indicates the scaling factor. This represents the offset factor.

[0059] In this optional embodiment, the modulated spatiotemporal features may be selected. An element-level residual connection is performed with the standard spatiotemporal feature h to obtain the fused spatiotemporal feature. The fused spatiotemporal feature is used as the intermediate layer feature. Based on the Model Agnostic Meta-Learning Framework (MAML), a small number of gradient updates are performed to fine-tune the basic motion generation model to achieve the final calibration of the model for the target robot, resulting in a motion generation model that matches the dual-arm robot. This embodiment is not limited.

[0060] As can be seen, implementing this optional embodiment can generate condition vectors by encoding the physical parameters of the dual-arm robot, determine the standard spatiotemporal features of the basic motion generation model, and generate corresponding scaling and offset factors based on the condition vectors. After performing layer normalization on the standard spatiotemporal features, it combines with the modulation factor to obtain the modulated spatiotemporal features. After residual connection, it obtains the fused spatiotemporal features and fine-tunes them based on gradient updates using a model-agnostic meta-learning framework, thus obtaining a motion generation model that matches the target dual-arm robot. This can quantify the robot's physical parameters into model-recognizable condition vectors, providing accurate physical attribute basis for dynamic model modulation. Through layer normalization combined with scaling, The offset factor enables adaptive modulation of the feature layer, allowing features to accurately adapt to the physical constraints of the target robot. The residual connection effectively preserves the core motion laws learned by the basic model, avoiding the loss of effective features during modulation. The model-agnostic meta-learning framework only requires a small amount of demonstration data to complete fine-tuning without the need for full model retraining, significantly shortening the cross-platform adaptation cycle and reducing adaptation costs. It enables cross-platform reuse of the basic motion generation model. The modulated model can generate joint angle sequences that fit the physical properties of the target robot, effectively avoiding motion faults such as joint limit overruns and self-collisions, further improving the stability of the robot in performing operational tasks.

[0061] Example 2 Please see Figure 2 , Figure 2 This is a flowchart illustrating a motion control method based on a vision and motion generation model disclosed in an embodiment of the present invention. Figure 2 The described motion control method based on vision and motion generation models can be applied to a dual-arm robot. This dual-arm robot may include a motion control device based on vision and motion generation models. This motion control device may include an intelligent server or intelligent platform for generating joint angle control sequences for the two arms of the dual-arm robot. The intelligent server may be a local server or a cloud server; this embodiment of the invention does not limit the scope. Figure 2 As shown, the motion control method based on vision and motion generation models can include the following operations: 201. Collect image information of the object to be operated corresponding to the dual-arm robot, and perform scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated.

[0062] 202. Obtain the physical parameters of the dual-arm robot, and dynamically modulate the preset basic motion generation model according to the physical parameters to obtain a motion generation model that matches the dual-arm robot.

[0063] 203. Obtain the task instructions for the object to be operated and the status of the two arms of the dual-arm robot.

[0064] 204. The feature fusion layer concatenates the features of the multimodal scene representation vector, task instructions and dual-arm state input to obtain the initial context features.

[0065] In this embodiment of the invention, optionally, the motion generation model is an end-to-end vision-motion mapping neural network. The motion generation model includes a feature fusion layer, a hybrid encoder-decoder, and an output layer. The hybrid encoder-decoder includes an encoder, a motion primitive library, and a decoder. The feature fusion layer is constructed using a cross-attention mechanism, with 8 attention heads. The multimodal scene representation vector, one-hot encoded task instructions, and dual-arm states can be dimensionally aligned. The feature fusion layer concatenates these three elements into a high-dimensional feature matrix. The cross-attention mechanism then performs weighted optimization on the feature matrix, assigning low weights to features unrelated to the operation task and high weights to core features. Finally, an initial context feature with unified dimensions is output. The initial context feature integrates scene information, task semantics, and the robot's current state.

[0066] 205. Extract the spatiotemporal features of the dual-arm robot's dual-arm collaboration from the initial context features using a multi-layer spatiotemporal convolutional network in the encoder.

[0067] In this embodiment of the invention, optionally, the initial context features are input into a multi-layer spatiotemporal convolutional network. The network first extracts global semantic features of the image through a visual network, then extracts spatiotemporal joint features of historical joint trajectories through a spatiotemporal graph convolutional network. After concatenation and cross-modal attention weighting, local spatial relationship features are extracted through a task-adaptive functional partitioning strategy. Finally, through multi-layer cascaded spatiotemporal convolution operations, spatiotemporal features containing spatial relationships, temporal evolution patterns, and task semantic information are output. The spatiotemporal features include the spatial relationships and temporal evolution patterns between the joints of the two arms of the dual-arm robot, as well as semantic information related to the operation tasks corresponding to the task instructions.

[0068] 206. Generate motion instructions that match the operation task based on the motion primitives stored in the motion primitive library and the spatiotemporal characteristics.

[0069] In this embodiment of the invention, optionally, the motion primitive library is a low-dimensional discrete motion primitive set built based on variational autoencoder (VAE), which is automatically segmented and extracted from human bi-arm operation demonstration data, and includes basic motion units such as joint swinging, grasping force, and posture adjustment. Based on the task semantic information in the spatiotemporal features, the cosine similarity between the spatiotemporal features and each basic motion primitive in the motion primitive library can be calculated, and control motion primitives with similarity higher than a preset threshold can be selected. Then, based on the spatial relationship and temporal evolution law in the spatiotemporal features, the control motion primitives are temporally sorted and combined to generate motion instructions that are completely matched with the current operation task. The motion instructions include bi-arm collaboration temporal logic, joint motion amplitude, and action execution duration.

[0070] 207. The decoder generates an initial joint angle sequence based on spatiotemporal characteristics and motion commands. The initial joint angle sequence includes joint angle control commands for multiple future time steps.

[0071] In this embodiment of the invention, optionally, the decoder is composed of a spatiotemporal Transformer network, adopting a 6-layer Transformer encoder structure, which has the ability to generate time-series motion sequences autoregressively. It can concatenate spatiotemporal features and motion commands and input them into the spatiotemporal Transformer decoder. The decoder adopts an autoregressive generation method, based on the joint angles of the current time step, and combines spatiotemporal features and motion commands to predict the joint angles of the next time step, and iteratively generates the initial joint angle sequence for the next N time steps. The time step length of the sequence is set according to the robot's underlying control frequency.

[0072] 208. The initial joint angle sequence is smoothed and filtered by the output layer to obtain the joint angle control sequence of both arms.

[0073] In this embodiment of the invention, optionally, the initial joint angle sequence is input to the fully connected layer of the output layer, and the sequence is dimension-mapped to ensure that the number of joint angles is consistent with the total number of joints of the target robot. Then, the mapped sequence is input to the smoothing filter layer, and the moving average filter algorithm is used to smooth the abrupt values ​​in the sequence to eliminate the jumps in joint angles. Finally, a continuous, smooth, and robot-compliant dual-arm joint angle control sequence is output.

[0074] 209. Control the dual-arm robot to execute the operation task corresponding to the task instruction according to the dual-arm joint angle control sequence.

[0075] In this embodiment of the invention, it should be noted that for other descriptions of steps 201-203 and step 209, please refer to the detailed description of steps 101-104 in Embodiment 1 of the invention. These descriptions will not be repeated in this embodiment of the invention.

[0076] It is evident that implementation Figure 1The described motion control method based on vision and motion generation models can acquire image information of the object to be manipulated by a dual-arm robot, perform scene perception analysis on the image information to obtain a multimodal scene representation vector corresponding to the object to be manipulated, obtain the physical parameters of the dual-arm robot, and dynamically modulate a preset basic motion generation model according to the physical parameters to obtain a motion generation model matching the dual-arm robot. It can also obtain task instructions for the object to be manipulated and the states of the two arms of the dual-arm robot, accurately perceive the 3D scene, and provide comprehensive and reliable visual input for motion generation. Dynamic modulation of the basic motion generation model based on the robot's physical parameters enables cross-platform reuse of the basic model, significantly reducing the adaptation cost and cycle time, and achieving end-to-end mapping from visual perception to motion control, improving the automation and intelligence level of motion control. The method uses a feature fusion layer to concatenate features from the multimodal scene representation vector, task instructions, and the states of the two arms to obtain initial context features, which are then utilized by multi-layer spatiotemporal convolution in the encoder. The network extracts the spatiotemporal features of the dual-arm robot's collaboration from the initial context features. Based on the motion primitives stored in the motion primitive library and the spatiotemporal features, it generates motion commands that match the operation task. The decoder generates an initial joint angle sequence based on the spatiotemporal features and motion commands. The output layer then performs smoothing filtering on the initial joint angle sequence to obtain the dual-arm joint angle control sequence. The dual-arm robot is controlled to execute the operation task corresponding to the task command according to the dual-arm joint angle control sequence. This makes the network architecture of the motion generation model clearer, the functions of each module more efficient, and accurately extracts the spatiotemporal features that fit the collaborative state of the dual-arm robot. Based on the motion primitive library, it generates motion commands that highly match the operation task, making the robot's movement more in line with the basic human movement patterns. At the same time, the smoothing filtering eliminates abrupt values ​​in the joint angle sequence, further improving the accuracy and smoothness of the dual-arm joint angle control sequence generation, effectively ensuring the stability of the robot's movement process, avoiding joint impact, and further improving the coordination of the robot's dual-arm movement and the execution effect of the operation task.

[0077] In an optional embodiment, the multilayer spatiotemporal convolutional network includes a visual network and a spatiotemporal graph convolutional network; Extracting spatiotemporal features of dual-arm robot collaboration from initial context features using a multi-layer spatiotemporal convolutional network in the encoder can include the following operations: The global semantic information of the image is captured in the initial context features by a visual network to obtain global visual features. The historical joint trajectory information corresponding to the state of both arms is input into the spatiotemporal graph convolutional network to obtain spatiotemporal joint features. The global visual features and spatiotemporal joint features are then concatenated to obtain the initial fusion features. Cross-modal spatiotemporal attention is calculated on the initial fused features to obtain a cross-modal attention weight set. The initial fused features are then weighted and optimized based on the cross-modal attention weight set to obtain cross-modal fused features. The cross-modal attention weight set includes the spatial attention weight matrix, the temporal attention weight matrix, and the correlation weights between global visual features and spatiotemporal joint features corresponding to the initial fused features. Based on the task instructions, at least one functional partition is determined for the two arms of the dual-arm robot, and based on each functional partition and cross-modal fusion features, the local spatial relationship features of the joints within each functional partition are determined. Each local spatial relationship feature is input into a preset multi-layer cascaded structure for multi-layer spatiotemporal convolution operations to obtain the spatiotemporal features of the dual-arm robot's dual-arm collaboration. The multi-layer cascaded structure consists of multiple adaptive graph convolutional network layers with multiple configuration partitions and time one-dimensional convolutional layers cascaded alternately.

[0078] In this optional embodiment, the multi-layer spatiotemporal convolutional network may include a visual network and a spatiotemporal graph convolutional network. The visual network is a visual Transformer (ViT) network, and the core of the spatiotemporal graph convolutional network is a multi-configuration partition adaptive graph convolutional network (MP-AGCN). Image semantic features can be extracted from the initial context features and input into the visual Transformer network. The global semantic information of the image is captured through a multi-head self-attention mechanism. Historical joint trajectory information corresponding to the state of both arms is extracted from the initial context features and input into the spatiotemporal graph convolutional network. Spatial graph convolution captures the spatial relationship between joints, and temporal convolution captures the temporal evolution law of joints. Spatiotemporal joint features are output. The historical joint trajectory information refers to the sequence of joint angles of both arms in the past multiple time steps contained in the state of both arms. Then, the global visual features and spatiotemporal joint features are dimensionally aligned and concatenated into an initial fusion feature. This embodiment is not limited to this.

[0079] In this optional embodiment, cross-modal spatiotemporal attention calculation refers to simultaneously considering attention weighting calculations in both spatial and temporal dimensions, combining spatial attention, temporal attention, and cross-attention to achieve deep fusion of visual features and joint features. Cross-modal spatiotemporal attention calculation can be performed on the initial fused features to obtain a cross-modal attention weight set. Specifically, spatiotemporal attention branches, temporal attention branches, and cross-attention branches can be constructed. Then, the initial fused features are input into the spatial attention branch, and the attention weights at each spatial location are calculated through convolutional layers to generate a spatial attention weight matrix. The initial fused features are input into the temporal attention branch, and the attention weights at each spatial location are calculated through temporal... The convolutional layer calculates the attention weights at each time step to generate a temporal attention weight matrix. The global visual features are used as query vectors, and the spatiotemporal joint features are used as key vectors. These are input to the cross-attention branch to calculate the correlation weights between the two. That is, the cross-modal attention weight set includes the spatial attention weight matrix, the temporal attention weight matrix, and the correlation weights between the global visual features and the spatiotemporal joint features corresponding to the initial fusion features. Then, the spatial attention weight matrix, the temporal attention weight matrix, and the correlation weights are multiplied by weight to obtain the comprehensive attention weights. The comprehensive attention weights are multiplied by the initial fusion features to obtain the cross-modal fusion features. This embodiment is not limited to this.

[0080] In this optional embodiment, functional partitioning refers to the functional division of the two arms of the dual-arm robot based on the semantic information of the task instructions. The partitioning method is strongly related to the operation task. For example, for grasping tasks, the two arms are divided into three functional partitions: shoulder, elbow, and wrist. For handling tasks, the two arms are divided into two functional partitions: proximal arm and distal arm. Each functional partition contains several joints. Local spatial relationship features refer to the spatial association features between joints within each functional partition, including the absolute position, relative position, and spatial coupling relationship of the joints within the partition. Based on the functional partitioning results, feature subsets corresponding to each functional partition can be extracted from cross-modal fusion features. Spatial graph convolution operations can be performed on each subset to capture the local spatial relationships between joints within the partition. This embodiment does not limit this.

[0081] In this optional embodiment, the multi-layer cascaded structure is optionally composed of multiple multi-configuration partition adaptive graph convolutional network layers and temporal one-dimensional convolutional layers alternately cascaded. For example, it is a network structure composed of 10 multi-configuration partition adaptive graph convolutional network layers (MP-AGCN) and 10 temporal one-dimensional convolutional layers (TCN) alternately cascaded. The local spatial relationship features of each functional partition are concatenated and input into the first MP-AGCN layer of the multi-layer cascaded structure to perform spatial graph convolution operation and extract spatial depth features. The output of the MP-AGCN layer is input into the corresponding TCN layer to perform temporal one-dimensional convolution operation and extract temporal depth features. The output dimension of each layer remains unchanged. After 10 sets of alternating cascaded operations of MP-AGCN and TCN, the local spatial relationship features are extracted by deep spatiotemporal convolution. The spatial relationship and temporal evolution law are fused to obtain the spatiotemporal features of the dual-arm robot's dual-arm collaboration. These features include the spatial relationship between the joints of the two arms, the temporal evolution law, and semantic information related to the operation task. This embodiment does not limit this.

[0082] As can be seen, implementing this optional embodiment can capture global visual features by using a visual network to capture global semantic information of the image in the initial context features. It then inputs the historical joint trajectory information corresponding to the state of both arms into a spatiotemporal graph convolutional network to obtain spatiotemporal joint features, which are then concatenated into initial fusion features. Cross-modal spatiotemporal attention calculation is performed on the initial fusion features to obtain cross-modal fusion features. Based on task instructions, the robot's two arms are divided into functional zones, and the local spatial relationship features of each zone are extracted. These local spatial relationship features are then input into a multi-layered cascaded structure for convolution operations to obtain the spatiotemporal features of the two-arm collaboration. This approach enables precise capture of both visual features and joint motion features. Cross-modal spatiotemporal attention calculation improves the fusion degree and effectiveness of different types of features. Task-instruction-oriented functional zones make feature extraction more targeted, while the multi-layered cascaded structure accurately mines the spatial relationships and temporal evolution of the two-arm collaboration, significantly improving the accuracy and comprehensiveness of spatiotemporal feature extraction. This lays a high-quality feature foundation for subsequent motion instruction generation and joint angle sequence generation, further enhancing the adaptability and accuracy of motion generation to the operational task and the robot's collaborative state.

[0083] In another alternative embodiment, the motion primitive library includes multiple basic motion primitives; Based on the motion primitives stored in the motion primitive library and spatiotemporal characteristics, generating motion instructions that match the operation task can include the following operations: Based on the semantic information related to the operation task corresponding to the task instruction, the similarity between the spatiotemporal features and each basic motion primitive is calculated, and basic motion primitives with similarity greater than a preset similarity threshold are selected as control motion primitives to obtain a set of control motion primitives. Based on the spatial relationship and temporal evolution of the joints of both arms, temporal constraints are constructed for the set of control motion primitives. The set of control motion primitives is then temporally sorted according to the temporal constraints to obtain the motion primitive sequence. The acceleration fluctuation value of the joint motion of both arms corresponding to the motion primitive sequence is minimized. The motion primitive sequence is mapped to the joint angle change, and motion commands matching the operation task are generated based on the joint angle change and the corresponding constraint parameters of the dual-arm robot. The constraint parameters include joint limit parameters and motion speed parameters, and the motion commands include dual-arm collaborative timing logic, joint motion amplitude, and action execution duration.

[0084] In this optional embodiment, the motion primitive library may include multiple basic motion primitives. The basic motion primitives represent low-dimensional discrete basic motion units extracted from human bi-arm operation demonstration data based on variational autoencoder (VAE). Each motion primitive corresponds to an independent basic bi-arm movement, such as shoulder joint swinging, wrist joint rotation, grasping force, posture adjustment, etc. Based on the semantic information of the operation task corresponding to the task instruction (such as grasping, placing, and carrying), the similarity between the spatiotemporal features and each basic motion primitive is calculated, and basic motion primitives with similarity greater than a preset similarity threshold are selected as control motion primitives to obtain a set of control motion primitives. For example, the similarity threshold can be set to 0.8, which is not limited in this embodiment.

[0085] In this optional embodiment, the timing constraints may optionally include hard constraints and soft constraints. Hard constraints represent temporal logic constraints that must be met, such as in a grasping task, the motion primitive "moving both arms to the vicinity of the object" must precede the motion primitive "grasping force" and the motion primitive "posture adjustment" must precede the motion primitive "object handling"; when both arms work synchronously, the basic motion primitives corresponding to the left and right arms must be executed simultaneously. Soft constraints represent optimization constraints that improve motion quality, such as the change in joint angle between adjacent motion primitives must be smooth to avoid abrupt changes in joint acceleration; the execution duration of motion primitives must be consistent with the timing of joint movement. The evolutionary law matching is performed. Taking the minimum acceleration fluctuation value of the bi-arm joint motion as the objective function, the dynamic programming algorithm is used to sort the control motion primitive set in time to obtain the motion primitive sequence. The motion primitive sequence corresponds to the minimum acceleration fluctuation value of the bi-arm joint motion. The motion primitive sequence is mapped to the joint angle change. Based on the joint angle change and the constraint parameters corresponding to the bi-arm robot, motion commands matching the operation task are generated. The constraint parameters include joint limit parameters and motion speed parameters. The motion commands include bi-arm collaborative timing logic, joint motion amplitude, and action execution duration. This embodiment does not limit these parameters.

[0086] As can be seen, implementing this optional embodiment can calculate the similarity between spatiotemporal features and basic motion primitives based on the semantic information of the task instructions, filter to obtain a set of control motion primitives, construct temporal constraints based on the spatial relationship and temporal evolution of the bi-arm joints, sort the set of control motion primitives to obtain the sequence of motion primitives with the smallest acceleration fluctuation value, map this sequence to the joint angle change and combine it with robot constraint parameters to generate motion instructions that match the operation task. This approach can quickly and accurately match motion primitives adapted to the operation task through similarity filtering, reducing subsequent computation and improving the efficiency of motion instruction generation. The temporal constraint conditions... The optimization objective of minimizing acceleration fluctuations ensures that the motion primitive sequence conforms to the task execution logic while guaranteeing the smoothness of robot motion. By combining robot physical constraint parameters to limit the changes in joint angles, the motion commands are ensured to meet the physical property requirements of the robot. The final generated motion commands contain rich collaborative logic and parameter information, providing clear and specific guidance for the decoder to generate joint angle sequences. This makes the joint angle sequences more closely match the operational task requirements and the robot's real-time collaborative state, further improving the rationality and accuracy of motion generation, effectively ensuring the coordination and stability of the robot's dual arm movements, and improving the execution quality of the operational task.

[0087] Example 3 Please see Figure 3 , Figure 3 This is a schematic diagram of a motion control device based on a vision and motion generation model disclosed in an embodiment of the present invention. Figure 3 The described motion control device based on vision and motion generation models may include an intelligent server or intelligent platform for generating joint angle control sequences for the two arms of a dual-arm robot. The intelligent server may be a local server or a cloud server; this embodiment of the invention does not limit the scope. Figure 3 As shown, the motion control device based on vision and motion generation models may include: The acquisition module 301 is used to acquire image information of the object to be operated corresponding to the dual-arm robot, and to perform scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated. The acquisition module 302 is used to acquire the physical parameters of the dual-arm robot; The modulation module 303 is used to dynamically modulate the preset basic motion generation model according to the physical parameters to obtain a motion generation model that matches the dual-arm robot. The physical parameters include at least one of the arm length parameters, mass distribution parameters, joint limit parameters, and joint transmission ratio parameters. The acquisition module 302 is also used to acquire the task instructions for the object to be operated and the state of the two arms of the dual-arm robot. The generation module 304 is used to input the multimodal scene representation vector, task instructions and dual-arm states into the motion generation model to generate dual-arm joint angle control sequences; The control module 305 is used to control the dual-arm robot to execute the operation task corresponding to the task instruction according to the dual-arm joint angle control sequence.

[0088] It is evident that implementation Figure 3 The described motion control device based on vision and motion generation models can acquire image information of the object to be manipulated by a dual-arm robot, perform scene perception analysis on the image information to obtain a multimodal scene representation vector corresponding to the object to be manipulated, obtain the physical parameters of the dual-arm robot, and dynamically modulate a preset basic motion generation model according to the physical parameters to obtain a motion generation model matching the dual-arm robot. It can also acquire task instructions for the object to be manipulated and the states of the two arms of the dual-arm robot, input the multimodal scene representation vector, task instructions, and arm states into the motion generation model to generate a joint angle control sequence for the two arms. Based on the joint angle control sequence, the device controls the dual-arm robot to execute the operation task corresponding to the task instruction. It can accurately perceive the 3D scene, providing comprehensive and reliable visual input for motion generation. Dynamic modulation of the basic motion generation model based on the robot's physical parameters enables cross-platform reuse of the basic model, significantly reducing the adaptation cost and cycle time. It achieves end-to-end mapping from visual perception to motion control, improving the automation and intelligence level of motion control, enhancing the naturalness, coordination, and accuracy of the robot's dual-arm movements, and increasing the success rate and stability of the robot's operation tasks.

[0089] In an optional embodiment, such as Figure 4 As shown, the motion generation model includes a feature fusion layer, a hybrid codec, and an output layer. The hybrid codec includes an encoder, a motion primitive library, and a decoder. The generation module 304 inputs the multimodal scene representation vector, task instructions, and dual-arm states into the motion generation model, and the specific methods for generating the dual-arm joint angle control sequence include: The feature fusion layer concatenates the features of the multimodal scene representation vector, task instructions, and dual-arm state input to obtain the initial context features. The spatiotemporal features of the dual-arm robot's dual-arm collaboration are extracted from the initial context features by a multi-layer spatiotemporal convolutional network in the encoder. The spatiotemporal features include the spatial relationship between the joints of the two arms of the dual-arm robot, the temporal evolution law, and the semantic information related to the operation task corresponding to the task instruction. Based on the motion primitives stored in the motion primitive library and the spatiotemporal characteristics, generate motion instructions that match the operation task. The decoder generates an initial joint angle sequence based on spatiotemporal features and motion commands. The initial joint angle sequence includes joint angle control commands for multiple future time steps. The initial joint angle sequence is smoothed and filtered by the output layer to obtain the joint angle control sequence of both arms.

[0090] It is evident that implementation Figure 4 The described motion control device, based on vision and motion generation models, can acquire image information of the object to be manipulated by a dual-arm robot, perform scene perception analysis on the image information to obtain a multimodal scene representation vector corresponding to the object to be manipulated, obtain the physical parameters of the dual-arm robot, and dynamically modulate a preset basic motion generation model according to the physical parameters to obtain a motion generation model matching the dual-arm robot. It can also acquire task instructions for the object to be manipulated and the states of the two arms of the dual-arm robot, accurately perceive the 3D scene, and provide comprehensive and reliable visual input for motion generation. Dynamic modulation of the basic motion generation model based on the robot's physical parameters enables cross-platform reuse of the basic model, significantly reducing the adaptation cost and cycle time, achieving end-to-end mapping from visual perception to motion control, and improving the automation and intelligence level of motion control. The device obtains initial context features by concatenating the multimodal scene representation vector, task instructions, and arm states through a feature fusion layer, and utilizes multi-layer spatiotemporal convolution in the encoder. The network extracts the spatiotemporal features of the dual-arm robot's collaboration from the initial context features. Based on the motion primitives stored in the motion primitive library and the spatiotemporal features, it generates motion commands that match the operation task. The decoder generates an initial joint angle sequence based on the spatiotemporal features and motion commands. The output layer then performs smoothing filtering on the initial joint angle sequence to obtain the dual-arm joint angle control sequence. The dual-arm robot is controlled to execute the operation task corresponding to the task command according to the dual-arm joint angle control sequence. This makes the network architecture of the motion generation model clearer, the functions of each module more efficient, and accurately extracts the spatiotemporal features that fit the collaborative state of the dual-arm robot. Based on the motion primitive library, it generates motion commands that highly match the operation task, making the robot's movement more in line with the basic human movement patterns. At the same time, the smoothing filtering eliminates abrupt values ​​in the joint angle sequence, further improving the accuracy and smoothness of the dual-arm joint angle control sequence generation, effectively ensuring the stability of the robot's movement process, avoiding joint impact, and further improving the coordination of the robot's dual-arm movement and the execution effect of the operation task.

[0091] In another alternative embodiment, such as Figure 4 As shown, multi-layer spatiotemporal convolutional networks include visual networks and spatiotemporal graph convolutional networks; The specific methods by which the generation module 304 extracts the spatiotemporal features of the dual-arm robot's collaboration from the initial context features through the multi-layer spatiotemporal convolutional network in the encoder include: The global semantic information of the image is captured in the initial context features by a visual network to obtain global visual features. The historical joint trajectory information corresponding to the state of both arms is input into the spatiotemporal graph convolutional network to obtain spatiotemporal joint features. The global visual features and spatiotemporal joint features are then concatenated to obtain the initial fusion features. Cross-modal spatiotemporal attention is calculated on the initial fused features to obtain a cross-modal attention weight set. The initial fused features are then weighted and optimized based on the cross-modal attention weight set to obtain cross-modal fused features. The cross-modal attention weight set includes the spatial attention weight matrix, the temporal attention weight matrix, and the correlation weights between global visual features and spatiotemporal joint features corresponding to the initial fused features. Based on the task instructions, at least one functional partition is determined for the two arms of the dual-arm robot, and based on each functional partition and cross-modal fusion features, the local spatial relationship features of the joints within each functional partition are determined. Each local spatial relationship feature is input into a preset multi-layer cascaded structure for multi-layer spatiotemporal convolution operations to obtain the spatiotemporal features of the dual-arm robot's dual-arm collaboration. The multi-layer cascaded structure consists of multiple adaptive graph convolutional network layers with multiple configuration partitions and time one-dimensional convolutional layers cascaded alternately.

[0092] It is evident that implementation Figure 4 The described motion control device based on vision and motion generation models can capture global semantic information of images from initial context features through a visual network to obtain global visual features. It inputs historical joint trajectory information corresponding to the states of both arms into a spatiotemporal graph convolutional network to obtain spatiotemporal joint features, which are then concatenated into initial fusion features. Cross-modal spatiotemporal attention calculation is performed on the initial fusion features to obtain cross-modal fusion features. Based on task instructions, the robot's two arms are divided into functional zones, and the local spatial relationship features of each zone are extracted. These local spatial relationship features are input into a multi-layered cascaded structure for convolution operations to obtain the spatiotemporal features of the two-arm collaboration. This enables precise capture of both visual features and joint motion features. Cross-modal spatiotemporal attention calculation improves the fusion degree and effectiveness of different types of features. Task-instruction-oriented functional partitioning makes feature extraction more targeted, while the multi-layered cascaded structure accurately mines the spatial relationships and temporal evolution of the two-arm collaboration, significantly improving the accuracy and comprehensiveness of spatiotemporal feature extraction. This lays a high-quality feature foundation for subsequent motion instruction generation and joint angle sequence generation, further enhancing the adaptability and accuracy of motion generation to operational tasks and robot collaborative states.

[0093] In yet another alternative embodiment, such as Figure 4 As shown, the motion control device based on vision and motion generation models may further include: Module 306 is used to build the basic motion generation model network architecture and training dataset, and to obtain the model prediction motion of the basic motion generation model network architecture based on the training dataset input to the basic motion generation model network architecture. The calculation module 307 is used to calculate the mean squared error loss of the model's predicted motion and the human demonstration motion in the training dataset based on the behavior cloning algorithm, to judge the model's predicted motion and the human demonstration motion based on the discriminator, to obtain the judgment result, and to construct a joint optimization loss function based on the mean squared error loss and the discrimination result, and to perform imitation learning pre-training on the basic motion generation model network architecture based on the joint optimization loss function to obtain the pre-trained model. The update module 308 is used to construct a weighted reward function by using a pre-trained model as the initial motion policy in a preset simulation environment, and to iteratively update the initial motion policy based on the proximal policy optimization algorithm and the weighted reward function to obtain a fine-tuned model. The verification module 309 is used to verify the fine-tuned model using a preset verification dataset and obtain the verification result. When the verification result indicates that the fine-tuned model meets the preset model requirements, the fine-tuned model is determined to be the basic motion generation model. The model requirements include task execution accuracy requirements, motion coordination requirements, and physical constraint requirements.

[0094] It is evident that implementation Figure 4 The described motion control device based on vision and motion generation models can construct a basic motion generation model network architecture and training dataset to obtain the model's predicted motion. It then combines the mean squared error loss of the behavior cloning algorithm with the discriminator's judgment results to construct a joint optimization loss function for imitation learning pre-training, resulting in a pre-trained model. In a simulation environment, using the pre-trained model as the initial motion policy, a weighted reward function is constructed and iteratively updated based on a proximal policy optimization algorithm to obtain a fine-tuned model. After verification with a validation dataset, the basic motion generation model is determined to meet the requirements. Through imitation learning, the model can quickly acquire basic vision-motion mapping capabilities. The joint optimization loss function makes the model's generated motion more closely resemble human demonstration patterns, improving the model's generalization ability. Reinforcement learning fine-tuning combined with a multi-dimensional weighted reward function allows the model to consider multiple indicators such as task execution and motion coordination. The validation process ensures the basic model's performance meets the standards. The final basic motion generation model possesses good generalization and robustness, providing high-quality and highly versatile foundational support for subsequent model dynamic modulation, significantly reducing the overall training cost and deployment difficulty of the model.

[0095] In yet another alternative embodiment, such as Figure 4 As shown, the image information includes binocular synchronized RGB image information; The acquisition module 301 performs scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated. The specific methods include: Instance segmentation is performed on the binocular synchronized RGB image information to obtain the 2D mask of the object corresponding to the dual-arm robot. Stereo matching is performed on the binocular synchronized RGB image information to obtain a dense depth map. Depth completion is performed on the dense depth map based on the 2D mask to obtain a 3D scene point cloud. The objects include the two arms of the dual-arm robot, the object to be operated, and obstacles. Based on the 3D scene point cloud, object information is identified, scene graph nodes are determined based on the object information, and an undirected scene graph is constructed based on the scene graph nodes and the spatial distance between each scene graph node. Based on the undirected scene graph, the task level and the cooperation relationship between the two arms are determined. RGB appearance features are extracted from binocular synchronized RGB image information, and the first weight coefficient of RGB appearance features and the second weight coefficient of 3D scene point cloud are calculated according to task level and binocular cooperation relationship. The RGB appearance features and the 3D scene point cloud are fused according to the first and second weight coefficients to obtain a fused feature vector. The fused feature vector is then normalized to obtain the multimodal scene representation vector corresponding to the object to be operated. The multimodal scene representation vector includes the pose of the object to be operated, obstacle information, scene context, task level, and the two-arm collaboration relationship.

[0096] It is evident that implementation Figure 4 The described motion control device based on vision and motion generation models can perform instance segmentation to obtain 2D masks, stereo matching to obtain dense depth maps, and depth completion to obtain 3D scene point clouds. Based on the 3D scene point clouds, an undirected scene graph is constructed, and the task hierarchy and dual-arm collaboration relationship are determined. RGB appearance features are extracted, and dual-weight coefficients are calculated according to the task hierarchy and dual-arm collaboration relationship. After feature fusion and normalization, a multimodal scene representation vector is obtained. It can overcome the perception limitations of monocular vision by utilizing the disparity information of binocular synchronized RGB images, accurately restore the 3D geometric structure of the operation scene, and mine the task hierarchy and dual-arm collaboration relationship through scene graph reasoning. This allows scene perception to have both geometric information and semantic guidance capabilities. Dynamic weight coefficients realize task-oriented fusion of features, and normalization processing improves the robustness of features, further improving the accuracy and comprehensiveness of the multimodal scene representation vector. This provides the motion generation model with visual input that is more in line with the actual operation task requirements, and greatly improves the support effect of scene perception for subsequent motion generation.

[0097] In yet another alternative embodiment, such as Figure 4 As shown, the modulation module 303 dynamically modulates the preset basic motion generation model according to physical parameters to obtain a motion generation model that matches the dual-arm robot. The specific methods include: Encode the physical parameters to generate condition vectors; Determine the standard spatiotemporal features corresponding to the basic motion generation model, and generate the scaling factor and offset factor of the standard spatiotemporal features based on the condition vector; A layer normalization operation is performed on the standard spatiotemporal features to obtain the normalization operation result. Based on the normalization operation result, scaling factor, and offset factor, the standard spatiotemporal features are dynamically modulated to obtain the modulated spatiotemporal features. The modulated spatiotemporal features are residually connected with the standard spatiotemporal features to obtain fused spatiotemporal features. Based on the model agnostic meta-learning framework, the basic motion generation model corresponding to the fused spatiotemporal features is fine-tuned by gradient update to obtain a motion generation model that matches the dual-arm robot.

[0098] It is evident that implementation Figure 4 The described motion control device, based on vision and motion generation models, can generate condition vectors by encoding the physical parameters of a dual-arm robot, determine the standard spatiotemporal features of the basic motion generation model, and generate corresponding scaling and offset factors based on the condition vectors. After performing layer normalization on the standard spatiotemporal features, it combines them with modulation factors to obtain modulated spatiotemporal features. These modulated features are then obtained through residual connections and fine-tuned using gradient updates based on a model-agnostic meta-learning framework, resulting in a motion generation model that matches the target dual-arm robot. This device can quantify the robot's physical parameters into model-recognizable condition vectors, providing precise physical attribute basis for dynamic model modulation. Through layer normalization... By combining scaling and offset factors, adaptive modulation of the feature layer is achieved, allowing features to accurately adapt to the physical constraints of the target robot. Residual connections effectively preserve the core motion laws learned by the basic model, avoiding the loss of effective features during modulation. The model-agnostic meta-learning framework only requires a small amount of demonstration data to complete fine-tuning without the need for full model retraining, significantly shortening the cross-platform adaptation cycle and reducing adaptation costs. It enables cross-platform reuse of the basic motion generation model. The modulated model can generate joint angle sequences that fit the physical properties of the target robot, effectively avoiding motion faults such as joint limit overruns and self-collisions, further improving the stability of the robot in performing operational tasks.

[0099] In yet another alternative embodiment, such as Figure 4 As shown, the motion primitive library includes multiple basic motion primitives; The generation module 304 generates motion instructions that match the operation task based on the motion primitives stored in the motion primitive library and the spatiotemporal characteristics. The specific methods include: Based on the semantic information related to the operation task corresponding to the task instruction, the similarity between the spatiotemporal features and each basic motion primitive is calculated, and basic motion primitives with similarity greater than a preset similarity threshold are selected as control motion primitives to obtain a set of control motion primitives. Based on the spatial relationship and temporal evolution of the joints of both arms, temporal constraints are constructed for the set of control motion primitives. The set of control motion primitives is then temporally sorted according to the temporal constraints to obtain the motion primitive sequence. The acceleration fluctuation value of the joint motion of both arms corresponding to the motion primitive sequence is minimized. The motion primitive sequence is mapped to the joint angle change, and motion commands matching the operation task are generated based on the joint angle change and the corresponding constraint parameters of the dual-arm robot. The constraint parameters include joint limit parameters and motion speed parameters, and the motion commands include dual-arm collaborative timing logic, joint motion amplitude, and action execution duration.

[0100] It is evident that implementation Figure 4 The described motion control device, based on a vision and motion generation model, can calculate the similarity between spatiotemporal features and basic motion primitives based on the semantic information of task commands, filter to obtain a set of control motion primitives, construct temporal constraints based on the spatial relationships and temporal evolution of the bi-arm joints, sort the set of control motion primitives to obtain the sequence of motion primitives with the smallest acceleration fluctuation value, map this sequence to joint angle changes, and combine it with robot constraint parameters to generate motion commands that match the operation task. It can quickly and accurately match motion primitives adapted to the operation task through similarity filtering, reducing subsequent computation and improving the efficiency of motion command generation. By combining constraints with the optimization objective of minimizing acceleration fluctuations, the motion primitive sequence is designed to both conform to the task execution logic and ensure the smoothness of robot motion. By limiting the changes in joint angles in conjunction with robot physical constraint parameters, the motion commands are ensured to meet the physical property requirements of the robot. The final generated motion commands contain rich collaborative logic and parameter information, providing clear and specific guidance for the decoder to generate joint angle sequences. This makes the joint angle sequences more closely match the operational task requirements and the robot's real-time collaborative state, further improving the rationality and accuracy of motion generation, effectively ensuring the coordination and stability of the robot's dual arm movements, and improving the execution quality of the operational task.

[0101] Example 4 Please see Figure 5 , Figure 5 This is a schematic diagram of another motion control device based on a vision and motion generation model disclosed in an embodiment of the present invention. (See diagram below.) Figure 5 As shown, the motion control device based on vision and motion generation models may include: Memory 401 storing executable program code; Processor 402 coupled to memory 401; The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the motion control method based on vision and motion generation model described in Embodiment 1 or Embodiment 2 of the present invention.

[0102] Example 5 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in any of the motion control methods based on vision and motion generation models disclosed in Embodiment 1 of this invention.

[0103] Example 6 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the motion control method based on vision and motion generation models described in Embodiment 1 or Embodiment 2.

[0104] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0106] Finally, it should be noted that the motion control method and apparatus based on vision and motion generation models disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motion control method based on vision and motion generation models, characterized in that, The method includes: Image information of the object to be operated corresponding to the dual-arm robot is collected, and scene perception analysis is performed on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated. The physical parameters of the dual-arm robot are obtained, and the preset basic motion generation model is dynamically modulated according to the physical parameters to obtain a motion generation model that matches the dual-arm robot. The physical parameters include at least one of the following: arm length parameter, mass distribution parameter, joint limit parameter, and joint transmission ratio parameter. Obtain the task instructions for the object to be operated and the state of the two arms of the dual-arm robot, and input the multimodal scene representation vector, the task instructions and the state of the two arms into the motion generation model to generate a control sequence for the joint angles of the two arms; The dual-arm robot is controlled to perform the operation task corresponding to the task instruction according to the dual-arm joint angle control sequence.

2. The motion control method based on vision and motion generation models according to claim 1, characterized in that, The motion generation model includes a feature fusion layer, a hybrid encoder-decoder, and an output layer. The hybrid encoder-decoder includes an encoder, a motion primitive library, and a decoder. The step of inputting the multimodal scene representation vector, the task instructions, and the states of both arms into the motion generation model to generate a control sequence for the joint angles of both arms includes: The feature fusion layer concatenates the features of the multimodal scene representation vector, the task instructions, and the dual-arm state input to obtain initial context features. The spatiotemporal features of the dual-arm robot's dual-arm collaboration are extracted from the initial context features by a multi-layer spatiotemporal convolutional network in the encoder. The spatiotemporal features include the spatial relationship between the joints of the dual-arm robot's two arms, the temporal evolution law, and semantic information related to the operation task corresponding to the task instruction. Based on the motion primitives stored in the motion primitive library and the spatiotemporal characteristics, generate motion instructions that match the operation task. The decoder generates an initial joint angle sequence based on the spatiotemporal features and the motion commands. The initial joint angle sequence includes joint angle control commands for multiple future time steps. The initial joint angle sequence is smoothed and filtered by the output layer to obtain the bi-arm joint angle control sequence.

3. The motion control method based on vision and motion generation models according to claim 2, characterized in that, The multi-layer spatiotemporal convolutional network includes a visual network and a spatiotemporal graph convolutional network; The step of extracting the spatiotemporal features of the dual-arm robot's dual-arm collaboration from the initial context features through a multi-layer spatiotemporal convolutional network in the encoder includes: The global semantic information of the image in the initial context features is captured by the visual network to obtain global visual features. The historical joint trajectory information corresponding to the state of the two arms is input into the spatiotemporal graph convolutional network to obtain spatiotemporal joint features. The global visual features and the spatiotemporal joint features are then concatenated to obtain the initial fusion features. Cross-modal spatiotemporal attention calculation is performed on the initial fusion features to obtain a cross-modal attention weight set, and the initial fusion features are weighted and optimized according to the cross-modal attention weight set to obtain cross-modal fusion features. The cross-modal attention weight set includes the spatial attention weight matrix, the temporal attention weight matrix, and the correlation weight between the global visual features and the spatiotemporal joint features corresponding to the initial fusion features. Based on the task instructions, at least one functional partition is determined for the two arms of the dual-arm robot, and based on each functional partition and the cross-modal fusion feature, the local spatial relationship features of the joints within each functional partition are determined; Each of the local spatial relationship features is input into a preset multi-layer cascaded structure for multi-layer spatiotemporal convolution operations to obtain the spatiotemporal features of the dual-arm robot's dual-arm collaboration. The multi-layer cascaded structure is composed of multiple multi-configuration partition adaptive graph convolutional network layers and time one-dimensional convolutional layers cascaded alternately.

4. The motion control method based on vision and motion generation models according to any one of claims 1-3, characterized in that, The method further includes: Construct a basic motion generation model network architecture and a training dataset, and input the training dataset into the basic motion generation model network architecture to obtain the model prediction motion of the basic motion generation model network architecture. The mean squared error loss between the model's predicted motion and the human demonstration motion in the training dataset is calculated based on the behavior cloning algorithm. The model's predicted motion and the human demonstration motion are judged based on the discriminator to obtain the judgment result. A joint optimization loss function is constructed based on the mean squared error loss and the judgment result. The basic motion generation model network architecture is pre-trained by imitation learning based on the joint optimization loss function to obtain the pre-trained model. In a preset simulation environment, the pre-trained model is used as the initial motion policy, a weighted reward function is constructed, and the initial motion policy is iteratively updated based on the proximal policy optimization algorithm and the weighted reward function to obtain a fine-tuned model. The fine-tuned model is validated using a preset validation dataset to obtain validation results. When the validation results indicate that the fine-tuned model meets preset model requirements, the fine-tuned model is determined to be a basic motion generation model. The model requirements include task execution accuracy requirements, motion coordination requirements, and physical constraint requirements.

5. The motion control method based on vision and motion generation models according to any one of claims 1-3, characterized in that, The image information includes binocular synchronized RGB image information; The step of performing scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated on includes: The binocular synchronized RGB image information is segmented to obtain a 2D mask of the object corresponding to the dual-arm robot. Stereo matching is performed on the binocular synchronized RGB image information to obtain a dense depth map. The dense depth map is then filled with depth based on the 2D mask to obtain a three-dimensional scene point cloud. The object includes the two arms of the dual-arm robot, the object to be operated, and obstacles. Based on the 3D scene point cloud, the object information of the object is identified, the scene graph nodes are determined based on the object information, and an undirected scene graph is constructed based on the scene graph nodes and the spatial distance between each scene graph node. Based on the undirected scene graph, the task level and the cooperation relationship between the two arms are determined. RGB appearance features are extracted from the binocular synchronized RGB image information, and a first weight coefficient of the RGB appearance features and a second weight coefficient of the three-dimensional scene point cloud are calculated according to the task level and the dual-arm cooperation relationship. The RGB appearance features and the 3D scene point cloud are fused according to the first weight coefficient and the second weight coefficient to obtain a fused feature vector. The fused feature vector is then normalized to obtain a multimodal scene representation vector corresponding to the object to be operated. The multimodal scene representation vector includes the pose of the object to be operated, obstacle information, scene context, the task level, and the dual-arm collaboration relationship.

6. The motion control method based on vision and motion generation models according to any one of claims 1-3, characterized in that, The step of dynamically modulating a preset basic motion generation model based on the physical parameters to obtain a motion generation model matching the dual-arm robot includes: The physical parameters are encoded to generate a condition vector; Determine the standard spatiotemporal features corresponding to the basic motion generation model, and generate the scaling factor and offset factor of the standard spatiotemporal features based on the condition vector; A layer normalization operation is performed on the standard spatiotemporal features to obtain the normalization operation result. Based on the normalization operation result, the scaling factor, and the offset factor, the standard spatiotemporal features are dynamically modulated to obtain modulated spatiotemporal features. The modulated spatiotemporal features are residually concatenated with the standard spatiotemporal features to obtain fused spatiotemporal features. Based on the model agnostic meta-learning framework, the basic motion generation model corresponding to the fused spatiotemporal features is fine-tuned by gradient update to obtain a motion generation model that matches the dual-arm robot.

7. The motion control method based on vision and motion generation models according to claim 2 or 3, characterized in that, The motion primitive library includes multiple basic motion primitives; The step of generating motion instructions matching the operation task based on the motion primitives stored in the motion primitive library and the spatiotemporal features includes: Based on the semantic information related to the operation task corresponding to the task instruction, the similarity between the spatiotemporal features and each basic motion primitive is calculated, and the basic motion primitives with similarity greater than a preset similarity threshold are selected as control motion primitives to obtain a set of control motion primitives. Based on the spatial relationship between the two arm joints and the temporal evolution law, the temporal constraints corresponding to the control motion primitive set are constructed, and the control motion primitive set is temporally sorted according to the temporal constraints to obtain the motion primitive sequence. The acceleration fluctuation value of the two arm joint motion corresponding to the motion primitive sequence is minimized. The motion primitive sequence is mapped to joint angle changes, and motion commands matching the operation task are generated based on the joint angle changes and the constraint parameters corresponding to the dual-arm robot. The constraint parameters include joint limit parameters and motion speed parameters, and the motion commands include dual-arm collaborative timing logic, joint motion amplitude, and action execution duration.

8. A motion control device based on vision and motion generation models, characterized in that, The device includes: The acquisition module is used to acquire image information of the object to be operated by the dual-arm robot, and to perform scene perception analysis on the image information to obtain the multimodal scene representation vector corresponding to the object to be operated. An acquisition module is used to acquire the physical parameters of the dual-arm robot; The modulation module is used to dynamically modulate the preset basic motion generation model according to the physical parameters to obtain a motion generation model that matches the dual-arm robot. The physical parameters include at least one of arm length parameters, mass distribution parameters, joint limit parameters, and joint transmission ratio parameters. The acquisition module is also used to acquire the task instructions for the object to be operated and the state of the two arms of the dual-arm robot. The generation module is used to input the multimodal scene representation vector, the task instructions, and the state of the two arms into the motion generation model to generate a control sequence for the joint angles of the two arms; The control module is used to control the dual-arm robot to execute the operation task corresponding to the task instruction according to the dual-arm joint angle control sequence.

9. A motion control device based on vision and motion generation models, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the motion control method based on vision and motion generation model as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the motion control method based on vision and motion generation models as described in any one of claims 1-7.

Images