Robot action generation method and related device

Abstract

The invention belongs to the technical field of robot operation and control, and discloses a robot action generation method and a related device. The robot action generation method comprises the steps of obtaining a task instruction, ontology perception of a to-be-operated robot at the current moment, a multi-view RGB-D image and an initialized current noise action; and utilizing the trained action consistency diffusion strategy model to perform iterative action denoising according to the obtained task instruction, the ontology perception of the to-be-operated robot at the current moment, the multi-view RGB-D image and the initialized current noise action, and generating the action of the next step of the to-be-operated robot. According to the technical scheme disclosed by the invention, the technical problems of high training modeling difficulty, insufficient action prediction precision, low action generation speed and the like of the existing robot action generation technology in a high-precision operation task of the robot can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot operation and control, and in particular relates to a robot motion generation method and related devices. Background Art

[0002] High-precision robotic tasks have broad potential applications in industrial manufacturing, healthcare, smart homes, and other fields. These tasks require robots to not only accurately perceive subtle changes in the environment but also quickly and reliably generate actions in complex operational scenarios. For example, in industrial manufacturing, robots need to precisely install components in their designated locations; in healthcare, robots must complete surgical procedures with extreme precision; and in smart homes, robots need to flexibly perform delicate operations such as grasping and placing.

[0003] Currently, existing robot motion generation schemes still face many challenges in high-precision robotic tasks, especially in terms of perception accuracy, computational efficiency, and motion generation speed. The following problems still exist: 1) The complexity of the diffusion model in modeling action strategies. Explanation: the diffusion model has different performances at different denoising steps, which means that building a time-dependent denoising model increases the difficulty of network learning. 2) The problem of low action generation efficiency; Explainably, existing diffusion model-based strategies usually require an iterative denoising process to generate actions, resulting in low computational efficiency and making it difficult to meet the fast response requirements of real-time robot control. Summary of the Invention

[0004] The present invention aims to provide a method and related apparatus for generating robot motions to address one or more of the aforementioned technical problems. The technical solution disclosed in the present invention can address technical difficulties associated with existing robot motion generation technologies for high-precision robot manipulation tasks, such as the difficulty in training and modeling, insufficient motion prediction accuracy, and slow motion generation speed.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: A first aspect of the present invention provides a method for generating robot motions, comprising the following steps: Obtain task instructions and the current proprioception of the robot to be operated, multi-view RGB-D images, and initialized current noise actions; Using the trained action consistency diffusion strategy model, iterative action denoising is performed based on the acquired task instructions, the robot's current proprioception, multi-view RGB-D images, and the initialized current noise action to generate the next action of the robot. Among them, the steps of iterative motion denoising include: first, based on the task instructions and the current proprioception and multi-view RGB-D images of the robot to be operated, the multimodal feature extraction module in the trained motion consistency diffusion strategy model is used to obtain the multimodal scene coding features; then, the motion denoising branch network in the trained motion consistency diffusion strategy model is used to output the noise prediction based on the multimodal scene coding features and the initialized current noise motion, and the current noise motion is corrected and updated according to the noise prediction, and this process is repeated until the noise in the noise motion is completely eliminated to obtain the action of the robot's next step.

[0006] A further improvement of the technical solution of the present invention is that in the step of updating the current noise action according to the noise prediction correction, the current noise action is updated in the following manner: at any iterative denoising round t, the current noise action is ; Noise action input action consistency diffusion strategy model and obtain noise prediction ; Update the current noise action for the next denoising round ; Among them, the initial iterative denoising round , the initialized current noise action is .

[0007] A further improvement of the technical solution of the present invention is that, in the process of repeating until the noise in the noisy action is completely eliminated and the action of the next step of the robot to be operated is obtained, an error threshold method is used as the key termination condition in the action iterative denoising algorithm; wherein, the error between the current iteration result and the previous iteration result is calculated and compared with a preset threshold to determine whether to continue iteration; if the error is less than the preset threshold, it is considered that the denoising process has converged and the iteration is stopped.

[0008] A further improvement of the technical solution of the present invention is that the action consistency diffusion strategy model includes: A multimodal feature extraction module for extracting multimodal scene coding features; wherein the multimodal feature extraction module includes: an image coding module for encoding image features of multi-view RGB-D images; a text coding module for encoding text features of task instructions; and an action coding module for encoding prior features of the robot's proprioception; An action recognition branch network for performing action rationality recognition; wherein the action recognition branch network includes: a first multimodal feature alignment module for mapping the multimodal scene encoding features extracted by the multimodal feature extraction module to the same feature semantic space and performing feature fusion to obtain a multimodal fusion feature of the scene; and an action rationality recognition module for determining whether an input action is a successful action based on the multimodal fusion feature of the scene; An action denoising branch network is used to remove noise; wherein the action denoising branch network includes: a second multimodal feature alignment module for mapping the multimodal scene encoding features extracted by the multimodal feature extraction module to the same feature semantic space, and performing feature fusion to obtain the multimodal fusion features of the scene; and an action consistency denoising module for judging the noise components in the input action based on the multimodal fusion features of the scene.

[0009] A further improvement of the technical solution of the present invention is that the network structures of the first multimodal feature alignment module and the second multimodal feature alignment module are the same.

[0010] A further improvement of the technical solution of the present invention is that the training step of the action consistency diffusion strategy model includes: Obtaining a training sample data set; wherein each training sample in the training sample data set includes a sample task instruction, proprioception and multi-view RGB-D images at the sample moment, and an action pose label for the next step at the sample moment; First, the sample task instructions, the proprioception at the sample moment, and the multi-view RGB-D image are encoded into feature vectors of the same length through the multimodal feature extraction module, and then noise is added to the action pose label of the next step at the sample moment. , get the noise action; Perform action recognition branch network training; the action recognition branch network predicts the discrimination score of noise action based on the input multimodal fusion features and noise action ; The minimum distance between the discriminant score label and the successful action The calculation result is: , It is used to adjust the hyperparameter of the successful action range; the discriminative loss Gradient feedback is used to optimize the parameters of the action recognition branch network; Perform motion denoising branch network training; according to the input multimodal fusion features and noise action, give the noise prediction of the noise action ; Noise label when sampling known noise action , the training weight of the noise prediction is calculated using the prediction score of the action recognition branch network , the calculation expression is: , is the sign function; the loss function based on noise prediction To calculate the loss and update the parameters, .

[0011] A further improvement of the technical solution of the present invention is that, in the training step of the action consistency diffusion strategy model, the model weights are smoothed using exponential moving average.

[0012] A second aspect of the present invention provides a robot motion generation system, comprising: The data acquisition module is used to obtain task instructions and the current proprioception of the robot to be operated, multi-view RGB-D images, and initialized current noise actions; The action generation module uses the trained action consistency diffusion strategy model to perform iterative action denoising based on the acquired task instructions, the robot's current proprioception, multi-view RGB-D images, and the initialized current noise action to generate the next action of the robot to be operated; Among them, the steps of iterative motion denoising include: first, based on the task instructions and the current proprioception and multi-view RGB-D images of the robot to be operated, the multimodal feature extraction module in the trained motion consistency diffusion strategy model is used to obtain the multimodal scene coding features; then, the motion denoising branch network in the trained motion consistency diffusion strategy model is used to output the noise prediction based on the multimodal scene coding features and the initialized current noise motion, and the current noise motion is corrected and updated according to the noise prediction, and this process is repeated until the noise in the noise motion is completely eliminated to obtain the action of the robot's next step.

[0013] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the robot motion generation method as described in any one of the first aspects of the present invention is implemented.

[0014] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the robot motion generation method as described in any one of the first aspects of the present invention is implemented.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention provides a method for robotic motion generation. By introducing a motion consistency diffusion strategy model, combining multimodal information (task instructions, proprioception, and multi-view RGB-D images) with an iterative motion denoising mechanism, it achieves significant results and advances in the field of robotic motion generation. Specifically, by integrating task instructions, proprioception (such as joint state and force feedback), and multi-view RGB-D images (spatial geometry and semantic information), the model achieves a more comprehensive understanding of the current task scenario and robot state. Furthermore, through an iterative denoising mechanism, the initial noisy motions are gradually corrected, ultimately generating a smooth, task-compliant motion sequence. Furthermore, by utilizing the motion denoising branch network within the motion consistency diffusion strategy model, the motions are gradually optimized through multiple iterations. Through a noise prediction and correction mechanism, the model can handle the uncertainty in the initial noisy motions and generate more robust motions. This optimizes the iterative denoising process of traditional diffusion models and significantly improves the speed of motion generation. In summary, the robot motion generation method of the present invention significantly improves the quality, robustness and generalization ability of robot motion generation through technical means such as multimodal information fusion, iterative motion denoising, and motion consistency diffusion strategy model; compared with existing traditional methods, the method of the present invention can better adapt to complex environments and generate smoother and more reasonable motions, thereby improving the task success rate. This technical solution has broad application prospects in the fields of robot operation, autonomous navigation, human-machine collaboration, etc.

[0016] In the preferred training step of the technical solution of the present invention, by introducing successful action perception information, high success rate action areas are pre-divided in the action space, providing prior guidance for the action diffusion denoising process, helping the action denoising module to focus on the successful action area, thereby improving the accuracy of the final generated action; through the time-consistent diffusion denoising strategy, the modeling of the action denoising space is simplified, and the difficulty of model training is reduced compared to the time-dependent method, while improving the efficiency and accuracy of action iterative denoising. In summary, in the new technical solution disclosed by the present invention, through the successful action perception ability, additional information is provided for the action denoising direction, combined with the time-consistent action diffusion strategy modeling, the iterative denoising process of the traditional diffusion model is optimized, and the action generation speed is significantly improved, which solves the technical problems of the existing robot action generation technology in the robot high-precision operation tasks, such as the difficulty of training modeling, insufficient action prediction accuracy, and slow action generation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a flow chart of a method for generating robot motions in an embodiment of the present invention; Figure 2 Schematic diagram of data information flow of a robot motion generation method based on deep learning in an embodiment of the present invention; Figure 3 is a schematic diagram of an exemplary structure of a diffusion strategy model based on temporal consistency and successful action perception in an embodiment of the present invention; Figure 4 1 is a schematic diagram of a training process of a diffusion strategy model based on temporal consistency and successful action perception in an embodiment of the present invention; Figure 5 is a schematic diagram of a scenario in which a robot operation task is completed in a simulation environment in an embodiment of the present invention; Figure 6 Schematic diagram of a robot motion generation system in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0020] All other embodiments obtained by persons of ordinary skill in the art based on the technical solutions disclosed in the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0021] See also Figure 1 , an embodiment of the present invention provides a robot action generation method, which specifically includes the following steps: Step 1: Obtain the task instructions and the current proprioception of the robot to be operated, multi-view RGB-D images (color images and depth information images), and the initialized current noise action; Step 2: Using the trained action consistency diffusion strategy model, iterative action denoising is performed based on the acquired task instructions, the current proprioception of the robot to be operated, the multi-view RGB-D image, and the initialized current noise action to generate the action of the next step of the robot to be operated. In the specific exemplary technical solution, the action of the next step can be described by the posture of the end effector.

[0022] In the exemplary technical solution of the embodiment of the present invention, the action consistency diffusion strategy model based on temporal consistency and successful action perception proposed in the embodiment of the present invention is trained according to a two-stage training process, and the successful task demonstration is used as training data to help the network model acquire the ability to generate successful actions.

[0023] See also Figure 2 In the embodiment of the present invention, the components of the proposed action consistency diffusion strategy model are as follows: Figure 2 As shown, specifically including: An image encoding module performs feature encoding on multi-view RGB-D images. One exemplary implementation utilizes a Contrastive Language-Image Pre-training image encoder coupled with a multi-scale feature pyramid structure for feature encoding. A text encoding module encodes the input task instructions; wherein a specific exemplary implementation scheme is to use a CLIP text encoder with a linear mapping network structure for encoding; An action encoding module is used to encode the current action of the robot (for example, a robotic arm) as perceived by the robot. A specific exemplary implementation is to use a rotational position encoding algorithm to encode the translation and rotation parameters representing the action. The first multimodal feature alignment module and the second multimodal feature alignment module are respectively used to map the multimodal features to the same feature semantic space and perform feature fusion to obtain fused features; The action rationality identification module is used to determine whether the input action is a successful action based on the multimodal characteristics of the scene. The successful action score is 1, otherwise the expected prediction score is 0; The action consistency denoising module is used to determine the noise components in the input action based on the multimodal characteristics of the scene. The present invention hopes to obtain a successful action by eliminating the noise in the input action.

[0024] See also Figure 3 and Figure 4 In the embodiment of the present invention, the training steps of the above-mentioned action consistency diffusion strategy model include: Obtaining a training sample data set; wherein each training sample in the training sample data set includes a sample task instruction, proprioception and multi-view RGB-D images at the sample moment, and an action pose label for the next step at the sample moment; Multimodal feature encoding. The multi-view RGB-D image, task instructions, and body motion perception are first encoded into feature vectors of the same length. To prevent overfitting, the multimodal feature encoding network model (image encoding module, text encoding module, and motion encoding module) in this part uses a pre-trained model, whose parameters are frozen during training. In a specific embodiment of the image encoding module, pixel-level RGB features are projected into the 3D scene space based on the depth map and camera extrinsics. Then, some RGB features are selected through farthest point sampling to filter out redundant information.

[0025] Noise action sampling. Successful action label according to the next step , add noise to it , thus obtaining the noise action Specifically and illustratively, a specific embodiment uses standard Gaussian noise.

[0026] Action recognition branch training. The action recognition branch consists of the first multimodal feature alignment module and the action rationality recognition module. It gives the discrimination score of the noise action based on the input multimodal features and noise action. The score label is the minimum distance from the noise action to the successful action Calculations show that: ,in is a hyperparameter used to adjust the range of successful actions. Finally, the loss Gradient feedback is used to optimize the parameters of the action recognition branch network; Motion denoising branch training. The motion denoising branch consists of the second multimodal feature alignment module and the motion consistency denoising module. It gives the noise prediction of the noise action based on the input multimodal features and the noise action. . Noise labels when sampling known noise actions , the training weight of the noise action is calculated using the prediction score of the action recognition branch ,in is a sign function. An embodiment of a loss function for noise prediction is: .

[0027] In a specific exemplary embodiment of the present invention, the AdamW optimizer is used for training, and the learning rate is set to 0.001; during the training process, each batch contains 32 training samples, and the total number of training steps is 10K; in a further preferred technical solution, in order to accelerate training and improve model performance, the exponential moving average is used to smooth the model weights.

[0028] In the technical solution disclosed above in the embodiment of the present invention, a diffusion strategy based on motion consistency is proposed, which utilizes motion recognition capabilities to construct a time-consistent denoising process to achieve efficient and accurate generation of robot motions. Specifically, first, the present invention introduces motion differentiation information to construct a time-consistent velocity field in the motion space, mapping noisy motions to accurate noise. By unifying all denoising time steps, the time-consistent velocity field reduces the difficulty of strategy learning and speeds up motion generation. Second, the present invention proposes an action-oriented training method that enables the identification of successful actions. Through action-oriented training, the strategy model learns to establish a time-consistent velocity field, thereby improving the performance of motion denoising.

[0029] In a specific embodiment of the present invention, the action consistency diffusion strategy model only involves the action denoising branch in the reasoning process, and the corresponding action generation step includes: Multimodal feature extraction. At any time, for the input multi-view RGB-D image, text task instruction and robot arm self-perception action, the current scene feature is obtained through the multimodal feature alignment module of the respective feature encoder and action denoising branch. ; Action iterative denoising process. The action consistency denoising module is based on the current scene characteristics. and noise action , output noise prediction ; According to the noise prediction, the new noise action can be corrected Repeating the above process can completely remove the noise in the noisy action and obtain the next action prediction of the robot arm; Termination Condition. The algorithm in this embodiment of the present invention uses an error threshold as the key termination condition in the action iterative denoising algorithm. The error (such as mean square error or absolute error) between the current iteration result and the previous result is calculated and compared with a preset threshold to determine whether to continue iteration. If the error is less than the threshold, the denoising process is considered to have converged and the iteration is terminated. In this embodiment, an iteration threshold of 0.001 is used.

[0030] See also Figure 5As shown in Table 1, the technical solution disclosed in the embodiment of the present invention improves the accuracy of robot manipulation: experiments show that the technical solution of the embodiment of the present invention performs well in 18 common RLBench simulation benchmark test tasks, and achieves the highest average task success rate while greatly accelerating the speed of action generation; it also achieves a higher task test success rate in 6 real-machine tasks. The technical solution disclosed in the embodiment of the present invention also speeds up the generation speed of robot actions: experimental results show that the technical solution of the present invention can maintain the accuracy of generated actions within 10 time steps, avoiding the need for longer iteration steps in existing methods. In summary, the technology of the present invention is suitable for scenarios such as high-precision operation, real-time control and multi-task learning of industrial robots, and has broad application prospects and economic benefits.

[0031] Table 1. Comparison results of different methods on 18 benchmark tasks of the RLBench dataset

[0032] Specifically, the 18 tasks listed in Table 1 cover a wide range of robot motion types and task requirements, enabling a comprehensive evaluation of the performance of robot motion strategies. The average success rate and variance for each task are used to generate the average success rate and average performance ranking for these 18 tasks. Our method achieves state-of-the-art performance compared to existing techniques, with the highest average success rate of 82.6% and the best average ranking of 1.9. Specifically, our method achieves the best performance in eight tasks and second-best performance in six.

[0033] In summary, the embodiment of the present invention discloses a method and system for generating motion diffusion based on temporal consistency and successful motion perception; wherein, the motion diffusion generation method includes the following reasoning steps: the current scene image from multiple camera perspectives, the text command describing the robot task, and the current posture of the robot arm self-perception are input into the robot arm motion planning framework in real time; the input data are processed respectively using a visual encoder, a text encoder, and a rotation position encoder to obtain a multimodal task-perceived scene representation; the scene representation is input into a diffusion strategy model with motion consistency, and the final robot arm motion planning is obtained through iterative motion denoising. It should be noted that the present invention proposes a method for training by action, which enables the identification of successful actions. By training by action, the diffusion strategy model learns and establishes a time-consistent velocity field, thereby improving the performance of motion denoising. The robot arm motion planning method and system provided by the embodiment of the present invention can simultaneously take into account the accuracy and efficiency of motion planning in many complex scenarios.

[0034] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0035] See also Figure 6 In an embodiment of the present invention, a robot motion generation system is provided, comprising: The data acquisition module is used to obtain task instructions and the current proprioception of the robot to be operated, multi-view RGB-D images, and initialized current noise actions; The action generation module uses the trained action consistency diffusion strategy model to perform iterative action denoising based on the acquired task instructions, the robot's current proprioception, multi-view RGB-D images, and the initialized current noise action to generate the next action of the robot to be operated; Among them, the steps of iterative motion denoising include: first, based on the task instructions and the current proprioception and multi-view RGB-D images of the robot to be operated, the multimodal feature extraction module in the trained motion consistency diffusion strategy model is used to obtain the multimodal scene coding features; then, the motion denoising branch network in the trained motion consistency diffusion strategy model is used to output the noise prediction based on the multimodal scene coding features and the initialized current noise motion, and the current noise motion is corrected and updated according to the noise prediction, and this process is repeated until the noise in the noise motion is completely eliminated to obtain the action of the robot's next step.

[0036] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, the computer program including program instructions, and the processor is configured to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor is the computing core and control core of the terminal and is adapted to implement one or more instructions, specifically, to load and execute one or more instructions in the computer storage medium to implement a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention may be used to perform operations in the robot motion generation method.

[0037] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device, used to store programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed random access memory (RAM) or a non-volatile memory, such as at least one disk drive. The processor may load and execute the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the robot motion generation method described in the above embodiment.

[0038] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) containing computer-usable program code.

[0039] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0040] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0041] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A robot motion generation method, characterized in that: The following steps are involved: Obtain task instructions and the current proprioception of the robot to be operated, multi-view RGB-D images, and initialized current noise actions; Using the trained action consistency diffusion strategy model, iterative action denoising is performed based on the acquired task instructions, the robot's current proprioception, multi-view RGB-D images, and the initialized current noise action to generate the next action of the robot. Among them, the steps of iterative motion denoising include: first, based on the task instructions and the current proprioception and multi-view RGB-D images of the robot to be operated, the multimodal feature extraction module in the trained motion consistency diffusion strategy model is used to obtain the multimodal scene coding features; then, the motion denoising branch network in the trained motion consistency diffusion strategy model is used to output the noise prediction based on the multimodal scene coding features and the initialized current noise motion, and the current noise motion is corrected and updated according to the noise prediction, and this process is repeated until the noise in the noise motion is completely eliminated to obtain the action of the robot's next step.

2. A robot motion generation method according to claim 1, characterized in that: In the step of updating the current noise action according to the noise prediction correction, the current noise action is updated in the following manner: at any iterative denoising round t, the current noise action is ; Noise action input action consistency diffusion strategy model and obtain noise prediction ; Update the current noise action for the next denoising round ; Among them, the initial iterative denoising round , the initialized current noise action is .

3. A robot motion generation method according to claim 1, characterized in that: The process of repeating until the noise in the noisy action is completely eliminated and the action of the next step of the robot to be operated is obtained adopts an error threshold method as a key termination condition in the action iterative denoising algorithm; wherein, the error between the current iteration result and the previous iteration result is calculated and compared with a preset threshold to determine whether to continue iteration; if the error is less than the preset threshold, it is considered that the denoising process has converged and the iteration is stopped.

4. A robot motion generation method according to claim 1, characterized in that: The action consistency diffusion strategy model includes: A multimodal feature extraction module for extracting multimodal scene coding features; wherein the multimodal feature extraction module includes: an image coding module for encoding image features of multi-view RGB-D images; a text coding module for encoding text features of task instructions; and an action coding module for encoding prior features of the robot's proprioception; An action recognition branch network for performing action rationality recognition; wherein the action recognition branch network includes: a first multimodal feature alignment module for mapping the multimodal scene encoding features extracted by the multimodal feature extraction module to the same feature semantic space and performing feature fusion to obtain a multimodal fusion feature of the scene; and an action rationality recognition module for determining whether an input action is a successful action based on the multimodal fusion feature of the scene; An action denoising branch network is used to remove noise; wherein the action denoising branch network includes: a second multimodal feature alignment module for mapping the multimodal scene encoding features extracted by the multimodal feature extraction module to the same feature semantic space, and performing feature fusion to obtain the multimodal fusion features of the scene; and an action consistency denoising module for judging the noise components in the input action based on the multimodal fusion features of the scene.

5. A robot motion generation method according to claim 4, characterized in that: The network structures of the first multimodal feature alignment module and the second multimodal feature alignment module are the same.

6. A robot motion generation method according to claim 4, characterized in that: The training steps of the action consistency diffusion strategy model include: Obtaining a training sample data set; wherein each training sample in the training sample data set includes a sample task instruction, proprioception and multi-view RGB-D images at the sample moment, and an action pose label for the next step at the sample moment; First, the sample task instructions, the proprioception at the sample moment, and the multi-view RGB-D image are encoded into feature vectors of the same length through the multimodal feature extraction module, and then noise is added to the action pose label of the next step at the sample moment. , get the noise action; Perform action recognition branch network training; the action recognition branch network predicts the discrimination score of noise action based on the input multimodal fusion features and noise action ; The minimum distance between the discriminant score label and the successful action The calculation result is: , It is used to adjust the hyperparameter of the successful action range; the discriminative loss Gradient feedback is used to optimize the parameters of the action recognition branch network; Perform motion denoising branch network training; according to the input multimodal fusion features and noise action, give the noise prediction of the noise action ; Noise label when sampling known noise action , the training weight of the noise prediction is calculated using the prediction score of the action recognition branch network , the calculation expression is: , is the sign function; the loss function based on noise prediction To calculate the loss and update the parameters, .

7. A robot motion generation method according to claim 6, characterized in that: In the training step of the action consistency diffusion strategy model, the exponential moving average is used to smooth the model weights.

8. A robot motion generation system, characterized in that: include: The data acquisition module is used to obtain task instructions and the current proprioception of the robot to be operated, multi-view RGB-D images, and initialized current noise actions; The action generation module uses the trained action consistency diffusion strategy model to perform iterative action denoising based on the acquired task instructions, the robot's current proprioception, multi-view RGB-D images, and the initialized current noise action to generate the next action of the robot to be operated; Among them, the steps of iterative motion denoising include: first, based on the task instructions and the current proprioception and multi-view RGB-D images of the robot to be operated, the multimodal feature extraction module in the trained motion consistency diffusion strategy model is used to obtain the multimodal scene coding features; then, the motion denoising branch network in the trained motion consistency diffusion strategy model is used to output the noise prediction based on the multimodal scene coding features and the initialized current noise motion, and the current noise motion is corrected and updated according to the noise prediction, and this process is repeated until the noise in the noise motion is completely eliminated to obtain the action of the robot's next step.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the robot motion generation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot motion generation method according to any one of claims 1 to 7 is implemented.

Images