A robot motion generation method and related apparatus

Abstract

The application belongs to the technical field of robot operation and control, and discloses a robot action generation method and related devices; wherein the robot action generation method comprises: obtaining a task instruction and a current time body perception and multi-view RGB-D image of a robot to be operated; based on the obtained task instruction, body perception and multi-view RGB-D image, using a trained region stream matching model to generate an action, and obtaining a next time action pose of the robot to be operated; the region stream matching model comprises: a feature encoding module for obtaining multi-modal encoding features, a multi-modal information fusion module for obtaining fused features, and an action generation module for generating an action using a conditional stream matching method. The technical scheme of the application can solve the technical problems of low calculation efficiency, insufficient perception accuracy and slow action generation speed in the prior art.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robot operation and control, and particularly relates to a robot action generation method and related device. BACKGROUND

[0002] High-precision robot operation tasks have wide application prospects in industrial automation, medical surgery, home service, etc. Such tasks require robots not only to accurately perceive geometric details in the environment, but also to quickly generate precise actions in complex operation scenarios. For example, in industrial automation, robots need to accurately assemble parts to designated positions; in medical surgery, robots need to perform surgical operations with extremely high precision; in home service, robots need to flexibly complete fine operations such as grasping and placing.

[0003] Existing robot action generation schemes still face many challenges in high-precision tasks, especially in terms of perception accuracy, computational efficiency, and action generation speed. There are the following problems:

[0004] (1) Insufficient perception accuracy: Existing robot operation techniques usually rely on global scene perception, which is difficult to accurately capture the geometric details of the task-related area in complex tasks; in high-precision robot operation tasks, the lack of local geometric information often leads to operation failure;

[0005] (2) Low computational efficiency: Existing action generation methods, especially those based on diffusion models, usually need to generate actions through an iterative denoising process, resulting in low computational efficiency and difficulty in meeting the real-time robot control requirements;

[0006] (3) Slow action generation speed: Traditional action generation methods often require multiple iterations when generating complex actions, resulting in slow action generation speed and inability to adapt to the requirements of fast response in high-precision tasks. SUMMARY

[0007] The present application aims to provide a robot action generation method and related device to solve one or more of the above technical problems. The technical solution disclosed in the present application is a robot action generation method based on dynamic radius planning and conditional flow matching, which can solve the technical problems of low computational efficiency, insufficient perception accuracy, and slow action generation speed in existing robot action generation techniques in high-precision robot operation tasks.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] In a first aspect of the present application, a robot action generation method is provided, comprising the following steps:

[0010] Obtain a task instruction and an ontology perception and multi-view RGB-D image of a current moment of a robot to be operated;

[0011] Based on the obtained task instruction, ontology perception and multi-view RGB-D image, an action generation is performed by using a trained regional flow matching model to obtain an action pose of a next moment of the robot to be operated;

[0012] The regional flow matching model comprises:

[0013] A feature encoding module is configured to input the task instruction, the ontology perception and the multi-view RGB-D image, encode text features of the task instruction, encode prior features of the ontology perception, encode image features of the multi-view RGB-D image and encode region features of the multi-view RGB-D image guided by a dynamic radius scheduling, and obtain multi-modal encoding features.

[0014] A multi-modal information fusion module is configured to input the multi-modal encoding features and perform information fusion to obtain fused features.

[0015] An action generation module is configured to input the fused features and perform action generation by using a conditional flow matching method, wherein the fused features are injected into a constructed parametric velocity field as conditional information when the action generation is performed by using the conditional flow matching method.

[0016] The further improvement of the present application is that,

[0017] The training step of the regional flow matching model comprises:

[0018] A training sample data set is obtained, wherein each training sample in the training sample data set comprises a sample task instruction, an ontology perception and multi-view RGB-D image of a sample moment, and an action pose label of a next moment of the sample moment.

[0019] Based on the training sample data set and a constructed overall loss function, the parameters of the regional flow matching model are iteratively updated in a supervised training manner, and the trained regional flow matching model is obtained after a preset convergence condition is reached.

[0020] The expression of the overall loss function is: .

[0021] In the formula, is the overall loss function; is a conditional flow matching loss; is a binary cross-entropy loss of end effector state prediction; is a weight hyperparameter.

[0022] The further improvement of the present application is that,

[0023] The calculation expression of the binary cross-entropy loss of the end effector state prediction is:

[0024] ;

[0025] In the formula, is the real state of the end effector, is the predicted state.

[0026] Further improvement of the application is that,

[0027] The step of performing dynamic radius scheduling guided region feature encoding on the multi-view RGB-D image in the feature encoding module comprises:

[0028] Based on the multi-view RGB-D image, original point cloud data is constructed, the original point cloud data is sampled, and the sampled points are encoded using a point cloud encoder;

[0029] The step of sampling the original point cloud data comprises: based on the original point cloud data, randomly obtaining noise points; connecting the noise points with the real state to be reached in the next step to construct a flow matching path, and discretizing the flow matching path into a plurality of noise disturbance position points according to time steps; for each noise disturbance position point, a perception area is constructed with the noise disturbance position point as the center, and a sampling result is obtained based on the perception area mask;

[0030] The perception area mask is represented as, ; wherein, represents the noise disturbance position point of the i-th time step, represents the noise disturbance position point of the i-th time step, represents the noise disturbance position point of the i-th time step corresponding to the perception radius; is the total number of time steps; N

[0031] ;

[0032] In the formula, is the initial perception radius, is the minimum perception radius, satisfying ≤ < .

[0033] Further improvement of the application is that,

[0034] The step of performing information fusion in the multi-modal information fusion module comprises:

[0035] The multi-modal encoded features are spliced in the channel dimension, and then the spliced features are processed using a Mamba model to obtain fused features. ​

[0036] Further improvements of the present application are,

[0037] In the action generation module, the step of generating actions by using the conditional flow matching method,

[0038] Conditional flow matching maps the initial distribution to the target action distribution by learning the parameterized velocity field;

[0039] Conditional flow matching describes the data transfer process through the following ordinary differential equation:

[0040] ;

[0041] ;

[0042] In the formula, is the state variable at time step t; t is the velocity field; is the conditional information including the fused features; is the sample of the target distribution is the sample of the initial distribution ; Conditional flow matching estimates the velocity field by minimizing the following loss function :

[0043]

[0044] ;

[0045] In the formula, is the expectation.

[0046] Further improvements of the present application are,

[0047] The action generation module samples the initial action pose from the noise distribution in the inference stage, and generates the target action pose through numerical integration; wherein the pose increment in numerical integration is expressed as:

[0048] ;

[0049] Wherein, is the pose increment; is the time increment.

[0050] In the second aspect of the present application, a robot action generation system is provided, comprising:

[0051] A data acquisition module is configured to acquire task instructions and current ontology perception and multi-view RGB-D images of a robot to be operated;

[0052] ​​An action generation module is configured to generate an action based on the obtained task instruction, the body perception, and the multi-view RGB-D image, and obtain a next action pose of the robot to be operated by using a trained regional flow matching model.

[0053] The regional flow matching model comprises:

[0054] A feature encoding module is configured to input the task instruction, the body perception, and the multi-view RGB-D image, encode text features of the task instruction, encode prior features of the body perception, encode image features of the multi-view RGB-D image, and encode region features of the multi-view RGB-D image by using a dynamic radius scheduling guide, and obtain multi-modal encoded features.

[0055] A multi-modal information fusion module is configured to input the multi-modal encoded features and perform information fusion to obtain fused features.

[0056] An action generation module is configured to input the fused features and generate an action by using a conditional flow matching method, wherein the fused features are injected into a constructed parametric velocity field as conditional information when the action is generated by using the conditional flow matching method.

[0057] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot action generation method according to any one of the first aspect.

[0058] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the robot action generation method according to any one of the first aspect.

[0059] Compared with the prior art, the present application has the following beneficial effects:

[0060] The present application specifically discloses a high-precision robot operation scheme based on dynamic radius planning and conditional flow matching, which can accurately perceive the geometric details of the task-related region by introducing a dynamic radius planning mechanism, significantly improving the perception accuracy; by using the conditional flow matching technology, the action generation process is simplified, and the calculation efficiency is greatly improved; by using multi-modal information fusion, visual, geometric, language, and body perception information are efficiently integrated, the action generation speed is significantly accelerated, and the requirement for fast response in high-precision tasks is met. In summary, in the technical scheme disclosed in the present application, the perception range is gradually reduced by using the dynamic radius planning technology, and the transition from global scene understanding to fine-grained geometric details is realized, and the action pose is directly generated by using the conditional flow matching technology, which simplifies the iterative denoising process of the traditional diffusion model and significantly improves the inference speed.

[0061] Further specifically and illustratively, in the feature encoding module, adaptive region perception is achieved through dynamic radius planning; in the multi-modal information fusion module, state space model is used to efficiently integrate RGB images, point clouds and language instructions; in the action generation module, target actions are generated through parameterized velocity field according to conditional flow matching.

[0062] Further specifically and illustratively, the present application improves operation precision: through the dynamic radius planning mechanism, the robot can accurately perceive the geometric details of the task-related region, significantly improving the execution success rate of high-precision operation tasks; experiments show that the present application performs well in the RLBench benchmark test, especially in high-precision tasks, with an average success rate improvement of 12.0%, significantly faster reasoning speed, suitable for industrial robot high-precision operation, real-time control and multi-task learning scenes, with wide application prospects and economic benefits. The present application improves computational efficiency: using conditional flow matching technology, the iterative denoising process of traditional diffusion models is avoided, significantly reducing reasoning time; experimental results show that the present application can generate physically feasible actions in less than 4 time steps, with significantly faster reasoning speed than existing methods. The present application enhances multi-modal information processing capability: through the state space model, multi-modal information is effectively integrated, which can reduce computational overhead while ensuring performance, suitable for complex robot operation tasks. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description; obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0064] Figure 1 is a flowchart of a robot action generation method in an embodiment of the present application;

[0065] Figure 2 is a processing logic diagram of a region flow matching model based on dynamic radius planning and conditional flow matching in an embodiment of the present application;

[0066] Figure 3 is an exemplary principle diagram of a region flow matching model based on dynamic radius planning and conditional flow matching in an embodiment of the present application;

[0067] Figure 4 is a result diagram of a perception region generated by dynamic radius planning in a real-world robot operation task in an embodiment of the present application;

[0068] Figure 5is a schematic diagram of a robot action generation system in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiment technical solutions are part of the embodiments of the present application, and are not all the embodiments.

[0070] Based on the technical solutions disclosed in the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0071] Referring to Figures 1 to 3 The robot action generation method provided in the embodiments of the present application comprises the following steps:

[0072] Step 1, obtaining task instructions and current ontology perception and multi-view RGB-D images of a robot to be operated (explanatorily, the RGB-D image is image data combined with a red, green and blue three-channel color image and a depth image); in a specific exemplary technical solution, the task instructions can be natural language instructions; the ontology perception can be information such as joint angle pose of a robot arm;

[0073] Step 2, based on the task instructions, ontology perception and multi-view RGB-D images obtained in step 1, using a trained region stream matching model to generate actions to obtain action poses of the robot to be operated at the next time;

[0074] The region stream matching model comprises:

[0075] A feature encoding module for inputting the task instructions, ontology perception and multi-view RGB-D images, encoding text features of the task instructions, encoding prior features of the ontology perception, respectively encoding image features of the multi-view RGB-D images and region features encoded by dynamic radius scheduling guidance, and obtaining multi-modal encoding features;

[0076] A multi-modal information fusion module for inputting the multi-modal encoding features and performing information fusion to obtain fused features;

[0077] The action generation module is configured to input the fused feature and generate an action by using a conditional flow matching method, wherein the fused feature is taken as condition information and injected into a constructed parametric velocity field when the action is generated by using the conditional flow matching method.

[0078] In the embodiment of the present application, a robot action generation method based on dynamic radius planning and conditional flow matching is specifically disclosed, which introduces a dynamic radius planning mechanism (DRS) and can accurately perceive the geometric details of the task-related area, significantly improving the perception accuracy. Through the conditional flow matching (CFM) technology, the action generation process is simplified, and the calculation efficiency is greatly improved. Through the multi-modal information fusion module, visual, geometric, instruction and proprioceptive perception information are efficiently integrated, which can significantly speed up the action generation speed and meet the requirements of high-precision robot operation tasks for fast response. In summary, the embodiment of the present application designs a robot action generation method that can accurately perceive the task-related area, efficiently generate actions and has fast reasoning ability, which has important significance for solving the existing technical problems in high-precision robot operation tasks. The combination of the dynamic radius planning mechanism and the conditional flow matching technology enables the present application to flexibly cope with the requirements of different tasks and has strong adaptability, which is suitable for various high-precision operation scenes. Through the conditional flow matching technology, the learning process of action generation is simplified, the complexity of model training is reduced, and the high-performance action generation ability is maintained.

[0079] In one specific embodiment of the present application, the training step of the region flow matching model comprises:

[0080] obtaining a training sample data set; wherein each training sample in the training sample data set comprises a sample task instruction, a sample moment proprioceptive perception and a multi-view RGB-D image, and an action pose label of the next moment of the sample moment;

[0081] Based on the training sample data set and the constructed overall loss function, the parameters of the region flow matching model are iteratively updated in a supervised training manner, and after the preset convergence condition is reached, a trained region flow matching model is obtained;

[0082] wherein the overall loss function in the training process comprises two loss functions, namely a conditional flow matching loss and a binary cross-entropy loss of end effector state prediction , and the expression is:

[0083] ;

[0084] wherein, is the overall loss function in the training process; is the conditional flow matching loss; Binary cross-entropy loss for end effector state prediction Weight hyperparameters

[0085]

[0086] wherein, is the real state of the end effector, is the predicted state;

[0087] Further illustratively, when the robot is a robotic arm, may be the real state of the gripper (0 or 1), is the predicted value of the gripper state.

[0088] In a specific illustrative embodiment of the application, the AdamW optimizer is used for training, and the learning rate is set to ; during the training process, each batch contains 320 training samples, and the total number of training steps is 300K; in a further preferred technical solution, in order to accelerate the training and improve the performance of the model, the model weights are smoothed using the exponential moving average.

[0089] In a preferred solution of the embodiment of the application, the specific steps of updating the model parameters are given, and a model with good performance can be obtained.

[0090] In an embodiment of the application, the step of performing dynamic radius scheduling guided region feature encoding on the multi-view RGB-D image in the feature encoding module of the region flow matching model comprises:

[0091] Based on the multi-view RGB-D image at the current time, original point cloud data is constructed, the original point cloud data is sampled, and the sampled points are encoded using a point cloud encoder;

[0092] The step of sampling the original point cloud data comprises:

[0093] Based on the original point cloud data, noise points are randomly obtained; the noise points are connected with the real state to be reached in the next step to construct a flow matching path, and the flow matching path is discretized into a plurality of noise disturbance position points according to the time step; for each noise disturbance position point, a perception area is constructed with it as the center, and a sampling result is obtained based on the perception area mask;

[0094] Region perception mask wherein represents the noise disturbance position point at the th time step, represents the perception radius corresponding to the noise disturbance position point at the th time step;

[0095] As the time step​ increasing, the perception radius gradually decreases, following the relationship:

[0096] ;

[0097] wherein, is the initial perception radius, is the minimum perception radius; satisfies ≤ < ;

[0098] In the technical solution provided by the embodiment of the application, dynamic adjustment of regional perception is realized through dynamic radius planning, which can significantly improve the perception accuracy; wherein, the core idea of dynamic radius planning is to gradually reduce the perception range in the flow matching generation process, thereby transitioning from global scene understanding to capturing of local geometric details; this dynamic adjustment mechanism enables the application to gradually focus on the task-related area during task execution, and capture the geometric details required for high-precision tasks.

[0099] In a specific exemplary technical solution, the point cloud encoder generates geometric features wherein, is the number of sampling points, is the feature dimension. By dynamically adjusting the perception radius, the application can perform high-resolution sampling in the task-related area, thereby capturing finer geometric features.

[0100] In an embodiment of the application, the information fusion step in the multi-modal information fusion module is specifically:

[0101] The multi-modal input has been processed in the feature encoding module, including point cloud, RGB image, task instruction and robot body perception information; for the RGB image, a CLIP image encoder is used to extract global semantic features , is the number of feature points extracted from the RGB image; the task instruction is mapped to a text feature , is the number of tokens extracted from the task instruction; the robot body perception information (such as the pose of the end effector) is projected as a feature ;

[0102] The embodiment of the application introduces a state space model as a multi-modal feature fusion module, including: first, position encoding is performed on the point cloud data to capture the position information in the unordered point cloud; then, the geometric features , the RGB features , the text features and the body perception features Concatenate in the channel dimension to obtain the fused feature ; Next, the fused feature is processed using a Mamba model, which efficiently models sequential data through a Selective SSM (Selective SSM) and outputs the fused feature .

[0103] The calculation process of the Mamba model is as follows:

[0104] ;

[0105] ;

[0106] ;

[0107] wherein, is the first layer feature representation, obtained by performing layer normalization (LN) on the input feature ; is the second layer feature representation, obtained by performing linear transformation, Conv1D, SiLU activation and SSM on ; represents element-wise multiplication, represents layer normalization, is an activation function, is a state space model.

[0108] In one specific embodiment of the present application, the step of performing action generation using a conditional flow matching method in the action generation module includes:

[0109] Initialize the action pose: sample the initial action pose from the Gaussian noise distribution x 0; Explanatorily, this initial pose is random, representing the possible starting point of the action;

[0110] Construct a parameterized velocity field: in the training stage, as many coordinate points as possible are sampled to cover the entire operation space to construct a comprehensive parameterized velocity field, which describes how the action changes over time (i.e., velocity) at different positions (coordinate points).

[0111] Conditional flow matching: the fused feature is injected into the constructed parameterized velocity field as conditional information, and the conditional flow matching ensures that the velocity field can generate the desired target action according to the conditional information.

[0112] Denoising along a probability flow path: Pose increments are acquired step-by-step along a probability flow path in the velocity field. This process can be viewed as a gradual "denoising" process in the time dimension, progressively approximating the final target motion from the initial random pose. At each time step, the pose for the next time step is calculated based on the current pose and the velocity information in the velocity field. Once the denoising process is complete, the final motion pose is output. This pose conforms to the fusion feature conditions and is the target motion generated through the parameterized velocity field.

[0113] In a specific exemplary technical solution, conditional flow matching is used to generate 6-DOF action poses; conditional flow matching learns parameterized velocity fields. , distribute noise Mapping to target action distribution .

[0114] Specifically, conditional flow matching describes the data transfer process through the following ordinary differential equation:

[0115] ;

[0116] , For time steps t The state variables represent intermediate states in the process, indicating the transition from the initial distribution. To target distribution Linear interpolation; The fused features output by the Mamba model;

[0117] Conditional flow matching estimates the velocity field by minimizing the following CFM loss function. :

[0118] ;

[0119] In the formula, For the initial distribution The sample, For the target distribution The sample, As expected.

[0120] During the inference phase, this invention samples the initial action posture from the noise distribution. The target action pose is generated through numerical integration. Specifically, the reasoning process is as follows:

[0121] ;

[0122] in, It is conditional information, which may include fused multimodal features, task instructions, and robot body perception information.

[0123] The technical scheme of the embodiment of the present application generates a target action by condition flow matching, can generate a high-precision action posture in a few reasoning steps, and can significantly improve the reasoning speed. Specifically, in the reasoning phase, the present application samples an initial action posture from a noise distribution and generates a target action posture by condition flow matching. The generated target action posture is executed by a BiRRT motion planner, ensuring that the robot can accurately complete the task.

[0124] In order to comprehensively evaluate the performance of the robot operation control system (the technical scheme of the present application) based on the region perception Flow Matching strategy and the Mamba framework proposed by the present application, experiments are performed on multiple data sets and tasks, and the average success rate (Average Success Rate) is used as an evaluation index. The detailed analysis of the experimental results is as follows:

[0125] In the embodiment of the present application, the performance analysis on real-world data is as follows: Table 1 shows the results of comparison of different methods on manually collected real-world data; the experimental results show that the performance of the present application in multiple real-world tasks is significantly better than that of existing methods.

[0126] Table 1. Comparison results of different tasks based on manually collected real-world data sets

[0127]

[0128] Specifically, the success rate of the present application in the “Put Fruit in Bowl” task is 100%, which is 10 percentage points higher than that of the existing optimal method; in the “Place Earphone on Block” task, the success rate of the present application is also 100%, which is 15 percentage points higher than that of the existing method; in particular, in the “Take Skewer to Tiny Box” task, the success rate of the present application is 80%, which is 20 percentage points higher than that of the existing method; in summary, the experimental results show that the technical scheme of the present application has strong robustness and generalization ability in processing complex tasks in the real world.

[0129] Please refer to Figure 4 In the embodiment of the present application, the perception region analysis of dynamic radius planning is as follows: Figure 4 is the perception region result generated by the technical scheme of the present application in the real-world robot operation task through dynamic radius planning; the experimental results show that dynamic radius planning can effectively adjust the perception region, and gradually transition from global scene understanding to local geometric detail capture.

[0130] Specifically, in the 'Insert in Pen Cap' task, the dynamic radius planning can accurately capture the geometric details of the pen cap, ensuring that the robot can accurately complete the task; in addition, in the 'Setup Chess' task, the dynamic radius planning can dynamically adjust the perception area, ensuring that the robot can capture the accurate position of the chess pieces, thereby improving the success rate of the task; in summary, the experimental results show that the dynamic radius planning has significant advantages in handling high-precision tasks.

[0131] In the embodiment of the application, the multi-task performance analysis on the RLBench dataset is as follows: Table 2 shows the comparison results of different methods on 10 common tasks on the RLBench dataset; the experimental results show that the average success rate of the technical solution of the application is 82.3%, which is significantly better than the existing method.

[0132] Specifically, in the 'Put Buttons' task, the success rate of the application is 80.3%, which is 11.0 percentage points higher than the existing method; in particular, in the 'Stack Blocks' task, the success rate of the technical solution of the application is 31.3%, which is 19.3 percentage points higher than the existing method (3D Diffuser Actor); in summary, the experimental results show that the application has significant advantages in handling tasks that require high-precision operations.

[0133] Table 2. Comparison results of different methods on 10 benchmark tasks on the RLBench dataset

[0134]

[0135] In the embodiment of the application, the high-precision task performance analysis on the RLBench dataset is as follows: Table 3 shows the comparison results of different methods on 7 high-precision tasks on the RLBench dataset; the experimental results show that the average success rate of the technical solution of the application in these tasks is 52.0%, which is significantly better than the existing method.

[0136] Specifically, in the 'Insert USB' task, the success rate of the application is 57.3%, which is 36.0 percentage points higher than the existing optimal method (RVT-2); in the 'Screw Nail' task, the success rate of the application is 54.7%, which is 4.7 percentage points higher than the existing method; in summary, the experimental results show that the application has strong robustness and accuracy in handling high-precision tasks.

[0137] Table 3. Comparison results of different methods on 7 high-precision tasks on the RLBench dataset

[0138]

[0139] In summary, based on the above experimental results, the technical scheme provided by the embodiment of the present application has the following significant progress:

[0140] (1) Excellent performance in real-world tasks: The performance of the present application on the real-world data collected manually is significantly better than existing methods, especially in the "put fruit into bowl" and "put earphone on block" tasks, with a success rate of 100%.

[0141] (2) Effectiveness of dynamic radius planning: Dynamic radius planning can effectively adjust the perception area, gradually transition from global scene understanding to capture of local geometric details, significantly improving the success rate of high-precision tasks.

[0142] (3) Outstanding performance in multiple tasks: The present application performs well in all 10 common tasks of the RLBench benchmark, with an average success rate of 82.3%, significantly better than existing methods.

[0143] (4) Excellent performance in high-precision tasks: In the 7 high-precision tasks of RLBench, the average success rate of the present application is 52.0%, especially in the "insert USB" and "screw" tasks, with a significantly higher success rate than existing methods.

[0144] In summary, the high-precision robot operation scheme based on dynamic radius planning and conditional flow matching proposed in the embodiment of the present application performs well in multiple tasks and scenarios, and has wide application prospects.

[0145] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment, please refer to the method embodiment of the present application.

[0146] Please refer to Figure 5 In the embodiment of the present application, a robot action generation system is provided, which includes:

[0147] A data acquisition module is configured to acquire task instructions and current body perception and multi-view RGB-D images of a robot to be operated;

[0148] An action generation module is configured to generate actions based on the acquired task instructions, body perception and multi-view RGB-D images, using a trained region flow matching model, to obtain the action pose of the robot to be operated at the next time;

[0149] The region flow matching model includes:

[0150] The feature coding module is used for inputting a task instruction, ontology perception and multi-view RGB-D images, coding text features of the task instruction, coding prior features of the ontology perception, coding image features of the multi-view RGB-D images respectively and coding region features of the multi-view RGB-D images guided by a dynamic radius scheduling, and obtaining multi-modal coding features.

[0151] The multi-modal information fusion module is used for inputting the multi-modal coding features and performing information fusion, and obtaining fused features.

[0152] The action generation module is used for inputting the fused features and generating actions by using a conditional flow matching method.

[0153] In an embodiment of the present application, a computer device is provided, which comprises a processor and a memory, the memory is used for storing a computer program, the computer program comprises program instructions, and the processor is used for executing the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement a corresponding method flow or a corresponding function. The processor in the embodiment of the present application can be used to execute the operation of the robot action generation method.

[0154] In an embodiment of the present application, a storage medium, specifically a computer readable storage medium, is provided. The computer readable storage medium is a memory device in a computer device, and is configured to store programs and data. It should be understood that the computer readable storage medium can include an internal storage medium of the computer device, and can also include an external storage medium supported by the computer device. The computer readable storage medium provides a storage space, and the storage space stores an operating system of the terminal. In addition, one or more instructions adapted to be loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be understood that the computer readable storage medium can be a high-speed RAM (Random Access Memory) memory, or a non-volatile memory such as at least one disk memory. The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the robot action generation method in the above embodiment.

[0155] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. In addition, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0156] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0157] These computer program instructions can also be stored in a computer readable storage medium capable of directing the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including an instruction device, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocksFigure 1 the function(s) specified in the block or blocks.

[0158] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flowchart Figure 1 the flowchart or flowcharts and / or a block Figure 1 the steps of the function(s) specified in the block or blocks.

[0159] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the above examples, those skilled in the art should understand: the specific embodiments of the present application can still be modified or replaced by the equivalent, without departing from the spirit and scope of the present application, any modification or equivalent replacement, which should be covered within the protection scope of the claims of the present application.​

Claims

1. A method for generating robot actions, characterized in that, Includes the following steps: Acquire task instructions and the robot's current body perception and multi-view RGB-D images; Based on the acquired task instructions, ontology perception, and multi-view RGB-D images, a trained region flow matching model is used to generate actions and obtain the next moment's action pose of the robot to be operated. The regional flow matching model includes: The feature encoding module is used to input task instructions, ontology perception, and multi-view RGB-D images. It performs text feature encoding on the task instructions, prior feature encoding on the ontology perception, and image feature encoding and dynamic radius scheduling-guided region feature encoding on the multi-view RGB-D images to obtain multimodal encoded features. The multimodal information fusion module is used to input multimodal coded features and perform information fusion to obtain fused features; The action generation module is used to input the fused features and generate actions using the conditional flow matching method. When generating actions using the conditional flow matching method, the fused features are injected as conditional information into the constructed parameterized velocity field. The feature encoding module includes the following steps for performing dynamic radius scheduling-guided region feature encoding on multi-view RGB-D images: constructing original point cloud data based on the multi-view RGB-D images, sampling the original point cloud data, and encoding the sampled points using a point cloud encoder; the sampling of the original point cloud data includes: randomly acquiring noise points based on the original point cloud data; connecting the noise points with the next real state to be reached to construct a flow matching path, and discretizing the flow matching path into several noise disturbance location points according to the time step; for each noise disturbance location point, constructing a perception region centered on it, and obtaining the sampling result based on the perception region mask; wherein, the perception region mask... Represented as, ;in, Indicates the first The location of noise disturbance at each time step. Indicates the first The sensing radius corresponding to the noise disturbance location point at each time step; N This represents the total number of time steps. In the formula, It is the initial sensing radius. It is the minimum sensing radius, satisfying ≤ < ; In the action generation module, during the step of generating actions using the conditional flow matching method, the conditional flow matching maps the initial distribution to the target action distribution by learning a parameterized velocity field. The conditional flow matching describes the data transfer process using the following ordinary differential equation: ; In the formula, For time step t State variables; For velocity field; This includes conditional information that incorporates the fused features; For the target distribution ; For the initial distribution The sample; conditional flow matching estimates the velocity field by minimizing the following loss function. : In the formula, As expected.

2. The robot motion generation method according to claim 1, characterized in that, The training steps of the region flow matching model include: Obtain the training sample dataset; wherein, each training sample in the training sample dataset includes a sample task instruction, an ontology perception and multi-view RGB-D image at the sample time, and an action pose label for the next time step at the sample time. Based on the training sample dataset and the constructed overall loss function, the parameters of the regional flow matching model are iteratively updated using supervised training. After reaching the preset convergence condition, the trained regional flow matching model is obtained. The expression for the overall loss function is as follows: ; In the formula, The overall loss function; For conditional flow matching loss; Binary cross-entropy loss for end effector state prediction; For weight hyperparameters.

3. The robot motion generation method according to claim 2, characterized in that, The formula for calculating the binary cross-entropy loss of the end effector state prediction is as follows: ； In the formula, This is the actual state of the end effector. This is a predicted state.

4. The robot motion generation method according to claim 1, characterized in that, The multimodal information fusion module includes the following steps for information fusion: Multimodal encoded features are concatenated along the channel dimension, and then the concatenated features are processed using the Mamba model to obtain fused features.

5. The robot motion generation method according to claim 1, characterized in that, During the inference phase, the action generation module samples the initial action pose from the noise distribution and generates the target action pose through numerical integration; wherein, the pose increment in the numerical integration is represented as: ； in, For pose increment; This is the time increment.

6. A robot motion generation system, characterized in that, include: The data acquisition module is used to acquire task instructions and the body perception and multi-view RGB-D images of the robot to be operated at the current moment; The motion generation module is used to generate motion based on the acquired task instructions, body perception and multi-view RGB-D images, using a trained region flow matching model to obtain the next moment's motion pose of the robot to be operated. The regional flow matching model includes: The feature encoding module is used to input task instructions, ontology perception, and multi-view RGB-D images. It performs text feature encoding on the task instructions, prior feature encoding on the ontology perception, and image feature encoding and dynamic radius scheduling-guided region feature encoding on the multi-view RGB-D images to obtain multimodal encoded features. The multimodal information fusion module is used to input multimodal coded features and perform information fusion to obtain fused features; The action generation module is used to input the fused features and generate actions using the conditional flow matching method. When generating actions using the conditional flow matching method, the fused features are injected as conditional information into the constructed parameterized velocity field. The feature encoding module includes the following steps for performing dynamic radius scheduling-guided region feature encoding on multi-view RGB-D images: constructing original point cloud data based on the multi-view RGB-D images, sampling the original point cloud data, and encoding the sampled points using a point cloud encoder; the sampling of the original point cloud data includes: randomly acquiring noise points based on the original point cloud data; connecting the noise points with the next real state to be reached to construct a flow matching path, and discretizing the flow matching path into several noise disturbance location points according to the time step; for each noise disturbance location point, constructing a perception region centered on it, and obtaining the sampling result based on the perception region mask; wherein, the perception region mask... Represented as, ;in, Indicates the first The location of noise disturbance at each time step. Indicates the first The sensing radius corresponding to the noise disturbance location point at each time step; N This represents the total number of time steps. In the formula, It is the initial sensing radius. It is the minimum sensing radius, satisfying ≤ < ; In the action generation module, during the step of generating actions using the conditional flow matching method, the conditional flow matching maps the initial distribution to the target action distribution by learning a parameterized velocity field. The conditional flow matching describes the data transfer process using the following ordinary differential equation: ; In the formula, For time step t State variables; For velocity field; This includes conditional information that incorporates the fused features; For the target distribution ; For the initial distribution The sample; conditional flow matching estimates the velocity field by minimizing the following loss function. : In the formula, As expected.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the robot motion generation method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot motion generation method as described in any one of claims 1 to 5.

Images