Robot action generation method and related device

Abstract

The invention belongs to the technical field of robot operation and control, and discloses a robot action generation method and a related device. The robot action generation method comprises the following steps: acquiring a task instruction and ontology perception and multi-view RGB-D images of a to-be-operated robot at the current moment; based on the acquired task instruction, ontology perception and the multi-view RGB-D image, utilizing the trained region flow matching model to perform action generation, and acquiring the action pose of the to-be-operated robot at the next moment; the regional flow matching model comprises a feature coding module used for acquiring multi-modal coding features, a multi-modal information fusion module used for acquiring fused features, and an action generation module used for performing action generation by adopting a conditional flow matching method. According to the technical scheme, the technical problems of low calculation efficiency, insufficient sensing precision, low action generation speed and the like in the prior art can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot operation and control, and in particular relates to a robot motion generation method and related devices. Background Art

[0002] High-precision robot operation tasks have broad application prospects in industrial automation, medical surgery, home services and other fields. Such tasks require robots to not only accurately perceive the geometric details in the environment, but also quickly generate precise movements in complex operation scenarios. For example, in industrial automation, robots need to accurately assemble parts to specified positions; in medical surgery, robots need to perform surgical operations with extremely high precision; in home services, robots need to flexibly complete delicate operations such as grasping and placing.

[0003] Existing robot motion generation schemes still face many challenges in high-precision tasks, especially in terms of perception accuracy, computational efficiency, and motion generation speed. (1) Insufficient perception accuracy: Existing robot manipulation technologies usually rely on the perception of the global scene, which makes it difficult to accurately capture the geometric details of the task-related areas in complex tasks. In high-precision robot manipulation tasks, the lack of local geometric information often leads to operation failures. (2) Low computational efficiency: Existing motion generation methods, especially those based on diffusion models, usually require an iterative denoising process to generate motions, which results in low computational efficiency and makes it difficult to meet the requirements of real-time robot control. (3) The speed of motion generation is relatively slow: Traditional motion generation methods often require multiple iterations when generating complex motions, resulting in a slow motion generation speed and being unable to adapt to the requirements for fast response in high-precision tasks. Summary of the invention

[0004] The purpose of the present invention is to provide a robot motion generation method and related devices to solve one or more of the above-mentioned technical problems. The technical solution disclosed in the present invention is specifically a robot motion generation method based on dynamic radius planning and conditional flow matching, which can solve the technical problems of low computational efficiency, insufficient perception accuracy and slow motion generation speed in the existing robot motion generation technology in the robot high-precision operation tasks.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: A first aspect of the present invention provides a method for generating robot actions, comprising the following steps: Obtain task instructions and the proprioception and multi-view RGB-D images of the robot to be operated at the current moment; Based on the acquired task instructions, proprioception, and multi-view RGB-D images, the trained regional flow matching model is used to generate actions and obtain the action posture of the robot to be operated at the next moment. Wherein, the regional flow matching model includes: The feature encoding module is used to input task instructions, proprioceptive perception and multi-view RGB-D images, perform text feature encoding on the task instructions, perform prior feature encoding on the proprioceptive perception, perform image feature encoding on the multi-view RGB-D images and regional feature encoding guided by dynamic radius scheduling, and obtain multi-modal encoding features; The multimodal information fusion module is used to input multimodal coding features and perform information fusion to obtain fused features; The action generation module is used to input the fused features and use the conditional flow matching method to generate actions; when the conditional flow matching method is used to generate actions, the fused features are injected into the constructed parameterized velocity field as conditional information.

[0006] A further improvement of the present invention is that The training steps of the regional flow matching model include: Acquire a training sample data set; wherein each training sample in the training sample data set includes a sample task instruction, proprioception and multi-view RGB-D images at the sample moment, and an action posture label at the next moment of the sample moment; Based on the training sample data set and the constructed overall loss function, the parameters of the regional flow matching model are iteratively updated in a supervised training manner, and after reaching a preset convergence condition, a trained regional flow matching model is obtained; Among them, the expression of the overall loss function is: ; In the formula, is the overall loss function; is the conditional flow matching loss; Binary cross entropy loss for end-effector state prediction; is the weight hyperparameter.

[0007] A further improvement of the present invention is that The calculation expression of the binary cross entropy loss for the end effector state prediction is: ; In the formula, is the real state of the end effector, is the predicted state.

[0008] A further improvement of the present invention is that In the feature encoding module, the step of performing regional feature encoding for dynamic radius scheduling guidance on the multi-view RGB-D image includes: Construct raw point cloud data based on multi-view RGB-D images, sample the raw point cloud data, and encode the sampled points using a point cloud encoder; The step of sampling the original point cloud data includes: randomly acquiring noise points based on the original point cloud data; connecting the noise points with the real state to be reached in the next step to construct a flow matching path, and discretizing the flow matching path into a number of noise disturbance position points according to the time step; for each noise disturbance position point, constructing a perception area with it as the center, and obtaining the sampling result based on the perception area mask; Among them, the perception area mask It is expressed as, ;in, Indicates The noise perturbation position points of time steps, Indicates The perception radius corresponding to the noise disturbance position point of time steps; N is the total number of time steps; ; In the formula, is the initial perception radius, is the minimum perception radius, satisfying ≤ < .

[0009] A further improvement of the present invention is that In the multimodal information fusion module, the step of performing information fusion includes: The multimodal coding features are concatenated in the channel dimension, and then the concatenated features are processed using the Mamba model to obtain the fused features.

[0010] A further improvement of the present invention is that In the action generation module, in the step of using the conditional flow matching method to generate actions, Conditional flow matching maps the initial distribution to the target action distribution by learning a parameterized velocity field; Conditional flow matching describes the data transfer process through the following ordinary differential equation: ; ; In the formula, is the time step t state variables; is the velocity field; To include the conditional information of the fused features; The target distribution Samples of The initial distribution Samples of Conditional flow matching estimates the velocity field by minimizing the following loss function : ; In the formula, For expectation.

[0011] A further improvement of the present invention is that In the inference stage, the action generation module samples the initial action posture from the noise distribution and generates the target action posture through numerical integration; wherein the posture increment in the numerical integration is expressed as: ; in, is the posture increment; is the time increment.

[0012] A second aspect of the present invention provides a robot motion generation system, comprising: The data acquisition module is used to obtain task instructions and the proprioception and multi-view RGB-D images of the robot to be operated at the current moment; The action generation module is used to generate actions based on the acquired task instructions, proprioception, and multi-view RGB-D images using the trained regional flow matching model to obtain the action posture of the robot to be operated at the next moment; Wherein, the regional flow matching model includes: The feature encoding module is used to input task instructions, proprioceptive perception and multi-view RGB-D images, perform text feature encoding on the task instructions, perform prior feature encoding on the proprioceptive perception, perform image feature encoding on the multi-view RGB-D images and regional feature encoding guided by dynamic radius scheduling, and obtain multi-modal encoding features; The multimodal information fusion module is used to input multimodal coding features and perform information fusion to obtain fused features; The action generation module is used to input the fused features and use the conditional flow matching method to generate actions; when the conditional flow matching method is used to generate actions, the fused features are injected into the constructed parameterized velocity field as conditional information.

[0013] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the robot motion generation method as described in any one of the first aspect of the present invention is implemented.

[0014] According to a fourth aspect of the present invention, there is provided a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the robot motion generation method as described in any one of the first aspect of the present invention is implemented.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention specifically discloses a high-precision robot operation scheme based on dynamic radius planning and conditional flow matching. By introducing a dynamic radius planning mechanism, it can accurately perceive the geometric details of the task-related area and significantly improve the perception accuracy; through the conditional flow matching technology, the action generation process is simplified and the calculation efficiency is greatly improved; through multimodal information fusion, visual, geometric, language and body perception information are efficiently integrated, and the action generation speed is significantly accelerated to meet the requirements for rapid response in high-precision tasks. In summary, in the technical solution disclosed by the present invention, the perception range is gradually narrowed through the dynamic radius planning technology, and the transition from global scene understanding to fine-grained geometric details is made. Combined with the conditional flow matching technology, the action posture is directly generated, which simplifies the iterative denoising process of the traditional diffusion model and significantly improves the reasoning speed.

[0016] To explain more specifically, in the feature encoding module, adaptive regional perception is achieved through dynamic radius planning; in the multimodal information fusion module, the state space model is used to efficiently integrate RGB images, point clouds and language instructions; in the action generation module, the target action is generated through parameterized velocity field according to conditional flow matching.

[0017] To further explain in detail, the present invention improves the operation accuracy: through the dynamic radius planning mechanism, the robot can accurately perceive the geometric details of the task-related area, significantly improving the execution success rate of high-precision operation tasks; experiments show that the technical solution of the present invention performs well in the RLBench benchmark test, especially in high-precision tasks, the average success rate is improved by 12.0%, and the reasoning speed is significantly accelerated. It is suitable for scenes such as high-precision operation, real-time control and multi-task learning of industrial robots, and has broad application prospects and economic benefits. The present invention improves the computational efficiency: the conditional flow matching technology is adopted to avoid the iterative denoising process of the traditional diffusion model and significantly reduce the reasoning time; experimental results show that the technical solution of the present invention can generate physically feasible actions in less than 4 time steps, and the reasoning speed is significantly better than the existing methods. The present invention enhances the multimodal information processing capability: through the state space model, the effective fusion of multimodal information can be achieved, which can reduce the computational overhead while ensuring performance, and is suitable for complex robot operation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a flow chart of a method for generating robot actions in an embodiment of the present invention; Figure 2 It is a processing logic diagram of a regional flow matching model based on dynamic radius planning and conditional flow matching in an embodiment of the present invention; Figure 3 is an exemplary principle schematic diagram of a regional flow matching model based on dynamic radius planning and conditional flow matching in an embodiment of the present invention; Figure 4 is a result schematic diagram of a perception area generated by dynamic radius planning in a robot operation task in the real world in an embodiment of the present invention; Figure 5 It is a schematic diagram of a robot motion generation system in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0021] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] See also Figures 1 to 3 , a robot action generation method provided by an embodiment of the present invention comprises the following steps: Step 1, obtaining task instructions and proprioception and multi-view RGB-D images of the robot to be operated at the current moment (explanatory, RGB-D images are image data that combines red, green, and blue three-channel color images and depth images); in a specific exemplary technical solution, the task instructions may be natural language instructions; proprioception may be information such as the joint angles and postures of the robot arm; Step 2: Based on the task instructions, proprioception, and multi-view RGB-D images obtained in step 1, the trained regional flow matching model is used to generate actions and obtain the action posture of the robot to be operated at the next moment; Wherein, the regional flow matching model includes: The feature encoding module is used to input task instructions, proprioceptive perception and multi-view RGB-D images, perform text feature encoding on the task instructions, perform prior feature encoding on the proprioceptive perception, perform image feature encoding on the multi-view RGB-D images and regional feature encoding guided by dynamic radius scheduling, and obtain multi-modal encoding features; The multimodal information fusion module is used to input multimodal coding features and perform information fusion to obtain fused features; The action generation module is used to input the fused features and use the conditional flow matching method to generate actions; when the conditional flow matching method is used to generate actions, the fused features are injected into the constructed parameterized velocity field as conditional information.

[0023] In the embodiment of the present invention, a robot action generation method based on dynamic radius planning and conditional flow matching is specifically disclosed, which introduces a dynamic radius planning mechanism (DRS), can accurately perceive the geometric details of the task-related area, and significantly improves the perception accuracy; through the conditional flow matching (CFM) technology, the action generation process is simplified, which can greatly improve the calculation efficiency; through the multimodal information fusion module, the visual, geometric, instruction and body perception information are efficiently integrated, which can significantly speed up the action generation speed, and can meet the requirements of high-precision robot operation tasks for rapid response. In summary, the embodiment of the present invention designs a robot action generation method that can accurately perceive the task-related area, efficiently generate actions and has rapid reasoning capabilities, which is of great significance for solving the existing technical problems in high-precision robot operation tasks; among them, the combination of the dynamic radius planning mechanism and the conditional flow matching technology enables the present invention to flexibly respond to the needs of different tasks, has strong adaptability, and is suitable for a variety of high-precision operation scenarios; through the conditional flow matching technology, the learning process of action generation is simplified, the complexity of model training is reduced, and the high-performance action generation capability is maintained.

[0024] In a specific embodiment of the present invention, the training step of the regional flow matching model includes: Acquire a training sample data set; wherein each training sample in the training sample data set includes a sample task instruction, proprioception and multi-view RGB-D images at the sample moment, and an action posture label at the next moment of the sample moment; Based on the training sample data set and the constructed overall loss function, the parameters of the regional flow matching model are iteratively updated by a supervised training method, and after reaching a preset convergence condition, a trained regional flow matching model is obtained; Among them, the overall loss function in the training process includes two loss functions, namely conditional flow matching loss and the binary cross entropy loss for end-effector state prediction , the expression is: ; In the formula, is the overall loss function during training; is the conditional flow matching loss; Binary cross entropy loss for end-effector state prediction; is the weight hyperparameter; ; In the formula, is the real state of the end effector, is the predicted state; By way of further example, when the robot is a robotic arm, Can be the actual state of the gripper (0 or 1), is the predicted value of the gripper state.

[0025] In a specific exemplary embodiment of the present invention, the AdamW optimizer is used for training, and the learning rate is set to ; During the training process, each batch contains 320 training samples, and the total number of training steps is 300K; in a further preferred technical solution, in order to accelerate training and improve model performance, the exponential moving average is used to smooth the model weights.

[0026] In the preferred solution of the embodiment of the present invention, specific steps for updating model parameters are given, so as to obtain a model with better performance.

[0027] In one embodiment of the present invention, in the feature encoding module of the regional flow matching model, the step of performing regional feature encoding guided by dynamic radius scheduling on the multi-view RGB-D image includes: Construct the original point cloud data based on the multi-view RGB-D image at the current moment, sample the original point cloud data, and encode the sampled points using a point cloud encoder; The steps of sampling the original point cloud data include: Based on the original point cloud data, randomly obtain noise points; connect the noise points with the real state to be reached in the next step, construct a flow matching path, and discretize the flow matching path into a number of noise disturbance position points according to the time step; for each noise disturbance position point, construct a perception area with it as the center, and obtain a sampling result based on the perception area mask; Region-aware mask ,in Indicates The noise perturbation position points of time steps, Indicates The perception radius corresponding to the noise disturbance position point of time steps; As time goes by Increase in perception radius Gradually decreases, following the following relationship: ; in, is the initial perception radius, is the minimum perception radius; ≤ < ; In the technical solution provided by the embodiment of the present invention, dynamic adjustment of regional perception is achieved through dynamic radius planning, which can significantly improve the perception accuracy; the core idea of ​​dynamic radius planning is to gradually narrow the perception range during the flow matching generation process, thereby transitioning from global scene understanding to capturing local geometric details; this dynamic adjustment mechanism enables the present invention to gradually focus on task-related areas during task execution and capture the geometric details required for high-precision tasks.

[0028] In a specific exemplary technical solution, the point cloud encoder generates geometric features ,in is the number of sampling points, By dynamically adjusting the perception radius, the present invention can perform high-resolution sampling in the task-related area, thereby capturing finer geometric features.

[0029] In one embodiment of the present invention, in the multimodal information fusion module, the steps of performing information fusion are specifically as follows: The feature encoding module has processed multimodal inputs, including point clouds, RGB images, task instructions, and robot proprioception information; for RGB images, the CLIP image encoder is used to extract global semantic features. , is the number of feature points extracted from the RGB image; the task instructions are mapped to text features through the CLIP language encoder , is the number of tokens extracted from the task instructions; the robot proprioceptive information (such as the posture of the end effector) is projected as features ; The embodiment of the present invention introduces a state space model as a multimodal feature fusion module, including: first, position encoding the point cloud data to capture the position information in the disordered point cloud; then, the geometric features are encoded. , RGB features , text features and proprioceptive features Splice in the channel dimension to obtain the fused features Next, the fused features are processed using the Mamba model, which efficiently models sequence data through the Selective State Space Mechanism (Selective SSM) and outputs the fused features. .

[0030] The calculation process of the Mamba model is as follows: ; ; ; in, It is the first layer feature representation, through the input feature Perform layer normalization (LayerNormalization, LN) to obtain; is the second-level feature representation. It is obtained by linear transformation, one-dimensional convolution (Conv1D), SiLU activation and state-space model (SSM); represents element-wise product, Representation layer normalization, is the activation function, is a state-space model.

[0031] In a specific embodiment of the present invention, in the action generation module, the step of performing action generation using the conditional flow matching method includes: Initialize action pose: Sample the initial action pose from Gaussian noise distribution x 0 ;Explanatory, this initial pose is random and represents a possible starting point for the action; Construct a parameterized velocity field: During the training phase, sample as many coordinate points as possible across the entire action space to construct a comprehensive parameterized velocity field that describes how the action changes over time (i.e., velocity) at different locations (coordinate points).

[0032] Conditional flow matching: The fused features are injected into the constructed parameterized velocity field as conditional information. Conditional flow matching ensures that the velocity field can generate the desired target action according to the conditional information.

[0033] Denoising along the probability flow path: On a probability flow path of the velocity field, the pose increments are obtained step by time. This process can be regarded as gradual "denoising" in the time dimension, gradually approaching the final target action from the initial random pose. At each time step, the pose of the next time step is calculated based on the current pose and the velocity information in the velocity field. When the denoising process is completed, the final action pose is output. This pose meets the fusion feature conditions and is the target action generated by the parameterized velocity field.

[0034] In a specific exemplary technical solution, conditional flow matching is used to generate 6-DOF motion postures; conditional flow matching is achieved by learning a parameterized velocity field. , the noise distribution Mapping to target action distribution .

[0035] Specifically, conditional flow matching describes the data transfer process through the following ordinary differential equation: ; , is the time step t The state variable represents the intermediate state in the process, indicating the transition from the initial distribution To target distribution Linear interpolation of ; The fused features output by the Mamba model; Conditional flow matching estimates the velocity field by minimizing the following CFM loss function : ; In the formula, The initial distribution A sample of The target distribution A sample of For expectation.

[0036] In the inference stage, the present invention samples the initial action posture from the noise distribution , and generate the target action posture through numerical integration Specifically, the reasoning process is as follows: ; in, It is conditional information, which may include fused multimodal features, task instructions, and robot proprioception information.

[0037] The technical solution of the embodiment of the present invention uses conditional flow matching to generate target actions, which can generate high-precision action postures within very few reasoning steps, and can significantly improve the reasoning speed. Specifically, in the reasoning stage, the present invention samples the initial action posture from the noise distribution, and generates the target action posture through conditional flow matching. The generated target action posture is executed by the BiRRT motion planner to ensure that the robot can accurately complete the task.

[0038] In order to comprehensively evaluate the performance of the region-aware Flow Matching strategy and the Mamba framework robot operation control system (the technical solution of the present invention), the present invention conducted experiments on multiple data sets and tasks, and used the average success rate as the evaluation indicator. The detailed analysis of the experimental results is as follows: In an embodiment of the present invention, the performance analysis on real-world data is as follows: Table 1 is a schematic diagram of the results of comparing different methods on manually collected real-world data; experimental results show that the performance of the present invention in multiple real-world tasks is significantly better than existing methods.

[0039] Table 1. Comparison results between different tasks based on manually collected real-world datasets

[0040] Specifically, the success rate of the present invention in the "Put Fruit in Bowl" task is 100%, which is 10 percentage points higher than the existing optimal method; in the "Place Earphone on Block" task, the success rate of the present invention is also 100%, which is 15 percentage points higher than the existing method; in the "Take Skewer to Tiny Box" task, the success rate of the present invention is 80%, which is 20 percentage points higher than the existing method; In summary, the experimental results show that the technical solution of the present invention has strong robustness and generalization ability in dealing with complex tasks in the real world.

[0041] See also Figure 4 , the perception area of ​​the embodiment of the present invention in dynamic radius planning is analyzed as follows: Figure 4 It is the perception area result generated by the technical solution of the present invention through dynamic radius planning in the robot operation task in the real world; the experimental results show that dynamic radius planning can effectively adjust the perception area and gradually transition from global scene understanding to the capture of local geometric details.

[0042] Specifically, in the "Insert in Pen Cap" task, dynamic radius planning can accurately capture the geometric details of the pen cap, ensuring that the robot can complete the task accurately; in addition, in the "Setup Chess" task, dynamic radius planning can dynamically adjust the perception area to ensure that the robot can capture the precise position of the chess pieces, thereby improving the success rate of the task; in summary, the experimental results show that dynamic radius planning has significant advantages in handling high-precision tasks.

[0043] In the embodiment of the present invention, the multi-task performance analysis on the RLBench dataset is as follows: Table 2 is a schematic diagram of the comparison results of different methods on the RLBench dataset on 10 common tasks; the experimental results show that the average success rate of the technical solution of the present invention is 82.3%, which is significantly better than the existing methods.

[0044] Specifically, in the "Put Buttons" task, the success rate of the present invention is 80.3%, which is 11.0 percentage points higher than the existing method. In particular, in the "Stack Blocks" task, the success rate of the technical solution of the present invention is 31.3%, which is 19.3 percentage points higher than the existing method (3D Diffuser Actor). In summary, the experimental results show that the present invention has significant advantages in processing tasks that require high-precision operations.

[0045] Table 2. Comparison results of different methods on 10 benchmark tasks of RLBench dataset

[0046] In the embodiment of the present invention, the performance analysis of high-precision tasks on the RLBench dataset is as follows. Table 3 is a schematic diagram of the comparison results of different methods on the RLBench dataset on 7 high-precision tasks; the experimental results show that the average success rate of the technical solution of the present invention in these tasks is 52.0%, which is significantly better than the existing methods.

[0047] Specifically, in the "Insert USB" task, the success rate of the present invention is 57.3%, which is 36.0 percentage points higher than the existing best method (RVT-2); in the "Screw Nail" task, the success rate of the present invention is 54.7%, which is 4.7 percentage points higher than the existing method; In summary, the experimental results show that the present invention has strong robustness and accuracy when processing high-precision tasks.

[0048] Table 3. Comparison results of different methods on 7 high-precision tasks of RLBench dataset

[0049] In summary, based on the above experimental results, it can be seen that the technical solution provided by the embodiment of the present invention has the following significant improvements: (1) Excellent performance on real-world tasks: The proposed method significantly outperforms existing methods on manually collected real-world data, especially in the tasks of “placing fruit in a bowl” and “placing headphones on building blocks”, with a success rate of 100%.

[0050] (2) Effectiveness of dynamic radius planning: Dynamic radius planning can effectively adjust the perception area and gradually transition from global scene understanding to capturing local geometric details, significantly improving the success rate of high-precision tasks.

[0051] (3) Outstanding performance in multiple tasks: The proposed method performs well in all 10 common tasks of the RLBench benchmark, with an average success rate of 82.3%, which is significantly better than existing methods.

[0052] (4) Excellent performance in high-precision tasks: Among the seven high-precision tasks of RLBench, the average success rate of the proposed method is 52.0%. In particular, in the "insert USB" and "tighten the screw" tasks, the success rate is significantly higher than that of the existing methods.

[0053] In summary, the high-precision robot operation scheme based on dynamic radius planning and conditional flow matching proposed in the embodiments of the present invention performs well in multiple tasks and scenarios and has broad application prospects.

[0054] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0055] See also Figure 5 In an embodiment of the present invention, a robot action generation system is provided, comprising: The data acquisition module is used to obtain task instructions and the proprioception and multi-view RGB-D images of the robot to be operated at the current moment; The action generation module is used to generate actions based on the acquired task instructions, proprioception, and multi-view RGB-D images using the trained regional flow matching model to obtain the action posture of the robot to be operated at the next moment; Wherein, the regional flow matching model includes: The feature encoding module is used to input task instructions, proprioceptive perception and multi-view RGB-D images, perform text feature encoding on the task instructions, perform prior feature encoding on the proprioceptive perception, perform image feature encoding on the multi-view RGB-D images and regional feature encoding guided by dynamic radius scheduling, and obtain multi-modal encoding features; The multimodal information fusion module is used to input multimodal coding features and perform information fusion to obtain fused features; The action generation module is used to input the fused features and use the conditional flow matching method to generate actions; when the conditional flow matching method is used to generate actions, the fused features are injected into the constructed parameterized velocity field as conditional information.

[0056] In one embodiment of the present invention, a computer device is provided, the computer device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the robot action generation method.

[0057] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM (Random Access Memory) memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the robot action generation method in the above embodiment.

[0058] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program codes.

[0059] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0060] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A robot motion generation method, characterized in that: The following steps are involved: Obtain task instructions and the proprioception and multi-view RGB-D images of the robot to be operated at the current moment; Based on the acquired task instructions, proprioception, and multi-view RGB-D images, the trained regional flow matching model is used to generate actions and obtain the action posture of the robot to be operated at the next moment. Wherein, the regional flow matching model includes: The feature encoding module is used to input task instructions, proprioceptive perception and multi-view RGB-D images, perform text feature encoding on the task instructions, perform prior feature encoding on the proprioceptive perception, perform image feature encoding on the multi-view RGB-D images and regional feature encoding guided by dynamic radius scheduling, and obtain multi-modal encoding features; The multimodal information fusion module is used to input multimodal coding features and perform information fusion to obtain fused features; The action generation module is used to input the fused features and use the conditional flow matching method to generate actions; when the conditional flow matching method is used to generate actions, the fused features are injected into the constructed parameterized velocity field as conditional information.

2. A robot motion generation method according to claim 1, characterized in that: The training steps of the regional flow matching model include: Acquire a training sample data set; wherein each training sample in the training sample data set includes a sample task instruction, proprioception and multi-view RGB-D images at the sample moment, and an action posture label at the next moment of the sample moment; Based on the training sample data set and the constructed overall loss function, the parameters of the regional flow matching model are iteratively updated in a supervised training manner, and after reaching a preset convergence condition, a trained regional flow matching model is obtained; Among them, the expression of the overall loss function is: ; In the formula, is the overall loss function; is the conditional flow matching loss; Binary cross entropy loss for end-effector state prediction; is the weight hyperparameter.

3. A robot motion generation method according to claim 2, characterized in that: The calculation expression of the binary cross entropy loss for the end effector state prediction is: ； In the formula, is the real state of the end effector, is the predicted state.

4. A robot motion generation method according to claim 1, characterized in that: In the feature encoding module, the step of performing regional feature encoding for dynamic radius scheduling guidance on the multi-view RGB-D image includes: Construct raw point cloud data based on multi-view RGB-D images, sample the raw point cloud data, and encode the sampled points using a point cloud encoder; The step of sampling the original point cloud data includes: randomly acquiring noise points based on the original point cloud data; connecting the noise points with the real state to be reached in the next step to construct a flow matching path, and discretizing the flow matching path into a number of noise disturbance position points according to the time step; for each noise disturbance position point, constructing a perception area with it as the center, and obtaining the sampling result based on the perception area mask; Among them, the perception area mask It is expressed as, ;in, Indicates The noise perturbation position points of time steps, Indicates The perception radius corresponding to the noise disturbance position point of time steps; N is the total number of time steps; ； In the formula, is the initial perception radius, is the minimum perception radius, satisfying ≤ < .

5. A robot motion generation method according to claim 1, characterized in that: In the multimodal information fusion module, the step of performing information fusion includes: The multimodal coding features are concatenated in the channel dimension, and then the concatenated features are processed using the Mamba model to obtain the fused features.

6. A robot motion generation method according to claim 1, characterized in that: In the action generation module, in the step of using the conditional flow matching method to generate actions, Conditional flow matching maps the initial distribution to the target action distribution by learning a parameterized velocity field; Conditional flow matching describes the data transfer process through the following ordinary differential equation: ； ； In the formula, is the time step t state variables; is the velocity field; The conditional information including the fused features; The target distribution Samples of The initial distribution Samples of Conditional flow matching estimates the velocity field by minimizing the following loss function : ； In the formula, For expectation.

7. A robot motion generation method according to claim 6, characterized in that: In the inference stage, the action generation module samples the initial action posture from the noise distribution and generates the target action posture through numerical integration; wherein the posture increment in the numerical integration is expressed as: ； in, is the posture increment; is the time increment.

8. A robot motion generation system, characterized in that: include: The data acquisition module is used to obtain task instructions and the proprioception and multi-view RGB-D images of the robot to be operated at the current moment; The action generation module is used to generate actions based on the acquired task instructions, proprioception, and multi-view RGB-D images using the trained regional flow matching model to obtain the action posture of the robot to be operated at the next moment; Wherein, the regional flow matching model includes: The feature encoding module is used to input task instructions, proprioceptive perception and multi-view RGB-D images, perform text feature encoding on the task instructions, perform prior feature encoding on the proprioceptive perception, perform image feature encoding on the multi-view RGB-D images and regional feature encoding guided by dynamic radius scheduling, and obtain multi-modal encoding features; The multimodal information fusion module is used to input multimodal coding features and perform information fusion to obtain fused features; The action generation module is used to input the fused features and use the conditional flow matching method to generate actions; when the conditional flow matching method is used to generate actions, the fused features are injected into the constructed parameterized velocity field as conditional information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the robot motion generation method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot motion generation method according to any one of claims 1 to 7 is implemented.

Images