Robot action generation method based on three-view strategy diffusion field and related device

Abstract

The application belongs to the field of robot perception, decision and control, and discloses a robot action generation method based on a three-view strategy diffusion field and related devices; the robot action generation method comprises the following steps: acquiring a task instruction and a multi-view RGB-D image observed by a robot; using a pre-trained action generation model to generate an action to obtain a robot action distribution; the action generation model comprises the following modules: a three-view projection module, which is used for inputting the multi-view RGB-D image and performing three-view feature mapping processing to obtain three-view features; a three-view transformer module, which is used for inputting the three-view features and the task instruction and performing feature fusion processing to obtain fused three-view features; and a denoising network module, which is used for inputting the fused three-view features and performing step-by-step denoising processing to obtain the robot action distribution. The technical scheme of the application solves the technical problems of the prior art in terms of perception accuracy, calculation efficiency and action generation speed.

Description

Technical Field

[0001] This invention belongs to the field of robot perception, decision-making and control, and specifically relates to a method and related apparatus for generating robot actions based on a three-view policy diffusion field. Background Technology

[0002] Multi-task robot operation is an important research direction in the field of robotics, and it is widely used in industrial manufacturing, logistics sorting, and home services. These tasks require robots to have comprehensive capabilities in environmental perception, decision-making and planning, and precise execution.

[0003] Currently, with the increasing complexity of tasks and the diversity of scenarios, existing technical solutions still face many challenges in terms of perception, action generation, and computational efficiency, specifically manifested in the following ways:

[0004] Environmental perception has limitations: most existing robot operation technologies rely on global scene perception and lack the ability to focus on the geometric details of task-related areas. This global perception method is easily interfered with in complex or dynamic environments, making it difficult for robots to obtain key task information, thereby affecting operational accuracy.

[0005] Insufficient flexibility in action generation: Existing traditional action generation methods usually rely on discretized spatial models or implicit representations. Although they can adapt to certain single tasks, they exhibit poor adaptability when dealing with multimodal inputs or generating diverse actions.

[0006] High computational complexity: Although existing diffusion-based methods have the ability to generate continuous action distributions, they require multiple iterative denoising processes in a high-dimensional space to generate the target action. Their high computational complexity severely restricts the real-time requirements in practical applications, and their performance is particularly inadequate in tasks that require rapid response.

[0007] Insufficient multimodal information fusion capability: With the introduction of multimodal inputs (such as vision, language, touch, etc.) into robot operation, how to effectively fuse this information has become a challenge. Existing technologies usually cannot ensure both information richness and computational efficiency, resulting in unsatisfactory task decision-making results. Summary of the Invention

[0008] The purpose of this invention is to provide a robot motion generation method and related apparatus based on a three-view strategy diffusion field, so as to solve one or more technical problems existing in the prior art in terms of perception accuracy, computational efficiency and motion generation speed.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a method for generating robot actions based on a three-view policy diffusion field, comprising the following steps:

[0011] Acquire multi-view RGB-D images of task instructions and robot observations;

[0012] Based on the acquired task instructions and multi-view RGB-D images, a pre-trained motion generation model is used to generate motions and obtain the robot motion distribution.

[0013] The action generation model includes:

[0014] The three-view projection module is used to input multi-view RGB-D images and perform three-view feature mapping processing to obtain three-view features.

[0015] The three-view converter module is used to input three-view features and task instructions and perform feature fusion processing to obtain fused three-view features.

[0016] The denoising network module is used to input the fused three-view features and perform progressive denoising to obtain the robot's motion distribution.

[0017] A further improvement of the present invention is that, in the three-view projection module, the step of performing input multi-view RGB-D images and three-view feature mapping processing to obtain three-view features includes:

[0018] The 3D point cloud features are mapped onto three orthogonal 2D view feature planes, and a convolutional neural network is used to encode each 2D view feature plane to generate a high-dimensional representation, thus obtaining three-view features.

[0019] Among them, the three orthogonal two-dimensional view feature planes are the top view, side view, and front view, respectively;

[0020] The steps for generating each 2D view feature plane are as follows: reconstruct the 3D point cloud from the multi-view RGB-D image, and project the features of the 3D point cloud onto the 2D plane using max pooling. The specific formula is as follows:

[0021] V WD (x,z) = MaxPool(P(x,y,z));

[0022] V DH (y,z) = MaxPool(P(x,y,z));

[0023] V HW (x,y) = MaxaPool(P(x,y,z));

[0024] In the formula, V represents the planar feature obtained by projection; W, D, and H are the width, height, and depth of the feature plane of the two-dimensional view, respectively; MaxPool represents max pooling; and x, y, and z represent the normalized discrete three-dimensional coordinates.

[0025] The high-dimensional representation generated by encoding each two-dimensional view feature plane using a convolutional neural network is as follows:

[0026] F WD ∈R W×D×C ;

[0027] F DH ∈R D×H×C ;

[0028] F HW ∈R H×W×C ;

[0029] In the formula, F represents the encoded feature map; C represents the number of feature channels.

[0030] A further improvement of the present invention is that, in the three-view converter module, the step of executing the input three-view features and task instructions and performing feature fusion processing to obtain the fused three-view features includes:

[0031] The three-view features are concatenated with the multimodal features containing task instructions. The concatenated features are processed using the standard Transformer architecture, and layer normalization and adaptive adjustment are used to enhance the modeling ability of robot state information, thus obtaining the fused three-view features.

[0032] Specifically, a 3D rotation position code is added to the three-view features before stitching, using the following formula:

[0033] PE3D(i,j,k)=cat(PE1D(i),PE1D(j),PE1D(k));

[0034] In the formula, PE3D(·) represents the three-dimensional rotational position encoding function; i, j, and k are the spatial indices of the feature plane, respectively; cat represents vector concatenation; and PE1D(·) represents the one-dimensional rotational position encoding function.

[0035] A further improvement of the present invention is that, in the denoising network module, the step of performing input fusion three-view features and progressive denoising processing to obtain the robot motion distribution includes:

[0036] The fused three-view features are input and interpolated to obtain a latent representation of the 6-DOF action distribution at any point in space. Specifically, at any position (h, w, d) in 3D space, the corresponding features are sampled from the three-view feature plane, as shown in the formula:

[0037] c h,w,d =A(S(F) HW ,h,w),S(F DH ,d,h),S(F WD ,w,d));

[0038] In the formula, c is the obtained feature vector; A is the summation; S is the bilinear interpolation sampling function; and h, w, and d represent continuous three-dimensional coordinates.

[0039] Using the latent representation of the 6-DOF action distribution at any point as a condition, a lightweight denoising network is employed to generate the target action distribution through an iterative process, as shown in the formula:

[0040]

[0041] In the formula, a t-1 This represents a noisy 6DoF action in step t-1; a t Represents a noisy 6DoF action in t steps; ∈ θ Describing a denoising network, α t and σ t For noise intensity scheduling, t represents the denoising time step, c represents the potential diffusion field, and δ represents random noise.

[0042] A further improvement of the present invention is that the lightweight denoising network is specifically a multilayer perceptron.

[0043] A further improvement of the present invention is that, when generating the target action distribution through an iterative process, the initial action is sampled from the Gaussian noise distribution.

[0044] A further improvement of the present invention is that the training steps of the action generation model include:

[0045] Obtain a training sample set; wherein each training sample in the training sample set includes: RGB-D image samples, task instruction samples, and 6-DOF action demonstration labels;

[0046] Using the acquired training sample set, the parameters of the action generation model are updated in a supervised training manner. When the preset convergence condition is met, the trained action generation model is obtained. Specifically, for the selected training samples, RGB-D image samples and task instruction samples are used as inputs and the action generation model is used to obtain the predicted actions. The loss is calculated and the model parameters are updated based on the predicted actions and 6-DOF action demonstration labels.

[0047] The expression for the overall loss function used to calculate the loss is as follows:

[0048] L=λ1L pose +λ2L State ;

[0049] In the formula, L is the overall loss function, L pose L is used to optimize the action denoising loss of the denoising network to improve the action prediction accuracy. State λ1 and λ2 are used to predict the opening and closing states of the end effector; L is the state prediction loss. pose L State Hyperparameters for the weights of the two-part loss.

[0050] In a second aspect, the present invention provides a robot motion generation system based on a three-view policy diffusion field, comprising:

[0051] The data acquisition unit is used to acquire multi-view RGB-D images from task instructions and robot observations;

[0052] The motion generation unit is used to generate motions based on the acquired task instructions and multi-view RGB-D images, using a pre-trained motion generation model, to obtain the robot motion distribution.

[0053] The action generation model includes:

[0054] The three-view projection module is used to input multi-view RGB-D images and perform three-view feature mapping processing to obtain three-view features.

[0055] The three-view converter module is used to input three-view features and task instructions and perform feature fusion processing to obtain fused three-view features.

[0056] The denoising network module is used to input the fused three-view features and perform progressive denoising to obtain the robot's motion distribution.

[0057] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot motion generation method based on a three-view strategy diffusion field as described in any one of the first aspects of the present invention.

[0058] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the robot motion generation method based on a three-view strategy diffusion field as described in any one of the first aspects of the present invention.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] This invention provides a robot motion generation method based on a three-view strategy diffusion field, addressing the shortcomings of existing technologies in terms of perception accuracy, computational efficiency, and motion generation speed. Specifically, this invention incorporates a three-view projection module, enabling the robot to focus on task-related key areas, thereby capturing finer geometric information and enhancing operational accuracy. Subsequent experiments demonstrate that this invention significantly improves the success rate of high-precision tasks in multi-task scenarios and effectively addresses environmental interference and detail requirements in complex tasks. This invention also incorporates a small denoising network module that can directly decode target actions from the latent diffusion field, eliminating the need for multiple iterations in traditional diffusion models and optimizing motion generation efficiency. This innovation greatly shortens inference time, providing technical support for real-time operation. Finally, this invention includes a three-view converter module that effectively integrates visual, linguistic, and robot state information, ensuring efficient and reliable motion generation capabilities even in complex multimodal input scenarios and enhancing multimodal information processing capabilities.

[0061] This invention employs a hybrid explicit and implicit representation design, preserving the flexible modeling capability for continuous motion distributions while avoiding computational bottlenecks in high-dimensional spaces, thus exhibiting excellent scalability and adaptability. This invention can dynamically adjust perception and motion generation strategies according to task requirements, demonstrating outstanding performance in industrial robot assembly, home service robots, and real-time control tasks. Furthermore, through lightweight model design, this invention significantly reduces model training complexity while minimizing computational resource requirements, thereby improving deployment efficiency in practical applications. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0063] Figure 1 This is a flowchart illustrating a robot motion generation method based on a three-view strategy diffusion field, as described in an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the framework structure of the action generation model in an embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of the processing logic of the action generation model in an embodiment of the present invention;

[0066] Figure 4 This is a schematic diagram showing the comparison of the running time of the method of the present invention with that of different existing methods in an embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram of a robot motion generation system based on a three-view strategy diffusion field in an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0069] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0070] Please see Figure 1 This invention provides a robot motion generation method based on a three-view policy diffusion field, comprising the following steps:

[0071] Step 1: Acquire multi-view RGB-D images of the task instructions and robot observations;

[0072] Step 2: Based on the task instructions and multi-view RGB-D images obtained in Step 1, generate actions using a pre-trained action generation model to obtain the robot action distribution;

[0073] The action generation model includes:

[0074] The three-view projection module is used to input multi-view RGB-D images and perform three-view feature mapping processing to obtain three-view features.

[0075] The three-view converter module is used to input three-view features and task instructions and perform feature fusion processing to obtain fused three-view features.

[0076] A denoising network module is used to input the fused three-view features and perform progressive denoising processing to obtain the robot motion distribution. In a further exemplary preferred embodiment, the initial motion can be sampled from a Gaussian noise distribution during the inference stage, and the final robot motion distribution can be generated through the denoising network module. Furthermore, the generated motion can be further executed by a motion planner (such as BiRRT) to ensure that the robot can complete high-precision tasks in multi-tasking operations.

[0077] In the technical solution disclosed in this invention, the three-view projection module maps a 3D point cloud into three sets of orthogonal feature planes using a three-view projection method, achieving efficient spatial perception and enhancing detail capture in task-related areas. The three-view transformer module combines vision, language commands, and robot state perception to generate a latent diffusion field, completing deep fusion of multimodal information. The denoising network module can directly decode the 6-DOF action distribution from the latent diffusion field using a lightweight denoising network, eliminating the need for iterative denoising processes in traditional diffusion models and significantly improving inference efficiency. In summary, the technical solution of this invention decomposes a 3D point cloud into three sets of orthogonal feature planes and combines them with a lightweight denoising network to quickly generate a 6-DOF action distribution aligned with the observation space, significantly improving operational accuracy and inference efficiency. In conclusion, this invention develops a robot operation framework capable of accurate perception of task-related areas, supporting multimodal information fusion, and possessing rapid action generation capabilities, which is of great significance for improving the multi-task operation capabilities of robots and meeting the needs of complex application scenarios.

[0078] Please see Figure 2 and Figure 3 In one specific embodiment of the present invention, the three-view projection module performs the following steps to obtain three-view features by inputting multi-view RGB-D images and performing three-view feature mapping processing:

[0079] (1) Map the 3D point cloud features to three orthogonal 2D view feature planes: top view, side view, and front view; wherein each view feature plane is generated by the following steps: reconstruct the 3D point cloud from multi-view RGB-D images, the point cloud containing the geometric coordinates and other features (such as color and depth) of each point; max pool the features of the 3D point cloud onto the 2D plane, the specific formula is as follows:

[0080] V wD (x,z) = MaxPool(P(x,y,z));

[0081] V DH (y,z) = MaxPool(P(x,y,z));

[0082] V HW (x,y) = MaxPool(P(x,y,z));

[0083] In the formula, V represents the three-plane feature obtained by projection, E, D, and H are the width, height, and depth of the view feature plane, respectively, MaxPool represents max pooling, and x, y, and z represent normalized discrete three-dimensional coordinates;

[0084] This step preserves key geometric features for each plane and effectively reduces computational complexity.

[0085] To further extract deeper information from the three-view features, this embodiment of the invention uses a convolutional neural network (CNN) to encode each view feature plane, generating a high-dimensional representation:

[0086] F WD ∈R W×D×C ;

[0087] F DH ∈R D×H×C ;

[0088] F HW ∈R H×W×C ;

[0089] In the formula, F represents the encoded feature map; W, D, and H are the width, height, and depth of the view feature plane, respectively; and C represents the number of feature channels.

[0090] These high-dimensional representations provide ample geometric context information for subsequent action generation; in addition, through an efficient point cloud feature projection mechanism, the three-view features can effectively capture the geometric details of the local task area and adapt to diverse operation scenarios.

[0091] Please see Figure 2 and Figure 3 In one specific embodiment of the present invention, the three-view converter module executes the steps of inputting three-view features and task instructions and performing feature fusion processing to obtain fused three-view features, including:

[0092] The three-view features are fused with multimodal inputs (such as visual features, task instructions, and robot state information) to generate a unified multimodal representation. The specific steps are as follows: 1) RGB-D visual feature extraction: global visual features are extracted from RGB-D images using a convolutional network; 2) Language instruction encoding: language instructions are encoded into text features using a pre-trained CLIP language encoder; 3) Robot state information (such as end effector position and posture) is encoded into features using a multilayer perceptron; 4) The above multimodal features are concatenated with the three-view feature plane to form a unified input representation.

[0093] To enhance the model's understanding of three-dimensional space, this embodiment of the invention adds three-dimensional rotational position encoding (3D RoPE) to the three-view features before stitching, specifically implemented as follows:

[0094] PE3D(i,j,k)=cat(PE1D(i),PE1D(j),PE1D(k));

[0095] In the formula, i, j, and k are the spatial indices of the feature plane, PE1D(·) represents the one-dimensional rotation position encoding function, PE3D(·) represents the three-dimensional rotation position encoding function, and cat represents vector concatenation. Through three-dimensional position encoding, the model can preserve the spatial context relationship between features and improve the accuracy and flexibility of action generation.

[0096] This invention uses a three-view transformer to process the fused features. The transformer adopts the standard Transformer architecture and enhances the modeling ability of robot state information through layer normalization and adaptive layer normalization. The final output fused features provide a unified representation of multimodal data, supporting subsequent action generation.

[0097] Please see Figure 2 and Figure 3 In a specific embodiment of the present invention, the step of performing input fusion three-view features and progressive denoising processing to obtain the robot motion distribution in the denoising network module includes:

[0098] The fused three-view features are input and interpolated to obtain a latent representation of the 6-DOF action distribution at any point in space. Specifically, at any position (h, w, d) in 3D space, the corresponding features are sampled from the three-view feature plane, as shown in the formula:

[0099] c h,w,d =A(S(F) HW ,h,w),S(F DH ,d,h),S(F WD ,w,d));

[0100] In the formula, S is the bilinear interpolation sampling function, A is the summation, c is the obtained feature vector, and h, w and d represent continuous three-dimensional coordinates used to integrate the three-view features into a unified latent representation; interpretably, the latent diffusion field explicitly models the action distribution, providing accurate spatial features for subsequent action generation.

[0101] Using the latent representation of the 6-DOF action distribution at any point as a condition, a lightweight denoising network MLP (Multilayer Perceptron) is employed to generate the target action distribution through an iterative process. The specific formula is as follows:

[0102]

[0103] In the formula, a t-1 This represents a noisy 6DoF action in step t-1; a t Represents a noisy 6DoF action in t steps; ∈ θ Describes a denoising network, α t and σ tFor noise intensity scheduling, t represents the denoising time step, c represents the potential diffusion field, and δ represents random noise; this process can generate a high-precision 6-DOF motion distribution within a few time steps.

[0104] In a specific embodiment of the present invention, the network training and optimization steps include:

[0105] Obtain a training sample set, in which each training sample includes: RGB-D image samples, task instruction samples, and 6-DOF action demonstration labels;

[0106] Using the acquired training sample set, supervised training is employed to update the parameters of the action generation model until the preset convergence condition is met, thus obtaining a well-trained action generation model.

[0107] Interpretatively, during supervised training, for selected training samples, RGB-D image samples and task instruction samples are used as inputs, and the model obtains predicted actions. The loss is calculated based on the predicted actions and 6-DOF action demonstration labels, and the model parameters are updated. Specifically, for example, the model uses the AdamW optimizer with an initial learning rate of 2.5 × 10⁻⁶. -4 The total number of training steps is 30k. By sampling the time step t multiple times, the stability and performance of the model training are improved.

[0108] In this embodiment of the invention, the training objective includes the following two loss functions:

[0109] Action denoising loss is used to optimize the action prediction accuracy of the denoising network.

[0110] L pose =E ∈,t [|∈-∈ θ (a t |t,c)| 2 ];

[0111] In the formula, E represents the mathematical expectation, ∈ represents the noise of random sampling, and ∈ θ Describing a denoising network, a t This represents a noisy 6DoF action with step t, where t represents the denoising time step and c represents the potential diffusion field.

[0112] State prediction loss is used to predict the opening and closing states of the end effector.

[0113]

[0114] In the formula, BCE represents the binary cross-entropy, a open and These represent the predicted terminal state and the actual terminal state, respectively.

[0115] The final loss function is:

[0116] L=λ1L pose +λ2L State ;

[0117] In the formula, λ1 and λ2 are hyperparameters used to adjust the weights of the two parts of the loss.

[0118] In specific embodiments of this invention, to comprehensively evaluate the performance of the multi-task robot operation framework based on a three-view policy diffusion field proposed in this invention (hereinafter referred to as "this invention"), extensive experiments were conducted in the RLBench simulation environment and real-world tasks, using average success rate (ASR) and inference efficiency as evaluation metrics. The following is a detailed analysis of the experimental results:

[0119] Real-world task performance analysis: Table 1 shows the performance of this invention in six real-world tasks and compares it with existing methods. These tasks include high-precision operation tasks such as "Put Fruit," "Push Buttons," and "Stack Blocks." Experimental results show that: in the "Put Fruit" task, this invention achieved a 100% success rate; in the "Push Buttons" task, the success rate reached 100%; and in the "Stack Blocks" task, despite the complex scene and the need for high-precision operation, this invention still achieved a 70% success rate. These results demonstrate that this invention has strong generalization ability and stability when dealing with complex and diverse task scenarios in the real world.

[0120] Table 1. Results on manually collected real-world data

[0121]

[0122] Multi-task performance analysis on the RLBench dataset: Table 2 shows the comparison results of the present invention with various existing methods in 18 multi-task experiments conducted in the RLBench simulation environment. The experiments demonstrate that the present invention exhibits superior performance in multi-task scenarios: the average success rate is 87.3%, which is 5.9 percentage points higher than the existing best method (RVT-2). In the "Push Buttons" task, the present invention achieves a success rate of 98.4%, a 14.4% improvement compared to 3D DiffuserActor. In the "Stack Blocks" task, the present invention achieves a success rate of 75.2%, a 19.3 percentage point improvement compared to existing methods. These results prove that the present invention can not only efficiently handle diverse task requirements but also maintain stable performance as task complexity increases.

[0123] Table 2. Comparison of different methods on 18 common tasks on the RLBench dataset.

[0124]

[0125] Reasoning efficiency analysis: Figure 4 This paper presents a comparison of the inference efficiency of this invention with other diffusion model methods. The results show that the inference speed of this invention is approximately 10 times faster than existing diffusion model methods while maintaining a high success rate. This result fully demonstrates that the hybrid explicit and implicit representation design of this invention significantly improves inference efficiency, making it more suitable for real-time operation scenarios.

[0126] Based on the above experimental results, the following conclusions can be drawn:

[0127] Excellent performance in real-world tasks: In multiple complex task scenarios, the success rate of this invention is significantly better than that of existing methods, especially in tasks such as "placing fruit" and "pressing a button", where it reaches 100%;

[0128] Strong multi-task learning ability: On the RLBench dataset, the average success rate of this invention is 87.3%, which significantly outperforms the existing best methods and performs outstandingly in high-complexity tasks;

[0129] High inference efficiency: Compared with existing diffusion models, this invention significantly reduces inference time while maintaining high-precision motion generation capability, making it suitable for real-time robot control scenarios;

[0130] In summary, this invention, through its innovative three-view strategy diffusion field design, achieves a high-precision, multi-tasking, and high-efficiency robot operation framework with broad application prospects and economic benefits. This framework decomposes a 3D point cloud into three sets of orthogonal feature planes using three-view projection technology and employs a three-view transformer to generate a potential diffusion field aligned with the observation space, thereby efficiently representing the probability distribution of 6-DOF actions. The framework innovatively combines the advantages of explicit and implicit representations, implementing action sampling through a lightweight denoising network, simplifying the iterative denoising process of traditional diffusion models while balancing inference efficiency and operational accuracy. The main technical features of this invention include: a three-view projection module, which significantly improves spatial perception capabilities; a three-view transformer module, which integrates visual, linguistic, and robot state information; and a denoising network module, which supports action decoding at arbitrary resolutions. This invention's framework performs excellently in RLBench benchmark tests, with an average success rate improvement of 5.9% and an inference speed ten times faster than existing methods. Its design can be widely applied to multi-task learning, complex scene operation, and real-time control of industrial robots, possessing significant practical application value and economic benefits.

[0131] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0132] Please see Figure 5 In this embodiment of the invention, a robot motion generation system based on a three-view strategy diffusion field is provided, comprising:

[0133] The data acquisition unit is used to acquire multi-view RGB-D images from task instructions and robot observations;

[0134] The motion generation unit is used to generate motions based on the acquired task instructions and multi-view RGB-D images, using a pre-trained motion generation model, to obtain the robot motion distribution.

[0135] The action generation model includes:

[0136] The three-view projection module is used to input multi-view RGB-D images and perform three-view feature mapping processing to obtain three-view features.

[0137] The three-view converter module is used to input three-view features and task instructions and perform feature fusion processing to obtain fused three-view features.

[0138] The denoising network module is used to input the fused three-view features and perform progressive denoising to obtain the robot's motion distribution.

[0139] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used to execute the operation of a robot motion generation method based on a three-view strategy diffusion field.

[0140] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the robot motion generation method based on a three-view policy diffusion field in the above embodiments.

[0141] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0142] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating robot actions based on a three-view strategy diffusion field, characterized in that, Includes the following steps: Acquire multi-view RGB-D images of task instructions and robot observations; Based on the acquired task instructions and multi-view RGB-D images, a pre-trained motion generation model is used to generate motions and obtain the robot motion distribution. The action generation model includes: The three-view projection module is used to input multi-view RGB-D images and perform three-view feature mapping processing to obtain three-view features. The three-view converter module is used to input three-view features and task instructions and perform feature fusion processing to obtain fused three-view features. The denoising network module is used to input the fused three-view features and perform progressive denoising to obtain the robot motion distribution; In the three-view projection module, the steps of inputting multi-view RGB-D images and performing three-view feature mapping processing to obtain three-view features include: The 3D point cloud features are mapped onto three orthogonal 2D view feature planes, and a convolutional neural network is used to encode each 2D view feature plane to generate a high-dimensional representation, thus obtaining three-view features. Among them, the three orthogonal two-dimensional view feature planes are the top view, side view, and front view, respectively; The steps for generating each 2D view feature plane are as follows: reconstruct the 3D point cloud from the multi-view RGB-D image, and project the features of the 3D point cloud onto the 2D plane using max pooling. The specific formula is as follows: ； ； ； In the formula, V This represents the planar features obtained by projection; , , These represent the width, height, and depth of the feature plane in the two-dimensional view, respectively. MaxPool Indicates max pooling; x , y and z Represents normalized discrete three-dimensional coordinates; The high-dimensional representation generated by encoding each two-dimensional view feature plane using a convolutional neural network is as follows: ； ； ； In the formula, F This represents the encoded feature map; Indicates the number of feature channels; In the denoising network module, the steps of performing input fusion of three-view features and progressive denoising to obtain the robot motion distribution include: Inputting the fused three-view features and using interpolation, a latent representation of the 6-DOF action distribution at any point in space is obtained; where, at any position in three-dimensional space ( h , w , d The corresponding features are sampled from the three-view feature plane, and the formula is: ； In the formula, c The resulting feature vector; For summation; It is a bilinear interpolation sampling function. h , w and d Represents continuous three-dimensional coordinates; Using the latent representation of the 6-DOF action distribution at any point as a condition, a lightweight denoising network is employed to generate the target action distribution through an iterative process, as shown in the formula: ； In the formula, express t -1 step of noisy 6DoF action; express t Noisy 6DoF actions; This represents a denoising network. and For noise intensity scheduling, t Indicates the denoising time step. c Represented as a potential diffusion field, It is random noise; The lightweight denoising network is specifically a multilayer perceptron; When generating the target action distribution through an iterative process, the initial action is sampled from the Gaussian noise distribution.

2. The robot motion generation method based on a three-view strategy diffusion field according to claim 1, characterized in that, In the three-view converter module, the steps of executing input three-view features and task instructions and performing feature fusion processing to obtain fused three-view features include: The three-view features are concatenated with the multimodal features containing task instructions. The concatenated features are processed using the standard Transformer architecture, and layer normalization and adaptive adjustment are used to enhance the modeling ability of robot state information, thus obtaining the fused three-view features. Specifically, a 3D rotation position code is added to the three-view features before stitching, using the following formula: ； In the formula, PE3D ( ) represents a three-dimensional rotational position encoding function; , , These are the spatial indices of the feature planes; cat Indicates vector concatenation; PE1D( ) represents a one-dimensional rotational position encoding function.

3. The robot motion generation method based on a three-view strategy diffusion field according to claim 1, characterized in that, The training steps of the action generation model include: Obtain a training sample set; wherein each training sample in the training sample set includes: RGB-D image samples, task instruction samples, and 6-DOF action demonstration labels; Using the acquired training sample set, the parameters of the action generation model are updated in a supervised training manner. When the preset convergence condition is met, the trained action generation model is obtained. Specifically, for the selected training samples, RGB-D image samples and task instruction samples are used as inputs and the action generation model is used to obtain the predicted action. The loss is calculated and the model parameters are updated based on the predicted action and the 6-DOF action demonstration label. The expression for the overall loss function used to calculate the loss is as follows: ； In the formula, For the overall loss function, The action denoising loss is used to optimize the action prediction accuracy of the denoising network. State prediction loss used to predict the opening and closing states of the end effector; 1. 2 is used for adjustment , Hyperparameters for the weights of the two-part loss.

4. A robot motion generation system based on a three-view strategy diffusion field, characterized in that, include: The data acquisition unit is used to acquire multi-view RGB-D images from task instructions and robot observations; The motion generation unit is used to generate motions based on the acquired task instructions and multi-view RGB-D images, using a pre-trained motion generation model, to obtain the robot motion distribution. The action generation model includes: The three-view projection module is used to input multi-view RGB-D images and perform three-view feature mapping processing to obtain three-view features. The three-view converter module is used to input three-view features and task instructions and perform feature fusion processing to obtain fused three-view features. The denoising network module is used to input the fused three-view features and perform progressive denoising to obtain the robot motion distribution; In the three-view projection module, the steps of inputting multi-view RGB-D images and performing three-view feature mapping processing to obtain three-view features include: The 3D point cloud features are mapped onto three orthogonal 2D view feature planes, and a convolutional neural network is used to encode each 2D view feature plane to generate a high-dimensional representation, thus obtaining three-view features. Among them, the three orthogonal two-dimensional view feature planes are the top view, side view, and front view, respectively; The steps for generating each 2D view feature plane are as follows: reconstruct the 3D point cloud from the multi-view RGB-D image, and project the features of the 3D point cloud onto the 2D plane using max pooling. The specific formula is as follows: ； ； ； In the formula, V This represents the planar features obtained by projection; , , These represent the width, height, and depth of the feature plane in the two-dimensional view, respectively. MaxPool Indicates max pooling; x , y and z Represents normalized discrete three-dimensional coordinates; The high-dimensional representation generated by encoding each two-dimensional view feature plane using a convolutional neural network is as follows: ； ； ； In the formula, F This represents the encoded feature map; Indicates the number of feature channels; In the denoising network module, the steps of performing input fusion of three-view features and progressive denoising to obtain the robot motion distribution include: Inputting the fused three-view features and using interpolation, a latent representation of the 6-DOF action distribution at any point in space is obtained; where, at any position in three-dimensional space ( h , w , d The corresponding features are sampled from the three-view feature plane, and the formula is: ； In the formula, c The resulting feature vector; For summation; It is a bilinear interpolation sampling function. h , w and d Represents continuous three-dimensional coordinates; Using the latent representation of the 6-DOF action distribution at any point as a condition, a lightweight denoising network is employed to generate the target action distribution through an iterative process, as shown in the formula: ； In the formula, express t -1 step of noisy 6DoF action; express t Noisy 6DoF actions; This represents a denoising network. and For noise intensity scheduling, t Indicates the denoising time step. c Represented as a potential diffusion field, It is random noise; The lightweight denoising network is specifically a multilayer perceptron; When generating the target action distribution through an iterative process, the initial action is sampled from the Gaussian noise distribution.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the robot motion generation method based on a three-view policy diffusion field as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot motion generation method based on a three-view policy diffusion field as described in any one of claims 1 to 3.

Images