A method and system for robot operation based on waypoint-aware world model

Abstract

The application relates to the field of robot operation and model algorithm design of deep learning, in particular to a robot operation method and system based on a waypoint perception world model, wherein operation information of a robot is collected first, including observation images, current state information of the robot and instructions given by a user, the operation information is input into a preset waypoint perception world model, the model generates path point information of the robot based on the input, the path point information and the state information of the robot are calculated through a preset action prediction module to obtain action information of the robot, the calculated action information of the robot and the path point information are combined, a complete action trajectory of the robot is generated and output, and the waypoint perception world model in the application captures key information and maintains good generalization ability in a changing scene.

Description

Technical Field

[0001] This invention relates to the fields of robot operation and deep learning model algorithm design, and more specifically, to a robot operation method and system based on a waypoint-aware world model. Background Technology

[0002] Language-guided robot manipulation is a long-standing research area within the field of artificial intelligence. In recent years, it has gained widespread research attention due to its flexibility and practicality in facilitating human-computer interaction. Numerous benchmarks have been established for this purpose, such as RLBench, CALVIN, and VLMBench. Early methods improved manipulation performance by introducing robust representations, carefully designed network architectures, or efficient training mechanisms. With the rapid development of visual-language models (VLMs), recent research has attempted to incorporate VLMs to improve manipulation accuracy and generalization to unseen scenes / objects. However, most previous methods tended to learn mappings from multimodal inputs directly to low-level actions, neglecting explicit modeling of environmental dynamics. This can lead to failures in executing actions or planning in complex environments, resulting in poor generalization capabilities.

[0003] World models aim to generate predictive models of their surrounding environment, taking into account uncertainty and dynamic changes. They have been extensively studied in fields such as video generation, navigation, and autonomous driving. For example, Genie introduced a spatiotemporal video segmenter and dynamic models to predict the next video frame in an autoregressive manner; DriveDreamer constructed world models from real driving scenarios to achieve reasonable driving strategy generation. Some studies have also introduced world models for robot manipulation tasks, such as Daydreamer and SWIM. However, these models typically perform world modeling and decision-making alternately, which poses significant challenges to model training and is inefficient. Summary of the Invention

[0004] To overcome at least one of the defects described in the prior art, the present invention provides a robot operation method and system based on a waypoint-aware world model.

[0005] The present invention aims to solve the above-mentioned technical problems to at least some extent.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A robot operation method based on a waypoint-aware world model includes the following steps:

[0008] S1: Obtain robot operation information, which includes observed images, robot status information, and user instructions;

[0009] S2: Input the observed images and user commands into the preset waypoint-aware world model, and the waypoint-aware world model obtains the robot's waypoint information;

[0010] S3: Obtain robot motion information through a preset motion prediction module based on robot path point information and robot state information;

[0011] S4: Combine the robot motion information and robot path point information to obtain the robot's motion trajectory.

[0012] Furthermore, the waypoint-aware world model includes a pre-trained visual language model and a scene prediction module, which are connected.

[0013] Further, in step S2, the observed image and user command are input into a preset waypoint-aware world model, and the waypoint-aware world model obtains robot waypoint information, including:

[0014] S2.1: Combine the robot's current observed image and the user's command as the first input, and input the first input into the pre-trained visual language model to obtain the robot's waypoint prompts;

[0015] S2.2: Encode the robot's path point prompts into text to obtain encoded path point prompts;

[0016] S2.3: Use the robot's historical observation images as the second input after image encoding;

[0017] S2.4: Input the encoded path point prompts and the second input into the scene prediction module to obtain robot path point information.

[0018] Furthermore, in step S3, the robot's motion information is obtained through a preset motion prediction module based on the robot path point information and robot state information, including:

[0019] The robot's state information is input into a preset multilayer perceptron to obtain an output result. The output result is combined with the second input as a third input and input into a preset motion prediction module. The motion prediction module then obtains the robot's motion information.

[0020] Furthermore, before the first and second inputs are input to the waypoint-aware world model or action prediction module, they also pass through the asynchronous hierarchical actuator.

[0021] Furthermore, the scene prediction module includes a first self-attention module, a first cross-attention module, and a first feedforward network. The first self-attention module, the first cross-attention module, and the first feedforward network are connected in sequence. The path point features of the historical observation image are input into the first self-attention module to obtain the output of the first self-attention module. The path point cues and the output of the first self-attention module are input into the first cross-attention module to obtain the output of the first cross-attention module. The output of the first cross-attention module is input into the first feedforward network to obtain the robot path point information.

[0022] Furthermore, the action prediction module includes a second self-attention module, a second cross-attention module, and a second feedforward network. The second self-attention module, the second cross-attention module, and the second feedforward network are connected in sequence. The robot operation information processed by the multilayer perceptron and the path point features of the historical observation images are input into the second self-attention module to obtain the output of the second self-attention module. The output of the second self-attention module and the path point features are input into the second cross-attention module to obtain the output of the second cross-attention module. The output of the second cross-attention module is input into the second feedforward network to obtain the action trajectory of the operating robot.

[0023] Furthermore, the asynchronous hierarchical executor executes the input information and user instruction parsing of the visual language model, scene prediction module, and action prediction module respectively. During the execution process, the asynchronous hierarchical executor returns the execution results of the previous round of user instruction parsing, scene prediction module, and action prediction module. After the execution is completed, the asynchronous hierarchical executor returns the execution results of the current user instruction parsing, scene prediction module, and action prediction module.

[0024] Furthermore, in step S4, the motion trajectory of the operating robot includes:

[0025] Tra={l,[O1,S1,A1,M1],...,[O T ,S T A T M T ]}

[0026] In the formula, Tra represents the motion trajectory of the robot, l represents the user command, and O1,...,O T For historical observation images, S1,...,S T For robot state information, A1,...,A T For the robot's motion information, M1,...,M T This represents the robot's pathpoint information, where T is the time step length.

[0027] A robot operating system based on a waypoint-aware world model, applied to the aforementioned robot operation method, includes:

[0028] The robot operation information acquisition module is used to acquire robot operation information, which includes observed images, robot status information, and user commands.

[0029] The robot waypoint information acquisition module is used to input the observed images and user commands into a preset waypoint perception world model, and the waypoint perception world model obtains the robot waypoint information.

[0030] The robot motion information acquisition module is used to obtain robot motion information based on robot path point information and robot state information through a preset motion prediction module.

[0031] The robot motion trajectory acquisition module is used to merge the robot motion information and robot path point information to obtain the motion trajectory of the operating robot.

[0032] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0033] This invention first collects robot operation information, including observed images, the robot's current state information, and user-given commands. This operation information is input into a pre-defined waypoint-aware world model to obtain the robot's waypoint information. The waypoint information and the robot's state information are then used by a pre-defined motion prediction module to calculate the robot's required actions. Finally, the calculated robot motion information is merged with the waypoint information to generate and output the complete motion trajectory of the operated robot. This invention, based on the waypoint-aware world model, significantly improves the robot's control capabilities, enabling a better understanding and prediction of dynamic environments. Attached Figure Description

[0034] Figure 1 This is a flowchart of the robot operation method described in this invention;

[0035] Figure 2 This is a structural diagram of the scene prediction module described in this invention;

[0036] Figure 3 This is a structural diagram of the motion prediction module described in this invention;

[0037] Figure 4 This is a structural diagram of the waypoint-aware world model described in this embodiment;

[0038] Figure 5 This is a schematic diagram of the robot operating system described in this invention. Detailed Implementation

[0039] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0040] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0041] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] Example 1

[0044] A robot manipulation method based on a waypoint-aware world model, such as Figure 1 As shown, it includes the following steps:

[0045] S1: Obtain robot operation information, which includes observed images, robot status information, and user instructions;

[0046] S2: Input the observed images and user commands into the preset waypoint-aware world model, and the waypoint-aware world model obtains the robot's waypoint information;

[0047] S3: Obtain robot motion information through a preset motion prediction module based on robot path point information and robot state information;

[0048] S4: Combine the robot motion information and robot path point information to obtain the robot's motion trajectory.

[0049] This invention first collects robot operation information, including observed images, the robot's current state information, and user-given commands. This operation information is input into a pre-defined waypoint-aware world model to obtain the robot's waypoint information. The waypoint information and the robot's state information are then used by a pre-defined motion prediction module to calculate the robot's required actions. Finally, the calculated robot motion information is merged with the waypoint information to generate and output the complete motion trajectory of the operated robot. This invention, based on the waypoint-aware world model, significantly improves the robot's control capabilities, enabling a better understanding and prediction of dynamic environments.

[0050] Example 2

[0051] This embodiment, based on Embodiment 1, continues to disclose the following content:

[0052] like Figure 4 As shown, the waypoint-aware world model includes a pre-trained visual language model and a scene prediction module, which are connected.

[0053] Step S2, the observed image and user command are input into a preset waypoint-aware world model, and the waypoint-aware world model obtains robot waypoint information, including;

[0054] S2.1: Combine the robot's current observed image and the user's command as the first input, and input the first input into the pre-trained visual language model to obtain the robot's waypoint prompts;

[0055] S2.2: Encode the robot's path point prompts into text to obtain encoded path point prompts;

[0056] S2.3: Use the robot's historical observation images as the second input after image encoding;

[0057] S2.4: Input the encoded path point prompts and the second input into the scene prediction module to obtain robot path point information.

[0058] The pathpoint-aware world model includes a powerful visual language model and a scene prediction module. Given user instructions, the visual language model parses the user instructions to provide task-related pathpoint cues. These cues guide the scene prediction module to predict critical pathpoints. At each time step, the cues are combined with the user instructions and the robot's observed image as input to the pre-trained visual language model to perform primitive motion parsing related to the manipulation task. The parsed primitive motions and primitive user instructions are combined to form the pathpoint indications for the scene prediction module.

[0059] In step S3, the robot's motion information is obtained through a preset motion prediction module based on the robot path point information and robot state information, including:

[0060] The robot's state information is input into a preset multilayer perceptron to obtain an output result. The output result is combined with the second input as a third input and input into a preset motion prediction module. The motion prediction module then obtains the robot's motion information.

[0061] Before the first and second inputs are input to the waypoint-aware world model or action prediction module, they also pass through the asynchronous hierarchical actuator.

[0062] Example 3

[0063] Based on Examples 1 and 2, this embodiment continues to disclose the following content:

[0064] like Figure 2As shown, the scene prediction module includes a first self-attention module, a first cross-attention module, and a first feedforward network. The first self-attention module, the first cross-attention module, and the first feedforward network are connected in sequence. The path point features of the historical observation image are input into the first self-attention module to obtain the output of the first self-attention module. The path point cues and the output of the first self-attention module are input into the first cross-attention module to obtain the output of the first cross-attention module. The output of the first cross-attention module is input into the first feedforward network to obtain the robot path point information.

[0065] For the scene prediction module, the path point cues related to the robot's manipulation task and the robot's historical observation images are used as input to predict the path point features of the robot's manipulation. The scene prediction module consists of 12 stacked TransformerDecoders.

[0066] like Figure 3 As shown, the action prediction module includes a second self-attention module, a second cross-attention module, and a second feedforward network. The second self-attention module, the second cross-attention module, and the second feedforward network are connected in sequence. The robot operation information processed by the multilayer perceptron and the path point features of the historical observation images are input into the second self-attention module to obtain the output of the second self-attention module. The output of the second self-attention module and the path point features are input into the second cross-attention module to obtain the output of the second cross-attention module. The output of the second cross-attention module is input into the second feedforward network to obtain the action trajectory of the operating robot.

[0067] In the specific implementation process, the action prediction module consists of three key components: the self-attention module, the cross-attention module, and the feedforward network. These components are interconnected from top to bottom, working together to extract features, capture dependencies, and generate prediction results from the input data. Within the action prediction module, the self-attention module first captures dependencies within the sequence; then, the cross-attention module enables information interaction and fusion across sequences or modalities; finally, the feedforward network performs nonlinear transformations and feature extraction on the fused information to generate the final prediction result.

[0068] For the action prediction module, the robot's control path point state features predicted by the scene prediction module are used as prompts, and the robot's historical observation images and robot state are used as inputs to predict the actions the robot will take at each time step. The action prediction module consists of three stacked Transformer Decoders.

[0069] Example 4

[0070] Based on Examples 1, 2, and 3, this embodiment continues to disclose the following content:

[0071] The asynchronous hierarchical executor separately executes the input information of the vision language model, the scene prediction module, and the action prediction module, as well as the parsing of user instructions. During the execution process, the asynchronous hierarchical executor returns the execution results of the previous round of user instruction parsing, the scene prediction module, and the action prediction module. After the execution ends, the asynchronous hierarchical executor returns the execution results of the current user instruction parsing, the scene prediction module, and the action prediction module.

[0072] The proposed asynchronous hierarchical executor significantly improves the execution efficiency of the model by setting different frequencies for different modules. It adopts an asynchronous hierarchical execution mode to separately execute the original action parsing, scene prediction, and action prediction. Different execution frequencies are used for these three parts as needed. For the original action parsing that requires a large amount of computing power, a lower execution frequency v1 is used. For the action prediction module, a higher execution frequency v3 is adopted. These three execution frequencies follow the following relationship: v1 < v2 < v3, where v2 is the execution frequency of the scene prediction module. If a module has not finished processing a new request, the asynchronous hierarchical executor will first return the previous results.

[0073] In the specific implementation process, the original actions input to the scene prediction module may be the same at different time steps. By using low-frequency original action parsing, the model avoids redundancy caused by alternately using the vision language model and the waypoint-aware world model, thereby improving the training and inference efficiency.

[0074] In step S4, the action trajectory of the operating robot includes:

[0075] Tra = {l, [O1, S1, A1, M1],..., [O T , S T , A T , M T}

[0076] In the formula, Tra is the action trajectory of the operating robot, l is the user instruction, O1,..., O T are the historical observation images, S1,..., S T [[ID=2,7]]are the robot state information, A1,..., A T are the robot action information, M1,..., M T are the robot waypoint information, and T is the time step length.

[0077] Embodiment 5

[0078] Based on Embodiments 1, 2, 3, and 4, this embodiment further discloses the following content:

[0079] A robot operating system based on a waypoint-aware world model, such as Figure 5As shown, the robot operation method includes:

[0080] The robot operation information acquisition module is used to acquire robot operation information, which includes observed images, robot status information, and user commands.

[0081] The robot waypoint information acquisition module is used to input the observed images and user commands into a preset waypoint perception world model, and the waypoint perception world model obtains the robot waypoint information.

[0082] The robot motion information acquisition module is used to obtain robot motion information based on robot path point information and robot state information through a preset motion prediction module.

[0083] The robot motion trajectory acquisition module is used to merge the robot motion information and robot path point information to obtain the motion trajectory of the operating robot.

[0084] In practical implementation, a trainable robot control model π combines the user's command l with a series of observed images O from time step th to the current time step t. t-h:t and robot state S t-h:t Mapping to Action A t The .h file represents the length of the history frames, which is set to 3. In addition, a scene prediction module is introduced to help the model build world knowledge.

[0085] π(VLM(l,O t ),O t-h:t ,S t-h:t )→M′ t ,A′ t

[0086] Where M' t ,A' t These are the path points and actions predicted by the model for robot control at time step t.

[0087] The training loss of PIVOT-R mainly includes the scene prediction loss L. scene And action prediction loss L act For scene prediction loss L scene Calculate the average L2 distance of features between the predicted pathpoint state M' and the true pathpoint state M, where M is encoded using a pre-trained CLIP image encoder. For the action prediction loss L... act Cross-entropy loss is used to calculate the loss between the predicted action A' and the actual action A. The total loss of PIVOT-R is L = L scene +L act .

[0088] The effects of the present invention are shown in Tables 1 and 2:

[0089] Table 1

[0090]

[0091] Experiments were conducted on four levels of tasks on the open-source benchmark SeaWave. The results are shown in Table 1. PIVOT-R achieved significant improvements across all tasks. Specifically, PIVOT-R's average success rate was 74.19%, 19.45% higher than the best baseline. Both maneuverability and instruction comprehension were greatly improved. This confirms the effectiveness of the proposed method.

[0092] Table 2

[0093]

[0094] Experiments were also conducted in different, unseen scenarios. These new scenarios included unseen backgrounds, varying lighting intensities, and more obstructions. The results are shown in Table 2. PIVOT-R maintained a significantly higher success rate than other models, indicating that the model captured key information and maintained good generalization ability in changing scenarios.

[0095] This demonstrates that waypoint world modeling enables models to benefit from dynamic environment modeling. It shows that modeling the world manipulated by the robot at key waypoints significantly improves the model's maneuverability, not just performance, but also its ability to better understand and predict dynamic environments.

[0096] The same or similar labels correspond to the same or similar parts;

[0097] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0098] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A robot operation method based on a waypoint-aware world model, characterized by, The method comprises the following steps: S1: obtaining robot operation information, the robot operation information comprising observation images, robot state information and user instructions; S2: inputting the observation images and the user instructions into a preset waypoint perception world model, the waypoint perception world model obtaining robot waypoint information, specifically comprising: S2.1: merging the current observation images of the robot and the user instructions as a first input, inputting the first input into a pre-trained visual language model to obtain a waypoint prompt of the robot; S2.2: text encoding the waypoint prompt of the robot to obtain an encoded waypoint prompt; S2.3: image encoding the historical observation images of the robot as a second input; S2.4: inputting the encoded waypoint prompt and the second input into a scene prediction module to obtain robot waypoint information; S3: obtaining robot action information through a preset action prediction module according to the robot waypoint information and the robot state information; S4: merging the robot action information and the robot waypoint information to obtain an action trajectory for operating the robot. 2.The method according to claim 1, wherein, The waypoint perception world model comprises a pre-trained visual language model and a scene prediction module, and the pre-trained visual language model and the scene prediction module are connected. 3.The method according to claim 1, wherein, In step S3, the robot action information is obtained through a preset action prediction module according to the robot waypoint information and the robot state information, comprising: inputting the robot state information into a preset multi-layer perception machine to obtain an output result, merging the output result and the second input as a third input and inputting the third input into a preset action prediction module, and obtaining the robot action information through the action prediction module. 4.The method according to claim 3, wherein, The first input and the second input are input into the waypoint perception world model or the action prediction module, and are further input into an asynchronous hierarchical executor. 5.The method according to claim 1, wherein, The scene prediction module comprises a first self-attention module, a first cross-attention module and a first feedforward network, the first self-attention module, the first cross-attention module and the first feedforward network being connected in sequence, the waypoint feature of the historical observation images being input into the first self-attention module to obtain an output of the first self-attention module, the waypoint prompt and the output of the first self-attention module being input into the first cross-attention module to obtain an output of the first cross-attention module, and the output of the first cross-attention module being input into the first feedforward network to obtain the robot waypoint information. 6.The method according to claim 5, wherein, The action prediction module comprises a second self-attention module, a second cross-attention module and a second feedforward network, the second self-attention module, the second cross-attention module and the second feedforward network being connected in sequence, the robot operation information processed by the multi-layer perception machine and the waypoint feature of the historical observation images being input into the second self-attention module to obtain an output of the second self-attention module, the output of the second self-attention module and the waypoint feature being input into the second cross-attention module to obtain an output of the second cross-attention module, and the output of the second cross-attention module being input into the second feedforward network to obtain the action trajectory for operating the robot. 7.The method according to claim 4, wherein, The asynchronous layered executor executes input information and user instruction analysis of the visual language model, the scene prediction module and the action prediction module respectively, returns execution results of the user instruction analysis, the scene prediction module and the action prediction module of the last round in the execution process, and returns execution results of the current user instruction analysis, the scene prediction module and the action prediction module after the execution ends. 8.The method according to claim 6, wherein, In step S4, the action trajectory of the operating robot comprises: In the formula, Tra is the action trajectory of the operating robot, and l is a user instruction, is a historical observation image, is robot state information, is action information of the robot, is robot path point information, and T is a time step length.

9. A robot operating system based on a waypoint-aware world model, characterized in that, The robot operation method is applied to the robot operation method in any one of claims 1 to 8, comprising: A robot operation information acquisition module is configured to acquire robot operation information, wherein the robot operation information comprises observation images, robot state information and user instructions; A robot path point information acquisition module is configured to input the observation images and the user instructions into a preset path point perception world model, and obtain robot path point information from the path point perception world model; A robot action information acquisition module is configured to obtain robot action information from the robot path point information and the robot state information through a preset action prediction module; An action trajectory acquisition module of the robot is configured to combine the robot action information and the robot path point information to obtain an action trajectory of the operating robot.

Images