Robot operation method and system based on waypoint perception world model

Abstract

The invention relates to the field of robot operation and deep learning model algorithm design, in particular to a robot operation method and system based on a waypoint perception world model.The robot operation method comprises the steps that firstly, operation information of a robot is collected, and the operation information comprises an observation image, current state information of the robot and an instruction given by a user; the operation information is input into a preset waypoint sensing world model, the model generates waypoint information of the robot based on input, the waypoint information and state information of the robot are used for calculating motion information of the robot through a preset motion prediction module, the calculated motion information of the robot is combined with the waypoint information, and the motion information of the robot is obtained. According to the method, the waypoint sensing world model captures key information, and good generalization ability is kept in a changing scene.

Description

Technical Field

[0001] The present invention relates to the field of robot operation and deep learning model algorithm design, and more specifically, to a robot operation method and system based on a waypoint perception world model. Background Art

[0002] Language-guided robotic manipulation is a long-standing research area in the field of embodied artificial intelligence. In recent years, language-guided robotic manipulation has gained extensive research attention due to its flexibility and practicality in facilitating human-robot interaction. Many benchmarks have been established for this purpose, such as RLBench, CALVIN, and VLMBench. Early methods improved manipulation performance by introducing powerful representations, carefully designed network architectures, or effective training mechanisms. With the rapid development of vision-language models (VLMs), recent studies have attempted to introduce VLMs to improve manipulation accuracy and generalization to unseen scenes / objects. However, most previous methods tend to directly learn mappings from multimodal inputs to low-level actions, ignoring explicit modeling of environmental dynamics, which may lead to failures in executing actions or plans in complex environments and poor generalization.

[0003] World models aim to generate predictive models of their surroundings, taking into account uncertainty and dynamic changes. They have been widely studied in areas such as video generation, navigation, and autonomous driving. For example, Genie introduced a spatiotemporal video segmenter and a dynamic model to predict the next video frame in an autoregressive manner; DriveDreamer built a world model from real driving scenes to achieve reasonable driving strategy generation. Some studies have also introduced world models for robot manipulation tasks, such as Daydreamer and SWIM. However, these models usually perform world modeling and decision making alternately, which brings great difficulties to model training and is not efficient. Summary of the invention

[0004] In order to overcome at least one of the defects of the prior art described above, the present invention provides a robot operation method and system based on a waypoint perception world model.

[0005] The present invention aims to solve the above technical problems at least to a certain extent.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A robot operation method based on a waypoint perception world model comprises the following steps:

[0008] S1: Acquire robot operation information, where the robot operation information includes observation images, robot state information, and user instructions;

[0009] S2: inputting the observed image and the user command into a preset waypoint perception world model, and the waypoint perception world model obtains the robot path point information;

[0010] S3: Obtain robot motion information through a preset motion prediction module according to the robot path point information and robot state information;

[0011] S4: Combining the robot motion information and the robot path point information to obtain a motion trajectory of the operating robot.

[0012] Furthermore, the waypoint perception world model includes a pre-trained visual language model and a scene prediction module, and the pre-trained visual language model and the scene prediction module are connected.

[0013] Further, in step S2, the observation image and the user command are input into a preset waypoint perception world model, and the waypoint perception world model obtains the robot path point information, including:

[0014] S2.1: Combine the robot's current observation image and the user's command as the first input, input the first input into the pre-trained visual language model, and obtain the robot's path point prompt;

[0015] S2.2: text encoding the path point prompt of the robot to obtain an encoded path point prompt;

[0016] S2.3: The robot's historical observation images are encoded as the second input;

[0017] S2.4: Input the encoded path point prompt and the second input into the scene prediction module to obtain the robot path point information.

[0018] Further, in step S3, the robot's motion information is obtained through a preset motion prediction module according to the robot's path point information and the robot's state information, including:

[0019] The robot state information is input into a preset multi-layer perceptron to obtain an output result, the output result is combined with the second input as a third input and input into a preset action prediction module, and the robot's action information is obtained through the action prediction module.

[0020] Furthermore, the first input and the second input pass through the asynchronous hierarchical executor before being input into the waypoint perception world model or the action prediction module.

[0021] Furthermore, the scene prediction module includes a first self-attention module, a first cross-attention module and a first feedforward network, wherein the first self-attention module, the first cross-attention module and the first feedforward network are connected in sequence, the path point features of the historical observation image are input into the first self-attention module to obtain the output of the first self-attention module, the path point prompts and the output of the first self-attention module are input into the first cross-attention module to obtain the output of the first cross-attention module, and the output of the first cross-attention module is input into the first feedforward network to obtain the robot path point information.

[0022] Furthermore, the action prediction module includes a second self-attention module, a second cross-attention module and a second feedforward network, and the second self-attention module, the second cross-attention module and the second feedforward network are connected in sequence, and the robot operation information processed by the multi-layer perceptron and the path point features of the historical observation image are input into the second self-attention module to obtain the output of the second self-attention module, and the output of the second self-attention module and the path point features are input into the second cross-attention module to obtain the output of the second cross-attention module, and the output of the second cross-attention module is input into the second feedforward network to obtain the action trajectory of the operating robot.

[0023] Furthermore, the asynchronous hierarchical executor respectively executes the input information and user instruction parsing of the visual language model, scene prediction module, and action prediction module. During the execution process, the asynchronous hierarchical executor returns the execution results of the previous round of user instruction parsing, scene prediction module, and action prediction module. After the execution is completed, the asynchronous hierarchical executor returns the execution results of the current user instruction parsing, scene prediction module, and action prediction module.

[0024] Further, in step S4, the motion trajectory of the operating robot includes:

[0025] Tra={l,[O1,S1,A1,M1],...,[O T ,S T ,A T ,M T ]}

[0026] Where Tra is the motion trajectory of the robot, l is the user command, O1,...,O T are historical observation images, S1,...,S T is the robot status information, A1,...,A T is the robot's motion information, M1,...,M T is the robot path point information, and T is the time step length.

[0027] A robot operating system based on a waypoint perception world model, applied to the robot operating method, comprises:

[0028] A robot operation information acquisition module, used to acquire robot operation information, wherein the robot operation information includes observation images, robot state information and user instructions;

[0029] A robot path point information acquisition module, used to input observation images and user instructions into a preset waypoint perception world model, and the waypoint perception world model obtains the robot path point information;

[0030] A robot motion information acquisition module is used to obtain robot motion information through a preset motion prediction module according to robot path point information and robot state information;

[0031] The robot motion trajectory acquisition module is used to combine the robot motion information and the robot path point information to obtain the motion trajectory of the operating robot.

[0032] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0033] The present invention first collects the robot's operating information, including the observed images, the robot's current state information, and the instructions given by the user. The operating information is input into a preset waypoint perception world model to obtain the robot's path point information. The path point information and the robot's state information are used to calculate the action information that the robot should take through a preset action prediction module. Finally, the calculated robot action information is combined with the path point information to generate and output the complete action trajectory of the operating robot. Based on the waypoint perception world model, the present invention significantly improves the robot's control ability and can better understand and predict dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of the robot operation method of the present invention;

[0035] Figure 2 This is a structural diagram of the scene prediction module of the present invention;

[0036] Figure 3 This is a structural diagram of the action prediction module of the present invention;

[0037] Figure 4 This is a structural diagram of the waypoint perception world model described in this embodiment;

[0038] Figure 5 This is a schematic diagram of the robot operating system of the present invention. DETAILED DESCRIPTION

[0039] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0040] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0041] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0042] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0043] Example 1

[0044] A robot operation method based on a waypoint-aware world model, such as Figure 1 As shown, the following steps are included:

[0045] S1: Acquire robot operation information, where the robot operation information includes observation images, robot state information, and user instructions;

[0046] S2: inputting the observed image and the user command into a preset waypoint perception world model, and the waypoint perception world model obtains the robot path point information;

[0047] S3: Obtain robot motion information through a preset motion prediction module according to the robot path point information and robot state information;

[0048] S4: Combining the robot motion information and the robot path point information to obtain a motion trajectory of the operating robot.

[0049] The present invention first collects the robot's operating information, including the observed images, the robot's current state information, and the instructions given by the user. The operating information is input into a preset waypoint perception world model to obtain the robot's path point information. The path point information and the robot's state information are used to calculate the action information that the robot should take through a preset action prediction module. Finally, the calculated robot action information is combined with the path point information to generate and output the complete action trajectory of the operating robot. Based on the waypoint perception world model, the present invention significantly improves the robot's control ability and can better understand and predict dynamic environments.

[0050] Example 2

[0051] This embodiment further discloses the following contents based on the embodiment 1:

[0052] like Figure 4 As shown, the waypoint perception world model includes a pre-trained visual language model and a scene prediction module, and the pre-trained visual language model and the scene prediction module are connected.

[0053] Step S2, inputting the observed image and the user instruction into a preset waypoint perception world model, wherein the waypoint perception world model obtains the robot path point information, including:

[0054] S2.1: Combine the robot's current observation image and the user's command as the first input, input the first input into the pre-trained visual language model, and obtain the robot's path point prompt;

[0055] S2.2: text encoding the path point prompt of the robot to obtain an encoded path point prompt;

[0056] S2.3: The robot's historical observation images are encoded as the second input;

[0057] S2.4: Input the encoded path point prompt and the second input into the scene prediction module to obtain the robot path point information.

[0058] The waypoint-aware world model includes a powerful visual language model and a scene prediction module. Given user instructions, the visual language model parses the user instructions to provide task-related waypoint prompts. These prompts are used to guide the scene prediction module to predict key waypoints. At each time step, the prompts are combined with the user instructions and the robot observation images as input to the pre-trained visual language model to perform raw action parsing related to the manipulation task. The parsed raw actions and the original user instructions are combined as waypoint instructions for the scene prediction module.

[0059] In step S3, the robot's motion information is obtained through a preset motion prediction module according to the robot's path point information and the robot's state information, including:

[0060] The robot state information is input into a preset multi-layer perceptron to obtain an output result, the output result is combined with the second input as a third input and input into a preset action prediction module, and the robot's action information is obtained through the action prediction module.

[0061] The first input and the second input also pass through the asynchronous hierarchical executor before being input into the waypoint perception world model or the action prediction module.

[0062] Example 3

[0063] This embodiment further discloses the following contents based on Embodiments 1 and 2:

[0064] like Figure 2As shown, the scene prediction module includes a first self-attention module, a first cross-attention module and a first feedforward network, and the first self-attention module, the first cross-attention module and the first feedforward network are connected in sequence, the path point features of the historical observation image are input into the first self-attention module to obtain the output of the first self-attention module, the path point prompts and the output of the first self-attention module are input into the first cross-attention module to obtain the output of the first cross-attention module, and the output of the first cross-attention module is input into the first feedforward network to obtain the robot path point information.

[0065] For the scene prediction module, the waypoint prompts related to the robot manipulation task and the historical observation images of the robot are used as input to predict the waypoint features of the robot manipulation. The scene prediction module is composed of 12 TransformerDecoders stacked together.

[0066] like Figure 3 As shown, the action prediction module includes a second self-attention module, a second cross-attention module and a second feedforward network. The second self-attention module, the second cross-attention module and the second feedforward network are connected in sequence. The robot operation information processed by the multi-layer perceptron and the path point features of the historical observation image are input into the second self-attention module to obtain the output of the second self-attention module. The output of the second self-attention module and the path point features are input into the second cross-attention module to obtain the output of the second cross-attention module. The output of the second cross-attention module is input into the second feedforward network to obtain the action trajectory of the operating robot.

[0067] In the specific implementation process, in the action prediction module, the self-attention module, the cross-attention module and the feedforward network are three key components. They are connected in sequence from top to bottom and work together on the input data to extract features, capture dependencies and generate prediction results. In the action prediction module, the self-attention module first captures the dependencies within the sequence, then the cross-attention module realizes cross-sequence or cross-modal information interaction and fusion, and finally the feedforward network performs nonlinear transformation and feature extraction on the fused information to generate the final prediction result.

[0068] For the action prediction module, the robot manipulation path point state features predicted by the scene prediction module are used as prompts, and the robot's historical observation images and robot states are used as input to predict the action taken by the robot at the time step. The action prediction module is composed of 3 stacked Transformer Decoders.

[0069] Example 4

[0070] This embodiment further discloses the following contents based on Embodiments 1, 2 and 3:

[0071] The asynchronous hierarchical executor separately executes the input information of the vision-language model, the scene prediction module, and the action prediction module, as well as the parsing of user instructions. During the execution process, the asynchronous hierarchical executor returns the execution results of the previous round of user instruction parsing, the scene prediction module, and the action prediction module. After the execution ends, the asynchronous hierarchical executor returns the execution results of the current user instruction parsing, the scene prediction module, and the action prediction module.

[0072] The proposed asynchronous hierarchical executor significantly improves the execution efficiency of the model by setting different frequencies for different modules, and adopts an asynchronous hierarchical execution mode to separately execute the original action parsing, scene prediction, and action prediction. Different execution frequencies are used for these three parts as needed. For the original action parsing that requires a large amount of computing power, a lower execution frequency v1 is used. For the action prediction module, a higher execution frequency v3 is adopted. These three execution frequencies follow the following relationship: v1 < v2 < v3, where v2 is the execution frequency of the scene prediction module. If a module has not finished processing a new request, the asynchronous hierarchical executor will first return the previous result.

[0073] In the specific implementation process, the original actions input to the scene prediction module may be the same at different time steps. By using low-frequency original action parsing, the model avoids redundancy caused by alternately using the vision-language model and the waypoint-aware world model, thereby improving the training and inference efficiency.

[0074] In step S4, the action trajectory of the operating robot includes:

[0075] Tra={l,[O1,S1,A1,M1],...,[O T ,S T ,A T ,M T}

[0076] In the formula, Tra is the action trajectory of the operating robot, l is the user instruction, O1,...,O T are historical observation images, S1,...,S T are robot state information, A1,...,A T are robot action information, M1,...,M T are robot waypoint information, and T is the time step length.

[0077] Embodiment 5

[0078] Based on Embodiments 1, 2, 3, and 4, this embodiment further discloses the following content:

[0079] A robot operating system based on a waypoint-aware world model, such as Figure 5As shown, the robot operation method applied to the robot includes:

[0080] A robot operation information acquisition module, used to acquire robot operation information, wherein the robot operation information includes observation images, robot state information and user instructions;

[0081] A robot path point information acquisition module, used to input observation images and user instructions into a preset waypoint perception world model, and the waypoint perception world model obtains the robot path point information;

[0082] A robot motion information acquisition module is used to obtain robot motion information through a preset motion prediction module according to robot path point information and robot state information;

[0083] The robot motion trajectory acquisition module is used to combine the robot motion information and the robot path point information to obtain the motion trajectory of the operating robot.

[0084] In the specific implementation process, a trainable robot manipulation model π is constructed, which combines the user's command l and a series of observation images O from time step th to the current time step t. t-h:t and robot status S t-h:t Mapped to action A t .h represents the length of the history frame, which is set to 3. In addition, a scene prediction module is introduced to help the model build world knowledge.

[0085] π(VLM(l,O t ),O t-h:t ,S t-h:t )→M′ t ,A′ t

[0086] Where M' t ,A' t are respectively the path points and actions of the robot manipulation predicted by the model at time step t.

[0087] The training loss of PIVOT-R mainly includes the scene prediction loss L scene and the action prediction loss L act , for the scene prediction loss L scene , calculate the average L2 distance between the features of the predicted waypoint state M' and the true waypoint state M, where M is encoded using the pre-trained CLIP image encoder. For the action prediction loss L act , using cross entropy loss to calculate the loss between the predicted action A' and the true action A. The total loss of PIVOT-R is L = L scene +L act .

[0088] The effects of the present invention are shown in Table 1 and Table 2:

[0089] Table 1

[0090]

[0091] Experiments were conducted on four levels of tasks on the open source test benchmark SeaWave. The results are shown in Table 1. PIVOT-R achieved significant improvements on all tasks. Specifically, PIVOT-R achieved an average success rate of 74.19%, 19.45% higher than the best baseline. Both the ability to manipulate and the ability to understand instructions were greatly improved. This confirms the effectiveness of the proposed method.

[0092] Table 2

[0093]

[0094] We also conduct experiments in different unseen scenes. The new scenes include unseen backgrounds, changing light intensity, and more distractors. The results are shown in Table 2. PIVOT-R still maintains a much higher success rate than other models, indicating that the model captures key information and maintains good generalization ability in changing scenes.

[0095] We show that modeling the waypoint world enables the model to benefit from modeling the dynamic environment. This suggests that modeling the world that the robot manipulates at key waypoints can significantly improve the model’s manipulation capabilities, not just improving performance, but also enabling better understanding and prediction of the dynamic environment.

[0096] The same or similar reference numerals correspond to the same or similar components;

[0097] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0098] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A robot operation method based on a waypoint perception world model, characterized in that: The following steps are involved: S1: Acquire robot operation information, where the robot operation information includes observation images, robot state information, and user instructions; S2: inputting the observed image and the user command into a preset waypoint perception world model, and the waypoint perception world model obtains the robot path point information; S3: Obtain robot motion information through a preset motion prediction module according to the robot path point information and robot state information; S4: Combining the robot motion information and the robot path point information to obtain a motion trajectory of the operating robot.

2. The robot operation method based on the waypoint perception world model according to claim 1 is characterized in that: The waypoint perception world model includes a pre-trained visual language model and a scene prediction module, and the pre-trained visual language model and the scene prediction module are connected.

3. The robot operation method based on the waypoint perception world model according to claim 2 is characterized in that: Step S2, inputting the observed image and the user instruction into a preset waypoint perception world model, wherein the waypoint perception world model obtains the robot path point information, including: S2.1: Combine the robot's current observation image and the user's command as the first input, input the first input into the pre-trained visual language model, and obtain the robot's path point prompt; S2.2: text encoding the path point prompt of the robot to obtain an encoded path point prompt; S2.3: The robot's historical observation images are encoded as the second input; S2.4: Input the encoded path point prompt and the second input into the scene prediction module to obtain the robot path point information.

4. The robot operation method based on the waypoint perception world model according to claim 3 is characterized in that: In step S3, the robot's motion information is obtained through a preset motion prediction module according to the robot's path point information and the robot's state information, including: The robot state information is input into a preset multi-layer perceptron to obtain an output result, the output result is combined with the second input as a third input and input into a preset action prediction module, and the robot's action information is obtained through the action prediction module.

5. The robot operation method based on the waypoint perception world model according to claim 4 is characterized in that: The first input and the second input also pass through the asynchronous hierarchical executor before being input into the waypoint perception world model or the action prediction module.

6. The robot operation method based on the waypoint perception world model according to claim 3 is characterized in that: The scene prediction module includes a first self-attention module, a first cross-attention module and a first feedforward network, wherein the first self-attention module, the first cross-attention module and the first feedforward network are connected in sequence, the path point features of the historical observation image are input into the first self-attention module to obtain the output of the first self-attention module, the path point prompts and the output of the first self-attention module are input into the first cross-attention module to obtain the output of the first cross-attention module, the output of the first cross-attention module is input into the first feedforward network to obtain the robot path point information.

7. The robot operation method based on the waypoint perception world model according to claim 6 is characterized in that: The action prediction module includes a second self-attention module, a second cross-attention module and a second feedforward network. The second self-attention module, the second cross-attention module and the second feedforward network are connected in sequence. The robot operation information processed by the multi-layer perceptron and the path point features of the historical observation image are input into the second self-attention module to obtain the output of the second self-attention module. The output of the second self-attention module and the path point features are input into the second cross-attention module to obtain the output of the second cross-attention module. The output of the second cross-attention module is input into the second feedforward network to obtain the action trajectory of the operating robot.

8. The robot operation method based on the waypoint perception world model according to claim 5 is characterized in that: The asynchronous hierarchical executor executes the input information and user instruction parsing of the visual language model, scene prediction module, and action prediction module respectively. During the execution process, the asynchronous hierarchical executor returns the execution results of the previous round of user instruction parsing, scene prediction module, and action prediction module. After the execution is completed, the asynchronous hierarchical executor returns the execution results of the current user instruction parsing, scene prediction module, and action prediction module.

9. The robot operation method based on the waypoint perception world model according to claim 7, characterized in that: In step S4, the motion trajectory of the operating robot includes: Tra={l,[O1,S1,A1,M1],...,[O T ,S T ,A T ,M T ]} Where Tra is the motion trajectory of the robot, l is the user command, O1,...,O T are historical observation images, S1,...,S T is the robot status information, A1,...,A T is the robot's motion information, M1,...,M T is the robot path point information, and T is the time step length.

10. A robot operating system based on a waypoint perception world model, characterized in that: A robot operation method as claimed in any one of claims 1 to 9, comprising: A robot operation information acquisition module, used to acquire robot operation information, wherein the robot operation information includes observation images, robot state information and user instructions; A robot path point information acquisition module, used to input observation images and user instructions into a preset waypoint perception world model, and the waypoint perception world model obtains the robot path point information; A robot motion information acquisition module is used to obtain robot motion information through a preset motion prediction module according to robot path point information and robot state information; The robot motion trajectory acquisition module is used to combine the robot motion information and the robot path point information to obtain the motion trajectory of the operating robot.

Images