A robot manipulation method based on world model and incremental reasoning

Abstract

The invention relates to the technical field of robot operation, and proposes a robot operation method based on a world model and progressive reasoning, comprising the following steps: obtaining an object library composed of objects in a real scene; selecting an object from the object library, and generating an instruction based on the selected object; sending the instruction to a robot, and the robot operates the selected object according to the instruction, and records the instruction and the robot's motion trajectory image, state and execution action corresponding to the instruction; using an encoder to encode the instruction, the robot's motion trajectory image and the robot's state features into text features, image features and robot state features respectively; based on the text features, image features, robot state features and the robot's execution action, using a world model to predict the robot's action, and performing scene prediction of the scene corresponding to the predicted action, to obtain an action prediction result and a scene prediction result; and using the action prediction result and the scene prediction result to realize robot operation.

Description

Technical Field

[0001] The present invention relates to the technical field of robot operation, and more particularly, to a robot operation method based on world model and incremental reasoning. Background Art

[0002] How to make robots perform actions that conform to the laws of the physical world based on their current state, language instructions, and external vision is a major challenge.

[0003] Some existing methods, such as DayDreamer, use reinforcement learning methods, which require a lot of interaction with the environment and take a long time to operate the robot. Other existing methods, such as RT-2 and RT-X, combine robot operation with large-scale visual language data to improve the generalization ability of the model, thereby avoiding frequent interaction with the environment.

[0004] However, none of the existing methods have studied the impact of complex visual language environments on robots. A complex visual language environment means that the language instructions are obscure, and the target object needs to be confirmed by combining the features of the object in the language instructions with the visual scene. DayDreamer does not use language instructions, and RT-2 and RT-X only use simple language instructions such as grabbing an apple. All of them are unable to operate the robot in a complex visual language environment.

[0005] In summary, existing methods are unable to operate robots in complex visual language environments, which limits the application of existing methods in real scenarios. Summary of the invention

[0006] In order to overcome the defect of the prior art that a robot cannot be operated in a complex visual language environment, the present invention provides a robot operation method based on a world model and progressive reasoning that is adaptable to a complex language environment.

[0007] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0008] Obtain an object library consisting of objects in real scenes;

[0009] Select an object from the object library and generate instructions based on the selected object;

[0010] Send the command to the robot, the robot operates the selected object according to the command, and records the command and the robot's motion trajectory image, status and execution action corresponding to the command;

[0011] Using an encoder, the instructions are encoded into text features, the motion trajectory image of the robot corresponding to the instructions is encoded into image features, and the state of the robot corresponding to the instructions is encoded into robot state features;

[0012] Based on text features, image features, robot state features and the robot's execution actions, the world model is used to predict the robot's actions, and scene prediction is performed for the scene corresponding to the predicted action to obtain action prediction results and scene prediction results;

[0013] The action prediction results and scene prediction results are used to implement robot operations.

[0014] The present invention also proposes a robot operating system based on a world model and incremental reasoning for implementing the above-mentioned robot operating method based on a world model and incremental reasoning. The system comprises:

[0015] An object library acquisition module is used to acquire an object library consisting of objects in real scenes;

[0016] An instruction generation module, used for selecting an object from an object library and generating instructions based on the selected object;

[0017] The recording module is used to send instructions to the robot, and the robot operates the selected object according to the instructions, and records the instructions and the robot's motion trajectory image, state and execution action corresponding to the instructions;

[0018] An encoding module, used to encode the instruction into a text feature, encode the motion trajectory image of the robot corresponding to the instruction into an image feature, and encode the state of the robot corresponding to the instruction into a robot state feature;

[0019] A prediction module is used to predict the robot's actions based on text features, image features, robot state features and the robot's execution actions using a world model, and to perform scene prediction of the scene corresponding to the predicted action, thereby obtaining action prediction results and scene prediction results;

[0020] The robot operation module is used to use the action prediction results and scene prediction results to realize robot operation.

[0021] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes the steps of the robot operation method based on world model and progressive reasoning proposed by the present invention.

[0022] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0023] Based on text features, image features, robot state features and the robot's execution actions, the world model is used to predict the robot's actions, and scene prediction is performed on the scenes corresponding to the predicted actions to obtain action prediction results and scene prediction results. The action prediction results and scene prediction results are used to implement robot operations, so that the robot can perform actions that conform to the laws of the physical world according to the current state, language instructions and external vision, and realize the operation of the robot in a complex visual language environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the robot operation method based on the world model and progressive reasoning proposed in Example 1;

[0025] Figure 2 This is a schematic diagram of the restaurant simulator proposed in Example 1;

[0026] Figure 3 This is a schematic diagram of the framework of the robot operation method proposed in Example 1;

[0027] Figure 4 This is a schematic diagram of the model framework proposed in Example 1;

[0028] Figure 5 This is an example diagram of the robot operation proposed in Example 2;

[0029] Figure 6 This is the overall framework diagram of the robot operating system based on world model and incremental reasoning proposed in Example 3. DETAILED DESCRIPTION

[0030] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0031] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0032] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0033] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0034] Example 1

[0035] This embodiment proposes a robot operation method based on a world model and progressive reasoning. Figure 1 A schematic diagram of the flow of robot operation based on world model and progressive reasoning proposed in this embodiment.

[0036] The robot operation method based on the world model and progressive reasoning proposed in this embodiment includes the following steps:

[0037] S1: Obtain an object library consisting of objects in real scenes;

[0038] S2: Select an object from the object library and generate instructions based on the selected object;

[0039] S3: Send the command to the robot, the robot operates the selected object according to the command, and records the command and the robot's motion trajectory image, state and execution action corresponding to the command;

[0040] S4: using an encoder, encode the instruction into a text feature, encode the motion trajectory image of the robot corresponding to the instruction into an image feature, and encode the state of the robot corresponding to the instruction into a robot state feature;

[0041] S5: Based on the text features, image features, robot state features and the robot's execution actions, the world model is used to predict the robot's actions, and scene prediction is performed for the scene corresponding to the predicted action to obtain action prediction results and scene prediction results;

[0042] S6: Use the action prediction results and scene prediction results to implement robot operations.

[0043] In the specific implementation process, based on text features, image features, robot state features and the robot's execution actions, the world model is used to predict the robot's actions, and scene prediction is performed on the scenes corresponding to the predicted actions to obtain action prediction results and scene prediction results. The action prediction results and scene prediction results are used to realize robot operation, so that the robot can perform actions that conform to the laws of the physical world according to the current state, language instructions and external vision, and realize the operation of the robot in a complex visual language environment.

[0044] In an optional embodiment, the object library includes n subsets, and each object in each subset records the object name, the position of the object relative to the robot, and the rotational posture of the robot corresponding to the position;

[0045] The expression of the object library is:

[0046]

[0047] In the formula, represents the i-th object in the N-th subset of the object library, x i ,y i and h i Represent objects The horizontal axis coordinate value, horizontal axis coordinate value and height value relative to the robot's position, yaw i Represents the rotational pose of the robot.

[0048] In an optional embodiment, an object is selected from an object library, and before generating instructions based on the selected object, an instruction pool is constructed, wherein the instruction pool initially contains a number of preset instructions; each instruction generated is added to the instruction library in the order of the time when the instruction is generated;

[0049] Select several objects from the object library, and the steps of generating instructions based on the selected objects include:

[0050] Generate the appearance description and function description corresponding to the selected object using the large language model, and calculate the positional relationship between the selected object and the robot;

[0051] Extract the latest several instructions from the instruction pool as instruction examples,

[0052] Based on the instruction example, the appearance description, the function description and the position relationship, a natural language processing tool is used to generate instructions in a preset format.

[0053] As an exemplary illustration, the large language model is MiniGPT-4, and the natural language processing tool is ChatGPT;

[0054] In an optional embodiment, the preset format includes four formats: a verb plus noun format, a natural language format that explicitly includes the name of the selected object, a natural language format that only provides expressions related to the function of the target object, and a natural language format that only provides expressions related to the function, appearance or location of the target object;

[0055] The instructions include four levels of operation complexity from low to high: level one, level two, level three and level four;

[0056] The number of selected objects corresponding to the level one instruction is equal to 1, and the preset format corresponding to the level one instruction is: a verb plus a noun format;

[0057] The number of selected objects corresponding to the level 2 instruction is greater than 1, and the preset format corresponding to the level 2 instruction is: a natural language format that explicitly includes the names of the selected objects;

[0058] The number of selected objects corresponding to the level 3 instruction is greater than 1, and the preset format corresponding to the level 3 instruction is: a natural language format that only provides expressions related to the functions of the target objects;

[0059] The number of selected objects corresponding to the level 4 instruction is greater than 1, and the preset format corresponding to the level 3 instruction is: natural language that only provides expressions related to the function, appearance or location of the target object;

[0060] When generating instructions based on the selected object, level one, level two, level three and / or level four instructions are generated according to actual needs, and each instruction is classified by level type and stored in the instruction pool.

[0061] In this optional embodiment, a set of progressive tasks is also designed. Natural language is one of the most direct and effective ways of human-computer interaction; however, due to the complexity and variability of external visual scenes and human natural language instructions, understanding and executing these instructions has become one of the key challenges in embodied intelligence research; in order to systematically analyze and study these challenges, the tasks are divided into four levels according to the complexity of the instructions and the convenience of operation; (i) Level 1: The scene contains only one object, and the robot accepts instructions in the form of verb + noun, which is used to evaluate the basic grasping ability of the model; (ii) Level 2: The scene contains multiple objects, the instructions are in natural language form, and explicitly include the name of the target object. This level is used to evaluate the model's ability to understand traditional natural language instructions; (iii) Level 3: The scene contains multiple objects, but the natural language instructions do not include the name of the target object, but only provide expressions related to the function of the target object; it is used to evaluate the model's ability to infer the intention of human instructions; (iv) The scene contains multiple objects; the natural language instructions do not include the name of the target object, but only provide expressions related to the function, appearance or location of the target object; this level requires the model to have strong visual and language information processing capabilities at the same time; it is used to evaluate the model's visual perception and decision-making capabilities;

[0062] As an example, to help better control the robot to perform tasks, an inverse kinematics plugin was added to the simulator SeaWave and a motion planner was implemented; it can help effectively control the robot for experiments and data collection; data collection was performed for training through imitation learning; as an example, 9 objects and 10 tables were selected and used Figure 3 The solution shown in was used to generate data; as an illustrative example, a total of five thousand trajectories were collected;

[0063] As an illustrative illustration, in order to conveniently and fairly compare the success rate of robots understanding and executing human natural language instructions under different models, a high-fidelity restaurant scene simulator SeaWave based on UE5 was built to study robot operation; Figure 2 This is a schematic diagram of the restaurant simulator proposed in this embodiment. Figure 2 An example showing the simulator's restaurant scene thumbnails and object library, where Figure 2 (a) shows the overall scene diagram of this restaurant simulator; it is built on Unreal Engine with a high-fidelity digital twin scene, including 139 object categories and 1,474 natural language instructions at four levels. Figure 2(b) shows a subset of the simulator object library, which contains objects of various shapes and functions; this provides a guarantee for building a general robot operation environment and designing complex and diverse human natural language commands;

[0064] Figure 3 This is a schematic diagram of the framework of the robot operation method proposed in this embodiment, such as Figure 3 As shown, it mainly includes three parts: automatic scene generation, instruction generation and robot operation;

[0065] Automatic scene generation: responsible for generating rich and diverse scenes for agents to train and test; e.g. Figure 3 As shown in the leftmost box, we start by randomly selecting a table. M = 10, and table T from the object library i The corresponding sub-library of the object library Select n objects from Thus, scenes are automatically generated to simulate the real restaurant environment;

[0066] Instruction generation: In order to collect enough high-quality, complex, multi-level human-like natural language instructions, ChatGPT is used as the instruction generation model; specifically, Figure 3 As shown in the box in the middle, MiniGPT-4 is first used to generate appearance and functional descriptions for each object, and the positional relationships between objects in the scene are calculated; then, new instructions are generated from a set of artificial instruction pools in a guided manner; specifically, in each step, 8 instructions are extracted from the instruction pool as instruction examples (6 of which are pre-written instructions, and the other two are instructions generated before this step); using contextual instructions, object descriptions, and positional relationships as inputs to ChatGPT, ChatGPT needs to generate corresponding instructions based on customized prompts; finally, the generated instructions are added to the instruction library to increase the diversity of the instruction library;

[0067] Robot operation: Figure 3 As shown in the rightmost box, the robot receives visual scenes and natural language instructions and performs corresponding operations according to the instructions.

[0068] In an optional embodiment, the expression of the motion trajectory image of the robot is:

[0069]

[0070] In the formula, Tra represents the robot's motion trajectory image, N represents the length of the motion trajectory, and I t The RGB image representing the trajectory corresponding to the robot's t-th execution action also represents the RGB image of the robot's t-th step trajectory;

[0071] The state of the robot includes the state corresponding to each action performed by the robot, wherein the state corresponding to the t-th action performed by the robot is recorded as the t-th step state S of the robot t , the robot's state S at step t t It includes six dimensions of the robot's palm motion: (x, y, z, pitch, roll, yaw), where x, y, and z represent the horizontal, vertical, and vertical coordinate values ​​of the robot's palm center of gravity, respectively; pitch represents the pitch angle, roll represents the roll angle, and yaw represents the yaw angle, which refers to the Euler angle of the robot's palm and also represents the position and posture of the robot's palm;

[0072] The steps of encoding the instruction into text features using an encoder, encoding the motion trajectory image of the robot corresponding to the instruction into image features, and encoding the state of the robot corresponding to the instruction into robot state features include:

[0073] Use the pre-trained CLIP text encoder to encode the instruction P into text features

[0074] Use the pre-trained CLIP visual encoder to convert the RGB image I of the t-th step trajectory corresponding to the action trajectory Tra t Encoded as image features

[0075] Use MLP to transform the robot's t-step state S t Encoded as robot state features

[0076] In an optional embodiment, the world model includes: an action prediction module E ap And scene prediction module E sp ;

[0077] The action prediction module E ap and the scene prediction module E sp They are all composed of L standard Transformer Decoders stacked together, where each Transformer Decoder contains a self-attention layer, a cross-attention layer, and a feed-forward network.

[0078] In this optional embodiment, the action prediction module and the scene prediction module receive the current state of the scene and language instructions for action prediction, and the latter receives the current state of the scene prediction and the predicted action, and uses the next action framework for self-supervision to help the model learn to perform actions that are consistent with the world's logical knowledge.

[0079] In an optional embodiment, based on text features, image features, robot state features and the robot's execution action, the robot's action is predicted using a world model, and the step of predicting a scene corresponding to the predicted action includes:

[0080] Text features Image features and robot status characteristics Input action prediction module E ap , action prediction module E ap Output the current execution action A t The corresponding next step of the robot's action prediction result A' t+1 ;

[0081] The action prediction result A′ t+1 and image features Input scene prediction module E sp , scene prediction module E sp Output the RGB image of the current trajectory I t The scene prediction result I of the RGB image of the corresponding next step trajectory t ' +1 ;

[0082] The action prediction result A′ t+1 The expression is:

[0083]

[0084] In the formula, express and Through the action prediction module E ap The cross attention layer and Combine;

[0085] The scene prediction result I t ' +1 The expression is:

[0086]

[0087] In the formula, express Through the scene prediction module E sp The cross attention layer and A′ t+1 Combine.

[0088] In this optional embodiment, the world model has attracted people's attention because of its ability to predict the future state of the environment from a time series and has been successfully applied in various fields. The world model helps the method based on this optional embodiment to make inferences consistent with world knowledge based on the external environment, thereby reducing learning costs; wherein, the method of this optional embodiment embeds the world model into the decision-making process of the method to help obtain an execution strategy consistent with world knowledge.

[0089] In an optional embodiment, before the action prediction result and the scene prediction result are used to implement the robot operation, based on the action prediction result A′ t+1 And the scene prediction result I′ t+1 Design a loss function Loss, and use the loss function Loss to optimize the action prediction results and scene prediction results;

[0090] The expression of the loss function Loss is:

[0091] Loss = MSE(A′ t+1 ,A t+1 )+MSE(I′ t+1 , I t+1 )

[0092] In the formula, A t+1 Indicates the current execution action A t The corresponding real next step robot execution action, I t+1 RGB image I representing the current trajectory t The corresponding RGB image of the actual next step trajectory, MSE represents the mean square error loss function;

[0093] Iterate and solve the loss function Loss until the loss function Loss is minimized or the number of iterations reaches a preset number, then stop the iteration to obtain the optimized action prediction result and the optimized scene prediction result. The final action prediction result is A′ when the iteration is stopped. t+1 ;

[0094] The optimized action prediction results and the optimized scene prediction results are used to implement robot operations.

[0095] In this optional embodiment, the goal of the present application is to establish an embodied intelligent model that can perform operations consistent with world logic knowledge under complex visual and language instructions; to this end, the present application proposes a robot model DamWorld based on world model reasoning for implementing the robot operation method based on world model and progressive reasoning described above; as an exemplary illustration, Figure 4 This is a schematic diagram of the model framework proposed in this embodiment; the overall framework of DamWorld is as follows Figure 4As shown; in general, Figure 4 As shown in a, for any given instruction P, there is a set of corresponding robot motion trajectories state and perform actions And, there is an action frame conversion relationship, For step t, I t , S t and the corresponding instructions P, and encode them through their respective encoders to obtain image features Robot Status Characteristics and text features Finally, the world model takes these features as input to make predictions and inferences to match the world's logical knowledge of the robot's actions and the next action frame; the following is a detailed introduction to the various modules of DamWorld:

[0096] Encoder: As Figure 4 As shown in a, the encoder of this application mainly includes three types of input: image {I t , I t+1}、Robot status S t and text P; for RGB image I, use the pre-trained CLIP visual encoder for encoding; in order to speed up training and reasoning, use TokenLearner to compress the number of tokens; specifically, TokenLearner compresses the 49 tokens output from the CLIP visual encoder into 8 tokens; for command P and robot state S t , encode them using a pre-trained CLIP text encoder and a simple MLP respectively;

[0097] World Model: The world model of this application mainly consists of two modules: action prediction E ap and scenario prediction E sp ;like Figure 4 As shown in Fig. 2, the action prediction module is composed of L standard Transformer Decoders stacked together; each standard Transformer Decoder consists of three modules that process query features in the following order: a self-attention module, a cross-attention module, and a feed-forward network (FFN); the action prediction module is composed of For input, and Through the cross-attention layer Combined; the above process can be simply expressed as: In addition, if Figure 4 As shown in c, based on the reasoning advantage of the world model, similar to the action prediction module, this application uses and predict action A′ t+1 As input, predict That is

[0098] Loss function: The model is optimized by action prediction loss and scene prediction loss; MSE Loss is used to calculate the loss, that is, Loss = MSE(A′ t+1 ,A t+1 )+MSE(I′ t+1 , I t+1 ).

[0099] In this embodiment, the present application constructs a UE5-based robot operation simulator that supports robot operation based on a physics engine and can meet the needs of robot operation research and evaluation in most daily scenarios. The present application proposes a general robot operation benchmark with progressive reasoning tasks, which can systematically evaluate the robot's operation capabilities under complex visual and language signal instructions.

[0100] Example 2

[0101] This embodiment applies the operability component perception method proposed in Embodiment 1 to discover common interactive attributes from a large model, and proposes the following performance comparison result example:

[0102] The model DamWorld corresponding to the method proposed in this application significantly outperforms other baselines in four levels of operation tasks; specifically, compared with RT-1, DamWorld has an average improvement of 5.6% in the four-level tasks defined in SeaWave proposed in Example 1, and an improvement of 9% in the fourth-level tasks that require close integration of vision and language for reasoning.

[0103] Figure 5 This is an example diagram of the robot operation proposed in this embodiment. Figure 5 Some examples of robot manipulation are shown in this application. This application compares the reasoning capabilities of RT-1 and DamWorld in terms of positional relationships, spatial size, and color appearance. The results show that DamWorld has good cross-modal reasoning and manipulation capabilities. Specifically, Figure 5 As shown, in the task related to color appearance, if the instructions require only green objects, the actions of RT-1 and DamWorld are correct, but, in fact, the instructions of this application require an object with a green lid, so in the end only the object grasped by DamWorld is correct.

[0104] Example 3

[0105] This embodiment proposes a robot operating system based on a world model and incremental reasoning, which is used to implement the robot operating method based on a world model and incremental reasoning proposed in Embodiment 1.

[0106] Figure 6 This is the overall framework diagram of the robot operating system based on world model and progressive reasoning proposed in this embodiment.

[0107] The robot operating system based on world model and progressive reasoning includes:

[0108] An object library acquisition module is used to acquire an object library consisting of objects in real scenes;

[0109] An instruction generation module, used for selecting an object from an object library and generating instructions based on the selected object;

[0110] The recording module is used to send instructions to the robot, and the robot operates the selected object according to the instructions, and records the instructions and the robot's motion trajectory image, state and execution action corresponding to the instructions;

[0111] An encoding module, used to encode the instruction into a text feature, encode the motion trajectory image of the robot corresponding to the instruction into an image feature, and encode the state of the robot corresponding to the instruction into a robot state feature;

[0112] A prediction module is used to predict the robot's actions based on text features, image features, robot state features and the robot's execution actions using a world model, and to perform scene prediction of the scene corresponding to the predicted action, thereby obtaining action prediction results and scene prediction results;

[0113] The robot operation module is used to use the action prediction results and scene prediction results to realize robot operation.

[0114] This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes the steps of the robot operation method based on world model and progressive reasoning proposed by the present invention.

[0115] It can be understood that the system and computer device of this embodiment are applied to the method of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described repeatedly here.

[0116] The same or similar reference numerals correspond to the same or similar components;

[0117] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0118] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for robot manipulation based on a world model and incremental reasoning, characterized in that: The following steps are involved: Obtain an object library consisting of objects in real scenes; Select an object from the object library and generate instructions based on the selected object; Send the command to the robot, the robot operates the selected object according to the command, and records the command and the robot's motion trajectory image, status and execution action corresponding to the command; Using an encoder, the instructions are encoded into text features, the motion trajectory image of the robot corresponding to the instructions is encoded into image features, and the state of the robot corresponding to the instructions is encoded into robot state features; Based on text features, image features, robot state features and the robot's execution actions, the world model is used to predict the robot's actions, and scene prediction is performed for the scene corresponding to the predicted action to obtain action prediction results and scene prediction results; The action prediction results and scene prediction results are used to realize robot operation; The object library includes subsets, each object in each subset records the object name, the position of the object relative to the robot, and the rotational posture of the robot corresponding to the position; The expression of the object library is: In the formula, Represents the object library The subset objects, , and Represent objects The horizontal axis coordinate value, vertical axis coordinate value and height value relative to the robot's position, represents the rotational position of the robot, Representing objects The name of the object; Select an object from the object library, and before generating instructions based on the selected object, build an instruction pool, wherein the instruction pool initially contains a plurality of preset instructions; each instruction generated is added to the instruction library in the order of the time when the instruction is generated; Select several objects from the object library, and the steps of generating instructions based on the selected objects include: Generate the appearance description and function description corresponding to the selected object using the large language model, and calculate the positional relationship between the selected object and the robot; Extract the latest several instructions from the instruction pool as instruction examples, Based on the instruction example, the appearance description, the function description and the position relationship, using a natural language processing tool to generate instructions in a preset format; The preset formats include four formats: a verb plus noun format, a natural language format that explicitly includes the name of the selected object, a natural language format that only provides expressions related to the function of the target object, and a natural language format that only provides expressions related to the function, appearance or location of the target object; The instructions include four levels of operation complexity from low to high: level one, level two, level three and level four; The number of selected objects corresponding to the level one instruction is equal to 1, and the preset format corresponding to the level one instruction is: a verb plus a noun format; The number of selected objects corresponding to the level 2 instruction is greater than 1, and the preset format corresponding to the level 2 instruction is: a natural language format that explicitly includes the names of the selected objects; The number of selected objects corresponding to the level 3 instruction is greater than 1, and the preset format corresponding to the level 3 instruction is: a natural language format that only provides expressions related to the functions of the target objects; The number of selected objects corresponding to the level 4 instruction is greater than 1, and the preset format corresponding to the level 4 instruction is: natural language that only provides expressions related to the function, appearance or location of the target object; When generating instructions based on the selected object, level one, level two, level three and / or level four instructions are generated according to actual needs, and each instruction is classified by level type and stored in the instruction pool.

2. The robot operation method based on world model and incremental reasoning according to claim 1, characterized in that: The expression of the robot's motion trajectory image is: In the formula, represents the robot's motion trajectory image, N represents the length of the motion trajectory, The RGB image representing the trajectory corresponding to the robot's t-th execution action also represents the RGB image of the robot's t-th step trajectory; The state of the robot includes the state corresponding to each action performed by the robot, wherein the state corresponding to the t-th action performed by the robot is recorded as the t-th step state of the robot , the robot's state at step t It includes six dimensions of the robot's palm motion: (x, y, z, pitch, roll, yaw), where x, y, and z represent the horizontal, vertical, and vertical coordinate values ​​of the robot's palm center of gravity, respectively; pitch represents the pitch angle, roll represents the roll angle, and yaw represents the yaw angle, which refers to the Euler angle of the robot's palm and also represents the position and posture of the robot's palm; The steps of encoding the instruction into text features using an encoder, encoding the motion trajectory image of the robot corresponding to the instruction into image features, and encoding the state of the robot corresponding to the instruction into robot state features include: Use the pre-trained CLIP text encoder to convert the command Encoding as text features ; Use the pre-trained CLIP visual encoder to convert the action trajectory The corresponding RGB image of the t-th step trajectory Encoded as image features ; Use MLP to transform the robot's state at step t Encoded as robot state features .

3. The robot operation method based on world model and incremental reasoning according to claim 1 or 2, characterized in that: The world model includes: an action prediction module and scene prediction module ; The action prediction module and the scene prediction module They are all composed of L standard Transformer Decoders stacked together, where each Transformer Decoder contains a self-attention layer, a cross-attention layer, and a feed-forward network.

4. The robot operation method based on world model and incremental reasoning according to claim 3, characterized in that: Based on text features, image features, robot state features and the robot's execution actions, the robot's actions are predicted using a world model, and the steps of performing scene prediction of the scene corresponding to the predicted action include: Text features , Image features and robot status characteristics Input action prediction module , action prediction module Output the current execution action The corresponding action prediction result of the next robot action ; The action prediction results and image features Input scene prediction module , scene prediction module Output the RGB image of the current trajectory The scene prediction result of the RGB image corresponding to the next step of the trajectory ; The action prediction result The expression is: In the formula, express and Through the action prediction module The cross attention layer and Combine; The scenario prediction results The expression is: In the formula, express Through the scene prediction module The cross attention layer and Combine.

5. The robot operation method based on world model and progressive reasoning according to claim 4, characterized in that: Before using the action prediction results and scene prediction results to implement robot operations, And the scene prediction results Design loss function , using the loss function Optimize action prediction results and scene prediction results; The loss function The expression is: In the formula, Indicates the current execution action The corresponding real next step robot execution action, RGB image representing the current trajectory The corresponding RGB image of the actual next step trajectory, represents the mean square error loss function; Iteratively solve the loss function , until the loss function When the minimum is reached or the number of iterations reaches the preset number, the iteration is stopped to obtain the optimized action prediction result and the optimized scene prediction result. The optimized action prediction result is the result when the iteration is stopped. ; The optimized action prediction results and the optimized scene prediction results are used to implement robot operations.

6. A robot operating system based on a world model and incremental reasoning, used to implement the robot operating method based on a world model and incremental reasoning as described in any one of claims 1 to 5, characterized in that: include: An object library acquisition module is used to acquire an object library consisting of objects in real scenes; An instruction generation module, used for selecting an object from an object library and generating instructions based on the selected object; The recording module is used to send instructions to the robot, and the robot operates the selected object according to the instructions, and records the instructions and the robot's motion trajectory image, state and execution action corresponding to the instructions; An encoding module, used to encode the instruction into a text feature, encode the motion trajectory image of the robot corresponding to the instruction into an image feature, and encode the state of the robot corresponding to the instruction into a robot state feature; A prediction module is used to predict the robot's actions based on text features, image features, robot state features and the robot's execution actions using a world model, and to perform scene prediction of the scene corresponding to the predicted action, thereby obtaining action prediction results and scene prediction results; The robot operation module is used to use the action prediction results and scene prediction results to realize robot operation.

7. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor executes the steps of the robot operation method based on the world model and progressive reasoning as described in any one of claims 1 to 5.

Images