Robot system and method based on feedback pre-training language and visual model

Abstract

The invention discloses a robot system and method based on a feedback pre-training language and a visual model. S1, sensing information of a robot operation environment; s2, prompting project Prompt template design, and taking information such as a robot action primitive function, personal preference and task feedback as Prompt generation input data; s3, on the basis of the input data and image information in the environment, a pre-trained large language model and a visual language model are adopted to obtain a code of a robot operation task and a starting target mask image; s4, according to the generated robot operation task code, through methods of inverse kinematics solution, coordinate conversion of a mechanical arm and a camera and the like, an operation track of the robot is generated, and an operation task is executed; and S5, after the execution of the robot is finished, inputting the finished target mask image into the task feedback model. The robot can complete the task target indicated by the language instruction without a large number of model training processes and data collection processes, and the method has the advantages of being high in operability and convenient and easy to use.

Description

Technical Field

[0001] The present invention relates to the field of autonomous decision-making and motion control of robots, and in particular to a robot system and method based on feedback pre-training language and visual models. Background Art

[0002] In the field of intelligent robots, task planning in unstructured environments has always been a challenging task. Traditional robot systems often rely on pre-defined task planning strategies, which are usually based on static environmental models and fixed task objectives, and are difficult to adapt to the diversity and dynamics of the real world. In unstructured scenarios such as home scenes and office scenes, robots need to have flexible task planning capabilities, be able to autonomously adjust action strategies according to environmental changes and task complexity, and achieve efficient and intelligent operations. In short, intelligent robots' task planning for unstructured environments and being able to understand language instructions represented in natural language, infer corresponding operation goals, and generate task execution sequences are difficult problems that still need to be solved by intelligent robots.

[0003] In intelligent robot operation, pre-trained models can be applied to all aspects of task planning. First, it can realize real-time perception and understanding of natural language instructions and visual images, so as to accurately capture and understand human intentions. Secondly, the task reasoning and execution module based on the pre-trained model can use language and visual information for end-to-end task planning and execution, and generate intelligent operation strategies suitable for the current environment and task objectives. Finally, through the task execution feedback module, the intelligent robot can monitor the progress and effect of task execution in real time, and dynamically adjust and optimize the task planning according to the feedback information, so as to achieve efficient operation in complex, dynamic and unstructured environments.

[0004] In summary, it is necessary to explore the use of pre-trained language models and visual models to solve task planning in unstructured scenarios in order to realize task operations of intelligent robots. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings and deficiencies of the prior art and to provide a robot system and method based on feedback pre-trained language and visual models, so as to enable the robot to complete variable, open-vocabulary human instructions, master a variety of operating skills, and adapt to a changing operating environment.

[0006] The present invention is achieved through the following technical solutions:

[0007] A robot operation method based on a feedback-based pre-trained language and visual model, the method comprising the following steps:

[0008] 1) Obtain real-time natural language instructions from humans and visual image information in the robot's operating environment.

[0009] 2) Design a prompt template, including robot action primitive library functions, personal preferences, task feedback and other information. Combined with the object information provided in step 3), generate input data.

[0010] 3) Input the input data to the pre-trained language and vision model to obtain the target mask map when the task starts, the code of the robot operation task, and the task operation steps. And interact with step 2) in real time to provide information about objects in the environment.

[0011] The code of the robot operation task is obtained by combining the function defined in step 1) with the code generation capability of the large language model. And the target mask image has the same resolution as the visual image of the operation environment obtained in step 2).

[0012] 4) According to the robot operation task code, the robot generates a task trajectory, and obtains the target mask image at the end after the robot executes the code.

[0013] 5) Based on the target mask image at the end and the target mask image at the beginning, combined with the task information, determine whether the target mask image at the end meets the task requirements, that is, whether the task is completed. The task information is generated by the pre-trained language model in step 3), including task understanding information, robot operation steps, and the location information of the target object when the task is completed.

[0014] Furthermore, the visual image information described in step 1) includes an RGB image and a depth image.

[0015] Furthermore, the prompt template is designed as follows:

[0016] 1-1) Design the robot action primitive library function, which can be expressed as:

[0017]

[0018] Among them, d represents the distance of movement; q represents the quaternion of the robot arm operation end.

[0019] 1-2) Design user personal preferences, which can be expressed as:

[0020] Personal Preference={'smoother','safer','changepose',...}

[0021] Among them, words such as smoother represent human preference settings for robot movement in order to optimize the robot's operation.

[0022] 1-3) Design task feedback information based on the task feedback prompt output by the task feedback module.

[0023] 1-4) Based on the pre-trained visual system, the img-to-text feature is used to provide the name, type, and location information of objects in the operating environment as prompts.

[0024] Furthermore, the code of the robot operation task is recombined by combining the robot action primitive function library in the predefined Prompt with the code generation capability of the large language model to generate the robot operation code.

[0025] Furthermore, the mask image when the task target starts to be executed is generated by a pre-trained visual model, and a target mask image with higher accuracy is generated by combining depth information and image processing technology.

[0026] Furthermore, the task feedback module is used to determine whether the task is completed, including a visual module and a pre-trained language module. The visual module aligns the input target mask image at the beginning of the task with the target mask image at the end of the task, and outputs information such as the offset distance of the mask center point and the target object posture through visual processing methods such as two-dimensional posture estimation. The pre-trained language module is responsible for comparing the information output by the visual module with the motion parameters obtained from the robot task operation code information, and determining whether the task is completed.

[0027] A robot operating system based on a feedback-based pre-trained language and visual model comprises: a visual language information real-time perception module, a task reasoning and execution module, a task execution status feedback module, and a robot control module.

[0028] The visual language information real-time perception module is used to obtain visual image information including RGB information and depth information in real time, and is used to obtain natural language information of human instructions in real time.

[0029] The task reasoning and execution module uses a pre-trained language and visual model based on the input data to generate codes and task execution steps suitable for robot operation, and also generates a target mask map so that the robot can perform tasks more accurately.

[0030] The task execution feedback module is used to determine whether the task is completed after the robot completes the task reasoning and execution modules, and to update the Prompt template information.

[0031] The robot control module is a key component in the robot system and is responsible for implementing the underlying motion control. Its functions include coordinate system conversion of the position and posture in the operation instructions, and converting these instructions into driving signals for the joint motors through inverse kinematics, so that the robot can accurately reach the target position.

[0032] The present invention also provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the robot operation method based on the feedback pre-trained language and visual model as described above is implemented.

[0033] Compared with the prior art, the present invention has the following advantages and effects:

[0034] Taking into account the constraints of unstructured scenarios and the difficulty of task planning, the present invention uses a pre-trained large language model for knowledge reasoning, task understanding, and code generation. It also uses a pre-trained visual model for target detection, target mask generation, and so on.

[0035] Considering the importance of prompt engineering to the pre-trained model, the present invention designs three types of prompts, including robot action primitive library functions, user personal preferences, and information fed back by the task feedback module, thereby improving the accuracy of the pre-trained model output.

[0036] The present invention takes into account the real-time feedback in the task execution stage, and through a task feedback module, uses a visual model to judge the completion of the task based on the target mask image before and after the task execution, and uses a pre-trained language model to adjust and generate new prompts in real time to achieve closed-loop control of the entire robot operation, thereby improving the completion rate of the robot task. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process of the present invention.

[0038] Figure 2 This is the design drawing of the prompt project. DETAILED DESCRIPTION

[0039] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0040] Prompt engineering can help explain the decision-making process and internal mechanism of the model. By designing appropriate prompts or inputs, the interpretability of model behavior can be enhanced, and the interpretability and credibility of the model can be improved. It is important to provide appropriate prompts for pre-trained large models.

[0041] Based on the technical problems existing in the above-mentioned prior art, the present invention realizes a set of prompt solutions with feedback for robots to perform operation tasks according to language instructions, such as Figure 2 As shown. The specific design of the prompt template is:

[0042] 1-1) Design the robot action primitive library function, which can be expressed as:

[0043]

[0044] Among them, d represents the distance of movement; q represents the quaternion of the robot arm operation end.

[0045] 1-2) Design user personal preferences, which can be expressed as:

[0046] Personal Preference={'smoother','safer','changepose',...}

[0047] Among them, words such as smoother represent human preference settings for robot movement in order to optimize the robot's operation.

[0048] 1-3) Design task feedback information and use the output of the task feedback module as the prompt.

[0049] 1-4) Based on the pre-trained visual system, the img-to-text feature is used to provide the name, type, and location information of objects in the operating environment as prompts.

[0050] The specific prompt example is as follows:

[0051] 1. Instruction: "Clean up the desktop";

[0052] 2. Scenario:

[0053] 2-1. There are baskets, tennis balls, cups, bread and books on the table.

[0054] 2-2. The bread is at (5,5) in the diagram, the cup is at (10,10), etc.

[0055] 3. Robot action primitive library functions: move_j(d,q), paw_open(d), etc.

[0056] 4. Personal preference: I only want to have books in the basket, and the cup only wants to be placed next to the basket.

[0057] 5. Task feedback information: (added after the task is completed).

[0058] Embodiment 1:

[0059] like Figure 1 This embodiment provides a robot operation method based on feedback-based pre-trained language and visual models, and its workflow is briefly described as follows:

[0060] S1. Obtain language instructions in natural language form and visual image information in the operating environment including RGB images and depth images.

[0061] S2. Based on the designed Prompt template rules, the environment information, robot operation task information, object information of the image obtained in S1 and processed in S3, user personal preference information, task feedback information, etc. are combined into a Prompt template to generate input data.

[0062] S3, based on the pre-trained large language model and visual language model, transmits object information in the operating environment with S2. When the task execution conditions are met, such as the location information and mask information of the task target, the model starts task reasoning, disassembles the task process, and obtains the task execution code. The specific code example is as follows:

[0063] Call the move function;

[0064] move_j(5,5,0.707,0,0.707,0) calls the gripper opening function;

[0065] paw_open(2);

[0066] Call the move function;

[0067] move_j(10,0,0.707,0,0.707,0);

[0068] Call the gripper closing function;

[0069] paw_close().

[0070] S4. Based on the robot operation code of S3, the robot generates a task trajectory, and obtains the target mask image at the end after the robot executes the code.

[0071] S5. According to the target mask image at the end and the target mask image at the start, combined with the task information, it is determined whether the target mask image at the end meets the task requirements, that is, whether the task is completed.

[0072] Preferably, in order to illustrate the improvements of the present invention, Example 2 is given as an example:

[0073] Embodiment 2:

[0074] 1. Obtain human language instructions "Please bring me the soda on the table", operating environment images, and depth information.

[0075] 2. Initialize the Prompt template:

[0076] 1) Instruction: "Please bring me the soda on the table";

[0077] 2) Scenario:

[0078] a. There are baskets, tennis balls, cups, soda and books on the table;

[0079] b.(object location information to be added);

[0080] 3) Robot action primitive library functions: move_j(d,q), paw_open(d), etc.

[0081] 4) Personal preferences: I only want to have books in the basket, I only want to put the cup next to the basket, I only want Pepsi as soda, etc.

[0082] 5) Task feedback information: (added after the task is completed).

[0083] 3. Input the Prompt template, environment image information, etc. into the pre-trained large language and vision model. On the one hand, generate the target mask image Image1 before the robot operation is executed; on the other hand, generate the robot operation code; on the other hand, improve the object position information of Prompt.

[0084] The generated partial execution code is as follows:

[0085] Call the move function:

[0086] move_j(5,5,0.707,0,0.707,0)

[0087] Call the gripper open function

[0088] paw_open(2)

[0089] Calling the move function

[0090] move_j(10,0,0.707,0,0.707,0)

[0091] Call the gripper closing function

[0092] paw_close()

[0093] The completed prompt is as follows:

[0094] 1-1. Instruction: "Please bring me the soda on the table";

[0095] 1-2.Scenario:

[0096] On the table there are baskets, tennis balls, cups, sodas, and books;

[0097] The soda can is at (5, 5), the tennis ball is at (5, 16), and so on;

[0098] 1-3. Robot action primitive library functions: move_j(d,q), paw_open(d), etc.

[0099] 1-4. Personal preferences: I only want to have books in the basket, I only want to put the cup next to the basket, I only want Pepsi as soda, etc.

[0100] 1-5. Task feedback information: (added after the task is completed).

[0101] 4. The robot executes the task operation code and generates the motion trajectory l1.

[0102] 5. The robot stops and obtains the target mask image Image2 when stopped. Image1, Image2 and Prompt are input to the task feedback module. If the task identifier task_get = False, it indicates that the task has failed, and the Prompt is updated to return to step 2 to continue the task.

[0103] The above method can be described as:

[0104] initialization:

[0105] Robot action primitive library function set: A;

[0106] Pre-trained LLM and VLM models;

[0107] Initialization prompt: P 0；

[0108] Main program:

[0109] Input: operating environment image information Img, depth information Dep, language command L

[0110] Using VLM, search for objects in the image and get: Object_nums, Object_species

[0111] for i in Object_nums:

[0112] Object_position[i] is updated to the corresponding Object_species

[0113] Add prompt: P0->P1

[0114] Input P1 to LLM&VLM, get Code: C1, Img_initial, specific operation steps E

[0115] Robot Execution

[0116] The robot execution is finished, and Img_finish is obtained

[0117] Input Img_finish, E, and Img_initial into the task feedback model to obtain the identifier

[0118] If(task_get=False)

[0119] Update prompt: P1->P2

[0120] Re-loop execution

[0121] Else:

[0122] Break.

[0123] The present invention also discloses a storage medium having computer instructions or programs for executing a robot operation based on a feedback-based pre-trained language and visual model provided by the present invention. When the computer instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0124] As described above, the present invention can be implemented well.

[0125] The implementation methods of the present invention are not limited to the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be equivalent replacement methods and shall be included in the protection scope of the present invention.

Claims

1. A robot operation method based on feedback pre-trained language and visual models, characterized in that: The following steps are involved: 1) Obtaining real-time natural language instructions from humans and visual image information in the robot’s operating environment; 2) Design a prompt template, including robot action primitive library functions, personal preferences, task feedback and other information; And combined with the object information provided in step 3), generate input data; 3) Inputting the input data into the pre-trained language and vision model to obtain the target mask map when the task starts, the code of the robot operation task, and the task operation steps; and interacting with step 2) in real time to provide information about objects in the environment; The code of the robot operation task is obtained by combining the function defined in step 1) with the code generation capability of the large language model; and the target mask image has the same resolution as the visual image of the operation environment obtained in step 2); 4) According to the robot operation task code, the robot generates a task trajectory, and obtains the target mask image at the end after the robot executes the code; 5) Based on the target mask image at the end and the target mask image at the beginning, combined with the task information, determine whether the target mask image at the end meets the task requirements, that is, whether the task is completed; wherein the task information is generated by the pre-trained language model in step 3), including task understanding information, robot operation steps, and the position information of the target object when the task is completed.

2. The robot operation method based on feedback pre-training language and visual model according to claim 1, characterized in that: The language instruction in step 1) is in the form of natural language based on an open vocabulary.

3. The robot operation method based on feedback pre-trained language and visual model according to claim 1, characterized in that: The visual image information described in step 1) includes RGB image information and depth image information.

4. The robot operation method based on feedback pre-trained language and visual model according to claim 1, characterized in that: The prompt template information design described in step 2) is specifically as follows: 2-1) Design the robot action primitive library function, which can be expressed as: Among them, d represents the distance of movement; q represents the quaternion of the end operation of the robot arm; 2-2) Design user personal preferences, which can be expressed as: Personal Preference={'smoother','safer','changepose',...}; Among them, words such as smoother represent human preference settings for robot motion to optimize the robot's operation; 2-3) Design task feedback information according to the task feedback prompt output by the task feedback module; 2-4) Based on the pre-trained visual system, the img-to-text feature is used to use the name, type, and location information of objects in the operating environment as prompts.

5. The robot operation method based on feedback pre-trained language and visual model according to claim 1, characterized in that: The pre-trained language and visual model described in step 3) is composed of a pre-trained large language model module and a pre-trained visual language model.

6. The robot operation method based on feedback pre-trained language and visual model according to claim 1, characterized in that: The robot operation code described in step 3) is generated based on the robot operation function in the Prompt template information designed in step 2) combined with the code generation capability of the large language model.

7. The robot operation method based on feedback pre-trained language and visual model according to claim 1, characterized in that: The target mask map when the task described in step 3) starts to be executed is generated by a pre-trained visual model, and a target mask map with higher accuracy is generated by combining the depth information obtained in step 1) and image processing technology.

8. The robot operation method based on feedback pre-trained language and visual model according to claim 1, characterized in that: The motion trajectory of the robot described in step 4) is generated by inverse kinematics solution, coordinate system transformation and other methods according to the execution function.

9. The robot operation method based on feedback pre-trained language and visual model according to claim 1, characterized in that: In step 5), whether the task is completed is determined by the task feedback module, which includes the visual module and the pre-trained language module; the visual module aligns the input target mask image at the beginning of the task with the target mask image at the end of the task, and outputs information such as the offset distance of the mask center point and the target object posture through visual processing methods such as two-dimensional posture estimation; The pre-trained language module is responsible for comparing the information output by the visual module with the robot task operation code obtained in step 3) and determining whether the task is completed.

10. A robot operating system based on feedback pre-trained language and visual models, characterized in that: include: Real-time perception module of visual language information, task reasoning and execution module, task execution feedback module, and robot control module; The visual language information real-time perception module is used to obtain visual image information including RGB information and depth information in real time, and is used to obtain natural language information of human instructions in real time; The task reasoning and execution module generates codes and task execution steps suitable for robot operation based on the input data using pre-trained language and visual models, and generates a target mask map so that the robot can perform tasks more accurately; The task execution feedback module is used to determine whether the task is completed after the robot completes the task reasoning and execution modules, and update the prompt template information; The robot control module is a key component in the robot system and is responsible for implementing the underlying motion control. Its functions include coordinate system transformation of the position and posture in the operation instructions, and converting these instructions into drive signals for the joint motors through inverse kinematics solutions, so that the robot can accurately reach the target position.

Images