A robotic system and method based on feedback pre-training language and vision models

Abstract

The present invention discloses a robot system and method based on feedback pre-trained language and visual models. S1 involves information perception of the robot's operating environment; S2 involves designing a prompt template, using information such as the robot's action primitive functions, personal preferences, and task feedback as prompt-generated input data; S3 utilizes a pre-trained large language model and a visual language model based on the input data and image information in the environment to obtain the robot's operating task code and the initial target mask image; S4 generates the robot's operating trajectory based on the generated robot operating task code through methods such as inverse kinematics solution and coordinate transformation between the robot arm and the camera, and executes the operating task; S5, after the robot completes the execution, the completed target mask image is input into the task feedback model. The present invention enables the robot to complete the task objectives indicated by language instructions without requiring extensive model training and data collection processes, and is highly operational and easy to use.

Description

Technical Field

[0001] The present invention relates to the field of autonomous decision-making and motion control of robots, and in particular to a robot system and method based on feedback pre-training language and vision models. Background Art

[0002] In the field of intelligent robotics, task planning in unstructured environments has always been a challenging task. Traditional robotic systems often rely on predefined task planning strategies, which are typically based on static environmental models and fixed task objectives, making them difficult to adapt to the diversity and dynamics of the real world. In unstructured scenarios such as homes and offices, robots need to have flexible task planning capabilities, able to autonomously adjust their action strategies based on environmental changes and task complexity, and achieve efficient and intelligent operation. In short, intelligent robots must be able to plan tasks in unstructured environments, understand language instructions represented in natural language, infer corresponding operation objectives, and generate task execution sequences. These are currently difficult problems that intelligent robots still need to solve.

[0003] In intelligent robot operations, pre-trained models can be applied to all aspects of task planning. First, they enable real-time perception and understanding of natural language commands and visual images, thereby accurately capturing and understanding human intent. Second, the task reasoning and execution module based on the pre-trained model can utilize language and visual information for end-to-end task planning and execution, generating intelligent operational strategies appropriate to the current environment and task objectives. Finally, through the task execution feedback module, the intelligent robot can monitor the progress and effectiveness of task execution in real time and dynamically adjust and optimize the task plan based on this feedback information, thereby achieving efficient operation in complex, dynamic, and unstructured environments.

[0004] In summary, it is necessary to explore the use of pre-trained language models and visual models to solve task planning in unstructured scenarios in order to realize task operations of intelligent robots. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the above-mentioned prior art and to provide a robot system and method based on feedback pre-trained language and visual models to enable the robot to complete human instructions with diverse and open vocabulary, master a variety of operating skills, and adapt to a changing operating environment.

[0006] The present invention is achieved through the following technical solutions:

[0007] A robot operation method based on a pre-trained language and vision model with feedback, the method comprising the following steps:

[0008] 1) Obtain real-time natural language instructions from humans and visual image information in the robot's operating environment.

[0009] 2) Design a prompt template, including information such as the robot action primitive library functions, personal preferences, and task feedback. Combined with the object information provided in step 3), generate input data.

[0010] 3) Input data to the pre-trained language and vision model to obtain the target mask at the start of the task, as well as the robot operation task code and task operation steps. This model also interacts with step 2) in real time to provide information about objects in the environment.

[0011] The code for the robot operation task is obtained by combining the function defined in step 1) with the code generation capability of the large language model. The target mask image has the same resolution as the visual image of the operation environment obtained in step 2).

[0012] 4) According to the robot operation task code, the robot generates a task trajectory, and obtains the target mask image at the end after the robot executes the code.

[0013] 5) Based on the final target mask image and the initial target mask image, combined with the task information, determine whether the final target mask image meets the task requirements, that is, whether the task is completed. The task information is generated by the pre-trained language model in step 3) and includes task understanding information, robot operation steps, and the location of the target object at task completion.

[0014] Furthermore, the visual image information in step 1) includes an RGB image and a depth image.

[0015] Furthermore, the Prompt template is designed as follows:

[0016] 1-1) Design the robot action primitive library function, which can be expressed as:

[0017]

[0018] Among them, d represents the distance of movement; q represents the quaternion of the end operation of the robotic arm.

[0019] 1-2) Design user personal preferences, which can be expressed as:

[0020] Personal Preference={'smoother','safer','changepose',...}

[0021] Among them, words such as smoother represent human preference settings for robot movement to optimize the robot's operation.

[0022] 1-3) Design task feedback information based on the task feedback prompt output by the task feedback module.

[0023] 1-4) Based on the pre-trained visual system, the image-to-text feature is used to provide the name, type, and location information of objects in the operating environment as prompts.

[0024] Furthermore, the code of the robot operation task is recombined by combining the robot action primitive function library in the predefined Prompt with the code generation capability of the large language model to generate the robot operation code.

[0025] Furthermore, the mask image when the task target starts to be executed is generated by a pre-trained visual model, which combines depth information and image processing technology to generate a target mask map with higher accuracy.

[0026] Furthermore, the task completion determination is performed by a task feedback module, which includes a vision module and a pre-trained language module. The vision module aligns the input target mask image at the start of the task with the target mask image at the end of the task. Using visual processing methods such as two-dimensional pose estimation, it outputs information such as the offset distance of the mask center point and the pose of the target object. The pre-trained language module is responsible for comparing the information output by the vision module with the motion parameters obtained from the robot's task operation code information to determine whether the task is completed.

[0027] A robot operating system based on a feedback-based pre-trained language and vision model includes: a real-time perception module of visual language information, a task reasoning and execution module, a task execution status feedback module, and a robot control module.

[0028] The visual language information real-time perception module is used to obtain visual image information including RGB information and depth information in real time, and is used to obtain natural language information of human instructions in real time.

[0029] The task reasoning and execution module uses pre-trained language and visual models based on the input data to generate code and task execution steps suitable for robot operation, and at the same time generates a target mask map so that the robot can perform tasks more accurately.

[0030] The task execution status feedback module is used to determine whether the task is completed after the robot completes the task reasoning and execution modules, and to update the Prompt template information.

[0031] The robot control module is a key component of the robotic system, responsible for low-level motion control. Its functions include transforming the position and posture in the operating instructions into coordinate systems and, through inverse kinematics, converting these instructions into drive signals for the joint motors, enabling the robot to accurately reach its target position.

[0032] The present invention also provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the robot operation method based on the feedback-based pre-trained language and visual model is implemented as described above.

[0033] Compared with the prior art, the present invention has the following advantages and effects:

[0034] Taking into account the constraints of unstructured scenarios and the difficulty of task planning, this paper uses a pre-trained large language model for knowledge reasoning, task understanding, and code generation. It also uses a pre-trained visual model for object detection and object mask generation.

[0035] Considering the importance of prompt engineering to pre-trained models, this paper designs three types of prompts, including robot action primitive library functions, user personal preferences, and information fed back by the task feedback module. This improves the accuracy of the pre-trained model output.

[0036] The present invention takes into account real-time feedback during the task execution phase. Through a task feedback module, the visual model is used to judge the completion of the task based on the target mask images before and after the task execution. The pre-trained language model is used to adjust and generate new prompts in real time to achieve closed-loop control of the entire robot operation, thereby improving the completion rate of the robot task. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process of the present invention.

[0038] Figure 2 This is the design drawing for the prompt project. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0040] Prompt engineering can help explain a model's decision-making process and underlying mechanisms. By designing appropriate prompts or inputs, we can enhance the explainability of model behavior, improving both interpretability and credibility. Providing appropriate prompts for large pre-trained models is crucial.

[0041] Based on the technical problems existing in the above-mentioned prior art, the present invention realizes a set of prompt solutions with feedback for robots to perform operation tasks according to language instructions, such as Figure 2 As shown. The specific design of the Prompt template is:

[0042] 1-1) Design the robot action primitive library function, which can be expressed as:

[0043]

[0044] Among them, d represents the distance of movement; q represents the quaternion of the end operation of the robotic arm.

[0045] 1-2) Design user personal preferences, which can be expressed as:

[0046] Personal Preference={'smoother','safer','changepose',...}

[0047] Among them, words such as smoother represent human preference settings for robot movement to optimize the robot's operation.

[0048] 1-3) Design task feedback information and use the output of the task feedback module as a prompt.

[0049] 1-4) Based on the pre-trained visual system, the image-to-text feature is used to provide the name, type, and location information of objects in the operating environment as prompts.

[0050] The specific prompt example is as follows:

[0051] 1. Instruction: "Clean up the desktop"

[0052] 2. Scenario:

[0053] 2-1. There is a basket, tennis ball, cup, bread and books on the table

[0054] 2-2. The bread is at (5,5) in the diagram, the cup is at (10,10), etc.

[0055] 3. Robot action primitive library functions: move_j(d,q), paw_open(d), etc.

[0056] 4. Personal preference: I only want books in the basket, and cups only want to be placed next to the basket.

[0057] 5.Task feedback information: (added after the task is completed).

[0058] Example 1:

[0059] like Figure 1 This embodiment provides a robot operation method based on feedback-based pre-trained language and visual models, and its workflow is briefly described as follows:

[0060] S1. Obtain language instructions in the form of natural language and visual image information in the operating environment, including RGB images and depth images.

[0061] S2. Based on the designed Prompt template rules, the environmental information, robot operation task information, object information of the image obtained in S1 processed in S3, user personal preference information, task feedback information, etc. are combined into a Prompt template to generate input data.

[0062] S3, based on the pre-trained large language model and visual language model, transmits object information in the operating environment to S2. When the task execution conditions are met, such as the location information and mask information of the task target, the model begins task reasoning, disassembles the task process, and obtains the task execution code. The specific code example is as follows:

[0063] Call the move function;

[0064] move_j(5,5,0.707,0,0.707,0) calls the gripper opening function;

[0065] paw_open(2);

[0066] Call the move function;

[0067] move_j(10,0,0.707,0,0.707,0);

[0068] Call the gripper closing function;

[0069] paw_close().

[0070] S4. Based on the robot operation code in S3, the robot generates a task trajectory. After the robot executes the code, it obtains the target mask image at the end.

[0071] S5. Based on the target mask image at the end and the target mask image at the start, combined with the task information, determine whether the target mask image at the end meets the task requirements, that is, whether the task is completed.

[0072] Preferably, in order to illustrate the improvements of the present invention, Example 2 is given as an example:

[0073] Example 2:

[0074] 1. Obtain human language instructions such as "Please bring me the soda on the table", operating environment images, and depth information.

[0075] 2. Initialize the Prompt template:

[0076] 1) Instruction: "Please bring me the soda on the table";

[0077] 2) Scenario:

[0078] a. There is a basket, tennis ball, cup, soda and books on the table;

[0079] b.(Object location information to be added);

[0080] 3) Robot action primitive library functions: move_j(d,q), paw_open(d), etc.

[0081] 4) Personal preferences: I only want books in the basket, cups to be placed next to the basket, only Pepsi soda, etc.

[0082] 5) Task feedback information: (added after the task is completed).

[0083] 3. Input the Prompt template and environment image information into the pre-trained large language and vision model. This generates the target mask image (Image1) before the robot operation is executed, generates the robot operation code, and improves the object position information in the Prompt.

[0084] The generated partial execution code is as follows:

[0085] Call the move function:

[0086] move_j(5,5,0.707,0,0.707,0)

[0087] Call the gripper open function

[0088] paw_open(2)

[0089] Calling the move function

[0090] move_j(10,0,0.707,0,0.707,0)

[0091] Call the gripper close function

[0092] paw_close()

[0093] The improved prompt is as follows:

[0094] 1-1. Instruction: "Please bring me the soda on the table";

[0095] 1-2. Scenario:

[0096] On the table were baskets, tennis balls, cups, sodas, and books;

[0097] The soda can is at (5, 5), the tennis ball is at (5, 16), and so on;

[0098] 1-3. Robot action primitive library functions: move_j(d,q), paw_open(d), etc.

[0099] 1-4. Personal preferences: I only want books in the basket, cups only to be placed next to the basket, only Pepsi soda, etc.

[0100] 1-5. Task feedback information: (added after the task is completed).

[0101] 4. The robot executes the task operation code and generates the motion trajectory l1.

[0102] 5. The robot stops and obtains the target mask image Image2 at the time of the stop. Image1, Image2, and Prompt are input into the task feedback module. If the task identifier task_get = False, it marks the task as failed. The Prompt is updated and returns to step 2 to resume the task.

[0103] The above method can be described as:

[0104] initialization:

[0105] Robot action primitive library function set: A;

[0106] Pre-trained LLM and VLM models;

[0107] Initialization prompt: P 0；

[0108] Main program:

[0109] Input: operating environment image information Img, depth information Dep, language command L

[0110] Using VLM, search for objects in the image and get: Object_nums, Object_species

[0111] for iin Object_nums:

[0112] Object_position[i] is updated to the corresponding Object_species

[0113] Add prompt: P0->P1

[0114] Input P1 to LLM&VLM, get Code: C1, Img_initial, specific operation steps E

[0115] Robot Execution

[0116] The robot execution is completed and Img_finish is obtained

[0117] Input Img_finish, E, and Img_initial into the task feedback model to obtain the identifier

[0118] If(task_get=False)

[0119] Update Prompt: P1->P2

[0120] Re-loop execution

[0121] Else:

[0122] Break.

[0123] The present invention also discloses a storage medium having computer instructions or programs for executing a robot operation based on a feedback-based pre-trained language and visual model provided by the present invention. When the computer instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0124] As described above, the present invention can be implemented well.

[0125] The implementation methods of the present invention are not limited to the above-mentioned embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A robot operation method based on feedback pre-trained language and visual models, characterized in that: The following steps are involved: 1) Obtaining real-time natural language instructions from humans and visual image information from the robot’s operating environment; 2) Design a prompt template, including information such as the robot action primitive library functions, personal preferences, and task feedback; Combined with the object information provided in step 3), input data is generated; 3) Input the input data into the pre-trained language and vision model to obtain the target mask map at the beginning of the task, the robot operation task code, and the task operation steps; And interact with step 2) in real time to provide information about objects in the environment; The code for the robot operation task is obtained by combining the function defined in step 1) with the code generation capability of the large language model; and the target mask image has the same resolution as the visual image of the operation environment obtained in step 2); 4) Based on the robot operation task code, the robot generates the task trajectory and obtains the target mask map at the end after executing the code; 5) Based on the target mask at the end and the target mask at the start, combined with the task information, determine whether the target mask at the end meets the task requirements, that is, whether the task is completed. The task information is generated by the pre-trained language model in step 3) and includes task understanding information, robot operation steps, and the location of the target object when the task is completed.

2. The robot operation method based on feedback pre-trained language and vision models according to claim 1, characterized in that: The language instruction in the form of natural language based on an open vocabulary as described in step 1).

3. The robot operation method based on feedback pre-trained language and vision models according to claim 1, characterized in that: The visual image information in step 1) includes RGB image information and depth image information.

4. The robot operation method based on feedback pre-trained language and vision models according to claim 1, characterized in that: The prompt template information design described in step 2) is specifically as follows: 2-1) Design the robot action primitive library function, which can be expressed as: ； Among them, d represents the distance of movement; q represents the quaternion of the end operation of the robot arm; 2-2) Design user personal preferences, which can be expressed as: ； Among them, words such as smoother represent human preferences for robot movement to optimize the robot's operation; 2-3) Design task feedback information based on the task feedback prompt output by the task feedback module; 2-4) Based on the pre-trained visual system, the image-to-text feature is used to provide the name, type, and location information of objects in the operating environment as prompts.

5. The robot operation method based on feedback pre-trained language and vision model according to claim 1, characterized in that: The pre-trained language and vision model in step 3) is composed of a pre-trained large language model module and a pre-trained visual language model.

6. The robot operation method based on feedback pre-trained language and vision model according to claim 1, characterized in that: The robot operation code described in step 3) is generated based on the robot operation function in the prompt template information designed in step 2) combined with the code generation capability of the large language model.

7. The robot operation method based on feedback pre-trained language and vision model according to claim 1, characterized in that: The target mask map at the start of the task in step 3) is generated by a pre-trained visual model, and is combined with the depth information obtained in step 1) and image processing technology to generate a target mask map with higher accuracy.

8. The robot operation method based on feedback pre-trained language and vision model according to claim 1, characterized in that: The motion trajectory of the robot in step 4) is generated by inverse kinematics solution, coordinate system transformation and other methods according to the execution function.

9. The robot operation method based on feedback pre-trained language and vision model according to claim 1, characterized in that: In step 5), the task feedback module determines whether the task is completed. It includes a visual module and a pre-trained language module. The visual module aligns the input target mask image at the beginning of the task with the target mask image at the end of the task, and outputs information such as the offset distance of the mask center point and the target object posture through visual processing methods such as two-dimensional posture estimation. The pre-trained language module is responsible for comparing the information output by the vision module with the robot task operation code obtained in step 3) and determining whether the task is completed.

10. A robot operating system based on feedback pre-trained language and visual models for the robot operating method according to claim 1, characterized in that: include: Real-time perception module of visual language information, task reasoning and execution module, task execution status feedback module, and robot control module; The visual language information real-time perception module is used to obtain visual image information including RGB information and depth information in real time, and is used to obtain natural language information of human instructions in real time; The task reasoning and execution module uses pre-trained language and vision models based on the input data to generate code and task execution steps suitable for robot operation, and also generates a target mask map so that the robot can perform the task more accurately; The task execution feedback module is used to determine whether the task is completed after the robot completes the task reasoning and execution modules, and update the prompt template information; The robot control module is a key component in the robot system and is responsible for implementing low-level motion control. Its functions include coordinate system transformation of the position and posture in the operation instructions, and converting these instructions into drive signals for the joint motors through inverse kinematics solution, so that the robot can accurately reach the target position.

Images