Training Method and System for Embodied Intelligent Task Performers Based on Multimodal Large Models

Abstract

This application relates to the field of embodied intelligence technology, specifically disclosing a training method and system for an embodied intelligence task executor based on a multimodal large model. The training method mainly includes: acquiring a visual image O of the current environment and an input text instruction I; generating a guiding instruction Prompt1 based on the visual image O and the text instruction I; inputting the visual image O, the text instruction I, and the guiding instruction Prompt1 into the execution large model to obtain an action instruction W1 output by the large model; and executing the action instruction W1 to identify the object to be operated on in the current environment. This application enables language models to better understand and adapt to knowledge of the real physical world, thereby providing reasonable planning, obtaining timely and effective feedback, adjusting the execution process through feedback, and handling complex instructions more flexibly. Furthermore, it allows for more diverse and complex action execution processes to be achieved through the action instructions output by the large model.

Description

Technical Field

[0001] This application relates to the field of embodied intelligence technology, and in particular to a training method and system for embodied intelligence task executors based on multimodal large models. Background Technology

[0002] Embodied execution is the process of transforming simple instructions obtained through perception and planning into a form that embodied intelligent robots can understand, enabling the robot to perform a series of operations to complete these simple instructions. It is the final step in the task completion process within the overall framework of embodied intelligent robots, and also the most important step in the implementation of embodied intelligent robot systems.

[0003] Inspired by the powerful performance of basic models, language-driven embodied intelligent robots have become a focus of attention. In the field of embodied intelligent robots, language not only provides a user-friendly interface, but also has the potential for cross-task skill generalization and long-term task reasoning. The current training method for embodied intelligent robots is generally to parse text instructions through language models, extract key information, and generate corresponding task execution instructions. Complex execution instructions that call functions in APIs are usually implemented by writing code sequences.

[0004] However, during the implementation of the relevant technical solutions, at least the following technical problems were discovered:

[0005] 1. Language models are difficult to adapt to the physical environment during robot execution: Language models are usually trained based on text data, which often has no direct connection with the physical environment. Therefore, language models have difficulty directly understanding concepts such as entities and spatial relationships in the physical environment, making it difficult to generate task execution instructions that are adapted to the physical environment.

[0006] 2. Language models lack direct visual perception of the real world: Language models are usually trained on text data, which cannot directly provide visual perception of the real world. As a result, it is difficult to obtain timely and effective feedback from the physical environment and adjust the execution process through feedback.

[0007] 3. When using code generation to represent the execution process of robot actions, the code sequence is limited by the functionality of the API used: code generation is usually based on API libraries, but the functionality of API libraries is limited. API libraries may have some limitations, such as only being able to implement certain basic operations, so they cannot handle complex task execution processes and it is difficult to implement complex instructions. Summary of the Invention

[0008] This application provides a training method and system for embodied intelligent task executors based on a multimodal large model, which solves the problem that the code sequence is limited by the functionality of the API used when using code generation to represent the execution process of robot actions in the prior art, and can handle complex instructions more flexibly.

[0009] In a first aspect, this application provides a method for training an embodied intelligent task executor based on a multimodal large model, which includes the following steps:

[0010] Step 1: Obtain the visual image O of the current environment and the input text instruction I, and generate a guidance instruction based on the visual image O and the text instruction I. ;

[0011] Visual image O, text instruction I, and guidance instruction Input is fed into the large model, and the action commands output by the large model are obtained. ;

[0012] Step 2: Execute the action command This allows us to identify the objects to be operated on in the current environment;

[0013] Step 3: Call the API based on text instruction I from the action API library, and generate a bootstrap instruction based on the called API, text instruction I, and the object to be operated on in the current environment. , will guide instructions Input is fed into the large model, and the action commands output by the large model are obtained. ;

[0014] Execute action instructions .

[0015] Furthermore, the action command and action commands All are string sequences W, where W = { , ,..., }; In the formula, The string w represents the nth step.

[0016] Furthermore, execute the large model generation step string. According to the step string The resulting string sequence W is obtained.

[0017] Furthermore, based on the step string Obtaining the string sequence W involves: calculating the generation probability of the string sequence W. , for: In the formula, This represents the probability of generating the string w in the j-th step, given that the strings w in the first j-1 steps are known.

[0018] Furthermore, execute the action instructions. include:

[0019] The SAM model is used to segment the visual image O, resulting in an image library composed of visual segments of all objects.

[0020] Text embeddings and image embeddings are generated based on text instruction I and an image library, respectively. The CLIP model is used to obtain the matching results of the text embeddings and image embeddings, thereby identifying the object to be operated on in the current environment.

[0021] Furthermore, the APIs called in step three include the Pick action API or the Place action API.

[0022] Furthermore, the API is a Python function presented as a string.

[0023] Furthermore, in executing action instructions During the process, simultaneously obtain and execute action instructions. The execution feedback includes success and failure. If the execution feedback is failure, then step one is executed.

[0024] Secondly, this application provides a training system for embodied intelligent task executors based on a multimodal large model, which adopts the training method for embodied intelligent task executors based on a multimodal large model as described in the first aspect, and includes:

[0025] The high-level step planning module is used to acquire the visual image O of the current environment and the input text instruction I, and generate guiding instructions based on the visual image O and the text instruction I. ;to combine visual image O, text instruction I, and guidance instruction Input is fed into the large model, and the action commands output by the large model are obtained. ;

[0026] The environmental perception and recognition module is used to execute action commands. This allows us to identify the objects to be operated on in the current environment;

[0027] The low-level execution control module is used to call text-based APIs from the action API library, and generate bootstrap instructions based on the called API, the text-based API, and the object to be operated on in the current environment. , will guide instructions Input is fed into the large model, and the action commands output by the large model are obtained. The low-level execution control module is also used to execute action instructions. .

[0028] Thirdly, this application provides a training device for an embodied intelligent task executor based on a multimodal large model, comprising:

[0029] Memory, used to store computer programs;

[0030] The processor is used to implement the steps of the first aspect of the training method for an embodied intelligent task executor based on a multimodal large model when executing the computer program.

[0031] The technical solution provided in this application has at least the following technical effects or advantages:

[0032] 1. By inputting visual images and text instructions into the execution model and generating action instructions based on visual images and text instructions, the problem of language models being difficult to adapt to the physical environment during robot execution in existing technologies is effectively solved. This allows language models to better understand and adapt to knowledge of the real physical world, thereby providing reasonable planning.

[0033] 2. By acquiring visual images of the current environment and combining them with text instructions to input into the execution model, and adjusting the execution process through feedback, the problem of language models lacking direct visual perception of the real world in existing technologies is effectively solved. This allows for timely and effective feedback, and the execution process can be adjusted through feedback.

[0034] 3. By calling the motion API library based on text instructions and combining it with the text instructions and the objects to be operated on in the current environment, the problem of code sequence being limited by the functions of the API used when using code generation to represent the execution process of robot actions in existing technologies is effectively solved. It can handle complex instructions more flexibly and achieve more diverse and complex motion execution processes through motion instructions output by large models. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of the large-scale model implemented in this application;

[0036] Figure 2 This is a schematic diagram of the environmental perception and recognition module of this application;

[0037] Figure 3 This is a flowchart of the training method for the embodied intelligent task executor based on a multimodal large model in this application. Detailed Implementation

[0038] This application designs a general-purpose robot task execution system, namely an embodied intelligent task executor system. It utilizes the context learning capability of a large language model to generate execution decision actions from natural language instructions. It has rich modal inputs and a flexible retrieval architecture to handle different types of complex instructions.

[0039] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0040] Example 1

[0041] like Figures 1-3 As shown, this embodiment provides a training method for an embodied intelligent task executor based on a multimodal large model, which includes the following steps:

[0042] Step 1: Obtain the visual image O of the current environment and the input text instruction I, and generate a guidance instruction based on the visual image O and the text instruction I. ;to combine visual image O, text instruction I, and guidance instruction Input is fed into the large model, and the action commands output by the large model are obtained. .

[0043] The execution large model is a self-trained actuator large model. Functionally, its input and output are consistent with those of a multimodal large model similar to GPT4-4v. The difference is that the self-trained actuator large model will have basic knowledge of the real physical world, thus adapting to the execution tasks of embodied intelligent robots. By combining the powerful semantic capabilities of the large language model with the physical environment state cognition capabilities obtained from the training data, it ultimately generates appropriate execution process planning steps.

[0044] Specifically, the system uses multimodal content consisting of visual images (O) and text instructions (I) as input. Visual images (O) provide the model with the basic function of perceiving the execution environment of the embodied intelligent robot, while text instructions (I) represent the task objectives for the robot to perform its tasks, guided by relevant APIs. It serves as a communication bridge between natural language and robot execution control, thereby enabling the use of the language capabilities of large-scale execution models to control the robot's actions.

[0045] The output is the next step that the robot needs to execute, the action command. and action commands All are string sequences W, where W = { , ,..., }; In the formula, The string 'w' represents the nth step, expressed as Python code. This executes the large model to generate the step strings. According to the step string The resulting string sequence W is obtained.

[0046] Based on the step string Obtaining the string sequence W involves: calculating the generation probability of the string sequence W. , for: In the formula, This represents the probability of generating the string w in the j-th step, given the strings w from the first j-1 steps. This is achieved by modeling a generation probability. This results in the final generation of a complete string sequence W.

[0047] like Figure 1 As shown, the execution big model mainly consists of three modules: an image encoder, a mapping layer, and a big language model. The main feature of this structure is that it extends the big language model to multimodal input. With the help of the added visual input, the big language model can perceive the physical environment during the execution of the embodied intelligent robot, thereby supporting the model to make appropriate feedback to environmental changes. This includes not only adjusting decisions, but also replanning steps after a single-step execution failure, which is very suitable for the task execution requirements in complex environments.

[0048] During training, the image encoder is frozen, and the main focus is on training the mapping layer and fine-tuning the large language model. The primary goal is to create a large execution model adapted to the embodied intelligence domain. The data source consists of task execution step data collected from real-world environments, meeting the training requirements for embodied specialization of the large execution model.

[0049] Step 2: Execute the action command This allows us to identify the objects to be operated on in the current environment.

[0050] like Figure 2 As shown, in the process of environmental perception, the action instructions given in step one are used... The system will execute the action command. Specifically, this includes: using the SAM model to segment the visual image O, resulting in an image library composed of visual segments of all objects. This image library will be used in a multimodal retrieval and matching process when searching for images of objects required for the operation of an embodied intelligent robot.

[0051] Multimodal matching is crucial in understanding task text instructions. Embodied intelligent robots do not possess semantic understanding capabilities. Therefore, for an object to be manipulated, the object needs to retrieve a corresponding image through its text description before its three-dimensional real-world spatial position can be obtained through methods such as camera calibration, and then handed over to the robot for execution.

[0052] In the process of multimodal retrieval, the scheme of this embodiment adopts the CLIP model, which has a trained and aligned text encoder and image encoder. After generating text embeddings and image embeddings according to the text instruction I and the image library respectively, it can identify the object to be operated in the current environment based on the matching results of the obtained text embeddings and image embeddings.

[0053] Step 3: Call the API based on text instruction I from the action API library, and generate a bootstrap instruction based on the called API, text instruction I, and the object to be operated on in the current environment. , will guide instructions Input is fed into the large model, and the action commands output by the large model are obtained. ; Execute action instructions .

[0054] To facilitate the completion of robotic tasks by large language models, this embodiment uses Python code to build a bridge between natural language and the action interface of the embodied intelligent robot. During the modeling process of the large model, APIs in the action API library that are highly relevant to the task, as well as calls to various modules, will be organized into text-based guiding instructions. The input is given to the execution model to assist it in calling code and completing the task execution process of the embodied intelligent robot.

[0055] The APIs called include the Pick action API or Place action API, and available APIs from the environmental perception and recognition module. The action reserves in the API library are sufficient to enrich the execution capabilities of the embodied intelligent robot to perform complex tasks, which is the significance of the embodied intelligent task executor.

[0056] API hints include Python application programming interfaces and contextual examples to demonstrate how to use them; in words, APIs are Python functions presented as strings that can be executed by the Python interpreter to drive the robot's actions.

[0057] To efficiently utilize the vast amount of API information in the action API library and ensure that large execution models can efficiently complete manageable tasks while possessing a large number of operable skills, the system combines guiding instructions. The system will then search for APIs based on the input command I, find the API that best matches the task, and finally combine them into appropriate boot commands. Input is given to the execution model so that it can efficiently utilize the boot instructions. The content provides an appropriate step plan based on instructions and contextual examples, enabling the robot to successfully complete the task.

[0058] Based on the above method, the robot strategy generated by the API for large models is very flexible. The simple API naming and highly readable Python code make the generated strategy code highly interpretable.

[0059] In addition, when executing action instructions During the process, simultaneously obtain and execute action instructions. The system provides execution feedback. Execution feedback includes success and failure; if the feedback is failure, it returns to step one. After calling the API, the actuator system provides task execution feedback based on the robot's interaction with the environment, dynamically determining whether a new perception and planning scheme needs to be generated. This ensures the system can effectively respond to environmental changes and maintain optimal performance during task execution.

[0060] Example 2

[0061] This embodiment provides a training system for an embodied intelligent task executor based on a multimodal large model. It adopts the training method for an embodied intelligent task executor based on a multimodal large model in Embodiment 1, which includes: a high-level step planning module, an environmental perception and recognition module, and a low-level execution control module.

[0062] The high-level step planning module is used to acquire the visual image O of the current environment and the input text instruction I, and generate guiding instructions based on the visual image O and the text instruction I. ;to combine visual image O, text instruction I, and guidance instruction Input is fed into the large model, and the action commands output by the large model are obtained. ;

[0063] The environmental perception and recognition module is used to execute action commands. This allows us to identify the objects to be operated on in the current environment;

[0064] The low-level execution control module is used to call text-based APIs from the action API library, and generate bootstrap instructions based on the called API, the text-based API, and the object to be operated on in the current environment. , will guide instructions Input is fed into the large model, and the action commands output by the large model are obtained. The low-level execution control module is also used to execute action instructions. .

[0065] The embodied intelligent task executor training system based on a multimodal large model in this embodiment has the advantages of the embodied intelligent task executor training method based on a multimodal large model in Embodiment 1. Moreover, the system in this embodiment can automate the embodied intelligent task executor training method.

[0066] Example 3

[0067] This embodiment provides a training device for an embodied intelligent task executor based on a multimodal large model, which includes a memory and a processor. The memory is used to store a computer program; the processor is used to execute the computer program to implement the steps of the first aspect of the training method for an embodied intelligent task executor based on a multimodal large model.

[0068] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0070] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0071] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present application, based on the technical solution and concept of the present application, should be covered within the scope of protection of the present application.

Claims

1. A training method for embodied intelligent task executors based on multimodal large models, characterized in that, It includes the following steps: Step 1: Obtain the visual image O of the current environment and the input text instruction I, and generate a guidance instruction based on the visual image O and the text instruction I. ; Visual image O, text instruction I, and guidance instruction Input is fed into the large model, and the action commands output by the large model are obtained. ; Step 2: Execute the action command This allows us to identify the objects to be operated on in the current environment; Step 3: Call the API based on text instruction I from the action API library, and generate a bootstrap instruction based on the called API, text instruction I, and the object to be operated on in the current environment. , will guide instructions Input is fed into the large model, and the action commands output by the large model are obtained. ; Execute action instructions ; in executing action instructions During the process, simultaneously obtain and execute action instructions. The execution feedback includes success and failure. If the execution feedback is failure, then proceed to step one. The API is a Python function presented as a string; the action instructions... and action commands All are string sequences W, where W = { , ,..., }; In the formula, The string w represents the nth step; the string representing the steps generated by executing the large model. According to the step string The resulting string sequence W is obtained; Based on the step string Obtaining the string sequence W involves: calculating the generation probability of the string sequence W. , for: In the formula, This represents the probability of generating the string w in the j-th step, given that the strings w in the first j-1 steps are known.

2. The training method for embodied intelligent task executors based on multimodal large models as described in claim 1, characterized in that, Execute action instructions include: The SAM model is used to segment the visual image O, resulting in an image library composed of visual segments of all objects. Text embeddings and image embeddings are generated based on text instruction I and an image library, respectively. The CLIP model is used to obtain the matching results of the text embeddings and image embeddings, thereby identifying the object to be operated on in the current environment.

3. The training method for embodied intelligent task executors based on multimodal large models as described in claim 1, characterized in that, The APIs called in step three include the Pick action API or the Place action API.

4. A training system for embodied intelligent task executors based on a multimodal large model, which employs the training method for embodied intelligent task executors based on a multimodal large model as described in claims 1-3, characterized in that... It includes: The high-level step planning module is used to acquire the visual image O of the current environment and the input text instruction I, and generate guiding instructions based on the visual image O and the text instruction I. ;to combine visual image O, text instruction I, and guidance instruction Input is fed into the large model, and the action commands output by the large model are obtained. ; The environmental perception and recognition module is used to execute action commands. This allows us to identify the objects to be operated on in the current environment; The low-level execution control module is used to call text-based APIs from the action API library, and generate bootstrap instructions based on the called API, the text-based API, and the object to be operated on in the current environment. , will guide instructions Input is fed into the large model, and the action commands output by the large model are obtained. ; Execute action instructions .

5. A training device for an embodied intelligent task executor based on a multimodal large model, characterized in that, It includes: Memory, used to store computer programs; A processor, configured to implement the steps of the training method for an embodied intelligent task executor based on a multimodal large model as described in any one of claims 1 to 3 when executing the computer program.

Images