Body-aware agent training system and method

Abstract

This invention discloses an embodied agent training system and method. The system includes: a task generation module, used to randomly generate a scene configuration file and input the configuration file into a virtual environment module to construct a three-dimensional scene, and generate tasks based on the current scene conditions using a large language model; a task solving module, used to generate and solve the tasks generated by the task generation module using a large language model; a virtual environment module, used to construct a three-dimensional scene based on the configuration file and obtain the environmental state of the scene, calculate the optimal control sequence of the agent based on the environmental state and the agent's plan, execute the agent's actions in the environment according to the optimal control sequence and obtain the agent's observations, and obtain labeled training data; the agent's plan is obtained from the task solving module; and a model training module, used to train the embodied agent model based on the training data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an embodied intelligent agent training system and method. Background Technology

[0002] Embodied agents are virtual intelligent agents or physical robots that can interact with their environment, perceive it, and make autonomous decisions, much like humans. The realization of embodied agents requires the integrated use of technologies from computer vision, natural language processing, and robotics, making it more complex than general artificial intelligence. For a long time, training systems for embodied agents have been geared towards specific tasks, such as moving designated objects, which has greatly limited their application scenarios. With large language models demonstrating general understanding and generation capabilities, and many studies using the Transformer architecture to unify the modeling of images, text, and action space, general embodied agents based on large models have become a new research task. Since natural language can express various tasks, the versatility of embodied agents is reflected in the fact that users can interact with the agent through natural language, and the agent can perform tasks expressed by the user or answer questions posed by the user. General embodied agents use large language models as their base model, are trained on embodied data, and take text and images as input, with control output as output.

[0003] Currently, there are two main technical approaches for training general embodied agents based on large models: 1. Given a language task, each task has a corresponding reward function, and reinforcement learning training is performed by sampling a large amount of data in a virtual environment without the need for labeled data. 2. Given a language task, the embodied agent is manually controlled to label data in the real world or a virtual environment, and then trained using the manually labeled data through imitation learning.

[0004] However, existing training methods cannot efficiently train general embodied agents. In prior art 1, the reward function corresponding to the task is difficult to write effectively, and reinforcement learning requires sampling a large amount of data for training. The high computational cost of large models makes the training process very inefficient and incurs huge computational overhead. In prior art 2, manually labeled data is required. Training general embodied agents requires a large amount of labeled data, and the cost of collecting and labeling data is expensive, especially in the real world. Training in the real world also faces security risks, lack of scalability, and difficulty in reproducing results. Existing virtual environments are not geared towards embodied agents based on large models, do not consider making embodied agents more general through dialogue, and lack data generation methods adapted for training large models. Summary of the Invention

[0005] This invention provides an embodied intelligent agent training system and method, which solves the problems of high training cost or high annotation cost of existing training systems.

[0006] An embodied agent training system, comprising:

[0007] The task generation module is used to randomly generate scene configuration files and pass the configuration files into the virtual environment module to construct a 3D scene, and generate tasks based on the current scene conditions using a large language model;

[0008] The task resolution module is used to plan the task generation and resolution process generated by the task generation module through a large language model.

[0009] The virtual environment module is used to construct a 3D scene based on the configuration file and obtain the environmental state of the scene. Based on the environmental state and the agent's planning, it calculates the optimal control sequence for the agent. Based on the optimal control sequence, it executes the agent's actions in the environment and obtains the agent's observations to obtain labeled training data. The agent's planning is obtained from the task solving module.

[0010] The model training module is used to train the embodied intelligent agent model based on the training data.

[0011] In one embodiment of the present invention, the virtual environment module includes: a scene construction unit for constructing a scene by reading in an environment configuration and 3D objects; a dynamic simulation unit for simulating the physical effects of the real three-dimensional world; an action execution unit for executing corresponding control code to change the state of the intelligent agent and the state of the environment after receiving control from the intelligent agent; and an observation acquisition unit for acquiring an image from the first perspective of the intelligent agent and rendering the image from the first perspective.

[0012] The optimal control computation unit is used to convert the agent's plan into an optimal control sequence based on the environmental state and relevant algorithms;

[0013] The training data acquisition unit is used to acquire observations in sequence, execute control sequences, and combine the observations and control sequences in sequence to generate labeled training data.

[0014] In one embodiment of the present invention, the task generation module includes: a scene random generation unit, used to generate a configuration file representing the scene based on prior information of objects in the real three-dimensional world; and a task random generation unit, used to generate a task based on input prompts and the current environmental state through a large language model.

[0015] In one embodiment of the present invention, the task solving module is specifically used to generate a planned sequence oriented towards the label based on the input prompts, environmental state, and task output using a large language model; wherein, for each step of the planning sequence, the optimal control path can be calculated by the virtual environment module.

[0016] In one embodiment of the present invention, the model training module is specifically used to train a multimodal large model based on the Transformer architecture into a vision-language-action model, modeling the mapping between language tasks and image observation control; and to train the vision-language-action model using a cross-entropy loss function based on the output of the vision-language-action model and the control output of the training data.

[0017] An embodied agent training method includes:

[0018] A scene configuration file is randomly generated. After constructing a 3D scene based on the configuration file, a task is generated based on the current scene conditions.

[0019] A solution plan is generated based on the task described;

[0020] The optimal control sequence of the agent is calculated based on the environmental state of the scene and the agent's planning. The agent's actions are executed in the environment according to the optimal control sequence and the agent's observations are obtained to obtain the labeled training data.

[0021] The embodied agent model is trained based on the training data.

[0022] In one embodiment of the present invention, the random generation of scene configuration files, after constructing a three-dimensional scene based on the configuration files, and generating a task based on the current scene conditions, specifically includes: generating scene configuration files based on prior information of objects using a scene generation algorithm; after constructing a three-dimensional scene based on the configuration files, inputting prompts and the current environmental state into a large language model based on a few-shot prompting method, and outputting the task through the large language model; the environmental state includes scene configuration information.

[0023] In one embodiment of the present invention, the step of generating a solution plan based on the task specifically includes: inputting the prompt, the current environmental state and the task into a large language model based on a few-shot prompting method, and outputting a label-oriented planning sequence through the large language model.

[0024] An embodied intelligent agent training device, comprising:

[0025] At least one processor; and,

[0026] The memory is communicatively connected to the at least one processor via a bus; wherein,

[0027] The memory stores instructions that can be executed by the at least one processor to implement the method as described in any of the above embodiments.

[0028] A non-volatile storage medium stores computer-executable instructions, which are executed by a processor to implement the method described in any of the above embodiments. This invention provides an embodied agent training system and method, which has at least the following advantages: through a virtual environment module capable of computable optimal control, a task generation module capable of randomly generating task scenarios and task dialogues, a task solving module for automatic annotation, planning, and control, and a model training module, training data can be acquired on a large scale at low cost, which is beneficial for better training of general embodied agents based on large language models and has good practicality. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0030] Figure 1 A structural diagram of an embodied intelligent agent training system provided in an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram illustrating the steps of an embodied intelligent agent training method provided in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the various modules and data generation process in the training system provided in the embodiments of the present invention;

[0033] Figure 4 This is a schematic diagram of an embodied intelligent agent training device provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described clearly and completely below in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] It should be noted that those skilled in the art will understand, explicitly and implicitly, that the embodiments described in this invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical or scientific terms used in this invention should be understood in their ordinary sense by those skilled in the art. The terms "a," "an," "an," "the," etc., used in this invention do not indicate quantity limitation and can represent singular or plural. The terms "comprising," "including," "having," and any variations thereof used in this invention are intended to cover non-exclusive inclusion; the terms "first," "second," "third," etc., used in this invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0036] This invention proposes an embodied agent training system and method, solving the problem of how to construct a suitable virtual environment and obtain a large amount of labeled data from the virtual environment at low cost for training a general embodied agent. It achieves the effect of low-cost training of embodied agents. The details are described below.

[0037] To address the aforementioned technical problems, embodiments of the present invention provide an embodied intelligent agent training system, such as... Figure 1 As shown, it includes:

[0038] The task generation module 110 is used to randomly generate scene configuration files and pass the configuration files into the virtual environment module to build a 3D scene, and generate tasks based on the current scene conditions through a large language model;

[0039] Task solving module 120 is used to plan the task generation and solution of task generation module 110 through a large language model;

[0040] The virtual environment module 130 is used to construct a three-dimensional scene according to the configuration file and obtain the environmental state of the scene. Based on the environmental state and the planning of the agent, it calculates the optimal control sequence of the agent. Based on the optimal control sequence, it executes the actions of the agent in the environment and obtains the observations of the agent to obtain the labeled training data. The planning of the agent is obtained from the task solving module 120.

[0041] The model training module 140 is used to train the embodied agent model based on the training data.

[0042] Specifically, the task generation module 110 can randomly generate scene configurations, which are then read into the virtual environment module 130 to instantiate the scene. The large language model, which acts as the user, then proposes possible tasks for the scene, and the tasks are expressed in the form of dialogue.

[0043] The task solving module 120 uses a large language model that acts as an annotator to formulate a solution plan for the task proposed by the task generation module 110. The plan is then fed into the virtual environment module 130 to calculate the optimal control sequence.

[0044] The virtual environment module 130 can input the agent's actions and output the agent's observations or environmental states. At the same time, it can calculate the agent's optimal control sequence based on the environmental state and the agent's planning. The virtual environment module 130 executes this sequence to obtain labeled training data.

[0045] The model training module 140 uses the training data obtained from the virtual environment module 130 to train an embodied intelligent agent model, the core of which is a vision-language-action model. The vision-language-action model takes images and dialogues as input and outputs control and responses, where the output is represented in the form of code.

[0046] In one embodiment of the present invention, the virtual environment module 130 includes: a scene construction unit 1302, used to construct a scene by reading in environment configuration and 3D objects; a dynamic simulation unit 1304, used to simulate the physical effects of the real three-dimensional world; an action execution unit 1306, used to execute corresponding control code after receiving control from the intelligent agent to change the state of the intelligent agent itself and the state of the environment; an observation acquisition unit 1308, used to acquire images from the first perspective of the intelligent agent and render the images from the first perspective; an optimal control calculation unit 1310, used to convert the planning of the intelligent agent into an optimal control sequence according to the environmental state and related algorithms; and a training data acquisition unit, used to acquire observations in sequence, execute control sequences, and combine observations and control sequences in sequence to generate labeled training data.

[0047] Specifically, the virtual environment module 130 is implemented as follows:

[0048] The virtual environment module, built on a 3D engine, simulates the visual and physical aspects of the three-dimensional world. The virtual environment module 130 needs to implement functions for scene construction, dynamic simulation, action execution, observation acquisition, and optimal control calculation. For scene construction, the virtual environment module 130 can create a scene by reading in environment configuration and 3D objects. For dynamic simulation, the environment needs to simulate real-world physical effects, such as gravity and collisions. For action execution, after receiving control from the intelligent agent, the environment can execute corresponding control code, thereby changing the state of the intelligent agent and the environment. For observation acquisition, a first-person view camera can be built on the intelligent agent to render the scene as seen by the agent, while also communicating via text messages.

[0049] To compute optimal control functions, environmental states and related algorithms can be used to transform a plan into a control sequence. For example, in a plan to move to a target location, the environment implements a pathfinding algorithm that can calculate the key points the agent needs to traverse to reach the target location. The control sequence required between these key points is the optimal control sequence. (Using s...) T P represents the state at time T. T Let c represent a certain plan at time T. Ti Indicates completion of dialogue P T If the required n-step control is the i-th step, then the optimal control function E can be expressed as:

[0050] E(s T ,P T ) = c T1 ,c T2 ,…,c Tn

[0051] It should be noted that the control uses continuous values, such as the number of meters forward. This is because as the agent moves forward continuously, the incremental information is very small due to the similarity of the images seen. Moving forward by a continuous value at a time plays a significant role in increasing the information density of the data, which can improve the training and inference efficiency of subsequent large models.

[0052] In one embodiment of the present invention, the task generation module 110 includes: a scene random generation unit 1102, used to generate a configuration file representing the scene based on prior information of objects in the real three-dimensional world; and a task random generation unit 1104, used to generate a task based on input prompts and the current environmental state through a large language model.

[0053] Specifically, the task generation module 110 is implemented as follows:

[0054] 1. Randomly generate scenes;

[0055] The scene is represented by a configuration file containing information about what all objects are and where they are located. This module generates the configuration file using a scene generation algorithm. The scene generation algorithm uses prior information about the objects, i.e., their usual placement locations, and then randomly places these objects without any overlap to generate a reasonable scene layout.

[0056] 2. Randomly generate tasks;

[0057] Using a large language model as the user, we employ a few-shot suggestion method, inputting suggestion p. u and the environment state s at the current time T T The large language model proposes task q TThe task is input into the environment in the form of a dialogue. The environment state includes scene configuration information. Inputting the current environment ensures that the task proposed by the large language model is feasible or meaningful under the current scene conditions. We use LLM. u The process of representing a large language model that acts as a user can be represented as follows:

[0058] LLM u (p u ,s T )=q T

[0059] In one embodiment of the present invention, the task solving module 120 is specifically used to plan a sequence of input prompts, environmental states and task outputs based on a large language model; wherein, for each step of the planning sequence, the optimal control path can be calculated by the virtual environment module 130.

[0060] Specifically, the task resolution module 120 is implemented as follows:

[0061] This module can acquire the observation-action sequence for completing a task, where observations include dialogue and images, and actions include dialogue and control. First, a large language model acts as an annotator. At the moment T when the user asks a question, a few-shot suggestion method is used to input suggestion p. a Environmental state s T And dialogue q T The input is a large language model, which outputs a labeled planning sequence. Using P... Ti Indicates completion of task q T The required m-step planning for the i-th step, LLM a Representing the large language model that acts as the annotator, this process can be expressed as:

[0062] LLM a (p a q T s T ) = P T1 P T2 , ..., P Tm

[0063] For each step of the planning P Ti The optimal control path can be calculated by inputting the data into the virtual environment module.

[0064] E(s T P Ti ) = c Ti1 c Ti2 ..., c Tin

[0065] Merging all controls will allow us to complete task q. T All required controls C:

[0066] C = c T11 c T12 c Tmn

[0067] In the virtual environment module 130, observations are acquired sequentially. Tij Execution of action c Toj Then the final training data D can be represented as:

[0068] D={q T o T11 c T11 o T12 c Ti(j-1) o Tij c Tij |c Tij ∈C}

[0069] In one embodiment of the present invention, the model training module 140 is specifically used to train a multimodal large model based on the Transformer architecture into a vision-language-action model, modeling the mapping between language tasks and image observation control; and to train the vision-language-action model using the cross-entropy loss function based on the output of the vision-language-action model and the control output of the training data.

[0070] Specifically, the model training module 140 is implemented as follows:

[0071] A large multimodal model based on the Transformer architecture is trained as a vision-language-action model, modeling the mapping between language tasks and image observations to control. The dialogue output by both control and the agent is represented using code text. Specifically, using C... Tijx c Tij At the x-th position of the code text, for each training sample in the training data D, in T ij The observation-action history obtained by the agent at any given moment is as follows:

[0072] h Tij =o T11 c T11 o T12 C Ti(j-1) o Tij

[0073] Modeling the control c Tij The probability is:

[0074]

[0075] The model is trained using the cross-entropy loss function, which is applied to the model output and the control output of the training data.

[0076] By using a large language model as the user and annotator to generate tasks and plans, and having the virtual environment module calculate and control the generation of labeled data, the cost of manual data labeling is reduced.

[0077] This invention, through its design suitable for large language models, enables the use of text and images as agent inputs and text and control as agent outputs, ensuring versatility; it enables continuous control, increasing the information density of samples and improving training and inference efficiency; and it provides automatic annotation functionality by calculating optimal control based on environmental conditions and planning.

[0078] The above describes an embodied agent training system provided by an embodiment of the present invention. Based on the same inventive concept, the present invention also proposes an embodied agent training method. Figure 2 The embodied agent training method shown can be executed by an embodied agent training system, which can be located on a server. For example, the server can be a physical server containing independent hosts, a virtual server hosted in a host cluster, a cloud server, etc., and this embodiment of the invention does not limit this. The embodied agent training method may include the following steps:

[0079] S210: Randomly generate scene configuration files, and after constructing a 3D scene based on the configuration files, generate tasks based on the current scene conditions.

[0080] S220: Generate a solution plan based on the task.

[0081] S230: Calculate the optimal control sequence for the agent based on the environmental state of the scene and the agent's planning. Execute the agent's actions in the environment according to the optimal control sequence and obtain the agent's observations to obtain labeled training data.

[0082] S240: Train the embodied agent model based on the training data.

[0083] In one embodiment of the present invention, a scene configuration file is randomly generated. After constructing a three-dimensional scene based on the configuration file, a task is generated based on the current scene conditions. Specifically, this includes: generating a scene configuration file based on prior information of objects using a scene generation algorithm; after constructing a virtual environment module of the real three-dimensional world based on the configuration file, and after constructing a three-dimensional scene based on the configuration file, inputting the prompts and the current environmental state into a large language model based on a few-shot prompting method, and outputting the task through the large language model; the environmental state includes scene configuration information.

[0084] In one embodiment of the present invention, generating a solution plan based on the task specifically includes: inputting the prompt, the current environmental state and the task into a large language model based on a few-shot prompting method, and outputting a label-oriented planning sequence through the large language model.

[0085] Figure 3 The diagram shows an example of the various modules and data generation process of this invention. The virtual environment module allows the intelligent agent to interact; the task generation module uses a large language model acting as the user to propose possible tasks for the generated scenario; the task solving module uses a large language model acting as the annotator to formulate a solution plan for the tasks proposed by the task generation module, and uses the virtual environment module to calculate the optimal control sequence to obtain training data; the model training module uses the data generated by the task generation module for training.

[0086] Compared with existing technologies, this invention proposes a system and method for training a general embodied agent based on automatically generated data from a virtual environment. Through a virtual environment module capable of computable optimal control, a task generation module capable of randomly generating task scenarios and dialogues, a task solving module for automatic annotation, planning, and control, and a model training module, training data can be acquired on a large scale at low cost. This facilitates better training of a general embodied agent based on a large language model and demonstrates good practicality.

[0087] This invention also provides a corresponding embodied intelligent agent training device, such as... Figure 4 As shown, it includes:

[0088] The system includes at least one processor 402, a communication interface 404, a memory 406, and a communication bus 408; wherein the processor 402, the communication interface 404, and the memory 406 communicate with each other through the communication bus 408; the processor 402 can call the logical instructions stored in the memory 406 to cause at least one processor 402 to execute the steps of the above embodiments.

[0089] Based on the same idea, some embodiments of the present invention also provide media corresponding to the above methods.

[0090] Some embodiments of the present invention provide a storage medium storing computer-executable instructions, which are executed by a processor to implement the steps of the embodiments described above.

[0091] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the embodied agent training method provided in the above embodiments.

[0092] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the device, medium, and program product embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0093] The devices, media, and program products and methods provided in the embodiments of the present invention are one-to-one correspondences. Therefore, the devices, media, and program products also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices, media, and program products will not be repeated here.

[0094] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process method or product that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process method or product. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process method or product that includes that element.

[0095] The above are merely embodiments of the present invention and are not intended to limit the invention. Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. An embodied intelligent agent training system, characterized in that, include: The task generation module is used to randomly generate scene configuration files and pass the configuration files into the virtual environment module to construct a 3D scene, and generate tasks based on the current scene conditions using a large language model; The task solving module is used to plan the task generation and solution generated by the task generation module through a large language model. Specifically, it is used to plan a sequence of labeled input prompts, environmental states, and task outputs through the large language model. For each step of the planning sequence, the optimal control path can be calculated by the virtual environment module. The virtual environment module is used to construct a 3D scene based on the configuration file and obtain the environmental state of the scene. Based on the environmental state and the agent's planning, it calculates the optimal control sequence for the agent. Based on the optimal control sequence, it executes the agent's actions in the environment and obtains the agent's observations to obtain labeled training data. The agent's planning is obtained from the task solving module. The model training module is used to train the embodied agent model based on the training data. Specifically, it is used to train a multimodal large model based on the Transformer architecture into a vision-language-action model, modeling the mapping between language tasks and image observation control; and to train the vision-language-action model using the cross-entropy loss function based on the output of the vision-language-action model and the control output of the training data.

2. The embodied agent training system of claim 1, wherein, The virtual environment module includes: Construct scene units, used to build scenes by reading in environment configurations and 3D objects; Dynamic simulation unit, used to simulate the physical effects of the real three-dimensional world; The action execution unit is used to execute the corresponding control code after receiving control from the agent, so as to change the state of the agent itself and the state of the environment. An observation acquisition unit is used to acquire and render images from the agent's first-view perspective. The optimal control computation unit is used to convert the agent's plan into an optimal control sequence based on the environmental state and relevant algorithms; The training data acquisition unit is used to acquire observations in sequence, execute control sequences, and combine the observations and control sequences in sequence to generate labeled training data.

3. The embodied agent training system of claim 1, wherein, The task generation module includes: The scene random generation unit is used to generate a configuration file representing the scene based on prior information about objects in the real three-dimensional world; The task random generation unit is used to generate tasks based on input prompts and the current environmental state using a large language model.

4. An embodied agent training method based on the embodied agent training system of any one of claims 1-3, characterized in that, include: A scene configuration file is randomly generated. After constructing a 3D scene based on the configuration file, a task is generated based on the current scene conditions. A solution plan is generated based on the task described; The optimal control sequence of the agent is calculated based on the environmental state of the scene and the agent's planning. The agent's actions are executed in the environment according to the optimal control sequence and the agent's observations are obtained to obtain the labeled training data. The embodied agent model is trained based on the training data.

5. The embodied intelligent agent training method according to claim 4, characterized in that, The randomly generated scene configuration file, after constructing a 3D scene based on the configuration file, generates tasks based on the current scene conditions, specifically including: A scene configuration file is generated based on the prior information of objects using a scene generation algorithm; After constructing a 3D scene based on the configuration file, the prompts and the current environmental state are input into a large language model based on a few-shot prompting method, and the task is output through the large language model. The environmental status includes scene configuration information.

6. The embodied intelligent agent training method according to claim 5, characterized in that, The step of generating a solution plan based on the task specifically includes: The few-shot prompting method inputs the prompts, the current environmental state, and the task into a large language model, and outputs a planning sequence oriented towards annotation through the large language model.

7. A training device for an embodied intelligent agent, characterized in that, include: At least one processor; as well as, The memory is communicatively connected to the at least one processor via a bus; wherein, The memory stores instructions executable by the at least one processor, which are executed to implement the method as described in any one of claims 4-6.

8. A non-volatile storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are executed by a processor to implement the method as described in any one of claims 4-6.

Images