Auto-reflection agent training method and device and storage medium

Abstract

The invention discloses a training method and device of an auto-reflection agent and a storage medium. The method comprises the steps that a plurality of first tracks recorded in a natural language text form are used for conducting supervision and fine tuning on a first large language model, a first intelligent agent based on the large language model is obtained, the plurality of first tracks are expert tracks successfully completing corresponding first tasks, and each step of completing the corresponding first tasks is included; constructing a plurality of second tracks which are recorded in a natural language text form and successfully complete corresponding second tasks by utilizing the first intelligent agent and a teacher model, wherein the second tracks comprise steps for completing the corresponding second tasks and an auto-reflection step for reflecting errors of the first intelligent agent; and utilizing the second track to supervise and finely adjust the second large language model to obtain a second intelligent agent with the auto-reflection capability.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device and storage medium for a self-reflective intelligent agent. Background Art

[0002] With the advancement of large language models (LLMs), autonomous agents that can perceive the environment and take actions to achieve goals are becoming more and more feasible. However, powerful agents currently often rely on complex prompt engineering and closed-source LLMs like GPT-4. Although training open-source LLMs through expert trajectories has improved the capabilities of agents to a certain extent, this approach still faces limitations such as performance stagnation and error propagation.

[0003] The comprehensive capabilities of large language models (LLMs) have been significantly improved, enabling them to perform intelligent agent tasks that require interaction with various environments for reasoning, planning, and execution. Recently, many studies have used GPT-4 or other powerful closed-source LLMs to build agents that can perform complex tasks, such as manipulating computer systems, performing embodied intelligence tasks, and even solving problems in actual GitHub repositories. These tasks are usually achieved through complex prompt engineering and closed-source LLMs like GPT-4. However, small LLMs usually perform poorly on intelligent agent tasks compared to closed-source models.

[0004] Previous studies on building agents based on small LLMs often rely on supervised fine-tuning (SFT). Due to the inherent lack of instruction following and reasoning capabilities of these small LLMs, these studies usually introduce correct problem-solving examples (i.e., successful expert trajectories that successfully complete tasks, or golden trajectories) as multi-round dialogue training data to fine-tune small LLMs, thereby building LLM-based LLM agents. However, there are several challenges in using only successful expert trajectories to train small LLMs to build LLM agents:

[0005] First, as the amount of training data for successful expert trajectories increases, the performance improvement trend of LLM agents tends to slow down and may eventually reach a plateau, which indicates that such methods have limitations in enhancing generalization capabilities.

[0006] Second, LLM agents are prone to making mistakes while performing tasks. Once an error occurs, incorrect actions may further propagate the error, potentially trapping the LLM agent in an error loop, and how to build an LLM agent with self-reflection capabilities similar to proprietary models such as GPT-4 from scratch has not yet been explored.

[0007] Currently, no effective solution has been proposed for the above-mentioned technical problems that the use of successful expert trajectories in the prior art to train LLM agents has limitations in enhancing generalization capabilities, and that the trained LLM agents cannot escape from error cycles during task execution. Summary of the invention

[0008] The embodiments of the present disclosure provide a method, device and storage medium for training a self-reflective agent, so as to at least solve the technical problems existing in the prior art that using successful expert trajectories to train LLM agents has limitations in enhancing generalization capabilities, and the trained LLM agents cannot escape from error cycles during task execution.

[0009] According to one aspect of an embodiment of the present disclosure, a method for training a self-reflective agent is provided, comprising: using a plurality of first trajectories recorded in the form of natural language text to perform supervised fine-tuning on a first large language model to obtain a first agent based on the large language model, wherein the plurality of first trajectories are expert trajectories that successfully complete a corresponding first task and include various steps for completing the corresponding first task; using the first agent and a teacher model to construct a plurality of second trajectories that are recorded in the form of natural language text and successfully complete a corresponding second task, the second trajectories include various steps for completing the corresponding second task and a self-reflective step for reflecting on the mistakes of the first agent; and using the second trajectories to perform supervised fine-tuning on a second large language model to obtain a second agent with self-reflective capability.

[0010] According to another aspect of an embodiment of the present disclosure, a method for training a self-reflective intelligent agent is also provided, the method comprising: constructing a plurality of self-reflective trajectories recorded in the form of natural language text and successfully completing corresponding tasks, the self-reflective estimation comprising the various steps for successfully completing the corresponding tasks and the self-reflective steps for reflecting on errors; and using the self-reflective trajectories to perform supervised fine-tuning on a large language model to obtain an intelligent agent with self-reflective capabilities.

[0011] According to another aspect of an embodiment of the present disclosure, a storage medium is further provided, the storage medium including a stored program, wherein when the program is running, a processor executes any one of the methods described above.

[0012] According to another aspect of an embodiment of the present disclosure, a training device for a self-reflective agent is also provided, including: a first agent training module, used to use multiple first trajectories recorded in the form of natural language text to supervise and fine-tune a first large language model to obtain a first agent based on the large language model, wherein the multiple first trajectories are expert trajectories that successfully complete the corresponding first task and include various steps to complete the corresponding first task; a self-reflective trajectory construction module, used to use the first agent and the teacher model to construct multiple second trajectories recorded in the form of natural language text and successfully complete the corresponding second task, the second trajectories include various steps to complete the corresponding second task and a self-reflective step to reflect on the mistakes of the first agent; and a second agent training module, used to use the second trajectories to supervise and fine-tune the second large language model to obtain a second agent with self-reflective ability.

[0013] According to another aspect of the embodiments of the present disclosure, a training device for a self-reflective agent is also provided, comprising: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: using a plurality of first trajectories recorded in the form of natural language text to perform supervised fine-tuning on a first large language model to obtain a first agent based on the large language model, wherein the plurality of first trajectories are expert trajectories that successfully complete the corresponding first task and include the various steps for completing the corresponding first task; using the first agent and the teacher model to construct a plurality of second trajectories that are recorded in the form of natural language text and successfully complete the corresponding second task, the second trajectories include the various steps for completing the corresponding second task and a self-reflective step for reflecting on the mistakes of the first agent; and using the second trajectories to perform supervised fine-tuning on a second large language model to obtain a second agent with self-reflective capability.

[0014] In the disclosed embodiment, the small open source LLM can be supervised and fine-tuned using a set of successful expert trajectories (i.e., successful expert trajectories) to obtain a basic LLM agent, and then the basic LLM agent and the teacher model are used to construct a self-reflection trajectory set based on the task instructions in another task instruction space (e.g., U2) and the corresponding environment. The self-reflection trajectory of the self-reflection trajectory set includes the steps of reflecting on errors (i.e., identifying incorrect error activity information) and correcting (i.e., generating correct corrective activity information) in the process of completing the corresponding task. Then, the small open source LLM is supervised and fine-tuned using the self-reflection trajectory to obtain a self-reflective LLM agent based on the small LLM and with reflective ability. In this way, it is possible to obtain a self-reflective LLM agent with self-reflection ability and excellent performance only through a small open source LLM without relying on complex prompt engineering and closed source large models such as GPT-4. Compared with the method of only using successful expert trajectories to train small LLMs to construct LLM agents, this embodiment uses self-reflection trajectories to train small LLMs, so that when the amount of training data increases, the performance improvement trend of the LLM agent can be prevented from slowing down, and it has a strong generalization ability. And the generated self-reflection LLM agent can effectively prevent the propagation of errors through the process of self-reflection during the execution of tasks. This solves the technical problems in the prior art that using successful expert trajectories to train LLM agents has limitations in enhancing generalization ability, and the trained LLM agent cannot get rid of the error cycle during the execution of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0016] Figure 1 is a hardware structure block diagram of a computing device for implementing the method according to Embodiment 1 of the present disclosure;

[0017] Figure 2 is a schematic diagram of the interaction process between the self-reflective intelligent agent and the environment according to Embodiment 1 of the present disclosure;

[0018] Figure 3 is a flowchart of a method for training a self-reflective agent according to the first aspect of Embodiment 1 of the present disclosure;

[0019] Figure 4 is a specific schematic diagram of the training method of the self-reflective agent according to Embodiment 1 of the present disclosure;

[0020] Figure 5is a schematic diagram of a prompt template of a teacher model according to Embodiment 1 of the present disclosure;

[0021] Figure 6 This is a schematic diagram of the prompts for the task ALFWorld used in the experiment;

[0022] Figure 7 This is a schematic diagram of the prompts of the task SciWorld used when conducting the experiment;

[0023] Figure 8 This is a schematic diagram of the prompts for the task WebShop used in the laboratory;

[0024] Fig. 9 A schematic diagram showing a comparison of the prior art and the method described in Example 1 of the present disclosure in terms of training data scale and average reward in different tasks;

[0025] Fig.10 A diagram showing the number of self-reflective trajectories and average rewards generated by the teacher model using different LLMs;

[0026] Fig.11 is a schematic diagram of a training device for a self-reflective agent according to Embodiment 2 of the present disclosure; and

[0027] Fig.12 It is a schematic diagram of a training device for a self-reflective intelligent agent according to Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0030] First, some nouns or terms that appear in the process of describing the embodiments of the present disclosure are subject to the following explanations:

[0031] Activity: The term "activity" as used in this application refers to the activity of an intelligent agent. That is, the activity of an intelligent agent interacting with the environment in the process of completing a corresponding task. An activity may include only actions or both actions and thoughts. Furthermore, activity information refers to information corresponding to an activity recorded in the form of natural language text. Activity information may include only action information or both thought information and action information.

[0032] Example 1

[0033] According to this embodiment, a method embodiment of a method for training an intelligent agent using trajectories is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] The method embodiment provided in this embodiment can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computing device for implementing a training method for a self-reflective intelligent agent. Figure 1 As shown, the computing device may include one or more processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, the transmission device, and the input / output interface are connected to the processor via a bus. In addition, it may also include: a display, a keyboard, and a cursor control device connected to the input / output interface. A person skilled in the art can understand that Figure 1The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0035] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computing device. As involved in the embodiments of the present disclosure, the data processing circuitry acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0036] The memory can be used to store software programs and modules of application software, such as a program instruction / data storage device corresponding to a method for training a self-reflective intelligent agent in an embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, a method for training a self-reflective intelligent agent for the above-mentioned application program. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computing device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computing device. In one example, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0038] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the computing device.

[0039] It should be noted that, in some optional embodiments, the above Figure 1The computing device shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computing devices described above.

[0040] The self-reflection described in this application refers to the ability of an agent to identify its own mistakes and self-correct during the interaction with the environment. The agent interacts with the environment in a dialogue, that is, the agent outputs activity information by outputting natural language text, and the environment outputs feedback observation information (i.e., observation) by outputting natural language text. The activity information may include thought information (i.e., thought) and action information (i.e., action). Figure 2 As shown, after executing the wrong action steps during the interaction with the environment, the self-reflective agent can reflect on its mistakes based on the observation information fed back by the environment and make corrections.

[0041] In order to facilitate understanding of the technical solution of this application, the background knowledge related to this application is as follows:

[0042] For an agent task that involves interacting with an environment e (in a dialogue that outputs natural language text), the agent is required to generate actions a according to the task instructions u, and continue to advance or adjust these actions according to the feedback from the environment e until the task is completed. The interaction process between the agent and the user environment e based on LLM can be represented as a multi-round dialogue. The agent task can also be formalized as: (U, S, A, O, T, R). The elements represent the task instruction space (U), state space (S), action space (A), observation space (O), state transition function (T) and reward function (R).

[0043] For each task instruction u∈U, agent π θ The action information a1∈A will be generated. Under the influence of a1, the initial task implicit state s0∈S is converted to s1∈S according to T. At the same time, the environment e of the task will provide an observation information for a1, generating observation information o1∈O. Subsequent action information a i (i≥2) will be represented by agent π θ Generate based on the task instruction u, the previous action information a1~ai-1 and the previous observation information o1~oi-1 fed back by the environment e, that is, ai=π θ (u,a1,o1,...,ai-1,oi-1). The environment e will be based on a iThe resulting implicit state space transformation provides the corresponding observation information o i This process will continue to iterate until task u is completed or the maximum number of steps is reached. The inventors represent the number of actions at this time as the trajectory length n. θ The trajectory τ for performing this task is expressed as:

[0044] τ=(u,a1,o1,...,a i ,o i ,...,o n ,o n ) (1)

[0045]

[0046] Among them, u is the task instruction corresponding to the task, a1~a n For the intelligent agent π θ Generated action information, o1~o n is the observation information generated for the environment e corresponding to the task. In this embodiment, the task instruction u, the action information a1~a n And observation information o1~o n can be expressed in natural language text. In addition, formula (2) represents the agent π θ Expression for the interaction between task instruction u and environment e to generate trajectory τ.

[0047] Once the task is completed, the reward r∈[0,1] is calculated according to the reward function R. r=1 indicates successful completion of the task instruction. In this example, the agent π θ The activity of only includes action a, and its activity information also only includes action information a.

[0048] In addition, in this application, the ReAct format can be further used to enhance the reasoning ability of the agent. Before taking action, the agent first generates a thought information t (specifically, for example, the large model agent can be prompted to generate a thought information t first by using prompt), so that the trajectory τ will be converted into the following form:

[0049] τ=(u,t1,a1,o1,...,t i , a i , o i , ..., t n , a n , o n ) (3).

[0050] Among them, t1~t n The thought information generated for the agent. In this example, the agent π θThe activity includes both action a and thought t, so the activity information also includes action information a and thought information t.

[0051] Therefore, in this application, the thinking and actions of the agent are collectively referred to as "activities", and the thinking information and action information of the agent are collectively referred to as "activity information". Therefore, in this application, the agent activity may include only actions, or may include both thinking and actions. For ease of understanding, the agent activity in this embodiment includes thinking and actions, and the agent activity information includes thinking information and action information. However, the case where the agent activity only includes actions is also applicable to this application.

[0052] Based on the above background knowledge, the technical solution of this embodiment is further described below. According to the first aspect of this embodiment, a method for training a self-reflective intelligent agent is provided. Figure 1 The computing device implementation shown in . Figure 3 A schematic diagram showing the process of the method is shown in FIG. Figure 3 As shown, the method includes:

[0053] S302: performing supervised fine-tuning on a first large language model using a plurality of first trajectories recorded in the form of natural language texts to obtain a first agent based on the large language model, wherein the plurality of first trajectories are expert trajectories for successfully completing corresponding first tasks and include various steps for completing the corresponding first tasks;

[0054] S304: constructing a plurality of second trajectories recorded in the form of natural language text and successfully completing the corresponding second task using the first agent and the teacher model, the second trajectories including steps for completing the corresponding second task and a self-reflection step for reflecting on the mistakes of the first agent; and

[0055] S306: Use the second trajectory to perform supervised fine-tuning on the second largest language model to obtain a second intelligent agent with self-reflection ability.

[0056] Specifically, Figure 4 A specific schematic diagram of implementing the training method of the self-reflective agent described in this embodiment is shown.

[0057] refer to Figure 4 As shown, according to the technical solution of this embodiment, in order to obtain correctly formatted and effective trajectories, the successful expert trajectories (i.e., golden trajectories) that successfully complete the task are used to supervise the fine-tuning (SFT) of the large language model π base1 (i.e. the first language model), create a basic LLM agent π θ1 (i.e. the first agent). The basic LLM agent π θ1It will serve as a solid platform for developing more powerful LLM agents (S202). In this application, LLM agents refer to agents based on large language models (LLM) obtained by fine-tuning the large language model through supervision.

[0058] Specifically, let D represent the successful expert trajectory τ1~τ m (For example, successful expert trajectory τ1~τ m The expert trajectory set D consists of the successful expert trajectories τ1~τ m Contains the interaction sequence of successfully completing each task instruction in the task instruction space U, which describes the interaction process between the agent and the environment of the task instruction in the form of multiple rounds of dialogue. Each successful expert trajectory τ1~τ m They are all organized in the ReAct format, including reasoning thoughts and related actions, referring to the form of the trajectory shown in formula (3).

[0059] A subset of the expert trajectory set D is randomly selected by the computing device, denoted as D1, which corresponds to the task instruction subset U1, including the successful expert trajectory τ 1 1~τ 1 m1 The remaining set of trajectories is labeled D2, corresponding to the task instruction subset U2, including the successful expert trajectory τ 2 1~τ 2 m2 Among them, the successful expert trajectory τ in D1 1 1~τ 1 m1 The trajectory lengths are n 1 1~n 1 m1 , the corresponding task instructions are u 1 1~u 1 m1 ∈U1, and its corresponding environment is e 1 1~e 1 m1 The successful expert trajectory τ in D2 2 1~τ 2 m2 The trajectory lengths are n 2 1~n 2 m2 , the corresponding task instructions are u 2 1~u 2 m2 ∈U2, and the corresponding environments are e 2 1~e 2 m2Then, the successful expert trajectory τ in D1 is utilized by the computing device 1 1~τ 1 m1 (i.e. the first track) to supervise the fine-tuning of the large language model π base1 (i.e. the first language model) to obtain the basic LLM agent π θ1 (i.e., the first agent) to enable it to acquire basic agent task knowledge and instruction following ability. About the large language model π base1 The specific method of supervised fine-tuning will be described in detail later.

[0060] Then, refer to Figure 4 As shown, in the process of synthesizing self-reflective trajectories, the basic LLM agent π is used by the computing device θ1 and the pre-set teacher model π teacher , based on the task instruction u in the task instruction subset U2 of the remaining trajectory set D2 2 1~u 2 m2 , construct a self-reflection trajectory set D3, whose corresponding task instruction set is U3. Among them, the self-reflection trajectory set D3 includes multiple self-reflection trajectories τ that complete the corresponding tasks 3 1~τ 3 m3 (i.e., the second trajectory) (S204). Wherein, the self-reflection trajectory τ 3 1~τ 3 m3 The trajectory lengths are n 3 1~n 3 m3 , the corresponding task instructions are u 3 1~u 3 m3 ∈U3, and its corresponding environment is e 3 1~e 3 m3 .

[0061] Among them, the self-reflection trajectory τ 3 1~τ 3 m3 It can be expressed in the following form (y = 1 ~ m3):

[0062]

[0063] Among them, τ 3 y represents the yth trajectory in the self-reflective trajectory set D3, and Transform thinking information into teacher models and action information The erroneous thinking information and erroneous action information generated by marking as errors (i.e., thinking information and action information containing error markings), where the erroneous thinking information and error action information It constitutes a wrong step. For task instructions Corresponding environment Generated and erroneous thinking information and error action information The corresponding observation information.

[0064] and Generated and incorrect thinking information for the teacher model and error action information The corresponding corrective thinking information and corrective action information, wherein the corrective thinking information and corrective action information It constitutes a corrective step corresponding to the erroneous step. Information for corrective thinking and corrective action information The corresponding observation information.

[0065] Moreover, the error step and the correction step together constitute the basic LLM agent π θ1 The self-reflection step is to reflect on the mistakes.

[0066] The specific method of generating self-reflection trajectories will be explained in detail below.

[0067] Then, refer to Figure 4 As shown, in the stage of "supervised fine-tuning to obtain a self-reflective agent", the self-reflective trajectory τ in the self-reflective trajectory set D3 can be used by the computing device 3 1~τ 3 m3 For another large language model π base2 (i.e., the second largest language model) is fine-tuned to obtain a self-reflective LLM agent π with self-reflection ability θ2 (i.e. the second agent).

[0068] Among them, the self-reflective LLM agent π θ2 Have the ability to reflect on oneself. Figure 2 As shown, the self-reflective LLM agent π θ2 In the process of interacting with the environment, it is possible to reflect on and correct the wrong steps in the task execution process based on the feedback from the environment. base2 Thus, we get the self-reflective LLM agent π θ2 The specific method will be described in detail later.

[0069] As described in the background, there are several challenges in building LLM agents by training small LLMs using only successful expert trajectories: First, as the amount of training data for successful expert trajectories increases, the performance improvement trend of LLM agents tends to slow down and may eventually reach a plateau. This shows that such methods have limitations in enhancing generalization capabilities; second, LLM agents are prone to errors in the process of performing tasks. Once an error occurs, incorrect actions may further propagate the error, potentially trapping the LLM agent in an error loop, and how to build an LLM agent from scratch with self-reflection capabilities similar to proprietary models such as GPT-4 has not yet been explored.

[0070] In view of this, in this embodiment, the small open source LLM can be supervised and fine-tuned using a set of successful expert trajectories (i.e., golden trajectories) to obtain a basic LLM agent, and then the basic LLM agent and the teacher model are used to construct a self-reflection trajectory set based on the task instructions in another task instruction space (e.g., U2) and the corresponding environment. The self-reflection trajectory of the self-reflection trajectory set includes the steps of reflecting on errors (i.e., identifying the wrong error activity information) and correcting (i.e., generating correct correction activity information) in the process of completing the corresponding task. Then, the small open source LLM is supervised and fine-tuned using the self-reflection trajectory to obtain a self-reflection LLM agent based on the small LLM and with reflection ability. In this way, it is possible to obtain a self-reflection LLM agent with self-reflection ability and excellent performance only through a small open source LLM without relying on complex prompt engineering and closed source large models such as GPT-4. Compared with the method of only using successful expert trajectories to train small LLMs to construct LLM agents, this embodiment uses self-reflection trajectories to train small LLMs, so that when the amount of training data increases, the performance improvement trend of the LLM agent can be prevented from slowing down, and it has a strong generalization ability. And the generated self-reflection LLM agent can effectively prevent the propagation of errors through the process of self-reflection during the execution of tasks. This solves the technical problems in the prior art that using successful expert trajectories to train LLM agents has limitations in enhancing generalization ability, and the trained LLM agent cannot get rid of the error cycle during the execution of tasks.

[0071] In addition, the "Appendix" at the end of this embodiment shows a schematic track of the self-reflective LLM agent reflecting on its own wrong steps and correcting them in the process of performing an actual task. The applicant marked the self-reflective steps in Chinese.

[0072] Of course, in this embodiment, since the ReAct format is used, each step includes two operations: thinking t and action a. However, if the ReAct format is not used, each step may only include action operation a but not thinking operation t. That is, in this application, the activity of the intelligent agent may only include actions. Thus, the self-reflection step may only include action information marked as errors. and corrective action informationa' j+1 In addition, the error step marked as an error may also include only the error action information marked as an error, and the correction step may also include only the correction action information, thereby jointly constituting a self-reflection step.

[0073] Optionally, the operation of constructing multiple second trajectories that are recorded in the form of natural language text and successfully complete the corresponding second tasks using the first agent and the teacher model includes: using the first agent to execute task instructions of multiple third tasks and interacting with the environment of the third task in the form of natural language text; when the first agent outputs an erroneous activity during the interaction process, using the teacher model to identify the erroneous activity of the first agent during the interaction process to generate corresponding erroneous activity information, using the teacher model to generate corrective activity information corresponding to the erroneous activity information, and merging the erroneous activity information and the corrective activity information as self-reflection steps into the third trajectory corresponding to the current third task; and screening out second trajectories that successfully complete the task and include self-reflection steps from the third trajectories corresponding to each third task.

[0074] Specifically, in constructing the self-reflective trajectory τ of the self-reflective trajectory set D3 3 1~τ 3 m3 (i.e., the second trajectory). The computing device can use the basic LLM agent π θ1 (i.e., the first agent) executes the task instruction u in the task instruction subset U2 of the trajectory set D2 2 1~u 2 m2 (i.e., the task instruction of the third task), and in executing the task instruction u 2 1~u 2 m2 In the process, the corresponding environment e 2 1~e 2 m2 Interact in natural language text. For example, the basic LLM agent π θ1 Execute task command u 2 x (x = 1 ~ m2), and thus the corresponding environment e 2 xThe interaction is performed in the form of natural language text. During the interaction process, the following interaction information (i.e., trajectory information) is generated:

[0075]

[0076] Among them, t 2 x,i (i=1~n 2 x ) is the basic LLM agent π θ1 In executing the task command u 2 x The thinking information generated in the process of 2 x,i As the basic LLM agent π θ1 The generated action information, o 2 x,i For the environment 2 x , corresponding to the thought information t 2 x,i and action informationa 2 x,i , the generated observation information o 2 x,i .

[0077] Then, the computing device passes the teacher model π teacher For the basic LLM agent π θ1 For example, in the basic LLM agent π θ1 In executing the task command u 2 x After outputting the activity information of the i-th step in the process, the teacher model π teacher Real-time determination 2 x Whether the activity of the i-th step in is an error activity.

[0078] 1. If the teacher model π teacher If the activity of the i-th step is determined not to be an error, or if the activity of the i-th step is determined to be correct, the activity information of the activity is retained without being changed. In other words, no corresponding self-reflection step is generated to replace the activity information of the step.

[0079] 2. If the teacher model π teacher If the activity of the i-th step is determined to be an incorrect activity, the incorrect activity is marked to generate corresponding incorrect activity information. teacher τ 2 x The activity of the i-th step in is judged as an incorrect activity, and the basic LLM agent π in the i-th stepθ1 Generated activity information 2 x,i and a 2 x,i Mark it as an error (i.e. add an error flag) to generate corresponding error activity information and in, It is the information of thinking 2 x,i Wrong thinking information generated by adding wrong tags, is the action information a 2 x,i Error action information generated by adding error markers.

[0080] And the computing device also passes the teacher model π teacher Build and error activity information and Corresponding corrective action information and in, Is related to wrong thinking information The corresponding corrective thinking information, Is with error action information Corresponding corrective action information.

[0081] The computing device then uses the error activity information and and corrective activity information and Generate a self-reflection step. The self-reflection step includes two steps:

[0082] 1) Wrong steps, including: as well as

[0083] 2) Corrective steps, including:

[0084] Finally, the computing device merges the above self-reflection steps into the current task u 2 x The corresponding trajectory (i.e., the third trajectory) is related to the current task instruction u 2 x The corresponding trajectory (i.e., the third trajectory) after incorporating the self-reflection step is as follows:

[0085]

[0086] Then, for each task instruction u from x=1 to m2 2 x , both use the basic LLM agent π θ1Execute the task instruction and interact with the corresponding environment 2 x Interact with each other. This generates activity information and observation information for each step. At the same time, the teacher model π teacher The activity of each step is judged, and the corresponding self-reflection steps are generated for the wrong activities and merged into the corresponding trajectory τ 2 x . So finally, for each task instruction u from x=1 to m2 2 x The corresponding trajectory τ is generated in the above way 2’ x (i.e. the third trajectory).

[0087] The computing device then receives the instructions u from each task 2 x The corresponding trajectory τ 2’ x (i.e., the third track), select the basic LLM agent π θ1 Trajectories that successfully complete the task and include the self-reflection step are used as a supervised fine-tuning of the large language model π base2 Thus, we get the self-reflective LLM agent π θ2 The self-reflection trajectory τ 3 y (ie the second trajectory), where y=1~m3.

[0088] Thus, through the above method, this application makes the teacher model π based on LLM teacher The process of learning the agent’s capabilities is more effective and efficient.

[0089] Optionally, based on the error activity information, the correction activity information and the corresponding observation information during the interaction process, an operation of generating a self-reflection step includes: taking the error activity information as the error step; taking the correction activity information as the correction step; and combining the error step and the correction step into a self-reflection step.

[0090] Specifically, as described above, the computing device uses the error activity information and and corrective activity information and Generate a self-reflection step. The self-reflection step includes two steps:

[0091] 1) Identify the steps of the wrong activity (i.e., the wrong step), including: and as well as

[0092] 2) The steps of corrective actions (i.e., corrective steps), including: and

[0093] Optionally, in the case where the first agent outputs an erroneous activity during the interaction process, the teacher model is used to identify the erroneous activity of the first agent during the interaction process to generate corresponding erroneous activity information, and the teacher model is used to generate corrective activity information corresponding to the erroneous activity information, including: during the interaction process, the teacher model is used to receive current activity information corresponding to the current activity of the first agent and the current observation information of the environment with respect to the current activity information as current interaction information; the teacher model is used to obtain historical interaction information during the interaction process and task instructions of the third task; the teacher model is used to fill in the task instructions of the third task, the historical interaction information and the current interaction information into a pre-set prompt template, thereby generating corresponding prompt information; the teacher model is used to input the prompt information into the third largest language model, and the third largest language model is used to determine whether the current activity is an erroneous activity, and if the current activity is an erroneous activity, the current activity information is identified as an error to generate corresponding erroneous activity information; and the third largest language model is used to generate corrective activity information corresponding to the current activity information.

[0094] In the basic LLM agent π θ1 (i.e. the first agent) and the environment e 2 x In the interaction process, the teacher model π teacher In the basic LLM agent π θ1 Evaluate the correctness and rationality of its behavior in real time during interaction with the environment e.

[0095] Specifically, during the interaction, the teacher model π teacher Receive the base LLM agent π θ1 The current activity information t of the current activity (i.e., the thinking and action of the i-th step) 2 x,i and a 2 x,i , and the environment 2 x The current observation information o generated corresponding to the current activity 2 x,i Among them, the current activity information t 2 x,i and a 2 x,i And the current observation information o 2 x , as the current interaction information.

[0096] Then, the teacher model π teacher Further obtain the historical interaction information of the interaction process before the current activity (t 2 x,1 ,a2 x,1 ,o 2 x,1 ,...t 2 x,i-1 ,a 2 x,i-1 ,o 2 x,i-1 ) and task instructions u 2 x .

[0097] Then, the teacher model π teacher The current interaction information (t 2 x,i ,a 2 x,i ,o 2 x,i ), historical interaction information (t 2 x,1 ,a 2 x,1 ,o 2 x,1 ,...t 2 x,i-1 ,a 2 x,i-1 ,o 2 x,i-1 ) and task instructions u 2 x Fill in the preset prompt template to generate the corresponding prompt information. The prompt template will be explained in detail below.

[0098] Then, the teacher model π teacher Input the prompt information into the pre-set large language model π base3 (i.e. the third largest language model), thus through the large language model π base3 Determine whether the current activity is an error activity, and if the current activity is determined to be an error activity, send the current activity information (t 2 x,i ,a 2 x,i )Add error identifiers to generate corresponding error activity information And the large language model π base3 Further generate corrective activity information corresponding to the current activity

[0099] In addition, if the large language model π base3If the activity of the i-th step is determined not to be an error, or if the activity of the i-th step is determined to be correct, the activity information of the activity is retained without being changed. In other words, no corresponding self-reflection step is generated to replace the activity information of the step.

[0100] Thus, through the above method, this application makes the teacher model π based on LLM teacher The process of learning the agent’s capabilities is more effective and efficient.

[0101] Optionally, the prompt template includes instructions for instructing the third language model to implement the following operations: determining whether the current activity information filled in is an erroneous activity based on the task instructions, historical interaction information and current activity information filled in the prompt template; and in the case where it is determined that the current activity information filled in is an erroneous activity, identifying the filled in current activity information as an error, generating corresponding erroneous activity information and correcting activity information.

[0102] Specifically, Figure 5 A schematic diagram of a prompt template of a teacher model according to the present application is shown. Thus, the teacher model π teacher Relevant information can be filled into the prompt template. The prompt template includes the following parts that need to be filled: "{description}", "{notes}", "{task}", "{trajectory}", "{cur_step}", "{cur_feedback}".

[0103] The teacher model can fill in {description} with the description of the task (if any);

[0104] Fill the pre-set notes for completing the task into {notes};

[0105] Fill the task instruction into {task}, for example, fill the task instruction u 2 x Fill in {task} to form {u 2 x};

[0106] Fill the historical interaction information into {trajectory}, for example, (t 2 x,1 ,a 2 x,1 ,o 2 x,1 ,...t 2 x,i-1 ,a 2 x,i-1 ,o 2 x,i-1) is filled into {trajectory} to form {t 2 x,1 ,a 2 x,1 ,o 2 x,1 ,...t 2 x,i-1 ,a 2 x,i-1 ,o 2 x,i-1};

[0107] Fill the current activity information into {cur_step}, for example, (t 2 x,i ,a 2 x,i ,o 2 x,i ) is filled into {cur_step} to form {t 2 x,i ,a 2 x,i ,o 2 x,i};as well as

[0108] Fill the current observation information into {cur_feedback}, for example, 2 x,i Fill in {cur_feedback} to form {o 2 x,i}.

[0109] Additionally, the prompt template includes the following information:

[0110] “You need to judge whether the current Action of you is correct based on the task description and the trajectory.

[0111] 1.If it is correct,there is no obvious problem,or does not advancethe problem but is a necessary attempt or retrieval,just output"YES".In fact,most actions fall into this category.Assume the first Search step is correctand does not need correction,just output"YES".

[0112] 2.If it is incorrect or there is room for improvement,give thereasons and thoughts for the error.

[0113] *Then,based on the error that has already occurred,provide the Actionthat may correct the error.

[0114] you can only take one Action.*

[0115] You must output the content in the following format:

[0116] ERROR:[Write in here:contenErrort and error reason.]

[0117] THOUGHT:[Write in here:What should you to do next?What should you payattention to in the future?]

[0118] ACTION:[Write in here:The correct action that you should take to correct the error in*The current step*.Action can only be selected from theallowed actions.Output the answer directly without any additional explanatoryinformation.]”

[0119] The corresponding Chinese translation is as follows:

[0120] You need to judge whether your current activity is correct based on the task description and trajectory.

[0121] 1. If it is correct, there is no obvious problem, or it is a necessary attempt or search that does not advance the problem, just output "YES". In fact, most activities belong to this category. Assuming that the first step of the search is correct and does not need to be corrected, just output "YES".

[0122] 2. If it is incorrect or there is room for improvement, please give the reasons and thinking behind the error.

[0123] Then, based on the error that has occurred, provide activities that might correct the error.

[0124] You can only take one step at a time.

[0125] You must output content in the following format:

[0126] ERROR: [Write here: the content and reason of the error.]

[0127] THOUGHT: [Write here: What should you do next? What should you pay attention to in the future?]

[0128] ACTION: [Write here: The correct action you should take to correct the error in the current step. Actions can only be selected from allowed actions. Output the answer directly without any additional explanation information.]

[0129] Thus, the filled prompt information is input into the large language model π base3 After that, the large model π base3 According to the task instructions filled in the prompt template, the historical interaction information and the current interaction information of the i-th step, it is determined whether the activity of the i-th step is an error.

[0130] If the large model πbase3 If the current activity is determined to be an error activity, the current activity information (t 2 x,i ,a 2 x,i )Add error identifiers to generate corresponding error activity information And the large language model π base3 Further generate corrective activity information corresponding to the current activity

[0131] In addition, if the large language model π base3 If the activity of the i-th step is determined not to be an error, or if the activity of the i-th step is determined to be correct, the activity information of the activity is retained without being changed. In other words, no corresponding self-reflection step is generated to replace the activity information of the step.

[0132] Thus, through the above method, this application makes the teacher model π based on LLM teacher The process of learning the agent’s capabilities is more effective and efficient.

[0133] Optionally, the second large language model is supervised fine-tuned using the second trajectory to obtain the operation of the second intelligent agent with self-reflection ability, including: the second large language model is supervised fine-tuned using the fourth trajectory to obtain the second intelligent agent with self-reflection ability, wherein the second trajectory is the entire trajectory or a part of the trajectory of the fourth trajectory.

[0134] Specifically, according to this embodiment, the self-reflection trajectory set D3 can be used to determine the large language model π base2 The training trajectory set D4 (i.e., the fourth trajectory set) for supervised fine-tuning. The training trajectory set D4 corresponds to the task instruction subset U4, including the training trajectory τ 4 1~τ 4 m4 (i.e., the fourth trajectory). Among them, the training trajectory τ in D4 4 1~τ 4 m4 The trajectory lengths are n 4 1~n 4 m4 , the corresponding task instructions are u 4 1~u 4 m4 ∈U4, and its corresponding environments are e 4 1~e 4 m4 .

[0135] Among them, the training trajectory set D4, for example, can be completely composed of the self-reflective trajectory set D3. That is to say, the self-reflective trajectory τ 31~τ 3 m3 The training trajectory τ 4 1~τ 4 m4 Of all. Among them, m4=m3

[0136] However, in this embodiment, the self-reflection trajectory set D3 and the successful expert trajectory set D1 can also be combined to form the training trajectory set D4. 4 1~τ 4 m4 Part of the self-reflection trajectory τ 3 1~τ 3 m3 , and the other part is the successful expert trajectory τ 1 1~τ 1 m1 . Among them, m4=m1+m3.

[0137] Due to the tendency of catastrophic forgetting during supervised fine-tuning, they may lose some of their original capabilities after being trained on downstream tasks. θ1 Multiple self-reflective trajectories generated 3 1~τ 3 m3 (i.e., the second trajectory) and is used to supervise the fine-tuning of the LLM agent π θ1 Success Expert Trajectory τ 1 1~τ 1 m1 (i.e., the first trajectory) are combined to form a training dataset for retraining the large language model π base2 (i.e., the second largest language model) is fine-tuned through supervised fine-tuning to obtain a self-reflective LLM agent, which can effectively alleviate the above problems.

[0138] Optionally, the erroneous activity information includes erroneous thinking information and erroneous action information, and the corrective activity information includes corrective thinking information and corrective activity information corresponding to the erroneous thinking information and the erroneous action information.

[0139] Specifically, as described above, in this embodiment, the activity of the agent includes thought t and action a, so the erroneous activity information includes erroneous thought information (or ) and error action information (or ); Corrective activity information includes corrective thinking information and corrective action information Therefore, using this form of trajectory to supervise and fine-tune the agent can enhance the agent's reasoning ability.

[0140] Optionally, the operation of using the fourth trajectory to fine-tune the second largest language model to obtain the second agent with self-reflection ability includes: constructing a first loss function for fine-tuning the second largest language model; and minimizing the first loss function to fine-tune the second largest language model using the fourth trajectory to obtain the second agent. The first loss function is:

[0141]

[0142] in,

[0143] Among them, π θ2 Represents a self-reflective LLM agent π θ2 (i.e., the second agent);

[0144] D4 is the set of trajectories for training;

[0145] D4[logπ θ2 (τ 4 |e 4 ,u 4 )] represents the self-reflective LLM agent π θ2 With each training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 corresponding losses;

[0146] -E~D4[] represents the self-reflective LLM agent π θ2 and each training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 The expected value of the corresponding loss, which can be calculated by summing up the losses and taking the average in actual calculation;

[0147] τ 4 represents any training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 ;

[0148] e 4 Denotes any training trajectory τ 4 1~τ 4 m4 The corresponding environment 4 1~e 4 m4 ;

[0149] u 4 Denotes any training trajectory τ 4 1~τ 4m4 The corresponding task instruction u 4 1~u 4 m4 ;

[0150] n 4 Denotes any training trajectory τ 4 1~τ 4 m4 The length n 4 1~n 4 m4 ;

[0151] Denotes any training trajectory τ 4 1~τ 4 m4 The thinking information of the i-th step; Denotes any training trajectory τ 4 1~τ 4 m4 The action information of the ith step; Denotes any training trajectory τ 4 1~τ 4 m4 The observation information corresponding to the i-th step; h i-1 Represents τ 4 1~τ 4 m4 The historical interaction information before the i-th step.

[0152] Optionally, the fourth trajectory is used to perform supervised fine-tuning on the second language model to obtain the operation of the second agent with self-reflection ability, including:

[0153] constructing a second loss function for supervised fine-tuning of a second large language model; and

[0154] By minimizing the second loss function, the second largest language model is fine-tuned using the fourth trajectory to obtain the second agent, where the second loss function is:

[0155]

[0156] in,

[0157] Among them, π θ2 Represents a self-reflective LLM agent π θ2 (i.e., the second agent);

[0158] D4 is the set of trajectories for training;

[0159] D4[logπ θ2 (τ 4 |e4 ,u 4 )] represents the self-reflective LLM agent π θ2 With each training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 corresponding losses;

[0160] -E~D4[] represents the self-reflective LLM agent π θ2 and each training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 The expected value of the corresponding loss, which can be calculated by summing up the losses and taking the average in actual calculation;

[0161] τ 4 represents any training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 ;

[0162] e 4 Denotes any training trajectory τ 4 1~τ 4 m4 The corresponding environment 4 1~e 4 m4 ;

[0163] u 4 Denotes any training trajectory τ 4 1~τ 4 m4 The corresponding task instruction u 4 1~u 4 m4 ;

[0164] n 4 Denotes any training trajectory τ 4 1~τ 4 m4 The length n 4 1~n 4 m4 ;

[0165] Denotes any training trajectory τ 4 1~τ 4 m4 The thinking information of the i-th step; Denotes any training trajectory τ 4 1~τ 4 m4 The action information of the ith step; Denotes any training trajectory τ 4 1~τ 4 m4 The observation information corresponding to the i-th step; h i-1 Represents τ 4 1~τ 4 m4 The historical interaction information of the ith step; and

[0166] δ i is the partial shielding coefficient. When the i-th step is an error step, δ i = 0, and when the i-th step is not an error step, δ i =1.

[0167] Specifically, according to the method of the present application, using the training trajectory τ 4 1~τ 4 m4 π base2 When the second largest language model is fine-tuned through supervision, the partial shielding coefficient δ can be set in the loss function i In the process of supervised fine-tuning, the function of partial shielding is realized. base2 When the second largest language model is fine-tuned through supervision, the training trajectory τ can be shielded by partially shielding coefficients 4 1~τ 4 m4 The error activity information in the CNN is removed, while the observation information corresponding to the error activity information is retained. Therefore, the loss of the error activity information is not calculated.

[0168] Since the training trajectory τ 4 1~τ 4 m4 The self-reflection trajectory τ 3 1~τ 3 m3 In the process of supervised fine-tuning, the large language model may learn the wrong steps in the self-reflection trajectory, which may impair the ability of the agent to take correct actions. Therefore, in the process of supervised fine-tuning, this application introduces partial masking (PM) by introducing a partial masking coefficient in the loss function to prevent the large language model from learning incorrect thinking and behavior.

[0169] Optionally, the first large language model is supervised fine-tuned using multiple first trajectories recorded in the form of natural language text to obtain an operation of a first intelligent agent based on the large language model, including: constructing a third loss function for supervised fine-tuning the first large language model; and obtaining the first intelligent agent by minimizing the third loss function, using the first trajectories to supervise fine-tune the first large language model.

[0170] Specifically, we can use the successful expert trajectory τ in D1 1 1~τ 1 m1 (i.e., the first trajectory), by minimizing the loss function shown below (i.e., the third loss function), the large language model π base1 (i.e., the first language model) is fine-tuned to obtain the basic LLM agent π θ1 (i.e. the first agent):

[0171]

[0172] in,

[0173] Among them, π θ1 represents the first agent;

[0174] D1 is the successful expert trajectory τ 1 1~τ 1 m1 The training trajectory set is constructed;

[0175] D1[logπ θ1 (τ 1 |e 1 ,u 1 )] represents the basic LLM agent π θ1 (i.e., the first agent) and each successful expert trajectory τ in the training trajectory set 1 1~τ 1 m1 corresponding losses;

[0176] -E~D1[] represents the basic LLM agent π θ1 and each successful expert trajectory τ in the training trajectory set 1 1~τ 1 m1 The expected value of the corresponding loss, which can be calculated by summing up the losses and taking the average in actual calculation;

[0177] τ 1 represents any successful expert trajectory τ in the training trajectory set 1 1~τ 1 m1 ;e1 Denotes any successful expert trajectory τ 1 1~τ 1 m1 The corresponding environment 1 1~e 1 m1 ;u 1 Denotes any successful expert trajectory τ 1 1~τ 1 m1 The corresponding task instruction u 1 1~u 1 m1 ;n 1 Denotes any successful expert trajectory τ 1 1~τ 1 m1 The length n 1 1~n 1 m1 ; Denotes any successful expert trajectory τ 1 1~τ 1 m1 The thinking information of the i-th step; Denotes any successful expert trajectory τ 1 1~τ 1 m1 The action information of the ith step; Denotes any successful expert trajectory τ 1 1~τ 1 m1 The observation information corresponding to the i-th step; h i-1 Represents the successful expert trajectory τ 1 1~τ 1 m1 The historical interaction information corresponding to the i-th step.

[0178] Furthermore, optionally, the erroneous activity information includes only erroneous action information, and the corrective activity information includes only corrective action information corresponding to the erroneous action information.

[0179] That is, although in this embodiment, the activity of the agent includes thought t and action a. However, the case where the activity of the agent only includes action a is also applicable to the present application. Thus, the error activity information only includes the error action information. (or ), corrective activity information only includes corrective action information

[0180] Optionally, the expert trajectory set D can be obtained by the following method:

[0181] Use a large language model (such as GPT-4) to generate ReAct thoughts (i.e., t) for each step in a task that inherently contains an expert trajectory, or to generate expert trajectories for tasks that lack such trajectories. Then select successful expert trajectories with a reward equal to 1 from these expert trajectories to form an expert trajectory set D.

[0182] Further optionally, the expert trajectory set D1 is a subset obtained by randomly selecting 50% from the expert trajectory set D.

[0183] In addition, according to a second aspect of this embodiment, a training method for a self-reflective agent is provided, the method comprising:

[0184] S402: constructing a plurality of self-reflection trajectories recorded in the form of natural language text and successfully completing corresponding tasks, wherein the self-reflection estimate includes each step of successfully completing the corresponding task and a self-reflection step of reflecting on errors; and

[0185] S404: Use the self-reflection trajectory to supervise and fine-tune the large language model to obtain an intelligent agent with self-reflection ability.

[0186] Specifically, refer to Figure 4 As shown, the basic LLM agent π can be used by computing devices θ1 and the pre-set teacher model π teacher , based on the task instruction u in the task instruction subset U2 of the remaining trajectory set D2 2 1~u 2 m2 , construct the self-reflection trajectory set D3, and its corresponding task instruction set is U3.

[0187] Then, refer to Figure 4 As shown, the self-reflective trajectory τ in the self-reflective trajectory set D3 can be used by the computing device 3 1~τ 3 m3 For large language models π base2 Perform supervised fine-tuning to obtain a self-reflective LLM agent π with self-reflection ability θ2 .

[0188] Among them, the self-reflective LLM agent π θ2 Have the ability to reflect on oneself. Figure 2 As shown, the self-reflective LLM agent π θ2 In the process of interacting with the environment, it is possible to reflect on and correct the wrong steps in the task execution process based on the feedback from the environment. base2 Thus, we get the self-reflective LLM agent π θ2 The specific method will be described in detail later.

[0189] In addition, reference Figure 1 As shown, according to the third aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0190] Thus, in this embodiment, the small open source LLM can be supervised and fine-tuned using a set of successful expert trajectories (i.e., successful expert trajectories) to obtain a basic LLM agent, and then the basic LLM agent and the teacher model are used to construct a self-reflection trajectory set based on the task instructions in another task instruction space (e.g., U2) and the corresponding environment. The self-reflection trajectory of the self-reflection trajectory set includes the steps of reflecting on errors (i.e., identifying the wrong error activity information) and correcting (i.e., generating correct correction activity information) in the process of completing the corresponding task. Then, the small open source LLM is supervised and fine-tuned using the self-reflection trajectory to obtain a self-reflection LLM agent based on the small LLM and with reflection ability. Thus, in this way, it is possible to obtain a self-reflection LLM agent with self-reflection ability and excellent performance only through a small open source LLM without relying on complex prompt engineering and closed source large models such as GPT-4. Compared with the method of only using successful expert trajectories to train small LLMs to construct LLM agents, this embodiment uses self-reflection trajectories to train small LLMs, so that when the amount of training data increases, the performance improvement trend of the LLM agent can be prevented from slowing down, and it has a strong generalization ability. And the generated self-reflection LLM agent can effectively prevent the propagation of errors through the process of self-reflection during the execution of tasks. This solves the technical problems in the prior art that using successful expert trajectories to train LLM agents has limitations in enhancing generalization ability, and the trained LLM agent cannot get rid of the error cycle during the execution of tasks.

[0191] The following is a detailed description of the experiment completed by the inventor based on the technical solution of the present application to verify the technical effect of the technical solution of the present application:

[0192] 1. Experiments used to verify the technical effects of this application:

[0193] The inventors conducted experiments on three different intelligent agent tasks, and the results demonstrated the effectiveness of the inventors' approach on these three tasks. This section first introduces the three intelligent agent tasks and their datasets, and then describes the training settings, baseline models, and evaluation methods.

[0194] 1. Dataset

[0195] The inventors implemented the method of the present application on three intelligent agent tasks, namely ALFWorld, WebShop and SciWorld. Figure 6 to Figure 8 The prompt information of the above three tasks is shown. These tasks have been widely studied and are representative. ALFWorld and SciWorld contain a training set, a validation set (belonging to the training set distribution) and a test set (not belonging to the training set distribution). The inventors call the last two sets "seen" and "unseen" test sets respectively. Unlike the first two tasks, WebShop only contains one training set and one seen test set because it only involves one type of task. The specific data set statistics are shown in Table 1.

[0196] Table 1: Statistics of each dataset:

[0197] Task Training set Test set (seen) Test set (not seen) ALFWorld 3119 140 134 SciWorld 1483 194 211 WebShop 1016 200 -

[0198] 2. Experimental Setup

[0199] The inventor's experiments are mainly based on LLaMA2-7B-chat. Using a teacher model for real-time reflection and correction consumes a lot of tokens. Due to budget constraints, GPT-4 is not considered for experiments. Instead, the inventors used several powerful open source LLMs as teacher models, including LLaMA3-70B-Instruct, Qwen1.5-110B-Chat, and Qwen2-72B-Instruct. Both the teacher model and the basic LLM agent are deployed using the vllm framework. In all reasoning stages, the inventors set the temperature to 0 to ensure the consistency of the results. Due to the difficulties in the reflection and correction process, even the LLM teacher model may make mistakes. Therefore, the inventors excluded self-reflection trajectories containing a large number of error steps. Specifically, a maximum of 2 error steps are allowed in ALFWorld and SciWorld, while it is limited to 1 error step in Webshop. Finally, the inventors chose Qwen1.5-110B-Chat as the teacher model, generating 708 self-reflection trajectories.

[0200] 3. Benchmark

[0201] In addition to STeP (Self-Reflected Trajectories and Parital Masking), the inventors evaluated the performance of other LLMs on three agent tasks. 1) Using only successful expert trajectories (i.e., golden trajectories): FireAct and AgentTuning, that is, only using successful expert trajectories to supervise fine-tune LLM. The inventors only used successful expert trajectory D to supervise fine-tune LLaMA2-7B-chat, called "LLaMA2-7B-chat+Golden Trajs". The inventors also tested AgentLM-7B from AgentTuning, which includes a general dataset in its training data. Since its training set only covers ALFWorld and WebShop, and the data is inconsistent with the inventors, its results are for reference only. The evaluation indicators for AgentLM are also calculated only based on ALFWorld and WebShop. 2) Basic LLM: The inventors also tested some models that were not specifically trained for these agent tasks, including GPT-4o-0513, GPT-4, GPT-3.5-turbo, LLaMA3-70B-Instruct, Qwen1.5-110B-Chat, Qwen2-72B-Instruct, LLaMA2-7B-Chat, and LLaMA3-8B-Instruct.

[0202] 4. Evaluation

[0203] The performance of an LLM-based LLM agent in completing a given task instruction can be represented by a reward r∈[0,1], where r=1 represents complete success. The reward r is automatically derived from the environment e corresponding to the agent's task. The main evaluation metric is the average reward, which is the average of all rewards obtained by the agent in a specific agent task.

[0204] Agent tasks have a maximum step limit. If the agent fails to complete the task instruction within a given step, the reward r = 0. Specifically, the maximum step number of ALFWorld tasks is 30, WebShop is 10, and the step limit of different subtasks in SciWorld ranges from 10 to 120. If the number of tokens in the trajectory exceeds the maximum context length of LLM, r is also set to 0.

[0205] During testing, the inventors set the temperature of all models to 0 and adopted greedy decoding to obtain deterministic results. The inventors mainly used zero-shot prompts to guide the LLM-based agents to step through the task instructions in the ReAct format. This is because the context window length is limited and the LLM has been trained on trajectories that conform to this format, enabling it to follow the task requirements to a certain extent. In addition, the inventors also conducted a one-sample experiment for comparative analysis. However, since the base LLM is not specifically trained for the agent task, there are challenges in the zero-shot setting. Therefore, when testing these LLMs, the inventors adopted a one-shot approach.

[0206] 2. Experimental results and analysis

[0207] According to the experimental setup described above, we tested our trained LLM-based self-reflective LLM agent and various benchmarks on three agent tasks, as shown in Table 2. Subsequently, in this section, we analyze the results in detail and conduct ablation experiments for further exploration.

[0208] 1. Main results

[0209] Table 2 shows the main results of the inventor's experiment. The values ​​in the WebShop, ALFWorld, and SciWorld columns represent the average rewards of each model on the corresponding tasks. The "Average" column represents the mean of the values ​​in the first five columns. The last column represents the completion rate of the model on these three tasks.

[0210] The results shown in Table 2 show that our self-reflective LLM agent has higher average rewards across all three tasks compared to most baselines, demonstrating the effectiveness of our approach.

[0211] Compared to LLM-based agents trained only on the golden track, agents trained with STeP (using self-reflective tracks + partial masking) achieve superior results on all test sets. LLaMA2-7B-chat+STeP (i.e., using self-reflective tracks + partial masking to supervise fine-tuning LLaMA2-7B-chat) achieves an average improvement of 9.2% on the test set of WebShop and the "unseen" test sets of ALFWorld and SciWorld. Specifically, STeP improves performance by 9.4% on WebShop, 4.1% and 10.3% on ALFWorld, and 2.2% and 8% on SciWorld, respectively. In addition, LLaMA2-7B-chat+STeP achieves these improvements while using less training data than methods using only the golden track. Fig. 9 shows this more clearly.

[0212] As shown in Table 2, LLMs that are not specifically trained for these agent tasks have difficulty completing them, especially smaller LLMs. For LLaMA3-8B-Instruct and LLaMA2-7B-chat, completing the task instructions of ALFWorld is particularly difficult. By leveraging trajectories for SFT, the inventors can achieve at least 50% rewards, approaching or exceeding larger or closed-source LLMs. Agents trained by STeP can easily surpass their corresponding teacher models in experimental results. Larger LLMs or closed-source models have an average reward of about 0.6 on Webshop. However, Qwen performs better on ALFWorld, while other models perform well on SciWorld, which the inventors believe is due to differences in their training data. The training of these open source LLMs may include data from specific agent datasets.

[0213] Table 2: Main results of the inventor's experiment

[0214]

[0215] In addition, the inventors also tested the completion rate of the LLM-based agent, which is defined as the proportion of tasks completed before reaching the maximum step limit and the maximum context window length. The results in the table show that although adding reflection and error correction may increase the number of steps and tags to some extent, the inventors' method still improves the completion rate.

[0216] 2. Simplify the experiment

[0217] 1) Effectiveness of self-reflective trajectories. Our training set combines self-reflective trajectories D3 with a portion of golden trajectories D1. To evaluate the impact of self-reflective trajectories, we train LLaMA2-7B using only D1, and obtain model LLaMA2-7B (without self-reflective trajectories). As shown in Table 3, not using any self-reflective trajectories results in a drop in performance. If we increase the number of successful expert trajectories and train with the trajectory set D, i.e., LLaMA2-7B+golden trajectories, the gains are minimal: an improvement of 12.9% on the “unseen” SciWorld test set, an improvement of 1.8% on WebShop, and even a decrease of 2.2% on the “unseen” test set of ALFWorld. In contrast, as shown in Table 4, our method only adds 708 self-reflective trajectories to D1, but achieves a 22% improvement on SciWorld, an 11.4% improvement on WebShop, and a 7.8% improvement on ALFWorld, with an average growth rate of 13.7%.

[0218] Table 3: Without using self-reflective trajectories + partial shielding, the overall performance of the agent decreases

[0219]

[0220] 2) Effectiveness of partial masking. The inventors replaced the loss function shown in formula (7) with the loss function shown in formula (8), allowing the model to learn from actions and thoughts that are labeled as wrong. The results show that partial masking effectively enhances the overall agent capabilities of the LLM agent (except for the seen test set of ALFWorld, which the inventors believe may be because the ALFWorld task has a higher tolerance for wrong actions, allowing the agent trained without partial masking to explore a wider range of error trajectories).

[0221] 3. Analysis

[0222] Regarding the selection of LLM teacher model. The inventors conducted multiple experiments on open source LLM to determine a more suitable LLM teacher model. In the trajectory enhancement stage, the inventors used LLaMA3-70B-Instruct, Qwen1.5-110B-Chat, and Qwen2-72B-Instruct as LLM teacher models to provide real-time reflection and correction for LLaMA2-7B-chat. For each LLM teacher model, the inventors filtered out the corresponding self-reflective trajectory according to the method described in this embodiment and the experiment. Subsequently, the inventors used these data to train the corresponding self-reflective LLM agents according to the method described in this embodiment, and finally obtained the average reward of these agents on all test sets. As Fig.10 As shown, it is clear that when used as a teacher model, Qwen1.5-110B-Chat produces the most self-reflective trajectories and achieves the best test results. Although the other LLM teachers do not perform as well as Qwen1.5-110B-Chat, they still achieve significant results. The LLM-based agents trained using these other LLM teachers outperform LLaMA2-7B-chat+golden trajectory.

[0223] 4. Further Experiments

[0224] The inventors conducted further experiments to explore, as shown in Tables 4 and 5.

[0225] The inventors used the self-reflection track D3 and half of all the golden tracks D1 to train Mistral-7B-Instruct V0.3 and LLaMA3-8B-Instruct according to the method proposed by the inventors. The experimental results show that although the performance of both models on the SciWorld task is lower than the corresponding SFT results, they surpass these results on the WebShop and ALFWorld tasks, showing overall superior performance.

[0226] The inventor combined D3 with D1 to retrain the large language model π base , to alleviate the catastrophic forgetting problem. To verify this, the inventors designed an experiment in which the inventors first trained on D1 and then further trained on D3. The inventors named the resulting agent LLaMA2-7B (STeP). The results show that the efficiency on the ALFWorld task is improved, while the results on other tasks are significantly reduced. The results show that better results are achieved on the ALFWorld task, while the results on other tasks are significantly reduced. This confirms that combining D3 with D1 is necessary to improve the overall performance of the agent.

[0227] Our main results for our agents are based on zero-shot prompts. Here, we conduct one-shot experiments to explore the impact of in-context learning (ICL) examples on our method. Using one-shot for reasoning significantly impacts the capabilities of our model agents compared to zero-shot prompts. We hypothesize that this is due to the limitation of the context window and the potential differences between the ICL examples and the target instructions.

[0228] Table 4: Using STep, agents trained on other LLMs all achieved better overall performance

[0229]

[0230] Table 5: Other simplified experiments. Retraining LLM by combining self-reflection traces with some golden traces helps alleviate catastrophic forgetting. The performance of one-shot samples is not ideal due to the limited context window. Training for a single task does not necessarily improve the performance of the model on the corresponding task.

[0231]

[0232] The inventors also investigated the difference between training on a single task and training on multiple tasks. The inventors used the golden tracks from the three agent tasks to train LLaMA2-7B-Chat separately. As shown in the results presented in Tables 4 and 2, training on only one task only benefits some agent tasks, while for WebShop and SciWorld, training on multiple tasks produces better results.

[0233] Therefore, the experiments described above on several representative intelligent agent task datasets have demonstrated the effectiveness of the method of this application.

[0234] Appendix: Example of a self-reflection trajectory

[0235]

[0236]

[0237]

[0238] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0239] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0240] Example 2

[0241] Fig.11 The apparatus 1100 for training a self-reflective agent according to this embodiment is shown, and the apparatus 1100 corresponds to the method according to the first aspect of embodiment 1. Fig.11As shown, the device 1100 includes: a first agent training module 1110, which is used to supervise and fine-tune the first large language model using multiple first trajectories recorded in the form of natural language text to obtain a first agent based on the large language model, wherein the multiple first trajectories are expert trajectories that successfully complete the corresponding first task and include various steps for completing the corresponding first task; a self-reflection trajectory construction module 1120, which is used to construct multiple second trajectories recorded in the form of natural language text and successfully complete the corresponding second task using the first agent and the teacher model, the second trajectories include various steps for completing the corresponding second task and a self-reflection step for reflecting on the mistakes of the first agent; and a second agent training module 1130, which is used to supervise and fine-tune the second large language model using the second trajectory to obtain a second agent with self-reflection ability.

[0242] Optionally, the self-reflection trajectory construction module 1120 includes: a third task execution sub-module, which is used to use the first intelligent agent to execute task instructions of multiple third tasks and interact with the environment of the third task in the form of natural language text; a self-reflection step generation sub-module, which is used to use the teacher model to identify the erroneous activities of the first intelligent agent during the interaction process and generate corresponding erroneous activity information, use the teacher model to generate corrective activity information corresponding to the erroneous activity information, and merge the erroneous activity information and the corrective activity information as self-reflective steps into the third trajectory corresponding to the current third task; and a self-reflection trajectory construction sub-module, which is used to filter out second trajectories that successfully complete the task and include self-reflective steps from the third trajectories corresponding to each third task.

[0243] Optionally, the self-reflection step generates a submodule, including: a current interaction information receiving unit, which is used to use the teacher model to receive current activity information corresponding to the current activity of the first intelligent agent and the current observation information of the environment with respect to the current activity information as current interaction information during the interaction process; a task instruction acquisition unit, which is used to use the teacher model to obtain historical interaction information during the interaction process and task instructions of the third task; a prompt information generation unit, which is used to use the teacher model to fill in the task instructions, historical interaction information and current interaction information of the third task into a pre-set prompt template, thereby generating corresponding prompt information; and an error activity information generation unit, which is used to use the teacher model to input the prompt information into the third largest language model, determine whether the current activity is an error activity through the third largest language model, and if the current activity is an error activity, identify the current activity information as an error to generate corresponding error activity information; and a correction activity information generation unit, which is used to generate correction activity information corresponding to the current activity information through the third largest language model.

[0244] Optionally, the prompt template includes instructions for instructing the third language model to implement the following operations: determining whether the current activity information filled in is an erroneous activity based on the task instructions, historical interaction information and current interaction information filled in the prompt template; and in the case where it is determined that the current activity information filled in is an erroneous activity, identifying the filled in current activity information as an error, generating corresponding erroneous activity information and correcting activity information.

[0245] Optionally, the second agent training module 1130 includes: a supervised fine-tuning submodule, which is used to use the fourth trajectory to supervise the fine-tuning of the second large language model to obtain a second agent with self-reflection ability, wherein the second trajectory is the entire trajectory or a part of the fourth trajectory.

[0246] Optionally, the erroneous activity information includes erroneous thinking information and erroneous action information, and the corrective activity information includes corrective thinking information and corrective activity information corresponding to the erroneous thinking information and the erroneous action information.

[0247] Optionally, the supervised fine-tuning submodule includes: a first loss function construction unit, used to construct a first loss function for supervised fine-tuning the second large language model; and a first supervised fine-tuning unit, used to supervise fine-tune the second large language model using a fourth trajectory by minimizing the first loss function to obtain a second agent. The first loss function is:

[0248]

[0249] in,

[0250] Among them, π θ2 represents the second agent; D4 is the set of training trajectories consisting of the fourth trajectory; D4[logπ θ2 (τ 4 |e 4 ,u 4 )] represents the loss of each fourth trajectory in the set of training trajectories executed by the second agent; -E~D4[] represents the expected value of the loss; τ 4 represents any fourth trajectory in the training trajectory set; e 4 represents the environment corresponding to any fourth trajectory; u 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 represents the length of any fourth trajectory; represents the thinking information of the ith step of any fourth trajectory; represents the action information of the ith step of any fourth trajectory; represents the observation information corresponding to the i-th step of any fourth trajectory; h i-1Represents the historical interaction information before the i-th step.

[0251] Optionally, the supervised fine-tuning submodule includes: a second loss function construction unit, used to construct a second loss function for supervised fine-tuning the second large language model; and a second supervised fine-tuning unit, used to supervise fine-tune the second large language model using a fourth trajectory by minimizing the second loss function to obtain a second agent. The second loss function is:

[0252]

[0253] in,

[0254] Among them, π θ2 represents the second agent; D4 is the set of training trajectories consisting of the fourth trajectory; D4[logπ θ22 (τ 4 |e 4 ,u 4 )] represents the loss of each fourth trajectory in the set of training trajectories executed by the second agent; -E~D4[] represents the expected value of the loss; τ 4 represents any fourth trajectory in the training trajectory set; e 4 represents the environment corresponding to any fourth trajectory; u 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 represents the length of any fourth trajectory; represents the thinking information of the ith step of any fourth trajectory; represents the action information of the ith step of any fourth trajectory; represents the observation information corresponding to the i-th step of any fourth trajectory; h i-1 represents the historical interaction information before the i-th step; and, δ i is the partial shielding coefficient. When the i-th step is an error step, δ i = 0, and when the i-th step is not an error step, δ i =1.

[0255] Optionally, the first agent training module 1110 includes: a third loss function construction submodule, used to construct a third loss function for supervised fine-tuning of the first large language model; and a first supervised fine-tuning submodule, used to supervise fine-tune the first large language model using the first trajectory by minimizing the third loss function to obtain the first agent, wherein the third loss function is:

[0256]

[0257] in,

[0258] Among them, π θ1 represents the first agent; D1 is the set of training trajectories consisting of the first trajectory; D1[logπ θ1 (τ 1 |e 1 ,u 1 )] represents the loss of each first trajectory in the set of training trajectories executed by the first agent; -E~D1[] represents the expected value of the loss; τ 1 represents any first trajectory in the training trajectory set; e 1 represents the environment corresponding to any first trajectory; u 1 Indicates the task instruction corresponding to any first trajectory; n 1 represents the length of any first trajectory; represents the thinking information of the ith step of any first trajectory; represents the action information of the ith step of any first trajectory; represents the observation information corresponding to the i-th step of any first trajectory; h i-1 Represents the historical interaction information before the i-th step.

[0259] Thus, according to this embodiment, a small open source LLM can be supervised and fine-tuned using a set of successful expert trajectories (i.e., successful expert trajectories) to obtain a basic LLM agent, and then a self-reflection trajectory set is constructed based on the task instructions in another task instruction space (e.g., U2) and the corresponding environment using the basic LLM agent and the teacher model. The self-reflection trajectory of the self-reflection trajectory set includes the steps of reflecting on errors (i.e., identifying incorrect error activity information) and correcting (i.e., generating correct corrective activity information) in the process of completing the corresponding task. Then, the small open source LLM is supervised and fine-tuned using the self-reflection trajectory to obtain a self-reflection LLM agent based on the small LLM and with reflection ability. Thus, in this way, it is possible to obtain a self-reflection LLM agent with self-reflection ability and excellent performance only through a small open source LLM without relying on complex prompt engineering and closed source large models such as GPT-4. Compared with the method of only using successful expert trajectories to train small LLMs to construct LLM agents, this embodiment uses self-reflection trajectories to train small LLMs, so that when the amount of training data increases, the performance improvement trend of the LLM agent can be prevented from slowing down, and it has a strong generalization ability. And the generated self-reflection LLM agent can effectively prevent the propagation of errors through the process of self-reflection during the execution of tasks. This solves the technical problems in the prior art that using successful expert trajectories to train LLM agents has limitations in enhancing generalization ability, and the trained LLM agent cannot get rid of the error cycle during the execution of tasks.

[0260] Example 3

[0261] Fig.12 The apparatus 1200 for training a self-reflective agent according to this embodiment is shown, and the apparatus 1200 corresponds to the method according to the first aspect of embodiment 1. Fig.12 As shown, the device 1200 includes: a processor 1210; and a memory 1220, which is connected to the processor 1210 and is used to provide the processor 1210 with instructions for processing the following processing steps: using multiple first trajectories recorded in the form of natural language text to supervise and fine-tune the first large language model to obtain a first intelligent agent based on the large language model, wherein the multiple first trajectories are expert trajectories that successfully complete the corresponding first task and include various steps for completing the corresponding first task; using the first intelligent agent and the teacher model to construct multiple second trajectories that are recorded in the form of natural language text and successfully complete the corresponding second task, the second trajectories include various steps for completing the corresponding second task and a self-reflection step for reflecting on the mistakes of the first intelligent agent; and using the second trajectories to supervise and fine-tune the second large language model to obtain a second intelligent agent with self-reflection ability.

[0262] Optionally, the operation of constructing multiple second trajectories that are recorded in the form of natural language text and successfully complete the corresponding second tasks using the first agent and the teacher model includes: using the first agent to execute task instructions of multiple third tasks and interacting with the environment of the third task in the form of natural language text; when the first agent outputs an erroneous activity during the interaction process, using the teacher model to identify the erroneous activity of the first agent during the interaction process to generate corresponding erroneous activity information, using the teacher model to generate corrective activity information corresponding to the erroneous activity information, and merging the erroneous activity information and the corrective activity information as self-reflection steps into the third trajectory corresponding to the current third task; and screening out second trajectories that successfully complete the task and include self-reflection steps from the third trajectories corresponding to each third task.

[0263] Optionally, in the case where the first agent outputs an erroneous activity during the interaction process, the teacher model is used to identify the erroneous activity of the first agent during the interaction process to generate corresponding erroneous activity information, and the teacher model is used to generate corrective activity information corresponding to the erroneous activity information, including: during the interaction process, the teacher model is used to receive current activity information corresponding to the current activity of the first agent and the current observation information of the environment with respect to the current activity information as current interaction information; the teacher model is used to obtain historical interaction information during the interaction process and task instructions of the third task; the teacher model is used to fill in the task instructions of the third task, the historical interaction information and the current interaction information into a pre-set prompt template, thereby generating corresponding prompt information; the teacher model is used to input the prompt information into the third largest language model, and the third largest language model is used to determine whether the current activity is an erroneous activity, and if the current activity is an erroneous activity, the current activity information is identified as an error to generate corresponding erroneous activity information; and the third largest language model is used to generate corrective activity information corresponding to the current activity information.

[0264] Optionally, the prompt template includes instructions for instructing the third language model to implement the following operations: determining whether the current activity information filled in is an erroneous activity based on the task instructions, historical interaction information and current interaction information filled in the prompt template; and in the case where it is determined that the current activity information filled in is an erroneous activity, identifying the filled in current activity information as an error, generating corresponding erroneous activity information and correcting activity information.

[0265] Optionally, the second large language model is supervised fine-tuned using the second trajectory to obtain the operation of the second intelligent agent with self-reflection ability, including: the second large language model is supervised fine-tuned using the fourth trajectory to obtain the second intelligent agent with self-reflection ability, wherein the second trajectory is the entire trajectory or a part of the trajectory of the fourth trajectory.

[0266] Optionally, the erroneous activity information includes erroneous thinking information and erroneous action information, and the corrective activity information includes corrective thinking information and corrective activity information corresponding to the erroneous thinking information and the erroneous action information.

[0267] Optionally, the operation of using the fourth trajectory to fine-tune the second largest language model to obtain the second agent with self-reflection ability includes: constructing a first loss function for fine-tuning the second largest language model; and minimizing the first loss function to fine-tune the second largest language model using the fourth trajectory to obtain the second agent. The first loss function is:

[0268]

[0269] in,

[0270] Among them, π θ2 represents the second agent; D4 is the set of training trajectories consisting of the fourth trajectory; D4[logπ θ2 (τ 4 |e 4 ,u 4 )] represents the loss of each fourth trajectory in the set of training trajectories executed by the second agent; -E~D4[] represents the expected value of the loss; τ 4 represents any fourth trajectory in the training trajectory set; e 4 represents the environment corresponding to any fourth trajectory; u 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 represents the length of any fourth trajectory; represents the thinking information of the ith step of any fourth trajectory; represents the action information of the ith step of any fourth trajectory; represents the observation information corresponding to the i-th step of any fourth trajectory; h i-1 Represents the historical interaction information before the i-th step.

[0271] Optionally, the operation of using the fourth trajectory to fine-tune the second largest language model to obtain the second agent with self-reflection ability includes: constructing a second loss function for fine-tuning the second largest language model; and minimizing the second loss function to fine-tune the second largest language model using the fourth trajectory to obtain the second agent. The second loss function is:

[0272]

[0273] in,

[0274] Among them, πθ2 represents the second agent; D4 is the set of training trajectories consisting of the fourth trajectory; D4[logπ θ2 (τ 4 |e 4 ,u 4 )] represents the loss of each fourth trajectory in the set of training trajectories executed by the second agent; -E~D4[] represents the expected value of the loss; τ 4 represents any fourth trajectory in the training trajectory set; e 4 represents the environment corresponding to any fourth trajectory; u 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 represents the length of any fourth trajectory; represents the thinking information of the ith step of any fourth trajectory; represents the action information of the ith step of any fourth trajectory; represents the observation information corresponding to the i-th step of any fourth trajectory; h i-1 represents the historical interaction information before the i-th step; and, δ i is the partial shielding coefficient. When the i-th step is an error step, δ i = 0, and when the i-th step is not an error step, δ i =1.

[0275] Optionally, using a plurality of first trajectories recorded in the form of natural language text to fine-tune the first large language model in a supervised manner to obtain an operation of a first agent based on the large language model includes: constructing a third loss function for fine-tuning the first large language model in a supervised manner; and minimizing the third loss function to fine-tune the first large language model in a supervised manner using the first trajectories to obtain the first agent. The third loss function is:

[0276]

[0277] in,

[0278] Among them, π θ1 represents the first agent; D1 is the set of training trajectories consisting of the first trajectory; D1[logπ θ1 (τ 1 |e 1 ,u 1 )] represents the loss of each first trajectory in the set of training trajectories executed by the first agent; -E~D1[] represents the expected value of the loss; τ 1 represents any first trajectory in the training trajectory set; e 1 represents the environment corresponding to any first trajectory; u 1Indicates the task instruction corresponding to any first trajectory; n 1 represents the length of any first trajectory; represents the thinking information of the ith step of any first trajectory; represents the action information of the ith step of any first trajectory; represents the observation information corresponding to the i-th step of any first trajectory; h i-1 Represents the historical interaction information before the i-th step.

[0279] Thus, according to this embodiment, a small open source LLM can be supervised and fine-tuned using a set of successful expert trajectories (i.e., successful expert trajectories) to obtain a basic LLM agent, and then a self-reflection trajectory set is constructed based on the task instructions in another task instruction space (e.g., U2) and the corresponding environment using the basic LLM agent and the teacher model. The self-reflection trajectory of the self-reflection trajectory set includes the steps of reflecting on errors (i.e., identifying incorrect error activity information) and correcting (i.e., generating correct corrective activity information) in the process of completing the corresponding task. Then, the small open source LLM is supervised and fine-tuned using the self-reflection trajectory to obtain a self-reflection LLM agent based on the small LLM and with reflection ability. Thus, in this way, it is possible to obtain a self-reflection LLM agent with self-reflection ability and excellent performance only through a small open source LLM without relying on complex prompt engineering and closed source large models such as GPT-4. Compared with the method of only using successful expert trajectories to train small LLMs to construct LLM agents, this embodiment uses self-reflection trajectories to train small LLMs, so that when the amount of training data increases, the performance improvement trend of the LLM agent can be prevented from slowing down, and it has a strong generalization ability. And the generated self-reflection LLM agent can effectively prevent the propagation of errors through the process of self-reflection during the execution of tasks. This solves the technical problems in the prior art that using successful expert trajectories to train LLM agents has limitations in enhancing generalization ability, and the trained LLM agent cannot get rid of the error cycle during the execution of tasks.

[0280] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0281] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0282] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0283] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0284] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0285] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0286] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for training a self-reflective agent, characterized in that: include: Performing supervised fine-tuning on a first large language model using a plurality of first trajectories recorded in the form of natural language texts to obtain a first agent based on the large language model, wherein the plurality of first trajectories are expert trajectories for successfully completing corresponding first tasks and include various steps for completing the corresponding first tasks; Constructing a plurality of second trajectories recorded in a natural language text form and successfully completing a corresponding second task using the first agent and the teacher model, wherein the second trajectories include steps for completing the corresponding second task and a self-reflection step for reflecting on the mistakes of the first agent; as well as The second trajectory is used to perform supervised fine-tuning on the second largest language model to obtain a second intelligent agent with self-reflection ability.

2. The method according to claim 1, characterized in that The operation of constructing a plurality of second trajectories recorded in the form of natural language text and successfully completing the corresponding second task using the first agent and the teacher model includes: Utilize the first agent to execute a plurality of task instructions of a third task, and interact with the environment of the third task in a natural language text manner; In the case where the first agent outputs an erroneous activity during the interaction, the teacher model is used to identify the erroneous activity of the first agent during the interaction to generate corresponding erroneous activity information, the teacher model is used to generate corrective activity information corresponding to the erroneous activity information, and the erroneous activity information and the corrective activity information are merged as the self-reflection step into a third trajectory corresponding to the current third task; The second trajectories that successfully complete the tasks and include the self-reflection step are screened out from the third trajectories corresponding to the respective third tasks.

3. The method according to claim 2, characterized in that In the case where the first agent outputs an erroneous activity during the interaction, the operation of using the teacher model to identify the erroneous activity of the first agent during the interaction to generate corresponding erroneous activity information, and using the teacher model to generate corrective activity information corresponding to the erroneous activity information includes: In the interaction process, using the teacher model to receive current activity information corresponding to the current activity of the first agent and current observation information of the environment with respect to the current activity information as current interaction information; Using the teacher model to obtain historical interaction information in the interaction process and a task instruction of the third task; Using the teacher model, the task instruction of the third task, the historical interaction information and the current interaction information are filled into a preset prompt template, thereby generating corresponding prompt information; Using the teacher model to input the prompt information into a third language model, determining whether the current activity is an erroneous activity by using the third language model, and if the current activity is an erroneous activity, marking the current activity information as an error and generating corresponding erroneous activity information; and Generating corrected activity information corresponding to the current activity information through the third largest language model, and wherein, The prompt template includes instructions for instructing the third language model to perform the following operations: Determining whether the filled current activity information is an erroneous activity according to the task instruction, historical interaction information and current interaction information filled into the prompt template; and In the case where it is determined that the filled current activity information is an erroneous activity, the filled current activity information is identified as an erroneous activity, and corresponding erroneous activity information and corrected activity information are generated. And wherein the second large language model is fine-tuned by using the second trajectory to obtain the operation of the second intelligent agent with self-reflection ability, including: The second large language model is fine-tuned by supervision using the fourth trajectory to obtain the second agent with self-reflection ability, wherein the second trajectory is the entire trajectory or a part of the fourth trajectory.

4. The method according to claim 3, characterized in that The erroneous activity information includes erroneous thinking information and erroneous action information, and the corrective activity information includes corrective thinking information and corrective activity information corresponding to the erroneous thinking information and the erroneous action information.

5. The method according to claim 4, characterized in that The second language model is fine-tuned by using the fourth trajectory to obtain the operation of the second agent with self-reflection ability, including: constructing a first loss function for supervised fine-tuning of the second largest language model; and The second agent is obtained by minimizing the first loss function and using the fourth trajectory to perform supervised fine-tuning on the second large language model, wherein the first loss function is: in, Among them, π θ2 represents the second agent; D4 is the set of training trajectories formed by the fourth trajectory; D4[logπ θ2 (τ 4 |e 4 ,u 4 )] represents the loss of the second agent executing each fourth trajectory in the training trajectory set; -E~D4[] represents the expected value of the loss; τ 4 represents any fourth track in the training track set; e 4 represents the environment corresponding to any fourth trajectory; 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 represents the length of any fourth trajectory; Representing the thought information of the ith step of any fourth trajectory; representing the action information of the ith step of any fourth trajectory; represents the observation information corresponding to the i-th step of any fourth trajectory; i-1 Represents the historical interaction information before the i-th step.

6. The method according to claim 4, characterized in that The second language model is fine-tuned by using the fourth trajectory to obtain the operation of the second agent with self-reflection ability, including: constructing a second loss function for supervised fine-tuning of the second largest language model; and The second agent is obtained by minimizing the second loss function and using the fourth trajectory to perform supervised fine-tuning on the second large language model, wherein the second loss function is: in, Among them, π θ2 represents the second agent; D4 is the set of training trajectories formed by the fourth trajectory; D4[logπ θ2 (τ 4 |e 4 ,u 4 )] represents the loss of the second agent executing each fourth trajectory in the training trajectory set; -E~D4[] represents the expected value of the loss; τ 4 represents any fourth track in the training track set; e 4 represents the environment corresponding to any fourth trajectory; 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 represents the length of any fourth trajectory; Representing the thought information of the ith step of any fourth trajectory; representing the action information of the ith step of any fourth trajectory; represents the observation information corresponding to the i-th step of any fourth trajectory; i-1 represents the historical interaction information before the i-th step; and, δ i is the partial shielding coefficient. When the i-th step is an error step, δ i = 0, and when the i-th step is not an error step, δ i =1.

7. The method according to claim 4, characterized in that The first large language model is fine-tuned by using a plurality of first trajectories recorded in the form of natural language texts to obtain an operation of the first agent based on the large language model, including: Constructing a third loss function for supervised fine-tuning of the first language model; and The first agent is obtained by minimizing the third loss function and using the first trajectory to perform supervised fine-tuning on the first large language model, wherein the third loss function is: in, Among them, π θ1 represents the first agent; D1 is the set of training trajectories formed by the first trajectory; D1[logπ θ1 (τ 1 |e 1 ,u 1 )] represents the loss of the first agent executing each first trajectory in the training trajectory set; -E~D1[] represents the expected value of the loss; τ 1 represents any first trajectory in the training trajectory set; e 1 represents the environment corresponding to any first trajectory; u 1 represents the task instruction corresponding to any first trajectory; n 1 represents the length of any first trajectory; Representing the thought information of the ith step of any first trajectory; representing the action information of the ith step of any first trajectory; represents the observation information corresponding to the i-th step of any first trajectory; h i-1 Represents the historical interaction information before the i-th step.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 7.

9. A training device for a self-reflective intelligent agent, characterized in that: include: A first agent training module is used to supervise and fine-tune the first large language model using a plurality of first trajectories recorded in the form of natural language texts to obtain a first agent based on the large language model, wherein the plurality of first trajectories are expert trajectories that successfully complete the corresponding first tasks and include various steps for completing the corresponding first tasks; A self-reflection trajectory construction module, used to construct a plurality of second trajectories recorded in a natural language text form and successfully completing the corresponding second task by using the first agent and the teacher model, wherein the second trajectories include various steps for completing the corresponding second task and a self-reflection step for reflecting on the mistakes of the first agent; as well as The second agent training module is used to use the second trajectory to supervise and fine-tune the second large language model to obtain a second agent with self-reflection ability.

10. A training device for a self-reflective intelligent agent, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Performing supervised fine-tuning on a first large language model using a plurality of first trajectories recorded in the form of natural language texts to obtain a first agent based on the large language model, wherein the plurality of first trajectories are expert trajectories for successfully completing corresponding first tasks and include various steps for completing the corresponding first tasks; Constructing a plurality of second trajectories recorded in a natural language text form and successfully completing a corresponding second task using the first agent and the teacher model, wherein the second trajectories include steps for completing the corresponding second task and a self-reflection step for reflecting on the mistakes of the first agent; as well as The second trajectory is used to perform supervised fine-tuning on the second largest language model to obtain a second intelligent agent with self-reflection ability.

Images