Training method and device of self-reflection intelligent agent and storage medium

Abstract

The application discloses a training method and device of a self-reflective intelligent agent and a storage medium. The method comprises: using a plurality of first trajectories recorded in the form of natural language text to supervise and fine-tune a first large language model to obtain a first intelligent agent based on the large language model, wherein the plurality of first trajectories are expert trajectories successfully completing corresponding first tasks and comprise steps of completing the corresponding first tasks; using the first intelligent agent and a teacher model to construct a plurality of second trajectories recorded in the form of natural language text and successfully completing corresponding second tasks, the second trajectories comprising steps of completing the corresponding second tasks and self-reflective steps of reflecting on errors of the first intelligent agent; and using the second trajectories to supervise and fine-tune a second large language model to obtain a second intelligent agent with a self-reflective ability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus and storage medium for a self-reflective intelligent agent. Background Technology

[0002] With advancements in Large Language Models (LLMs), autonomous agents capable of perceiving their environment and taking actions to achieve goals are becoming increasingly feasible. However, powerful agents currently rely heavily on complex cueing engineering and closed-source LLMs like GPT-4. While training open-source LLMs using expert trajectories has improved agent capabilities to some extent, this approach still faces limitations such as performance stagnation and error propagation.

[0003] The comprehensive capabilities of large language models (LLMs) have been significantly improved, enabling them to perform agent tasks that require interaction with diverse environments for reasoning, planning, and execution. Recently, much research has leveraged GPT-4 or other powerful closed-source LLMs to build agents capable of performing complex tasks, such as manipulating computer systems, performing embodied intelligence tasks, and even solving problems in real-world GitHub repositories. These tasks are typically achieved through sophisticated hint engineering and closed-source LLMs like GPT-4. However, smaller LLMs generally perform worse on agent tasks compared to closed-source models.

[0004] Previous research on building agents based on small LLMs has often relied on supervised fine-tuning (SFT). Due to the inherent lack of instruction following and reasoning capabilities in these small LLMs, these studies typically introduce correct problem-solving instances (i.e., successful expert trajectories, or golden trajectories, for successfully completing tasks) as multi-turn dialogue training data to fine-tune the small LLMs, thereby building LLM-based agents. However, there are several challenges to using only successful expert trajectories to train small LLMs and thus build LLM agents:

[0005] First, as the amount of training data on successful expert trajectories increases, the performance improvement of LLM agents tends to slow down, eventually reaching a plateau. This indicates that such methods have limitations in enhancing generalization capabilities.

[0006] Secondly, LLM agents are prone to errors during task execution. Once an error occurs, incorrect actions may further propagate the error, potentially trapping the LLM agent in an error loop. Furthermore, how to build an LLM agent with the self-reflective capabilities of proprietary models such as GPT-4 from scratch has not yet been explored.

[0007] To address the limitations of existing technologies in enhancing generalization ability by training LLM agents using successful expert trajectories, and the technical problem that trained LLM agents cannot escape from error loops during task execution, no effective solution has yet been proposed. Summary of the Invention

[0008] The embodiments of this disclosure provide a training method, apparatus, and storage medium for a self-reflective agent, which at least solves the technical problems in the prior art where training an LLM agent using successful expert trajectories has limitations in enhancing generalization ability, and where the trained LLM agent cannot escape from error loops during task execution.

[0009] According to one aspect of the present disclosure, a method for training a self-reflective intelligent agent is provided, comprising: supervising and fine-tuning a first large language model using multiple first trajectories recorded in natural language text to obtain a first intelligent agent based on the large language model, wherein the multiple first trajectories are expert trajectories that have successfully completed corresponding first tasks and include each step of completing the corresponding first task; constructing multiple second trajectories recorded in natural language text and successfully completing corresponding second tasks using the first intelligent agent and a teacher model, wherein the second trajectories include each step of completing the corresponding second task and a self-reflection step of reflecting on the errors of the first intelligent agent; and supervising and fine-tuning a second large language model using the second trajectories to obtain a second intelligent agent with self-reflection capabilities.

[0010] According to another aspect of the embodiments of this disclosure, a training method for a self-reflective agent is also provided. The method includes: constructing multiple self-reflective trajectories recorded in the form of natural language text and successfully completing corresponding tasks; the self-reflective estimation includes each step of successfully completing the corresponding task and the self-reflective steps of reflecting on errors; and using the self-reflective trajectories to perform supervised fine-tuning of a large language model to obtain an agent with self-reflective capabilities.

[0011] According to another aspect of the present disclosure, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0012] According to another aspect of the present disclosure, a training apparatus for a self-reflective intelligent agent is also provided, comprising: a first intelligent agent training module, configured to supervise and fine-tune a first large language model using multiple first trajectories recorded in natural language text form to obtain a first intelligent agent based on the large language model, wherein the multiple first trajectories are expert trajectories that have successfully completed corresponding first tasks and include various steps for completing the corresponding first tasks; a self-reflective trajectory construction module, configured to construct multiple second trajectories recorded in natural language text form and successfully completing corresponding second tasks using the first intelligent agent and a teacher model, wherein the second trajectories include various steps for completing the corresponding second tasks and self-reflection steps for reflecting on the errors of the first intelligent agent; and a second intelligent agent training module, configured to supervise and fine-tune a second large language model using the second trajectories to obtain a second intelligent agent with self-reflection capabilities.

[0013] According to another aspect of the present disclosure, a training apparatus for a self-reflective intelligent agent is also provided, comprising: a processor; and a memory connected to the processor, configured to provide the processor with instructions for processing the following steps: supervising and fine-tuning a first large language model using a plurality of first trajectories recorded in natural language text form to obtain a first intelligent agent based on the large language model, wherein the plurality of first trajectories are expert trajectories that have successfully completed corresponding first tasks and include various steps for completing the corresponding first tasks; constructing a plurality of second trajectories recorded in natural language text form and successfully completing corresponding second tasks using the first intelligent agent and a teacher model, wherein the second trajectories include various steps for completing the corresponding second tasks and self-reflection steps for reflecting on errors of the first intelligent agent; and supervising and fine-tuning a second large language model using the second trajectories to obtain a second intelligent agent with self-reflection capabilities.

[0014] In this embodiment, a base LLM agent is first obtained by supervised fine-tuning a small open-source LLM using a set of successful expert trajectories. Then, a set of self-reflective trajectories is constructed using this base LLM agent and a teacher model based on task instructions in another task instruction space (e.g., U2) and the corresponding environment. The self-reflective trajectories in this set include steps of reflecting on errors (i.e., identifying erroneous activity information) and correcting errors (i.e., generating correct corrective activity information) during the completion of the corresponding task. The small open-source LLM is then supervisedly fine-tuned using these self-reflective trajectories to obtain a self-reflective LLM agent based on the small LLM and possessing reflective capabilities. In this way, a self-reflective LLM agent with excellent performance and self-reflective capabilities can be obtained using only a small open-source LLM, without relying on complex prompting engineering or large closed-source models like GPT-4. Furthermore, compared to methods that rely solely on successful expert trajectories to train small-scale LLMs and construct LLM agents, this embodiment utilizes self-reflective trajectories to train small-scale LLMs. This prevents the performance improvement trend of the LLM agent from slowing down even with increased training data, demonstrating strong generalization ability. Moreover, the generated self-reflective LLM agent can effectively prevent error propagation through self-reflection during task execution. This solves the limitations of existing technologies that use successful expert trajectories to train LLM agents in enhancing generalization ability, and the technical problem that trained LLM agents cannot escape error loops during task execution. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation thereof. In the drawings:

[0016] Figure 1 This is a hardware structure block diagram of a computing device for implementing the method described in Embodiment 1 of this disclosure;

[0017] Figure 2 This is a schematic diagram of the interaction process between the self-reflecting intelligent agent and the environment according to Embodiment 1 of this disclosure;

[0018] Figure 3 This is a flowchart illustrating the training method for a self-reflective intelligent agent according to the first aspect of Embodiment 1 of this disclosure;

[0019] Figure 4 This is a schematic diagram of a specific training method for a self-reflective agent according to Embodiment 1 of this disclosure;

[0020] Figure 5This is a schematic diagram of a prompt template for a teacher model according to Embodiment 1 of this disclosure;

[0021] Figure 6 This is a diagram illustrating the prompts from the ALFWorld task used during the experiment;

[0022] Figure 7 This is a diagram illustrating the prompts from the SciWorld task used during the experiment;

[0023] Figure 8 This is a diagram illustrating the prompts from the WebShop used in the laboratory.

[0024] Figure 9 The diagram illustrates a comparison of the prior art and the method described in Embodiment 1 of this disclosure in terms of training data scale and average reward in different tasks.

[0025] Figure 10 The diagram illustrates the number of self-reflective trajectories generated and the average reward for teacher models using different LLMs;

[0026] Figure 11 This is a schematic diagram of a training device for a self-reflective intelligent agent according to Embodiment 2 of this disclosure; and

[0027] Figure 12 This is a schematic diagram of a training device for a self-reflective intelligent agent according to Embodiment 3 of this disclosure. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0031] Activity: In this application, "activity" refers to the activity of an intelligent agent. That is, the activity in which an intelligent agent interacts with the environment while completing a corresponding task. Activity may include only actions, or it may include both actions and thoughts. Furthermore, activity information refers to information corresponding to the activity recorded in natural language text form. Activity information may include only action information, or it may include both thought and action information.

[0032] Example 1

[0033] According to this embodiment, a method embodiment for training an agent using trajectories is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] The method embodiments provided in this example can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Figure 1 A hardware block diagram of a computing device for implementing a training method for a self-reflective agent is shown. Figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), memory for storing data, transmission devices for communication functions, and input / output interfaces. The memory, transmission devices, and input / output interfaces are connected to the processor via a bus. In addition, it may also include a display, keyboard, and cursor control device connected to the input / output interfaces. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, a computing device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element in a computing device. As involved in the embodiments of this disclosure, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0036] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to a training method for a self-reflective intelligent agent in an embodiment of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the training method for a self-reflective intelligent agent of the aforementioned application. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the computing device's communications provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0038] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of the computing device.

[0039] It should be noted here that, in some optional embodiments, the above... Figure 1The computing device shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computing devices.

[0040] The self-reflection described in this application refers to the ability of an intelligent agent to identify its own errors and self-correct during interaction with the environment. The agent interacts with the environment through dialogue; that is, the agent outputs activity information in the form of natural language text, and the environment outputs feedback observation information (i.e., observation) in the same way. This activity information can include thought information and action information. (See reference) Figure 2 As shown, a self-reflective agent, after performing an incorrect action during its interaction with the environment, can reflect on its mistake based on the observational information fed back by the environment and correct it.

[0041] To facilitate understanding of the technical solution of this application, the relevant background knowledge is as follows:

[0042] For an agent task involving interaction with an environment e (through dialogue via outputting natural language text), the agent is required to generate action a based on task instructions u, and continuously advance or adjust these actions based on feedback from the environment e until the task is completed. The interaction process between an LLM-based agent and the user environment e can be represented as a multi-turn dialogue. An agent task can also be formalized as: (U, S, A, O, T, R), where the elements represent the task instruction space (U), state space (S), action space (A), observation space (O), state transition function (T), and reward function (R), respectively.

[0043] For each task instruction u∈U, the agent π θ Action information a1∈A will be generated. Under the influence of a1, the initial task implicit state s0∈S will transform to s1∈S according to T. Simultaneously, the environment e of this task will provide observation information regarding a1, generating observation information o1∈O. Subsequent action information a i (i≥2) will be handled by agent π θ It is generated based on the task instruction u, previous action information a1~ai-1, and previous environmental feedback observation information o1~oi-1, i.e., ai=π θ (u, a1, o1, ..., ai-1, oi-1). The environment e will be determined according to a. iThe resulting implicit state-space transitions provide corresponding observation information. i This process will continue iteratively until task u is completed or the maximum number of steps is reached. The inventors represent the number of actions at this point as the trajectory length n. Wherein, the agent π... θ The trajectory τ for performing this task is represented as:

[0044] τ=(u,a1,o1,...,a i ,o i ,...,o n ,o n (1)

[0045]

[0046] Where u is the task instruction corresponding to the task, and a1~a n For the intelligent agent π θ The generated action information, o1~o n This refers to the observation information generated for the environment e corresponding to the task. In this embodiment, the task instruction u and action information a1 to a2 are included. n And observation information o1~o n All of these can be represented using natural language text. Furthermore, formula (2) represents the intelligent agent π. θ An expression relating the interaction between the task instruction u and the environment e to generate the trajectory τ.

[0047] Once the task is completed, a reward r∈[0,1] will be calculated according to the reward function R. r=1 indicates that the task instruction was successfully completed. In this example, agent π θ The activity only includes action a, and its activity information only includes action information a.

[0048] Furthermore, in this application, the ReAct format can be further employed to enhance the agent's reasoning ability. Before taking action, the agent first generates a thought information t (specifically, for example, a prompt can be used to prompt a large model agent to first generate a thought information t), thus the trajectory τ will be transformed into the following form:

[0049] τ=(u,t1,a1,o1,...,t i a i o i , ..., t n a n o n (3).

[0050] Among them, t1~t n The thought information generated for the agent. In this example, the agent π θThe activity includes both action a and thought t, and therefore the activity information also includes action information a and thought information t.

[0051] Therefore, in this application, the agent's thoughts and actions are collectively referred to as "activities," and the agent's thought information and action information are collectively referred to as "activity information." Thus, in this application, agent activities may include only actions, or they may include both thoughts and actions. For ease of understanding, in this embodiment, agent activities include both thoughts and actions, and agent activity information includes both thought information and action information. However, the case where agent activities only include actions also applies to this application.

[0052] Based on the aforementioned background knowledge, the technical solution of this embodiment is further described below. According to a first aspect of this embodiment, a training method for a self-reflective intelligent agent is provided, the method comprising... Figure 1 The computing device shown is implemented. Figure 3 A flowchart illustrating the method is shown below. (Refer to...) Figure 3 As shown, the method includes:

[0053] S302: Supervised fine-tuning of the first large language model is performed using multiple first trajectories recorded in the form of natural language text to obtain the first intelligent agent based on the large language model. The multiple first trajectories are expert trajectories that successfully complete the corresponding first task and include each step of completing the corresponding first task.

[0054] S304: Construct multiple second trajectories, recorded in natural language text form and successfully completing corresponding second tasks, using a first agent and a teacher model. Each second trajectory includes the steps for completing the corresponding second task and a self-reflection step involving reflection on the errors of the first agent; and

[0055] S306: The second trajectory is used to perform supervised fine-tuning of the second large language model to obtain a second intelligent agent with self-reflection capabilities.

[0056] Specifically, Figure 4 A schematic diagram illustrating a specific implementation of the training method for a self-reflective agent according to this embodiment is shown.

[0057] refer to Figure 4 As shown, according to the technical solution of this embodiment, in the agent initialization phase, in order to obtain a correctly formatted and effective trajectory, the successful expert trajectory (i.e., the golden trajectory) that successfully completes the task is used to supervise the fine-tuning (SFT) of the large language model π. base1 (i.e., the first major language model), creating a basic LLM agent π θ1 (i.e., the first intelligent agent). This basic LLM intelligent agent π θ1This will serve as a robust platform for developing more powerful LLM agents (S202). In this application, an LLM agent refers to an agent based on a large language model (LLM) obtained through supervised fine-tuning of a large language model.

[0058] Specifically, let D represent the trajectory τ1~τ of successful experts who successfully completed the task. m (For example, the trajectory of successful experts τ1~τ1) m This can be referred to as the golden trajectory, which is a set of expert trajectories. The successful expert trajectories τ1~τ2 in the expert trajectory set D are... m This contains the interaction sequence for each task instruction in the task instruction space U that is successfully completed. This sequence describes the interaction process between the agent and the environment of the task instructions in the form of a multi-turn dialogue. Each successful expert trajectory τ1~τ m All are organized in ReAct format, including reasoning and related actions, in the form of the trajectory shown in formula (3).

[0059] A subset of the expert trajectory set D is randomly selected by the computing device, denoted as D1, which corresponds to the task instruction subset U1, including the successful expert trajectory τ. 1 1~τ 1 m1 The set of remaining trajectories is denoted as D2, corresponding to the task instruction subset U2, including the successful expert trajectory τ. 2 1~τ 2 m2 Among them, the successful expert trajectory τ in D1 1 1~τ 1 m1 The trajectory lengths are n 1 1~n 1 m1 The corresponding task instructions are u 1 1~u 1 m1 ∈U1, and their corresponding environments are e 1 1~e 1 m1 The successful expert trajectory τ in D2 2 1~τ 2 m2 The trajectory lengths are n 2 1~n 2 m2 The corresponding task instructions are u 2 1~u 2 m2 ∈U2, and their corresponding environments are e 2 1~e 2 m2Then, the successful expert trajectory τ in D1 is utilized via computing devices. 1 1~τ 1 m1 (i.e., the first trajectory) is used to supervise the fine-tuning of the large language model π. base1 (i.e., the first major language model) yields the basic LLM agent π. θ1 (i.e., the first intelligent agent), enabling it to acquire basic intelligent agent task knowledge and instruction following ability. Regarding the large language model π... base1 The specific methods for monitoring and fine-tuning will be explained in detail later.

[0060] Then, refer to Figure 4 As shown, during the synthesis of self-reflective trajectories, the basic LLM agent π is utilized through a computing device. θ1 and the pre-set teacher model π teacher Based on the task instruction u in the task instruction subset U2 of the remaining trajectory set D2 mentioned above 2 1~u 2 m2 A set of self-reflective trajectories D3 is constructed, with a corresponding set of task instructions U3. The set of self-reflective trajectories D3 includes multiple self-reflective trajectories τ that complete the corresponding tasks. 3 1~τ 3 m3 (i.e., the second trajectory) (S204). Among them, the self-reflective trajectory τ 3 1~τ 3 m3 The trajectory lengths are n 3 1~n 3 m3 The corresponding task instructions are u 3 1~u 3 m3 ∈U3, and their corresponding environments are e 3 1~e 3 m3 .

[0061] Among them, the self-reflection trajectory τ 3 1~τ 3 m3 It can be expressed in the following form (y = 1 ~ m3):

[0062]

[0063] Where, τ 3 y This represents the y-th trajectory in the set of self-reflective trajectories D3. and To provide teachers with thinking information and action information Error-generated thought information and error-action information (i.e., thought information and action information containing error labels) are marked as errors. Among these, error-thought information... and error action information This constitutes an incorrect step. To match the task instructions Corresponding environment The generated information related to erroneous thinking and error action information The corresponding observation information.

[0064] and Information on incorrect thinking generated for teacher models and error action information The corresponding corrective thought information and corrective action information, among which corrective thought information and corrective action information This constitutes a corrective step corresponding to the erroneous step. In order to correct thinking information and corrective action information The corresponding observation information.

[0065] Furthermore, the error step and the correction step together constitute the basic LLM agent π. θ1 The self-reflection step involves reflecting on one's mistakes.

[0066] The specific method for generating self-reflective trajectories will be explained in detail below.

[0067] Then, refer to Figure 4 As shown, in the stage of "supervised fine-tuning to obtain a self-reflective agent", the self-reflective trajectory τ in the self-reflective trajectory set D3 can be utilized by computing devices. 3 1~τ 3 m3 For another large language model π base2 (i.e., the second largest language model) is subjected to supervised fine-tuning to obtain a self-reflective LLM agent π with self-reflective capabilities. θ2 (i.e., the second intelligent agent).

[0068] Among them, the self-reflective LLM agent π θ2 Possesses the ability to self-reflect. (Reference) Figure 2 As shown, the self-reflective LLM agent π θ2 During interaction with the environment, it can reflect on and correct erroneous steps taken during task execution based on environmental feedback. As for supervised fine-tuning of the large language model π... base2 Thus, a self-reflective LLM agent π is obtained. θ2 The specific methods will be explained in detail later.

[0069] As described in the background section, there are several challenges to building an LLM agent by simply training a small LLM using successful expert trajectories: First, the performance improvement of the LLM agent tends to slow down as the amount of training data for successful expert trajectories increases, and may eventually plateau. This indicates that such methods have limitations in enhancing generalization ability. Second, LLM agents are prone to errors during task execution. Once an error occurs, incorrect actions may further propagate the error, potentially trapping the LLM agent in an error loop. Furthermore, how to build an LLM agent with the self-reflective capabilities of proprietary models like GPT-4 from scratch has not yet been explored.

[0070] In view of this, in this embodiment, a basic LLM agent can first be obtained by supervised fine-tuning of a small open-source LLM using a set of successful expert trajectories (i.e., golden trajectories). Then, a set of self-reflective trajectories can be constructed using this basic LLM agent and a teacher model based on task instructions in another task instruction space (e.g., U2) and the corresponding environment. The self-reflective trajectories in this set include steps of reflecting on errors (i.e., identifying erroneous activity information) and correcting errors (i.e., generating correct corrective activity information) during the completion of the corresponding task. Then, the small open-source LLM is supervisedly fine-tuned using these self-reflective trajectories to obtain a self-reflective LLM agent based on the small LLM and possessing reflective capabilities. Thus, in this way, a self-reflective LLM agent with excellent performance and self-reflective capabilities can be obtained using only a small open-source LLM, without relying on complex prompting engineering or large closed-source models like GPT-4. Furthermore, compared to methods that rely solely on successful expert trajectories to train small-scale LLMs and construct LLM agents, this embodiment utilizes self-reflective trajectories to train small-scale LLMs. This prevents the performance improvement trend of the LLM agent from slowing down even with increased training data, demonstrating strong generalization ability. Moreover, the generated self-reflective LLM agent can effectively prevent error propagation through self-reflection during task execution. This solves the limitations of existing technologies that use successful expert trajectories to train LLM agents in enhancing generalization ability, and the technical problem that trained LLM agents cannot escape error loops during task execution.

[0071] Furthermore, the "Appendix" at the end of this embodiment illustrates a schematic trajectory of a self-reflective LLM agent reflecting on and correcting its erroneous steps during the execution of a practical task. The applicant has labeled the self-reflective steps in Chinese.

[0072] Of course, in this embodiment, because the ReAct format is used, each step includes two operations: thought 't' and action 'a'. However, if the ReAct format is not used, each step may only include action operation 'a' and not thought operation 't'. That is, in this application, the agent's activities may only include actions. Therefore, the self-reflection step may only include action information marked as erroneous. and corrective action information a' j+1 Furthermore, the error steps marked as errors may only include the error action information marked as errors, and the correction steps may only include the correction action information, thus together constituting the self-reflection steps.

[0073] Optionally, the operation of constructing multiple second trajectories in the form of natural language text, which are recorded using a first intelligent agent and a teacher model and have successfully completed the corresponding second task, includes: using the first intelligent agent to execute task instructions for multiple third tasks and interacting with the environment of the third task in the form of natural language text; when the first intelligent agent outputs erroneous activities during the interaction, using the teacher model to identify the erroneous activities of the first intelligent agent during the interaction and generate corresponding erroneous activity information, using the teacher model to generate correction activity information corresponding to the erroneous activity information, and merging the erroneous activity information and correction activity information as self-reflection steps into the third trajectory corresponding to the current third task; and selecting the second trajectory that has successfully completed the task and includes self-reflection steps from the third trajectories corresponding to each third task.

[0074] Specifically, in constructing the self-reflective trajectory τ of the self-reflective trajectory set D3 3 1~τ 3 m3 (i.e., the second trajectory) during the process. The computing device can utilize the basic LLM agent π. θ1 (i.e., the first intelligent agent) executes task instructions u in the task instruction subset U2 of the trajectory set D2. 2 1~u 2 m2 (i.e., the task instruction for the third task), and in executing the task instruction u 2 1~u 2 m2 During the process, each corresponds to its environment e 2 1~e 2 m2 Interacting via natural language text. For example, the basic LLM agent π. θ1 Execute task instruction u 2 x (x = 1 ~ m2), thus relating to the corresponding environment e 2 xInteraction is conducted via natural language text. During the interaction, the following interaction information (i.e., trajectory information) is generated:

[0075]

[0076] Among them, t 2 x,i (i = 1 to n) 2 x Based on LLM intelligent agents π θ1 Executing task instruction u 2 x The thought information generated during the process, a 2 x,i Based on LLM intelligent agent π θ1 The generated action information, o 2 x,i For the environment e 2 x Corresponding to the thought information t 2 x,i and action information a 2 x,i The generated observation information o 2 x,i .

[0077] Then, the computing device uses the teacher model π. teacher For basic LLM agents π θ1 The activity is determined in real time whether it is an erroneous activity. For example, in the basic LLM agent π θ1 Executing task instruction u 2 x After outputting the activity information of the i-th step during the process, the teacher model π teacher Real-time determination of τ 2 x Is the activity in the i-th step an erroneous activity?

[0078] 1. If the teacher model π teacher If the activity in step i is determined to be neither an error nor a correct action, then the activity information for that step is retained and not changed. In other words, no corresponding self-reflective step is generated to replace the activity information for that step.

[0079] 2. If the teacher model is π teacher If the activity in step i is determined to be an erroneous activity, then the erroneous activity is identified, and corresponding erroneous activity information is generated. For example, in the teacher model π... teacher τ 2 x The activity in the i-th step is judged as an erroneous activity, and the basic LLM agent π in the i-th step is removed.θ1 Generated activity information t 2 x,i and a 2 x,i Identify it as an error (i.e. add an error identifier) ​​to generate corresponding error activity information. and in, It is about thinking information t 2 x,i Error-generated thought information by adding error markers. Information about actions a 2 x,i Error action information generated by adding error markers.

[0080] Furthermore, the computing device also utilizes the teacher model π. teacher Generation and error activity information and Corresponding corrective action information and in, It is related to incorrect thinking information Corresponding corrective thinking information, It is related to incorrect action information Corresponding corrective action information.

[0081] Then, the computing device based on the error activity information and and information on corrective activities and Generate a self-reflection step. This self-reflection step includes two steps:

[0082] 1) Incorrect steps, including: as well as

[0083] 2) Corrective steps, including:

[0084] Finally, the computing device incorporates the above self-reflection steps into the current task u. 2 x The corresponding trajectory (i.e., the third trajectory), which corresponds to the current task instruction u 2 x The corresponding trajectory (i.e., the third trajectory) after the merging self-reflection step is as follows:

[0085]

[0086] Then, for each task instruction u of x = 1 to m2 2 x Both utilize the basic LLM agent π θ1Execute the task instructions and interact with the corresponding environment. 2 x Interactive learning is then implemented, generating activity and observation information for each step. Simultaneously, the teacher model π is utilized. teacher The activity at each step is evaluated, and for erroneous activities, corresponding self-reflective steps are generated and merged into the corresponding trajectory τ. 2 x Therefore, ultimately, for each task instruction u of x = 1 to m2... 2 x All trajectories τ are generated in the manner described above. 2’ x (i.e., the third trajectory).

[0087] Then the computing device receives the various task instructions u 2 x The corresponding trajectory τ 2’ x (i.e., the third trajectory) selects the basic LLM agent π. θ1 A trajectory of successful task completion, including self-reflection steps, serves as a tool for supervised fine-tuning of the large language model π. base2 Thus, a self-reflective LLM agent π is obtained. θ2 The trajectory of self-reflection τ 3 y (i.e., the second trajectory), where y = 1 to m3.

[0088] Thus, through the above methods, this application enables the transformation of the LLM-based teacher model π teacher The process of learning the capabilities of intelligent agents is more effective and efficient.

[0089] Optionally, the operation of generating self-reflection steps based on error activity information, correction activity information, and corresponding observation information during the interaction includes: taking error activity information as error steps; taking correction activity information as correction steps; and combining error steps and correction steps into self-reflection steps.

[0090] Specifically, as mentioned above, the computing device bases its information on error activity. and and information on corrective activities and Generate a self-reflection step. This self-reflection step includes two steps:

[0091] 1) The steps for identifying erroneous activities (i.e., erroneous steps) include: and as well as

[0092] 2) The steps of the corrective action (i.e., the corrective steps) include: and

[0093] Optionally, when the first agent outputs an erroneous activity during the interaction, the operation of using the teacher model to identify the erroneous activity of the first agent during the interaction and generate corresponding erroneous activity information, and using the teacher model to generate corrective activity information corresponding to the erroneous activity information, includes: during the interaction, using the teacher model to receive current activity information corresponding to the current activity of the first agent and the current observation information of the environment regarding the current activity information, as current interaction information; using the teacher model to obtain historical interaction information and the task instructions of the third task during the interaction; using the teacher model to fill the task instructions of the third task, historical interaction information, and current interaction information into a pre-set prompt template, thereby generating corresponding prompt information; using the teacher model to input the prompt information into a third language model, using the third language model to determine whether the current activity is an erroneous activity, and if the current activity is an erroneous activity, identifying the current activity information as erroneous and generating corresponding erroneous activity information; and using the third language model to generate corrective activity information corresponding to the current activity information.

[0094] In the basic LLM agent π θ1 (i.e., the first intelligent agent) and the environment e 2 x During the interaction process, the teacher model π teacher In the basic LLM agent π θ1 The correctness and rationality of its behavior are assessed in real time during interaction with the environment.

[0095] Specifically, during the interaction process, the teacher model π teacher Receive basic LLM agent π θ1 Current activity information t (i.e., the thought and action of the i-th step) 2 x,i and a 2 x,i and environment e 2 x The generated current observation information corresponding to the current activity. 2 x,i Among them, the current activity information t 2 x,i and a 2 x,i And current observation information o 2 x , as the current interactive information.

[0096] Then, teacher model π teacher Further obtain historical interaction information from the interaction process prior to the current activity (t) 2 x,1 ,a2 x,1 ,o 2 x,1 ,...t 2 x,i-1 ,a 2 x,i-1 ,o 2 x,i-1 ) and task instructions u 2 x .

[0097] Then, teacher model π teacher The current interaction information (t) 2 x,i ,a 2 x,i ,o 2 x,i ), historical interaction information (t) 2 x,1 ,a 2 x,1 ,o 2 x,1 ,...t 2 x,i-1 ,a 2 x,i-1 ,o 2 x,i-1 ) and task instructions u 2 x Enter the information into the pre-set prompt template to generate the corresponding prompt message. The prompt template will be explained in detail below.

[0098] Then, teacher model π teacher Input the prompt message into the pre-set large language model π base3 (i.e., the third largest language model), thereby utilizing the large language model π base3 Determine whether the current activity is an erroneous activity, and if so, transfer the current activity information (t) to the appropriate location. 2 x,i ,a 2 x,i Add error flags to generate corresponding error activity information. And the large language model π base3 It also generates corrective action information corresponding to the current activity.

[0099] Furthermore, if the large language model π base3If the activity in step i is determined to be neither an error nor a correct action, then the activity information for that step is retained and not changed. In other words, no corresponding self-reflective step is generated to replace the activity information for that step.

[0100] Thus, through the above methods, this application enables the transformation of the LLM-based teacher model π teacher The process of learning the capabilities of intelligent agents is more effective and efficient.

[0101] Optionally, the prompt template includes instructions for instructing the third language model to perform the following operations: determine whether the current activity information filled in is an erroneous activity based on the task instructions, historical interaction information, and current activity information filled in the prompt template; and if the current activity information filled in is determined to be an erroneous activity, mark the current activity information filled in as an error and generate corresponding erroneous activity information and corrective activity information.

[0102] Specifically, Figure 5 A schematic diagram of a prompt template for a teacher model according to this application is shown. Thus, the teacher model π teacher You can populate the prompt template with relevant information. The prompt template includes the following sections that need to be filled in: "{description}", "{notes}", "{task}", "{trajectory}", "{cur_step}", and "{cur_feedback}".

[0103] The teacher model can populate the {description} field with a description of the task (if any);

[0104] Fill in the pre-set notes for completing the task into {notes};

[0105] Fill the task instructions into {task}, for example, fill the task instruction u 2 x Fill it into {task} to form {u 2 x};

[0106] Populate the historical interaction information into {trajectory}, for example, by adding (t 2 x,1 ,a 2 x,1 ,o 2 x,1 ,...t 2 x,i-1 ,a 2 x,i-1 ,o 2 x,i-1Fill the {trajectory} with the t 2 x,1 ,a 2 x,1 ,o 2 x,1 ,...t 2 x,i-1 ,a 2 x,i-1 ,o 2 x,i-1};

[0107] Populate the current activity information into {cur_step}, for example, (t 2 x,i ,a 2 x,i ,o 2 x,i Fill the blanks into {cur_step} to form {t} 2 x,i ,a 2 x,i ,o 2 x,i};as well as

[0108] Populate {cur_feedback} with the current observation information, for example, o 2 x,i Fill it into {cur_feedback} to form {o 2 x,i}

[0109] In addition, the prompt template also includes the following information:

[0110] “You need to judge whether the current Action of you is correct based on the task description and the trajectory.

[0111] 1.If it is correct,there is no obvious problem,or does not advancethe problem but is a necessary attempt or retrieval,just output"YES".In fact,most actions fall into this category.Assume the first Search step is correctand does not need correction,just output"YES".

[0112] 2.If it is incorrect or there is room for improvement,give thereasons and thoughts for the error.

[0113] *Then,based on the error that has already occurred,provide the Actionthat may correct the error.

[0114] you can only take one Action.*

[0115] You must output the content in the following format:

[0116] ERROR:[Write in here:contenErrort and error reason.]

[0117] THOUGHT:[Write in here:What should you to do next?What should you payattention to in the future?]

[0118] ACTION:[Write in here:The correct action that you should take to correct the error in*The current step*.Action can only be selected from theallowed actions.Output the answer directly without any additional explanatoryinformation.]”

[0119] The corresponding Chinese translation is as follows:

[0120] You need to determine whether your current activity is correct based on the task description and trajectory.

[0121] 1. If correct, with no obvious problems, or if it's a necessary attempt or retrieval that doesn't advance the problem, simply output "YES". In fact, most activities fall into this category. Assuming the first step of the search is correct and requires no correction, simply output "YES".

[0122] 2. If this is incorrect or has room for improvement, please explain the reasoning and thought process behind the error.

[0123] Then, based on the error that has occurred, provide activities that may correct the error.

[0124] You can only take one step in the activity.

[0125] You must output the content in the following format:

[0126] ERROR: [Write the error details and reason here.]

[0127] THOUGHT: [Write here: What should you do next? What should you pay attention to in the future?]

[0128] ACTION: [Write here: The correct action you should take to correct the error in the current step. Only select actions from the allowed options. Output the answer directly without any additional explanation.]

[0129] Thus, the filled-in prompts are input into the large language model π. base3 After that, the large model π base3 Based on the task instructions, historical interaction information, and current interaction information of the i-th step filled in the prompt template, determine whether the activity of the i-th step is an error.

[0130] If the large model πbase3 If the current activity is determined to be an erroneous activity, then the current activity information (t) will be stored. 2 x,i ,a 2 x,i Add error flags to generate corresponding error activity information. And the large language model π base3 It also generates corrective action information corresponding to the current activity.

[0131] Furthermore, if the large language model π base3 If the activity in step i is determined to be neither an error nor a correct action, then the activity information for that step is retained and not changed. In other words, no corresponding self-reflective step is generated to replace the activity information for that step.

[0132] Thus, through the above methods, this application enables the transformation of the LLM-based teacher model π teacher The process of learning the capabilities of intelligent agents is more effective and efficient.

[0133] Optionally, the second large language model is supervised and fine-tuned using the second trajectory to obtain the operation of the second intelligent agent with self-reflection capability, including: supervising and fine-tuning the second large language model using the fourth trajectory to obtain the second intelligent agent with self-reflection capability, wherein the second trajectory is all or part of the fourth trajectory.

[0134] Specifically, according to this embodiment, the method for analyzing the large language model π can be determined based on the self-reflective trajectory set D3. base2 The training trajectory set D4 (i.e., the fourth trajectory set) is used for supervised fine-tuning. The training trajectory set D4 corresponds to the task instruction subset U4, including the training trajectory τ. 4 1~τ 4 m4 (i.e., the fourth trajectory). The training trajectory τ in D4... 4 1~τ 4 m4 The trajectory lengths are n 4 1~n 4 m4 The corresponding task instructions are u 4 1~u 4 m4 ∈U4, and their corresponding environments are e 4 1~e 4 m4 .

[0135] Regarding the training trajectory set D4, it can, for example, be entirely composed of the self-reflective trajectory set D3. That is, the self-reflective trajectory τ... 31~τ 3 m3 This constitutes the training trajectory τ 4 1~τ 4 m4 All of them. Where m4 = m3

[0136] However, in this embodiment, the self-reflection trajectory set D3 and the successful expert trajectory set D1 can also be combined to form the training trajectory set D4. That is, the training trajectory τ 4 1~τ 4 m4 Part of it is the self-reflective trajectory τ 3 1~τ 3 m3 The other part is the trajectory of successful experts. 1 1~τ 1 m1 Where m4 = m1 + m3.

[0137] Due to the tendency for catastrophic forgetting during supervised fine-tuning, these algorithms may lose some of their original capabilities after training on downstream tasks. This can be addressed by using an LLM-based agent π... θ1 Multiple self-reflective trajectories τ generated 3 1~τ 3 m3 (i.e., the second trajectory) and π used for supervising fine-tuning of the LLM agent θ1 The trajectory of successful experts τ 1 1~τ 1 m1 (i.e., the first trajectory) are combined to form the training dataset for re-training the large language model π. base2 (i.e., the second largest language model) is subjected to supervised fine-tuning to obtain a self-reflective LLM agent, which can effectively alleviate the above problems.

[0138] Optionally, the error activity information includes error thinking information and error action information, and the corrective activity information includes corrective thinking information and corrective activity information corresponding to the error thinking information and error action information.

[0139] Specifically, as described above, in this embodiment, the agent's activities include thought t and action a, thus erroneous activity information includes erroneous thought information. (or ) and error action information (or Corrective activity information includes corrective thinking information. and corrective action information Therefore, using this type of trajectory to supervise and fine-tune the agent can enhance the agent's reasoning ability.

[0140] Optionally, the operation of supervising fine-tuning the second large language model using the fourth trajectory to obtain a second intelligent agent with self-reflective capabilities includes: constructing a first loss function for supervising fine-tuning the second large language model; and obtaining the second intelligent agent by supervising fine-tuning the second large language model using the fourth trajectory by minimizing the first loss function. The first loss function is:

[0141]

[0142] in,

[0143] Where, π θ2 Represents a self-reflective LLM agent π θ2 (i.e., the second intelligent agent);

[0144] D4 is the set of trajectories used for training;

[0145] D4[logπ θ2 (τ 4 |e 4 ,u 4 )] represents a self-reflective LLM agent π θ2 With each training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 The corresponding losses;

[0146] -E~D4[] represents a self-reflective LLM agent π θ2 With each training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 The expected value of the corresponding loss can be calculated in practice by summing up the individual losses and then taking the average.

[0147] τ 4 Represents any training trajectory τ in the training trajectory set D4. 4 1~τ 4 m4 ;

[0148] e 4 Represents any training trajectory τ 4 1~τ 4 m4 The corresponding environment e 4 1~e 4 m4 ;

[0149] u 4 Represents any training trajectory τ 4 1~τ 4m4 The corresponding task instruction u 4 1~u 4 m4 ;

[0150] n 4 Represents any training trajectory τ 4 1~τ 4 m4 Length n 4 1~n 4 m4 ;

[0151] Represents any training trajectory τ 4 1~τ 4 m4 The thought process information of the i-th step; Represents any training trajectory τ 4 1~τ 4 m4 Action information for the i-th step; Represents any training trajectory τ 4 1~τ 4 m4 The observation information corresponding to the i-th step; h i-1 Represents τ 4 1~τ 4 m4 The historical interaction information prior to the i-th step.

[0152] Optionally, the second large language model is fine-tuned under supervision using the fourth trajectory to obtain the operations of the second intelligent agent with self-reflective capabilities, including:

[0153] Construct a second loss function for supervised fine-tuning of the second largest language model; and

[0154] By minimizing the second loss function and using the fourth trajectory to perform supervised fine-tuning of the second large language model, a second agent is obtained, where the second loss function is:

[0155]

[0156] in,

[0157] Where, π θ2 Represents a self-reflective LLM agent π θ2 (i.e., the second intelligent agent);

[0158] D4 is the set of trajectories used for training;

[0159] D4[logπ θ2 (τ 4 |e4 ,u 4 )] represents a self-reflective LLM agent π θ2 With each training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 The corresponding losses;

[0160] -E~D4[] represents a self-reflective LLM agent π θ2 With each training trajectory τ in the training trajectory set D4 4 1~τ 4 m4 The expected value of the corresponding loss can be calculated in practice by summing up the individual losses and then taking the average.

[0161] τ 4 Represents any training trajectory τ in the training trajectory set D4. 4 1~τ 4 m4 ;

[0162] e 4 Represents any training trajectory τ 4 1~τ 4 m4 The corresponding environment e 4 1~e 4 m4 ;

[0163] u 4 Represents any training trajectory τ 4 1~τ 4 m4 The corresponding task instruction u 4 1~u 4 m4 ;

[0164] n 4 Represents any training trajectory τ 4 1~τ 4 m4 Length n 4 1~n 4 m4 ;

[0165] Represents any training trajectory τ 4 1~τ 4 m4 The thought process information of the i-th step; Represents any training trajectory τ 4 1~τ 4 m4 Action information for the i-th step; Represents any training trajectory τ 4 1~τ 4 m4 The observation information corresponding to the i-th step; h i-1 Represents τ 4 1~τ 4 m4 The historical interaction information of the i-th step; and,

[0166] δ i δ is a partial shielding coefficient; when the i-th step is an erroneous step, δ i =0, and when the i-th step is not an erroneous step, δ i =1.

[0167] Specifically, according to the method of this application, when using the training trajectory τ 4 1~τ 4 m4 The pair of π base2 When performing supervised fine-tuning on the (i.e., the second largest language model), a partial masking coefficient δ can be set in the loss function. i During the monitoring and fine-tuning process, partial shielding is achieved. That is, during the adjustment of π... base2 When performing supervised fine-tuning of (i.e., the second largest language model), the training trajectory τ can be masked using a partial masking coefficient. 4 1~τ 4 m4 The system retrieves erroneous activity information but retains the observation information corresponding to the erroneous activity information. Therefore, the loss due to erroneous activity information is not calculated.

[0168] Due to the training trajectory τ 4 1~τ 4 m4 The trajectory of self-reflection τ 3 1~τ 3 m3 The process includes erroneous steps, and directly applying supervised fine-tuning methods during the agent stage may cause large language models to learn erroneous steps in the self-reflective trajectory, which could impair the agent's ability to take correct actions. Therefore, this application introduces partial masking (PM) into the loss function during supervised fine-tuning to prevent large language models from learning incorrect thoughts and behaviors.

[0169] Optionally, the operation of the first intelligent agent based on the large language model is obtained by supervising the fine-tuning of the first large language model using multiple first trajectories recorded in the form of natural language text, including: constructing a third loss function for supervising the fine-tuning of the first large language model; and obtaining the first intelligent agent by supervising the fine-tuning of the first large language model using the first trajectories by minimizing the third loss function.

[0170] Specifically, the successful expert trajectory τ in D1 can be utilized. 1 1~τ 1 m1 (i.e., the first trajectory), by minimizing the loss function shown below (i.e., the third loss function), the large language model π is... base1 (i.e., the first major language model) is subjected to supervised fine-tuning to obtain the basic LLM agent π. θ1 (i.e., the first intelligent agent):

[0171]

[0172] in,

[0173] Where, π θ1 Represents the first intelligent agent;

[0174] D1 represents the trajectory of successful experts (τ) 1 1~τ 1 m1 The set of training trajectories constituted;

[0175] D1[logπ θ1 (τ 1 |e 1 ,u 1 )] represents the basic LLM agent π θ1 (i.e., the first intelligent agent) and each successful expert trajectory τ in the training trajectory set 1 1~τ 1 m1 The corresponding losses;

[0176] -E~D1[] represents the basic LLM agent π. θ1 Compared with the successful expert trajectories τ in the training trajectory set 1 1~τ 1 m1 The expected value of the corresponding loss can be calculated in practice by summing up the individual losses and then taking the average.

[0177] τ 1 Represents any successful expert trajectory τ in the training trajectory set. 1 1~τ 1 m1 ;e1 Represents the trajectory τ of any successful expert 1 1~τ 1 m1 The corresponding environment e 1 1~e 1 m1 ;u 1 Represents the trajectory τ of any successful expert 1 1~τ 1 m1 The corresponding task instruction u 1 1~u 1 m1 ;n 1 Represents the trajectory τ of any successful expert 1 1~τ 1 m1 Length n 1 1~n 1 m1 ; Represents the trajectory τ of any successful expert 1 1~τ 1 m1 The thought process information of the i-th step; Represents the trajectory τ of any successful expert 1 1~τ 1 m1 Action information for the i-th step; Represents the trajectory τ of any successful expert 1 1~τ 1 m1 The observation information corresponding to the i-th step; h i-1 Represents the trajectory of successful experts τ 1 1~τ 1 m1 The historical interaction information corresponding to the i-th step.

[0178] In addition, optionally, the error activity information includes only error action information, and the correction activity information includes only the correction action information corresponding to the error action information.

[0179] In other words, although the agent's activities in this embodiment include thought t and action a, the application also applies to situations where the agent's activities include only action a. Therefore, the error activity information only includes error action information. (or Corrective action information includes only corrective action information.

[0180] Alternatively, the expert trajectory set D can be obtained by the following method:

[0181] A large language model (e.g., GPT-4) is used to generate ReAct thoughts (i.e., t) for each step in a task that inherently contains expert trajectories, or to generate expert trajectories for tasks lacking such trajectories. Successful expert trajectories with a reward of 1 are then selected from these expert trajectories to form the expert trajectory set D.

[0182] Alternatively, the expert trajectory set D1 is a subset obtained by randomly selecting 50% of the expert trajectory set D.

[0183] Furthermore, according to a second aspect of this embodiment, a method for training a self-reflective agent is provided, the method comprising:

[0184] S402: Construct multiple self-reflective trajectories, recorded in natural language text form, that successfully complete the corresponding task. The self-reflective estimation includes each step of successfully completing the task and the self-reflective steps of reflecting on errors; and

[0185] S404: Supervised fine-tuning of a large language model using self-reflective trajectories to obtain an intelligent agent with self-reflective capabilities.

[0186] Specifically, refer to Figure 4 As shown, the basic LLM agent π can be utilized through computing devices. θ1 and the pre-set teacher model π teacher Based on the task instruction u in the task instruction subset U2 of the remaining trajectory set D2 mentioned above 2 1~u 2 m2 Construct a self-reflective trajectory set D3, whose corresponding task instruction set is U3.

[0187] Then, refer to Figure 4 As shown, the self-reflective trajectory τ in the set of self-reflective trajectories D3 can be utilized by a computing device. 3 1~τ 3 m3 For large language model π base2 Supervised fine-tuning is performed to obtain a self-reflective LLM agent π with self-reflective capabilities. θ2 .

[0188] Among them, the self-reflective LLM agent π θ2 Possesses the ability to self-reflect. (Reference) Figure 2 As shown, the self-reflective LLM agent π θ2 During interaction with the environment, it can reflect on and correct erroneous steps taken during task execution based on environmental feedback. As for supervised fine-tuning of the large language model π... base2 Thus, a self-reflective LLM agent π is obtained. θ2 The specific methods will be explained in detail later.

[0189] In addition, refer to Figure 1 As shown, according to a third aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0190] Therefore, in this embodiment, a basic LLM agent can first be obtained by supervised fine-tuning a small open-source LLM using a set of successful expert trajectories. Then, a set of self-reflective trajectories can be constructed using this basic LLM agent and a teacher model based on task instructions in another task instruction space (e.g., U2) and the corresponding environment. The self-reflective trajectories in this set include steps of reflecting on errors (i.e., identifying erroneous activity information) and correcting errors (i.e., generating correct corrective activity information) during the completion of the corresponding task. Then, the small open-source LLM is supervisedly fine-tuned using these self-reflective trajectories to obtain a self-reflective LLM agent based on the small LLM and possessing reflective capabilities. In this way, a self-reflective LLM agent with excellent performance and self-reflective capabilities can be obtained using only a small open-source LLM, without relying on complex prompting engineering or large closed-source models like GPT-4. Furthermore, compared to methods that rely solely on successful expert trajectories to train small-scale LLMs and construct LLM agents, this embodiment utilizes self-reflective trajectories to train small-scale LLMs. This prevents the performance improvement trend of the LLM agent from slowing down even with increased training data, demonstrating strong generalization ability. Moreover, the generated self-reflective LLM agent can effectively prevent error propagation through self-reflection during task execution. This solves the limitations of existing technologies that use successful expert trajectories to train LLM agents in enhancing generalization ability, and the technical problem that trained LLM agents cannot escape error loops during task execution.

[0191] The following will describe in detail the experiments conducted by the inventors based on the technical solution of this application, which are used to verify the technical effects of the technical solution of this application:

[0192] I. Experiments related to verifying the technical effects of this application:

[0193] The inventors conducted experiments on three different agent tasks, and the results demonstrated the effectiveness of their method on these three tasks. This section first introduces the three agent tasks and their datasets, followed by a description of the training setup, benchmark model, and evaluation method.

[0194] 1. Dataset

[0195] The inventors implemented the method of this application on three intelligent agent tasks: ALFWorld, WebShop, and SciWorld. Figures 6-8 The prompts for the three tasks above are shown. These tasks have been extensively studied and are representative. ALFWorld and SciWorld contain a training set, a validation set (within the training set distribution), and a test set (not within the training set distribution). The inventors refer to the last two sets as the "seen" and "unseen" test sets, respectively. Unlike the first two tasks, WebShop contains only one training set and one seen test set because it involves only one type of task. Specific dataset statistics are shown in Table 1.

[0196] Table 1: Statistical information for each dataset:

[0197] ALFWorld 3119 140 134 SciWorld 1483 194 211 WebShop 1016 200 -

[0198] 2. Experimental Setup

[0199] The inventors' experiments were primarily based on LLaMA2-7B-chat. Using a teacher model for real-time reflection and correction would consume a large number of tokens. Due to budget constraints, GPT-4 was not considered for the experiments. Instead, the inventors employed several robust open-source LLMs as teacher models, including LLaMA3-70B-Instruct, Qwen1.5-110B-Chat, and Qwen2-72B-Instruct. Both the teacher models and the underlying LLM agents were deployed using the vllm framework. The inventors set the temperature to 0 during all inference phases to ensure consistency of results. Even LLM teacher models can make errors due to the difficulties in the reflection and correction process. Therefore, the inventors excluded self-reflective trajectories containing a large number of erroneous steps. Specifically, a maximum of two erroneous steps were allowed in ALFWorld and SciWorld, while only one erroneous step was allowed in Webshop. Ultimately, the inventors chose Qwen1.5-110B-Chat as the teacher model, generating 708 self-reflective trajectories.

[0200] 3. Benchmark

[0201] Besides STeP (Self-Reflected Trajectories and Parital Masking), the inventors evaluated the performance of other LLMs on three agent tasks. 1) Using only successful expert trajectories (i.e., golden trajectories): FireAct and AgentTuning, i.e., supervising fine-tuning of LLMs using only successful expert trajectories. The inventors used only successful expert trajectory D to supervise the fine-tuning of LLaMA2-7B-chat, called "LLaMA2-7B-chat+Golden Trajs". The inventors also tested AgentLM-7B from AgentTuning, which included a general dataset in its training data. Since its training set only covers ALFWorld and WebShop, and the data is inconsistent with the inventors', its results are for reference only. The evaluation metrics for AgentLM were also calculated only based on ALFWorld and WebShop. 2) Basic LLM: The inventors also tested several models that were not specifically trained for these agent tasks, including GPT-4o-0513, GPT-4, GPT-3.5-turbo, LLaMA3-70B-Instruct, Qwen1.5-110B-Chat, Qwen2-72B-Instruct, LLaMA2-7B-Chat, and LLaMA3-8B-Instruct.

[0202] 4. Evaluation

[0203] The performance of an LLM-based agent in completing a given task instruction can be represented by a reward r∈[0,1], where r=1 represents complete success. The reward r is automatically derived from the environment e corresponding to the agent's task. The main evaluation metric is the average reward, which is the mean of all rewards obtained by the agent in a specific agent task.

[0204] The agent task has a maximum step limit. If the agent fails to complete the task instruction within a given number of steps, the reward r = 0. Specifically, the maximum number of steps is 30 for the ALFWorld task, 10 for WebShop, and the step limit for different subtasks in SciWorld varies from 10 to 120. If the number of tokens in the trajectory exceeds the maximum context length of the LLM, r is also set to 0.

[0205] During testing, the inventors set the temperature of all models to 0 and employed a greedy decoding method to obtain deterministic results. The inventors primarily used zero-shot cues to guide the LLM-based agent to complete task instructions step-by-step in a ReAct format. This is because the context window length is limited and the LLM has already been trained on trajectories conforming to this format, enabling it to adhere to task requirements to a certain extent. Furthermore, the inventors conducted one-shot experiments for comparative analysis. However, since the underlying LLM was not specifically trained for agent tasks, zero-shot setups present challenges. Therefore, the inventors employed a one-shot approach when testing these LLMs.

[0206] II. Experimental Results and Analysis

[0207] Based on the experimental setup described above, the inventors tested their trained LLM-based self-reflective LLM agent on three agent tasks and various benchmarks, as shown in Table 2. Subsequently, in this section, the inventors provided a detailed analysis of the results and conducted ablation experiments for further exploration.

[0208] 1. Main Results

[0209] Table 2 shows the main results of the inventors' experiments. The values ​​in the WebShop, ALFWorld, and SciWorld columns represent the average reward of each model on the corresponding task. The "Average" column represents the mean of the values ​​in the first five columns. The last column represents the model's completion rate on these three tasks.

[0210] The results shown in Table 2 demonstrate that, compared to most baselines, the inventors’ self-reflective LLM agent achieved higher average rewards across all three tasks, proving the effectiveness of the method presented in this application.

[0211] Compared to LLM-based agents trained solely on the golden trajectory, agents trained via STeP (using a reflexive trajectory with partial masking) achieved superior results across all test sets. LLaMA2-7B-chat+STeP (i.e., supervising fine-tuning of LLaMA2-7B-chat using a reflexive trajectory with partial masking) achieved an average improvement of 9.2% on the WebShop test set, and on the "unseen" test sets of ALFWorld and SciWorld. Specifically, STeP achieved a 9.4% performance improvement on WebShop, 4.1% and 10.3% improvements on ALFWorld, and 2.2% and 8% improvements on SciWorld, respectively. Furthermore, LLaMA2-7B-chat+STeP achieved these improvements using less training data than the method using only the golden trajectory. Figure 9 This demonstrates the point more clearly.

[0212] As shown in Table 2, LLMs that are not specifically trained for these agent tasks struggle to complete them, especially smaller LLMs. Completing the ALFWorld task instructions is particularly difficult for LLaMA3-8B-Instruct and LLaMA2-7B-chat. By utilizing trajectories for SFT, the inventors achieved at least 50% of the rewards, approaching or exceeding those of larger or closed-source LLMs. Agents trained with STeP easily outperformed their corresponding teacher models in experimental results. Larger LLMs or closed-source models achieved an average reward of approximately 0.6 on Webshop. However, Qwen performed better on ALFWorld, while other models performed well on SciWorld, which the inventors attribute to differences in their training data. The training of these open-source LLMs may have included data from specific agent datasets.

[0213] Table 2: Main Results of the Inventors' Experiments

[0214]

[0215] Furthermore, the inventors tested the completion rate of the LLM-based agent, defined as the proportion of tasks completed before reaching the maximum number of steps and the maximum context window length. The results in the table show that, although adding reflection and error correction may increase the number of steps and tags to some extent, the inventors' method still improves the completion rate.

[0216] 2. Simplify the experiment

[0217] 1) Effectiveness of Reflective Trajectories. The inventors' training set combines the reflective trajectory D3 with a subset of the golden trajectories D1. To evaluate the impact of the reflective trajectories, the inventors trained LLaMA2-7B using only D1, resulting in the model LLaMA2-7B (without reflective trajectories). As shown in Table 3, not using any reflective trajectories leads to a performance degradation. Increasing the number of successful expert trajectories and training with the trajectory set D (LLaMA2-7B + golden trajectories) yields negligible gains: a 12.9% improvement on the "unseen" SciWorld test set, a 1.8% improvement on WebShop, and even a 2.2% decrease on the "unseen" ALFWorld test set. In contrast, as shown in Table 4, the inventors' method, by adding only 708 reflective trajectories to D1, achieved a 22% improvement on SciWorld, an 11.4% improvement on WebShop, and a 7.8% improvement on ALFWorld, with an average growth rate of 13.7%.

[0218] Table 3: The overall performance of the agent decreases when not using self-reflective trajectories and partial masking.

[0219]

[0220] 2) Effectiveness of partial masking. The inventors replaced the loss function shown in Equation (7) with the loss function shown in Equation (8), enabling the model to learn from actions and thoughts that were marked as errors. The results show that partial masking effectively enhances the overall agent capability of the LLM agent (in addition to the existing test set of ALFWorld, the inventors believe this may be because the ALFWorld task has a higher tolerance for erroneous actions, allowing the agent trained without partial masking to explore a wider range of error trajectories).

[0221] 3. Analysis

[0222] Regarding the selection of LLM teacher models, the inventors conducted multiple experiments with open-source LLM models to determine the most suitable teacher model. During the trajectory enhancement phase, the inventors used LLaMA3-70B-Instruct, Qwen1.5-110B-Chat, and Qwen2-72B-Instruct as LLM teacher models, providing real-time reflection and correction for LLaMA2-7B-chat. For each LLM teacher model, the inventors filtered out the corresponding self-reflective trajectories according to the methods described in this embodiment and the experiments. Subsequently, the inventors trained the corresponding self-reflective LLM agents using these data according to the methods described in this embodiment, and ultimately obtained the average reward of these agents across all test sets. Figure 10 As shown, it is clear that Qwen1.5-110B-Chat generated the most self-reflective trajectories and achieved the best test results when used as a teacher model. Although other LLM teachers did not perform as well as Qwen1.5-110B-Chat, they still achieved significant results. LLM-based agents trained using these other LLM teachers outperformed LLaMA2-7B-chat+Golden Trajectory.

[0223] 4. Further experiments

[0224] The inventors conducted further experiments to explore the possibilities, as shown in Tables 4 and 5.

[0225] The inventors trained Mistral-7B-Instruct V0.3 and LLaMA3-8B-Instruct using the self-reflective trajectory D3 and half of all golden trajectories D1, according to their proposed method. Experimental results show that although both models outperform the corresponding SFT results on the SciWorld task, they surpass these results on the WebShop and ALFWorld tasks, demonstrating superior overall performance.

[0226] The inventors combined D3 and D1 to retrain the large language model π. base To mitigate the catastrophic problem of forgetting, the inventors designed an experiment in which they first trained the agent on D1 and then further trained it on D3. The resulting agent was named LLaMA2-7B (STeP). Results showed improved efficiency on the ALFWorld task, while results on other tasks significantly decreased. This demonstrates that combining D3 with D1 is necessary to improve the overall performance of the agent.

[0227] The inventors' main results for their agent are based on zero-shot prompts. Here, the inventors conducted a one-shot experiment to explore the impact of learned context (ICL) examples on their method. Using one-shot examples for reasoning significantly impacted the model agent's capabilities compared to zero-shot prompts. The inventors hypothesize that this is due to limitations of the context window and the potential discrepancies between ICL examples and the target instruction.

[0228] Table 4: Using STep, agents trained based on other LLMs achieved better overall performance.

[0229]

[0230] Table 5: Other simplified experiments. Retraining the LLM by combining the self-reflective trajectory with a portion of the golden trajectory helps mitigate catastrophic forgetting. Performance on a single sample is not ideal due to the limited context window. Training for a single task does not necessarily improve the model's performance on that task.

[0231]

[0232] The inventors also investigated the difference between training on a single task and training on multiple tasks. They trained LLaMA2-7B-Chat separately using golden trajectories from three agent tasks. As the results presented in Tables 4 and 2 show, training on only one task only benefits some agent tasks, while training on multiple tasks yields better results for WebShop and SciWorld.

[0233] Thus, the experiments described above on several representative agent task datasets demonstrate the effectiveness of the proposed method.

[0234] Appendix: Examples of Self-Reflection Trajectories

[0235]

[0236]

[0237]

[0238] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0239] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0240] Example 2

[0241] Figure 11 A training apparatus 1100 for a self-reflective agent according to this embodiment is shown, which corresponds to the method described according to the first aspect of Embodiment 1. Reference Figure 11As shown, the device 1100 includes: a first agent training module 1110, used to supervise and fine-tune a first large language model using multiple first trajectories recorded in natural language text to obtain a first agent based on the large language model, wherein the multiple first trajectories are expert trajectories that have successfully completed the corresponding first task and include each step of completing the corresponding first task; a self-reflective trajectory construction module 1120, used to construct multiple second trajectories recorded in natural language text and successfully completed the corresponding second task using the first agent and a teacher model, wherein the second trajectories include each step of completing the corresponding second task and a self-reflective step of reflecting on the errors of the first agent; and a second agent training module 1130, used to supervise and fine-tune a second large language model using the second trajectories to obtain a second agent with self-reflective capabilities.

[0242] Optionally, the self-reflective trajectory construction module 1120 includes: a third task execution submodule, used to execute multiple third task instructions using the first agent and interact with the environment of the third task in the form of natural language text; a self-reflective step generation submodule, used to identify the first agent's erroneous activities during the interaction process using a teacher model to generate corresponding erroneous activity information, generate corrective activity information corresponding to the erroneous activity information using the teacher model, and merge the erroneous activity information and corrective activity information as self-reflective steps into the third trajectory corresponding to the current third task; and a self-reflective trajectory construction submodule, used to select a second trajectory that has successfully completed the task and includes self-reflective steps from the third trajectories corresponding to each third task.

[0243] Optionally, the self-reflection step generation submodule includes: a current interaction information receiving unit, used to receive, during the interaction process, current activity information corresponding to the current activity of the first agent and current observation information of the environment in response to the current activity information using a teacher model, as current interaction information; a task instruction acquisition unit, used to acquire historical interaction information and task instructions of the third task during the interaction process using a teacher model; a prompt information generation unit, used to fill the task instructions of the third task, historical interaction information and current interaction information into a pre-set prompt template using a teacher model, thereby generating corresponding prompt information; an error activity information generation unit, used to input the prompt information into a third language model using a teacher model, determine whether the current activity is an error activity through the third language model, and if the current activity is an error activity, mark the current activity information as an error and generate corresponding error activity information; and a correction activity information generation unit, used to generate correction activity information corresponding to the current activity information through the third language model.

[0244] Optionally, the prompt template includes instructions for instructing the third language model to perform the following operations: determine whether the current activity information filled in is an erroneous activity based on the task instructions, historical interaction information, and current interaction information filled in the prompt template; and if the current activity information filled in is determined to be an erroneous activity, mark the current activity information filled in as an error and generate corresponding erroneous activity information and corrective activity information.

[0245] Optionally, the second agent training module 1130 includes: a supervised fine-tuning submodule, used to supervise the fine-tuning of the second large language model using the fourth trajectory to obtain a second agent with self-reflection ability, wherein the second trajectory is all or part of the fourth trajectory.

[0246] Optionally, the error activity information includes error thinking information and error action information, and the corrective activity information includes corrective thinking information and corrective activity information corresponding to the error thinking information and error action information.

[0247] Optionally, the supervised fine-tuning submodule includes: a first loss function construction unit for constructing a first loss function for supervised fine-tuning of the second large language model; and a first supervised fine-tuning unit for supervised fine-tuning of the second large language model using a fourth trajectory by minimizing the first loss function to obtain the second agent. The first loss function is:

[0248]

[0249] in,

[0250] Where, π θ2 D4 represents the second intelligent agent; D4 is the training trajectory set consisting of the fourth trajectory; D4[logπ] θ2 (τ 4 |e 4 ,u 4 )] represents the loss of the second agent executing each fourth trajectory in the training trajectory set; -E~D4[] represents the expected value of the loss; τ 4 Represents any fourth trajectory in the training trajectory set; e 4 Indicates the environment corresponding to any fourth trajectory; u 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 Indicates the length of any fourth trajectory; This represents the thought information of the i-th step in any fourth trajectory; This represents the action information of the i-th step of any fourth trajectory; h represents the observation information corresponding to the i-th step of any fourth trajectory. i-1This represents the historical interaction information prior to the i-th step.

[0251] Optionally, the supervised fine-tuning submodule includes: a second loss function construction unit for constructing a second loss function for supervised fine-tuning of the second large language model; and a second supervised fine-tuning unit for supervised fine-tuning of the second large language model using the fourth trajectory by minimizing the second loss function to obtain the second agent. The second loss function is:

[0252]

[0253] in,

[0254] Where, π θ2 D4 represents the second intelligent agent; D4 is the training trajectory set consisting of the fourth trajectory; D4[logπ] θ22 (τ 4 |e 4 ,u 4 )] represents the loss of the second agent executing each fourth trajectory in the training trajectory set; -E~D4[] represents the expected value of the loss; τ 4 Represents any fourth trajectory in the training trajectory set; e 4 Indicates the environment corresponding to any fourth trajectory; u 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 Indicates the length of any fourth trajectory; This represents the thought information of the i-th step in any fourth trajectory; This represents the action information of the i-th step of any fourth trajectory; h represents the observation information corresponding to the i-th step of any fourth trajectory. i-1 Represents the historical interaction information prior to the i-th step; and δ i δ is a partial shielding coefficient; when the i-th step is an erroneous step, δ i =0, and when the i-th step is not an erroneous step, δ i =1.

[0255] Optionally, the first agent training module 1110 includes: a third loss function construction submodule, used to construct a third loss function for supervised fine-tuning of the first large language model; and a first supervised fine-tuning submodule, used to supervised fine-tuning of the first large language model using a first trajectory by minimizing the third loss function to obtain the first agent, wherein the third loss function is:

[0256]

[0257] in,

[0258] Where, π θ1 D1 represents the first intelligent agent; D1 is the training trajectory set composed of the first trajectory; D1[logπ] θ1 (τ 1 |e 1 ,u 1 )] represents the loss of the first agent executing each first trajectory in the training trajectory set; -E~D1[] represents the expected value of the loss; τ 1 e represents any first trajectory in the set of training trajectories; 1 Represents the environment corresponding to any first trajectory; u 1 Indicates the task instruction corresponding to any first trajectory; n 1 Indicates the length of any first trajectory; Represents the thought information of the i-th step of any first trajectory; This represents the action information of the i-th step of any first trajectory; h represents the observation information corresponding to the i-th step of any first trajectory; i-1 This represents the historical interaction information prior to the i-th step.

[0259] Therefore, according to this embodiment, a basic LLM agent can first be obtained by supervised fine-tuning a small open-source LLM using a set of successful expert trajectories. Then, a set of self-reflective trajectories can be constructed using this basic LLM agent and a teacher model based on task instructions in another task instruction space (e.g., U2) and the corresponding environment. The self-reflective trajectories in this set include steps of reflecting on errors (i.e., identifying erroneous activity information) and correcting errors (i.e., generating correct corrective activity information) during the completion of the corresponding task. Then, the small open-source LLM is supervisedly fine-tuned using these self-reflective trajectories to obtain a self-reflective LLM agent based on the small LLM and possessing reflective capabilities. In this way, a self-reflective LLM agent with excellent performance and self-reflective capabilities can be obtained using only a small open-source LLM, without relying on complex prompting engineering or large closed-source models like GPT-4. Furthermore, compared to methods that rely solely on successful expert trajectories to train small-scale LLMs and construct LLM agents, this embodiment utilizes self-reflective trajectories to train small-scale LLMs. This prevents the performance improvement trend of the LLM agent from slowing down even with increased training data, demonstrating strong generalization ability. Moreover, the generated self-reflective LLM agent can effectively prevent error propagation through self-reflection during task execution. This solves the limitations of existing technologies that use successful expert trajectories to train LLM agents in enhancing generalization ability, and the technical problem that trained LLM agents cannot escape error loops during task execution.

[0260] Example 3

[0261] Figure 12 A training apparatus 1200 for a self-reflective agent according to this embodiment is shown, which corresponds to the method described according to the first aspect of Embodiment 1. Reference Figure 12 As shown, the device 1200 includes: a processor 1210; and a memory 1220 connected to the processor 1210, for providing the processor 1210 with instructions to perform the following processing steps: supervising and fine-tuning a first large language model using multiple first trajectories recorded in natural language text form to obtain a first agent based on the large language model, wherein the multiple first trajectories are expert trajectories that have successfully completed corresponding first tasks and include each step of completing the corresponding first task; constructing multiple second trajectories recorded in natural language text form and successfully completing corresponding second tasks using the first agent and a teacher model, wherein the second trajectories include each step of completing the corresponding second task and a self-reflection step of reflecting on the errors of the first agent; and supervising and fine-tuning a second large language model using the second trajectories to obtain a second agent with self-reflection capabilities.

[0262] Optionally, the operation of constructing multiple second trajectories in the form of natural language text, which are recorded using a first intelligent agent and a teacher model and have successfully completed the corresponding second task, includes: using the first intelligent agent to execute task instructions for multiple third tasks and interacting with the environment of the third task in the form of natural language text; when the first intelligent agent outputs erroneous activities during the interaction, using the teacher model to identify the erroneous activities of the first intelligent agent during the interaction and generate corresponding erroneous activity information, using the teacher model to generate correction activity information corresponding to the erroneous activity information, and merging the erroneous activity information and correction activity information as self-reflection steps into the third trajectory corresponding to the current third task; and selecting the second trajectory that has successfully completed the task and includes self-reflection steps from the third trajectories corresponding to each third task.

[0263] Optionally, when the first agent outputs an erroneous activity during the interaction, the operation of using the teacher model to identify the erroneous activity of the first agent during the interaction and generate corresponding erroneous activity information, and using the teacher model to generate corrective activity information corresponding to the erroneous activity information, includes: during the interaction, using the teacher model to receive current activity information corresponding to the current activity of the first agent and the current observation information of the environment regarding the current activity information, as current interaction information; using the teacher model to obtain historical interaction information and the task instructions of the third task during the interaction; using the teacher model to fill the task instructions of the third task, historical interaction information, and current interaction information into a pre-set prompt template, thereby generating corresponding prompt information; using the teacher model to input the prompt information into a third language model, using the third language model to determine whether the current activity is an erroneous activity, and if the current activity is an erroneous activity, identifying the current activity information as erroneous and generating corresponding erroneous activity information; and using the third language model to generate corrective activity information corresponding to the current activity information.

[0264] Optionally, the prompt template includes instructions for instructing the third language model to perform the following operations: determine whether the current activity information filled in is an erroneous activity based on the task instructions, historical interaction information, and current interaction information filled in the prompt template; and if the current activity information filled in is determined to be an erroneous activity, mark the current activity information filled in as an error and generate corresponding erroneous activity information and corrective activity information.

[0265] Optionally, the second large language model is supervised and fine-tuned using the second trajectory to obtain the operation of the second intelligent agent with self-reflection capability, including: supervising and fine-tuning the second large language model using the fourth trajectory to obtain the second intelligent agent with self-reflection capability, wherein the second trajectory is all or part of the fourth trajectory.

[0266] Optionally, the error activity information includes error thinking information and error action information, and the corrective activity information includes corrective thinking information and corrective activity information corresponding to the error thinking information and error action information.

[0267] Optionally, the operation of supervising fine-tuning the second large language model using the fourth trajectory to obtain a second intelligent agent with self-reflective capabilities includes: constructing a first loss function for supervising fine-tuning the second large language model; and obtaining the second intelligent agent by supervising fine-tuning the second large language model using the fourth trajectory by minimizing the first loss function. The first loss function is:

[0268]

[0269] in,

[0270] Where, π θ2 D4 represents the second intelligent agent; D4 is the training trajectory set consisting of the fourth trajectory; D4[logπ] θ2 (τ 4 |e 4 ,u 4 )] represents the loss of the second agent executing each fourth trajectory in the training trajectory set; -E~D4[] represents the expected value of the loss; τ 4 Represents any fourth trajectory in the training trajectory set; e 4 Indicates the environment corresponding to any fourth trajectory; u 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 Indicates the length of any fourth trajectory; This represents the thought information of the i-th step in any fourth trajectory; This represents the action information of the i-th step of any fourth trajectory; h represents the observation information corresponding to the i-th step of any fourth trajectory. i-1 This represents the historical interaction information prior to the i-th step.

[0271] Optionally, the operation of supervising fine-tuning the second large language model using the fourth trajectory to obtain a second agent with self-reflective capabilities includes: constructing a second loss function for supervising fine-tuning the second large language model; and obtaining the second agent by supervising fine-tuning the second large language model using the fourth trajectory by minimizing the second loss function. The second loss function is:

[0272]

[0273] in,

[0274] Where, πθ2 D4 represents the second intelligent agent; D4 is the training trajectory set consisting of the fourth trajectory; D4[logπ] θ2 (τ 4 |e 4 ,u 4 )] represents the loss of the second agent executing each fourth trajectory in the training trajectory set; -E~D4[] represents the expected value of the loss; τ 4 Represents any fourth trajectory in the training trajectory set; e 4 Indicates the environment corresponding to any fourth trajectory; u 4 Indicates the task instruction corresponding to any fourth trajectory; n 4 Indicates the length of any fourth trajectory; This represents the thought information of the i-th step in any fourth trajectory; This represents the action information of the i-th step of any fourth trajectory; h represents the observation information corresponding to the i-th step of any fourth trajectory. i-1 Represents the historical interaction information prior to the i-th step; and δ i δ is a partial shielding coefficient; when the i-th step is an erroneous step, δ i =0, and when the i-th step is not an erroneous step, δ i =1.

[0275] Optionally, the operation of the first intelligent agent based on the large language model is obtained by supervising the fine-tuning of the first large language model using multiple first trajectories recorded in natural language text. This includes: constructing a third loss function for supervising the fine-tuning of the first large language model; and obtaining the first intelligent agent by supervising the fine-tuning of the first large language model using the first trajectories and minimizing the third loss function. The third loss function is:

[0276]

[0277] in,

[0278] Where, π θ1 D1 represents the first intelligent agent; D1 is the training trajectory set composed of the first trajectory; D1[logπ] θ1 (τ 1 |e 1 ,u 1 )] represents the loss of the first agent executing each first trajectory in the training trajectory set; -E~D1[] represents the expected value of the loss; τ 1 e represents any first trajectory in the set of training trajectories; 1 Represents the environment corresponding to any first trajectory; u 1Indicates the task instruction corresponding to any first trajectory; n 1 Indicates the length of any first trajectory; Represents the thought information of the i-th step of any first trajectory; This represents the action information of the i-th step of any first trajectory; h represents the observation information corresponding to the i-th step of any first trajectory; i-1 This represents the historical interaction information prior to the i-th step.

[0279] Therefore, according to this embodiment, a basic LLM agent can first be obtained by supervised fine-tuning a small open-source LLM using a set of successful expert trajectories. Then, a set of self-reflective trajectories can be constructed using this basic LLM agent and a teacher model based on task instructions in another task instruction space (e.g., U2) and the corresponding environment. The self-reflective trajectories in this set include steps of reflecting on errors (i.e., identifying erroneous activity information) and correcting errors (i.e., generating correct corrective activity information) during the completion of the corresponding task. Then, the small open-source LLM is supervisedly fine-tuned using these self-reflective trajectories to obtain a self-reflective LLM agent based on the small LLM and possessing reflective capabilities. In this way, a self-reflective LLM agent with excellent performance and self-reflective capabilities can be obtained using only a small open-source LLM, without relying on complex prompting engineering or large closed-source models like GPT-4. Furthermore, compared to methods that rely solely on successful expert trajectories to train small-scale LLMs and construct LLM agents, this embodiment utilizes self-reflective trajectories to train small-scale LLMs. This prevents the performance improvement trend of the LLM agent from slowing down even with increased training data, demonstrating strong generalization ability. Moreover, the generated self-reflective LLM agent can effectively prevent error propagation through self-reflection during task execution. This solves the limitations of existing technologies that use successful expert trajectories to train LLM agents in enhancing generalization ability, and the technical problem that trained LLM agents cannot escape error loops during task execution.

[0280] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0281] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0282] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0283] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0284] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0285] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0286] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A training method for a self-reflective intelligent agent, characterized in that, include: The first large language model is supervised and fine-tuned using multiple first trajectories recorded in natural language text form to obtain the first intelligent agent based on the large language model. The multiple first trajectories are expert trajectories that successfully complete the corresponding first task and include each step of completing the corresponding first task. Using the first intelligent agent and the teacher model, multiple second trajectories are constructed in the form of natural language text, which record and successfully complete the corresponding second task. The second trajectory includes each step of completing the corresponding second task and a self-reflection step of reflecting on the errors of the first intelligent agent. as well as The second trajectory is used to supervise and fine-tune the second large language model to obtain a second agent with self-reflection capabilities. The operation of constructing multiple second trajectories, recorded in natural language text and successfully completing corresponding second tasks, using the first agent and the teacher model includes: The first intelligent agent executes multiple third task instructions and interacts with the environment of the third task in the form of natural language text. When the first agent outputs an erroneous activity during the interaction, the teacher model is used to identify the erroneous activity of the first agent during the interaction and generate corresponding erroneous activity information. The teacher model is used to generate corrective activity information corresponding to the erroneous activity information, and the erroneous activity information and the corrective activity information are merged into the third trajectory corresponding to the current third task as the self-reflection step. The second trajectory that successfully completed the task and includes the self-reflection step is selected from the third trajectories corresponding to each third task.

2. The method according to claim 1, characterized in that, When the first agent outputs an erroneous activity during the interaction, the operation of using the teacher model to identify the erroneous activity of the first agent during the interaction and generate corresponding erroneous activity information, and using the teacher model to generate correction activity information corresponding to the erroneous activity information, includes: During the interaction, the teacher model receives current activity information corresponding to the current activity of the first agent and current observation information of the environment in response to the current activity information, which serves as the current interaction information. The teacher model is used to obtain historical interaction information during the interaction process and the task instructions for the third task. The teacher model is used to fill the task instructions of the third task, the historical interaction information, and the current interaction information into a pre-set prompt template, thereby generating corresponding prompt information; The teacher model is used to input the prompt information into a third language model. The third language model determines whether the current activity is an erroneous activity. If the current activity is an erroneous activity, the current activity information is marked as erroneous, and corresponding erroneous activity information is generated. Correction activity information corresponding to the current activity information is generated through the third language model, and wherein, The prompt template includes instructions for directing the third language model to perform the following operations: Based on the task instructions, historical interaction information, and current interaction information filled into the prompt template, determine whether the filled current activity information is an erroneous activity; and If the entered current activity information is determined to be an erroneous activity, the entered current activity information is marked as erroneous, and corresponding erroneous activity information and corrective activity information are generated. Furthermore, the second trajectory is used to supervise and fine-tune the second large language model to obtain the operations of a second intelligent agent with self-reflective capabilities, including: The second large language model is supervised and fine-tuned using the fourth trajectory to obtain the second intelligent agent with self-reflection ability, wherein the second trajectory is all or part of the fourth trajectory.

3. The method according to claim 2, characterized in that, The erroneous activity information includes erroneous thinking information and erroneous action information, and the corrective activity information includes corrective thinking information and corrective activity information corresponding to the erroneous thinking information and the erroneous action information.

4. The method according to claim 3, characterized in that, The second large language model is supervised and fine-tuned using the fourth trajectory to obtain the operations of the second intelligent agent with self-reflection capabilities, including: Construct a first loss function for supervised fine-tuning of the second largest language model; and By minimizing the first loss function, the second large language model is supervised and fine-tuned using the fourth trajectory to obtain the second agent, wherein the first loss function is: ， Among them, h i-1 =( , , , ..., , , ), in, D4 represents the second intelligent agent; D4 is the training trajectory set composed of the fourth trajectory; D4[ ] represents the loss of the second agent executing each of the fourth trajectories in the training trajectory set; -E~D4[ ] represents the expected value of the loss; Represents any fourth trajectory in the set of training trajectories; This represents the environment corresponding to any of the fourth trajectories; This represents the task instruction corresponding to any of the fourth trajectories; Indicates the length of any of the fourth trajectories; This represents the thought information of the i-th step of any of the fourth trajectories; This represents the action information of the i-th step of any of the fourth trajectories; h represents the observation information corresponding to the i-th step of any of the fourth trajectories. i-1 This represents the historical interaction information prior to the i-th step.

5. The method according to claim 3, characterized in that, The second large language model is supervised and fine-tuned using the fourth trajectory to obtain the operations of the second intelligent agent with self-reflection capabilities, including: Construct a second loss function for supervised fine-tuning of the second largest language model; and By minimizing the second loss function and using the fourth trajectory to perform supervised fine-tuning of the second large language model, the second agent is obtained, wherein the second loss function is: ， Among them, h i-1 =( , , , ..., , , ), in, D4 represents the second intelligent agent; D4 is the training trajectory set composed of the fourth trajectory; D4[ ] represents the loss of the second agent executing each of the fourth trajectories in the training trajectory set; -E~D4[ ] represents the expected value of the loss; Represents any fourth trajectory in the set of training trajectories; This represents the environment corresponding to any of the fourth trajectories; This represents the task instruction corresponding to any of the fourth trajectories; Indicates the length of any of the fourth trajectories; This represents the thought information of the i-th step of any of the fourth trajectories; This represents the action information of the i-th step of any of the fourth trajectories; h represents the observation information corresponding to the i-th step of any of the fourth trajectories. i-1 Represents the historical interaction information prior to the i-th step; and δ i δ is a partial shielding coefficient; when the i-th step is an erroneous step, δ i =0, and when the i-th step is not an erroneous step, δ i =1.

6. The method according to claim 3, characterized in that, Supervised fine-tuning of the first large language model is performed using multiple first trajectories recorded in natural language text form to obtain the operations of the first intelligent agent based on the large language model, including: Construct a third loss function for supervised fine-tuning of the first large language model; and By minimizing the third loss function, the first large language model is subjected to supervised fine-tuning using the first trajectory to obtain the first agent, wherein the third loss function is: ， Among them, h i-1 =( , , , ..., , , ), in, D1 represents the first intelligent agent; D1 is the training trajectory set composed of the first trajectory; D1[ ] represents the loss of the first agent executing each first trajectory in the training trajectory set; -E~D1[ ] represents the expected value of the loss; Represents any first trajectory in the set of training trajectories; This represents the environment corresponding to any of the first trajectories; This represents the task instruction corresponding to any of the first trajectories; Indicates the length of any of the first trajectories; The thought information represents the i-th step of any of the first trajectories; This represents the action information of the i-th step of any of the first trajectories; h represents the observation information corresponding to the i-th step of any of the first trajectories. i-1 This represents the historical interaction information prior to the i-th step.

7. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, a processor performs the method according to any one of claims 1 to 6.

8. A training device for a self-reflective intelligent agent, characterized in that, include: The first intelligent agent training module is used to supervise and fine-tune the first large language model using multiple first trajectories recorded in the form of natural language text to obtain the first intelligent agent based on the large language model. The multiple first trajectories are expert trajectories that have successfully completed the corresponding first task and include each step of completing the corresponding first task. The self-reflection trajectory construction module is used to construct multiple second trajectories in the form of natural language text, which are recorded and successfully completed in the form of natural language text, using the first intelligent agent and the teacher model. The second trajectory includes each step of completing the corresponding second task and a self-reflection step of reflecting on the errors of the first intelligent agent. as well as The second agent training module is used to supervise and fine-tune the second large language model using the second trajectory to obtain a second agent with self-reflection capabilities. The module includes operations such as constructing multiple second trajectories, recorded in natural language text and successfully completing corresponding second tasks, using the first agent and the teacher model. The first intelligent agent executes multiple third task instructions and interacts with the environment of the third task in the form of natural language text. When the first agent outputs an erroneous activity during the interaction, the teacher model is used to identify the erroneous activity of the first agent during the interaction and generate corresponding erroneous activity information. The teacher model is used to generate corrective activity information corresponding to the erroneous activity information, and the erroneous activity information and the corrective activity information are merged into the third trajectory corresponding to the current third task as the self-reflection step. The second trajectory that successfully completed the task and includes the self-reflection step is selected from the third trajectories corresponding to each third task.

9. A training device for a self-reflective intelligent agent, characterized in that, include: processor; as well as A memory, connected to the processor, for providing the processor with instructions to perform the following processing steps: The first large language model is supervised and fine-tuned using multiple first trajectories recorded in natural language text form to obtain the first intelligent agent based on the large language model. The multiple first trajectories are expert trajectories that successfully complete the corresponding first task and include each step of completing the corresponding first task. Using the first intelligent agent and the teacher model, multiple second trajectories are constructed in the form of natural language text, which record and successfully complete the corresponding second task. The second trajectory includes each step of completing the corresponding second task and a self-reflection step of reflecting on the errors of the first intelligent agent. as well as The second trajectory is used to supervise and fine-tune the second large language model to obtain a second agent with self-reflection capabilities. The operation of constructing multiple second trajectories, recorded in natural language text and successfully completing corresponding second tasks, using the first agent and the teacher model includes: The first intelligent agent executes multiple third task instructions and interacts with the environment of the third task in the form of natural language text. When the first agent outputs an erroneous activity during the interaction, the teacher model is used to identify the erroneous activity of the first agent during the interaction and generate corresponding erroneous activity information. The teacher model is used to generate corrective activity information corresponding to the erroneous activity information, and the erroneous activity information and the corrective activity information are merged into the third trajectory corresponding to the current third task as the self-reflection step. The second trajectory that successfully completed the task and includes the self-reflection step is selected from the third trajectories corresponding to each third task.

Images