Intelligent agent training method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an intelligent agent training method, device and equipment and a computer readable storage medium. The agent is associated with a large language model, the method comprising: obtaining expert trajectory data about an inference chain, the inference chain comprising a plurality of inference steps; generating agent trajectory data about the inference chain by the agent based on the expert trajectory data; the expert trajectory data and the agent trajectory data are progressively compared in an inference step order to generate an inference step-by-inference step reward, and the agent is trained based on the inference step-by-inference step reward. According to the method for training the intelligent agent, the reinforcement learning process of the intelligent agent can be optimized by rewarding one by one reasoning steps. Specifically, the method for training the agent according to the present disclosure can compare expert actions in an expert trajectory with agent actions in an agent trajectory, thereby automatically generating an intermediate reward for fine-grained optimization.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a method, apparatus, device and storage medium for training an intelligent agent. Background Art

[0002] Recently, the outstanding capabilities of large language models (LLMs) have led researchers to explore adopting these models as core controllers of intelligent agents and building artificial intelligence (AI) agents. In addition, large language models have begun a revolutionary era in general artificial intelligence (AGI) because of their extraordinary ability to handle complex interactive tasks with the environment and tools. These interactive tasks can involve multiple fields, for example, including web browsing, online shopping, household tasks, and complex question answering. The outstanding capabilities of large language models make them an important component in various autonomous agent systems.

[0003] While relying on the inherent knowledge of large language models, for example, these large language models are endowed with extensive knowledge during pre-training on large corpora, they show a tendency to produce hallucinated content when handling the complex interaction tasks mentioned above. To address this issue and further align the output content of large oracle models with human preferences, researchers introduced reinforcement learning (RL) to train agents on large oracle models to enhance their ability to plan and solve complex tasks. While traditional methods do not perform fine-tuning, recent methods have turned to reinforcement learning strategies to further enhance the ability of agents to solve complex interaction tasks with the environment and tools.

[0004] Currently, the development of existing intelligent agent systems can be roughly divided into two main categories: prompt-based methods and fine-tuning-based methods.

[0005] Prompt-based methods focus on carefully designing prompts and directly leverage closed-source large language models for task planning and reasoning, which can include, for example, ChatGPT or GPT-4. Chain of Thoughts (CoT) prompts can be the basis of most prompt-based methods, which introduce intermediate reasoning steps in demonstrations to improve the ability to make complex reasoning.

[0006] Inheriting the spirit of CoT prompts, ReAct designs a think-and-act format prompt to inspire the large language model to generate reasoning trajectories and task-specific actions in an intertwined manner. ToT is further generalized to a tree structure to ensure that various reasoning paths are explored and global decisions are made through lookahead or backtracking when necessary. Driven by human revision behavior, the SELF-REFINE method uses a large language model as a generator, refiner, and feedback provider. In addition, the Reflexion method uses language feedback stored in a memory buffer to strengthen the agent and induce better decisions.

[0007] While prompt-based methods achieve promising results without training, they rely heavily on carefully designed prompts and advanced closed-source models (e.g., ChatGPT and GPT-4), which are expensive to use. To address these challenges, recent studies have constructed expert trajectory data for teacher agents (e.g., GPT-4 or humans) and performed supervised fine-tuning on open-source large language models (e.g., LLaMA and Mistral). Furthermore, NAT and ETO introduced negative samples during the fine-tuning process to reduce model hallucinations and enhance robustness.

[0008] In previous methods for training intelligent agents, the reward function is either manually annotated or limited to the final reward feedback from the environment. However, previous methods for training intelligent agents are limited by problems such as sparse rewards. For example, existing datasets only provide the final scalar reward for each reasoning chain that includes multiple reasoning steps, potentially leading to ineffective and inefficient policy learning.

[0009] Therefore, an effective method for training intelligent agents is needed to improve the generation ability, adaptability and performance of large language models so that they can better meet the needs of specific tasks and scenarios. Summary of the invention

[0010] In order to solve the above problems, the method for training an intelligent agent according to the present disclosure can optimize the reinforcement learning process of the intelligent agent by rewarding each reasoning step. Specifically, the method for training an intelligent agent according to the present disclosure can compare the expert actions in the expert trajectory and the agent actions in the agent trajectory, thereby automatically generating intermediate rewards for fine-grained optimization. In addition, the method for training an intelligent agent according to the present disclosure also provides implicit rewards and reverse reinforcement learning techniques to promote reflection and strategy adjustment of the intelligent agent.

[0011] Embodiments of the present disclosure provide a method, apparatus, device, and computer-readable storage medium for training an agent.

[0012] An embodiment of the present disclosure provides a method for training an agent, wherein the agent is associated with a large language model, the method comprising: obtaining expert trajectory data about an inference chain, wherein the inference chain comprises a plurality of inference steps; generating, by the agent, agent trajectory data about the inference chain based on the expert trajectory data; comparing the expert trajectory data and the agent trajectory data step by step in the order of the inference steps to generate a reward for each inference step, and training the agent based on the reward for each inference step.

[0013] According to the method of an embodiment of the present disclosure, the reasoning step-by-reasoning step reward includes an implicit reinforcement learning reward, which is generated by gradually determining the direct preference optimization loss of the expert trajectory data and the agent trajectory data in the order of reasoning steps.

[0014] According to the method described in an embodiment of the present disclosure, the reasoning step-by-reasoning step reward includes a reverse reinforcement learning reward, which is generated based on the following items: a trained discriminator network gradually compares the occupancy metrics of the expert trajectory data and the occupancy metrics of the agent trajectory data in the order of reasoning steps to generate a reverse reinforcement learning reward.

[0015] According to the method of an embodiment of the present disclosure, training the agent based on the reasoning step-by-step reward includes: training the agent using a nearest policy optimization rule based on the inverse reinforcement learning reward.

[0016] According to the method of an embodiment of the present disclosure, the trained discriminator network is trained based on expert trajectory data and agent trajectory data.

[0017] According to a method of an embodiment of the present disclosure, the expert trajectory data includes the 1st to nth expert observations and the 1st to nth expert actions generated for the 1st to nth expert observations, wherein n is an integer greater than or equal to 1, and wherein generating the agent trajectory data about the reasoning chain includes: the agent takes the 1st expert observation to the i-th expert observation and the 1st expert action to the i-1th expert action as states, and generates the i-th agent action based on the states, wherein i is an integer from 1 to n.

[0018] According to the method of an embodiment of the present disclosure, the expert trajectory data is not manually annotated with rewards step by step in the order of reasoning steps.

[0019] The method according to an embodiment of the present disclosure further includes performing supervised fine-tuning on the agent before training the agent based on the reasoning step-by-step reward.

[0020] According to the method of an embodiment of the present disclosure, the intelligent agent is applied to one or more of the following fields: an online shopping assistant; a web browsing assistant; a scientific experiment assistant; a home task assistant; and a multi-hop question-answering assistant.

[0021] An embodiment of the present disclosure provides a device for training an agent, wherein the agent is associated with a large language model, and the device includes: an expert trajectory data acquisition module, configured to obtain expert trajectory data about an inference chain, wherein the inference chain includes multiple inference steps; an agent trajectory data generation module, configured to operate the agent to generate agent trajectory data about the inference chain based on the expert trajectory data; a reward generation module for each inference step, configured to compare the expert trajectory data and the agent trajectory data step by step in the order of the inference steps to generate a reward for each inference step; and an agent training module, configured to train the agent based on the reward for each inference step.

[0022] An embodiment of the present disclosure provides a device for training an intelligent agent, comprising: one or more processors; and one or more memories, wherein a computer executable program is stored, and when the computer executable program is executed by the processor, the above method is performed.

[0023] An embodiment of the present disclosure provides a computer program product, which is stored on a computer-readable storage medium and includes computer instructions. When the computer instructions are executed by a processor, the computer device executes the above method.

[0024] An embodiment of the present disclosure provides a computer-readable storage medium on which computer-executable instructions are stored. When the instructions are executed by a processor, the above method is implemented.

[0025] The method provided by the embodiments of the present disclosure can integrate supervision of each reasoning step into reinforcement learning to optimize the agent strategy. The method provided by the embodiments of the present disclosure can effectively construct supervision of each reasoning step without additional manual annotation to address the ability gap between large language agents and experts. According to the method provided by the embodiments of the present disclosure, even in the absence of a clear process supervision reward signal, the novice agent strategy can still gradually approach the expert strategy and respond quickly to external stimuli. The method according to the present disclosure makes the training process of the agent more efficient and flexible.

[0026] Through a large number of experiments conducted on multiple interactive task scenarios, the agent training method according to the embodiment of the present disclosure is superior to the current most advanced large language model agent training method. This clearly shows the superiority of applying the reasoning step-by-step reward reinforcement learning to the large language model agent strategy learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0028] Figure 1 This is a diagram of training an agent by maximizing the token-level generation probability of expert actions.

[0029] Figure 2 This is a schematic diagram of training an agent through a sparse reward method;

[0030] Figure 3 is a method for training an intelligent agent according to an embodiment of the present disclosure;

[0031] Figure 4 is a schematic diagram of a method for training an intelligent agent according to an embodiment of the present disclosure;

[0032] Figure 5 is a schematic diagram of a method for training an intelligent agent according to an embodiment of the present disclosure;

[0033] Figure 6 is a schematic diagram showing an apparatus for training an intelligent agent according to an embodiment of the present disclosure;

[0034] Figure 7 A schematic diagram showing a device for training an intelligent agent according to an embodiment of the present disclosure; and

[0035] Figure 8 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0037] In this specification and the drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. Meanwhile, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0038] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0040] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0041] The method of training an intelligent agent disclosed in the present invention may be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. An intelligent agent can be software, hardware or a system, for example, an intelligent agent model, and the intelligent agent can have autonomy, adaptability and interaction capabilities. For example, an intelligent agent can be implemented by a large language model or the intelligent agent can be a large language model itself.

[0042] The method for training an intelligent agent disclosed in the present invention can be based on a reinforcement learning (RL) method. Reinforcement learning, also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of an intelligent agent learning strategies to maximize rewards or achieve specific goals during the interaction with the environment. Reinforcement learning is a learning method in which an intelligent agent learns in a "trial and error" manner, and the behavior is guided by rewards obtained through interaction with the environment. The goal is to enable the intelligent agent to obtain the maximum reward. Reinforcement learning is different from supervised learning in connectionist learning, which is mainly manifested in reinforcement signals. The reinforcement signal provided by the environment in reinforcement learning is an evaluation of the quality of the action (usually a scalar signal), rather than telling the reinforcement learning system RLS (reinforcement learning system) how to generate the correct action. Since the external environment provides little information, RLS must rely on its own experience to learn. In this way, RLS acquires knowledge in an action-evaluation environment and improves action plans to adapt to the environment.

[0043] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence and reinforcement learning. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0044] Figure 1 A schematic diagram of training an agent by maximizing the token-level generation probability of expert actions.

[0045] like Figure 1 As shown in , initial efforts to develop large language model agents focused on maximizing the word-level generation probability of expert actions. Figure 1 The expert trajectory may include expert observations (e.g., Figure 1 ) and expert actions (e.g., Figure 1 The LLM agent can be trained by supervised fine-tuning (SFT) using the expert action at the end of the expert trajectory. Figure 1 The approach shown in , while straightforward and non-rewarding, falls short when faced with a shortage of training data and has difficulty generalizing outside the training data distribution.

[0046] Figure 2 Schematic diagram of training an agent using sparse rewards.

[0047] In realizing that Figure 1Following the limitations of the training methods described above, researchers have turned to using manually annotated preferences or final environment feedback as additional reward signals and performing reinforcement learning (RL) training on top of supervised fine-tuning (SFT) models. Figure 2 The expert trajectory may include expert observations (e.g., Figure 2 ) and expert actions (e.g., Figure 2 ), the agent trajectory can include agent observations (e.g., Figure 2 ) and agent actions (e.g., Figure 2 Rectangular shape in the agent).

[0048] However, if Figure 2 The described approach is limited by the sparsity and delay of reward signals. Figure 2 As shown in , the existing reward signal is represented as a single scalar reward for each generated agent-observation-agent-action trajectory. Such sparse feedback makes it challenging for the model to discern the quality of each action, especially for tasks with long reasoning chains. Therefore, Figure 2 The methods shown have difficulty in accurately refining low-quality actions, which in turn leads to inefficient learning or training of the agent. In addition, delayed reward feedback prevents the agent from making timely corrections, potentially leading to suboptimal responses.

[0049] Figure 3 A method for training an intelligent agent according to an embodiment of the present disclosure. Figure 4 It is a schematic diagram of a method for training an intelligent agent according to an embodiment of the present disclosure.

[0050] like Figure 3 As shown, the method for training an agent according to an embodiment of the present disclosure includes steps S310-S340. The agent can be associated with a large language model.

[0051] exist Figure 3 In S310, expert trajectory data about an inference chain may be obtained, the inference chain including a plurality of inference steps. The expert may be a human expert or other more advanced intelligent agent. For example, the expert may complete a task by executing the inference chain. These tasks may involve multiple fields, for example, the intelligent agent may be used as an online shopping assistant, a web browsing assistant, a scientific experiment assistant, a home task assistant, and a multi-hop question answering assistant.

[0052] For example, an agent that acts as a web shopping assistant can shop online. The user or the environment can give the web shopping assistant instructions on what to do. The web shopping assistant must follow the instructions. Each turn, the user or the environment can give the web shopping assistant a list of observations and available actions, and the web shopping assistant must react based on the state and instructions. If search is available, the web shopping assistant can use the search action. The web shopping assistant can click one of one or more clickable buttons.

[0053] For example, an agent that acts as a web browsing assistant may be adept at website design, navigation, and may perform tasks for the user, thereby making the web (e.g., web pages or websites, etc.) more accessible.

[0054] For example, an agent acting as a scientific experiment assistant can be a scientific experiment assistant doing some scientific experiments in an environment. In the environment, there can be several rooms: kitchen, foundry, workshop, bathroom, outdoor, living room, bedroom, greenhouse, art studio, corridor, etc. The agent acting as a scientific experiment assistant needs to explore the environment and find the items needed to complete the experiment. The agent acting as a scientific experiment assistant can be teleported to any room in one step. All containers in the environment are already opened, and the agent acting as a scientific experiment assistant can directly get items from the containers.

[0055] For example, an agent as a home task assistant may be a home task assistant that interacts with residents to solve tasks. An agent as a home task assistant may be an agent in a home environment. The goal of an agent as a home task assistant may be to complete a task goal by performing actions. At the beginning of the interaction, the agent as a home task assistant may obtain a detailed description of the current environment and the goals to be achieved by the user or the environment. In each round of the home task assistant, the home task assistant will be given the observations of the previous round. The home task assistant should first think about the current situation and plan for the home task assistant's future actions, and then output the actions of the home task assistant in sequence.

[0056] For example, an agent acting as a multi-hop question answering assistant can be an expert at answering multi-hop questions. An agent acting as a multi-hop question answering assistant can answer questions as simply and clearly as possible. Each round, the user or environment can give the agent acting as a multi-hop question answering assistant an observation, and the agent acting as a multi-hop question answering assistant can respond with interleaved thought and action steps.

[0057] A chain of reasoning can include multiple intermediate reasoning steps. Figure 4 As shown, the expert can perform the push chain by executing the reasoning steps one by one to complete the task and generate expert trajectory data. The expert trajectory can include expert observations (e.g., Figure 4 ) and expert actions (e.g., Figure 4 (rectangular shapes in the expert trajectory).

[0058] exist Figure 3 In S320, the agent may generate agent trajectory data about the reasoning chain based on the expert trajectory data. For example, the agent may generate agent trajectory data about the reasoning chain by checking the expert trajectory. Figure 4 As shown, checking can consist of the agent initially observing the expert's behavior and attempting to independently reproduce those behaviors in the same situation, which will be referred to below. Figure 5 The purpose of the agent's inspection operation is to recognize the gap between the agent's ability and the expert, thereby promoting the agent's subsequent policy improvement.

[0059] exist Figure 3 S330 may compare the expert trajectory data and the agent trajectory data step by step in the order of reasoning steps to generate a reward for each reasoning step. For example, progress toward expert-level performance may require the agent to reflect on the agent's interaction trajectory. The purpose of such reflection is to summarize and internalize experience, ultimately leading to the development of personalized behavior patterns and strategies for the agent. Figure 4 As shown, a reward can be generated for each reasoning step of the agent. The specific process of reflection will be referred to below. Figure 5 Detailed description.

[0060] exist Figure 3 S340 can train the agent based on rewards for each reasoning step. Figure 1-Figure 2 As described above, previous methods for training large language model agents focus on observing and imitating complete expert trajectories and optimizing using the final environmental reward feedback. However, due to the complexity of the agent's tasks, the large language model agent needs to constantly interact with the environment and conduct repeated trials to obtain the final reasoning results. The inherent multi-step reasoning characteristics of the agent's tasks bring dual challenges of efficiency and effectiveness to the agent's self-attempt to complete the trajectory. First, simulating the complete expert trajectory and obtaining the final environmental feedback requires the agent to constantly interact with the environment. Such interactions are sequential and cannot be parallelized, resulting in a large consumption of computing time and resources. In addition, the agent needs to understand every action of the expert at the same time, which may lead to information overload. Such information overload makes it complicated for the agent to understand and grasp the details of each behavior of the expert, often resulting in an inefficient learning process. Therefore, in previous methods, the agent may require additional training data or more iterations to fully grasp the observations from expert experience. However, the training method according to an embodiment of the present disclosure can automatically construct intermediate reward feedback for each reasoning step. Further, as Figure 4As shown, the training method according to the embodiment of the present disclosure can train the agent based on the reward of each reasoning step, thereby efficiently optimizing the agent strategy by each reasoning step. Finally, the agent strategy can be aligned with the expert strategy.

[0061] Figure 5 is a schematic diagram of a method for training an agent according to an embodiment of the present disclosure. Figure 5 The expert trajectory may include expert observations (e.g., Figure 5 ) and expert actions (e.g., Figure 5 The agent trajectory can include agent checks and agent actions (e.g., Figure 5 ).

[0062] As mentioned above, large language model agents have shown remarkable capabilities in handling complex interactive tasks by leveraging reinforcement learning strategies to align agent policies with human preferences. However, existing research on large language model agents encounters significant challenges from the sparsity of reward signals and the complexity associated with the reasoning process. To address these limitations, Figure 5 In the method shown, a per-reasoning step reinforcement learning framework is introduced to optimize the agent policy without manually annotating the rewards of the reasoning steps. By constantly monitoring the behavior of the expert and spontaneous practice, the large language model agent can accumulate experience and eventually improve from a novice to an expert level.

[0063] For example, we can first formulate an agent task that includes a reasoning chain. The process of the agent interacting with the environment to solve the task can be formalized as a partially observable Markov decision process (POMDP), where the state set is S and the action set is The observation set is O, and the transfer function is The reward function is Initially, the environment provides general task instructions as system prompts Prompt sys Together with the agent’s initial observation o1∈O as a specific task input, the agent needs to interact with the environment multiple times to complete the task and generate a response.

[0064] Specifically, at time step t, the large language model agent parameterized by θ receives an observation o from the environment t ∈O, and decides according to the strategy π θ (·|s t ) Take Action in, is the current state of the environment. The interaction process is repeated until the task is completed or the maximum number of steps is exceeded. Then the final trajectory (Prompt sys , o1, a1, …, on , a n ) calculates the reward r∈[0,1], where r=1 indicates task success and r=0 indicates failure. In the following description, Prompt is omitted for brevity. sys The whole process θ (a n The conditional probability distribution of |o1) can be expressed by the following equation (1):

[0065]

[0066] Alternatively or additionally, supervised fine-tuning can be performed on the agent before training it based on per-inference step rewards. Supervised fine-tuning requires leveraging relatively small amounts of labeled expert data to better adapt the pre-trained large language model to a specific domain or downstream task, thereby providing a solid foundation for creating a powerful agent.

[0067] Given a set of expert trajectories Expert trajectories in the interaction between experts and the environment The autoregressive loss can be used to fine-tune the initial large language model and obtain the basic agent as follows:

[0068]

[0069] According to equation (1), in, First, the trajectory t e The instruction prompt, action and observation cascade is a word segmentation sequence w=(w1, ..., w l ). Then, the probability in equation (2) It can be expressed as the following equation (3):

[0070]

[0071] Where w<k indicates the participle before the kth participle, and is an indicator function that indicates w k Is it a segmentation of an action generated by the agent? Here, the observation segmentation is masked and only the probability is calculated for the action segmentation.

[0072] like Figure 5 As shown, training an agent can include two main stages: (1) inspection and (2) reflection.

[0073] During the inspection phase, the agent can carefully observe and imitate the expert's actions step by step in the order of reasoning steps. This enables the agent to identify shortcomings in their behavior and promote the mastery of key skills.

[0074] Specifically, an expert trajectory with n reasoning steps can be obtained The expert trajectory may include the 1st to nth expert observations (e.g., ) and the 1st to nth expert actions generated for the 1st to nth expert observations (e.g., ), where n is an integer greater than or equal to 1. The agent can take the 1st expert observation to the i-th expert observation and the 1st expert action to the i-1th expert action as the state, and generate the i-th agent action based on the state, where i is an integer from 1 to n. Specifically, the expert trajectory can be split after each action, and each action can be regarded as a short-term learning goal of the agent:

[0075]

[0076] When the agent establishes a learning goal (e.g., the short-term learning goal mentioned above), the agent triggers the practice phase of the training process of the reasoning steps. Such spontaneous practice is intended to identify the behavioral differences between novice agents and experts, thereby allowing experience to be accumulated through repeated practice and gradually developing the behavior patterns of the novice agent. The core of such spontaneous practice is that the novice agent generates actions based on the previously established learning goals. Specifically, for For each learning goal in is regarded as a prompt, and the agent πθ parameterized by θ generates the appropriate action as in equation (5), and obtains the corresponding agent trajectory

[0077]

[0078] Reference Figure 5 Describe the above inspection process. Figure 5 As shown, expert trajectory data can be obtained, and the expert trajectory can include the following:

[0079] “Context: I want a shelf-stable cracker, spicy beef, and cheese combo. It has to be keto-friendly, gluten-free, and under $40.00.

[0080] Expert: Thinking: I need to find...

[0081] Action: Search for [cookies, spicy beef, and cheese set]

[0082] environment: B0978Q1KK9 , Spicy Beef Wrap Set..., $14.49; B0978P4L31 ,......

[0083] Expert: Thoughts: The Spicy Beef Wrap Set (B0978Q1KK9) seems to match my criteria and is within my budget. Action: Click [B0978Q1KK9]

[0084] Environment: Flavor Name: Original Beef Wrap Kit; Spicy Beef Wrap Kit… Price: $14.49. Level: Not Available. Description. Features. View. Buy Now

[0085] Expert: Now that I've clicked on the Spicy Beef Wrap, I need to confirm that it's spicy beef and cheese flavored, shelf-stable, keto-friendly, and gluten-free as requested. Action: Click on [Spicy Beef Wrap Kit] ......

[0087] Expert: Thinking: …, I will proceed with the purchase. Action: Click [Buy Now]”.

[0088] For example, the LLM agent can observe the expert trajectory during the inspection phase This may include:

[0089] “Context: I want a shelf-stable cracker, spicy beef, and cheese combo. It has to be keto-friendly, gluten-free, and under $40.00.

[0090] Expert: Thinking: I need to find...

[0091] Action: Search for [cookies, spicy beef, and cheese set]".

[0092] The LLM agent can be trained to generate agent actions based on observed states. For example, the LLM agent can perform the following exercises:

[0093] "Agent: Thinking: The Spicy Beef Wrap Set (B0978Q1KK9) seems to match my criteria and is within my budget. Action: Click [B0978Q1KK9]."

[0094] The generated agent actions can be reflected upon, which is further described in the reflection phase below.

[0095] Furthermore, if Figure 5 As shown, for example, the LLM agent can also observe the expert trajectory during the inspection phase This may include:

[0096] “Context: I want a shelf-stable cracker, spicy beef, and cheese combo. It has to be keto-friendly, gluten-free, and under $40.00.

[0097] Expert: Thinking: I need to find...

[0098] Action: Search for [cookies, spicy beef, and cheese set]

[0099] environment: B0978Q1KK9 , Spicy Beef Wrap Set..., $14.49; B0978P4L31 ,......

[0100] Expert: Thoughts: The Spicy Beef Wrap Set (B0978Q1KK9) seems to match my criteria and is within my budget. Action: Click [B0978Q1KK9]

[0101] Environment: Flavor Name: Original Beef Wrap Kit; Spicy Beef Wrap Kit… Price: $14.49. Level: Not Available. Description. Features. View. Buy Now”.

[0102] The LLM agent can practice based on the observed states to generate agent actions. For example, the LLM agent can practice the following:

[0103] “Agent: Thought: This is a spicy beef and cheese kit, and it is shelf-stable. I should verify that it is keto-friendly and gluten-free. But since the price is affordable and it seems to meet my requirements, I will just buy it. Action: Click [Buy Now]”.

[0104] The generated agent actions can be reflected upon, which is further described in the reflection phase below.

[0105] like Figure 5 As shown, in the reflection stage, the novice agent needs to reflect on the interaction trajectory of the agent according to the expert trajectory. The purpose of such reflection is to summarize and internalize experience, which ultimately leads to the development of personalized behavior patterns and strategies of the novice agent. Therefore, in the reflection stage, the present invention designs two different reflection strategies, for example, including implicit reward reinforcement learning and reverse reinforcement learning.

[0106] In one embodiment, the per-reasoning step reward may include an implicit reinforcement learning reward. For example, the implicit reinforcement learning reward may be determined step by step for the expert trajectory data and the agent trajectory data in the order of the reasoning steps. The implicit reinforcement learning reward may be generated by determining the direct preference optimization loss for the expert trajectory data and the agent trajectory data step by step in the order of the reasoning steps.

[0107] Specifically, we can directly compare the expert actions in the expert trajectory data with the agent actions of the agent without introducing explicit reward estimates. Given a trajectory pair (t sample , tθ ),in is the expert trajectory, and is the corresponding agent trajectory. The agent can be trained using direct preference optimization loss. The direct preference optimization loss can be defined as follows:

[0108]

[0109] Among them, π θ is the current agent strategy to be optimized, π e is the expert strategy, π ref is the reference model initialized with the agent’s policy and β is a hyperparameter.

[0110] In another embodiment, the per-reasoning step reward may include an inverse reinforcement learning reward. For example, the occupancy metric of the expert trajectory data and the occupancy metric of the agent trajectory data may be compared step by step by the trained discriminator network in the order of the inference steps to generate an inverse reinforcement learning reward. Training the agent based on the per-reasoning step reward may include training the agent using a proximal policy optimization rule (PPO) based on the generated inverse reinforcement learning reward.

[0111] Considering the lack of reward signals for each reasoning step in existing datasets, the present invention can introduce inverse reinforcement learning (IRL) technology. Inverse reinforcement learning technology can first derive the reward function for each reasoning step based on the expert trajectory and the agent trajectory, and then use the reward function to perform fine-grained optimization on the agent strategy. The occupancy measure ρ of the strategy π can be first defined as π , which indicates the normalized distribution of state-action pairs when the agent adopts policy π to explore the environment:

[0112]

[0113] Among them, 1-γ is the normalization factor, P π (s t =S) represents the probability that the agent is in state S at time t when adopting strategy π.

[0114] In order to accurately imitate the expert policy, we must ensure that the policy distribution generated by the agent is as similar as possible to the policy distribution generated by the expert. This can be achieved by maintaining the agent’s occupancy metric ρ πθ As close as possible to the expert πe The Jensen-Shannon (JS) divergence can be used to measure the distance between two distributions.

[0115]

[0116] Among them, λ is a hyperparameter, is the γ-discounted causal entropy of the agent’s policy.

[0117] Jensen-Shannon divergence JS(ρ πθ , ρ πe ) can be regularized by a convex cost function Represents, up to constant translation and scaling. Convex cost function regularization term Defined as:

[0118]

[0119] The above regularization term The optimal solution is expressed as follows:

[0120]

[0121] Therefore, the optimization problem of equation (8) can be transformed into finding the saddle point (π, D) of the following equation:

[0122]

[0123] The discriminator network can be trained directly using data sampled from expert trajectory data and agent trajectory data The main goal of D is to distinguish between the data distributions generated by the agent policy πθ and the expert policy πe. When D cannot distinguish between the data generated by the agent and the expert, then the agent ρ π The occupancy measure successfully matches the expert ρ πe The discriminator network D can be interpreted as an implicit reward model that provides a learning signal for the agent policy at each reasoning step. The complete learning process at each reasoning step can be found in Algorithm 1 in Table 1 below.

[0124] Table 1

[0125]

[0126] Through the large prediction model learning framework for each reasoning step provided by the embodiments of the present disclosure, it is possible to simulate the learning process from novice to expert by automatically constructing a supervision signal for reinforcement learning for each reasoning step, thereby approaching the expert strategy. The learning process from novice to expert for each reasoning step can be described as two different stages in the context of the agent task, such as a check stage and a reflection stage. Specifically, for the check stage, the goal of the present invention is to identify the strategic differences between the agent and the expert. It can start by observing the behavior pattern of the expert, and then allow the agent to practice independently at each reasoning step. This helps to understand the decision-making process of the expert more deeply and meticulously, and spontaneously provide reward feedback for each reasoning step. Next, the present invention designs a reflection stage to effectively adjust and improve the agent strategy according to the practice results. The present invention designs two agent reflection strategies, for example, including implicit reward reinforcement learning and reverse reinforcement learning. The main contributions of the present invention include but are not limited to the following three points: (1) The present invention proposes a reinforcement learning framework for each reasoning step, which can automatically construct intermediate feedback and efficiently optimize the agent strategy for each reasoning step to ultimately maintain consistency with the expert strategy; (2) The inspection phase and the reflection phase are introduced to construct process supervision training data without manual annotation, which facilitates novice agents to become experts; and (3) The present invention designs two reflection strategies for optimization for each reasoning step, including implicit rewards and inverse reinforcement learning.

[0127] Figure 6 is a schematic diagram showing an apparatus 600 for training an agent according to an embodiment of the present disclosure.

[0128] The agent may be associated with a large language model. According to an embodiment of the present disclosure, the apparatus 600 for training an agent may include an expert trajectory data acquisition module 601 , an agent trajectory data generation module 602 , a reasoning step-by-step reward generation module 603 , and an agent training module 604 .

[0129] The expert trajectory data acquisition module 601 may be configured to obtain expert trajectory data about a reasoning chain, the reasoning chain comprising a plurality of reasoning steps. Optionally, the expert trajectory data acquisition module 601 may perform the following steps: Figure 3 Step S310 described.

[0130] The agent trajectory data generation module 602 may be configured to operate the agent to generate agent trajectory data about the reasoning chain based on the expert trajectory data. Optionally, the agent trajectory data generation module 602 may perform the following steps as described in reference to Figure 3 Step S320 described.

[0131] The reasoning step-by-step reward generation module 603 may be configured to compare the expert trajectory data and the agent trajectory data step by step in the order of the reasoning steps to generate the reasoning step-by-step reward. Figure 3 Step S330 described.

[0132] The agent training module 604 can be configured to train the agent based on the reward for each reasoning step. Optionally, the agent training module 604 can perform the following steps as shown in FIG. Figure 3 Step S340 described.

[0133] According to yet another aspect of the present disclosure, a device for training an intelligent agent is also provided. Figure 7 A schematic diagram of a device 7000 for training an intelligent agent according to an embodiment of the present disclosure is shown.

[0134] like Figure 7 As shown, the device 7000 for training an agent may include one or more processors 7100 and one or more memories 7200. The memory 7200 stores a computer-readable code, and when the computer-readable code is run by the one or more processors 7100, the method for training an agent as described above may be executed.

[0135] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may be an X86 architecture or an ARM architecture.

[0136] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0137] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 8The architecture of the computing device 3000 shown in FIG. Figure 8 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method for training an intelligent agent provided by the present disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 8 One or more components of a computing device are shown.

[0138] According to another aspect of the present disclosure, a computer-readable storage medium is also provided. Computer-readable instructions are stored on the computer storage medium. When the computer-readable instructions are executed by a processor, the method for training a reward model according to an embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronously linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0139] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the method for training an intelligent agent according to the embodiments of the present disclosure.

[0140] Embodiments of the present disclosure provide a method, apparatus, device, and computer-readable storage medium for training an agent.

[0141] Throughout the process of the agent executing the reasoning chain, the reward for each intermediate reasoning step can help identify poorly performing strategies in a timely manner, allowing the agent to quickly improve its capabilities. Directly implementing step-by-step supervision of large oracle model agents may bring a series of challenges. First, current agent interaction datasets with multiple reasoning steps usually lack value assessments of individual reasoning steps, leaving only the final evaluation. Second, even for human annotators, it is a huge challenge to fully understand the contribution of each reasoning step to the final result. In other words, the manual annotation process is expensive and labor-intensive, and the effect is unclear. In addition, large language model agents need to interact with dynamically changing environments, which makes the situation more complicated.

[0142] The method provided by the embodiments of the present disclosure can integrate supervision of each reasoning step into reinforcement learning to optimize the agent strategy. The method provided by the embodiments of the present disclosure can effectively construct supervision of each reasoning step without additional manual annotation to address the ability gap between large language agents and experts. According to the method provided by the embodiments of the present disclosure, even in the absence of a clear process supervision reward signal, the novice agent strategy can still gradually approach the expert strategy and respond quickly to external stimuli. The method according to the present disclosure makes the training process of the agent more efficient and flexible.

[0143] Through a large number of experiments conducted on multiple interactive task scenarios, the agent training method according to the embodiment of the present disclosure is superior to the current most advanced large language model agent training method. This clearly shows the superiority of applying the reasoning step-by-step reward reinforcement learning to the large language model agent strategy learning.

[0144] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0146] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for training an agent, the agent being associated with a large language model, the method comprising: obtaining expert trajectory data about a reasoning chain, the reasoning chain comprising a plurality of reasoning steps; generating, by the agent, agent trajectory data about the reasoning chain based on the expert trajectory data; Compare the expert trajectory data and the agent trajectory data step by step in the order of reasoning steps to generate a reward for each reasoning step, The agent is trained based on the reasoning step-by-step rewards.

2. The method according to claim 1, wherein: The per-reasoning-step reward includes an implicit reinforcement learning reward generated by gradually determining a direct preference optimization loss for expert actions in the expert trajectory data and agent actions in the agent trajectory data in a sequence of reasoning steps.

3. The method according to claim 1, wherein: The reasoning step-by-step reward includes a reverse reinforcement learning reward, which is generated based on: The occupancy metrics of the expert trajectory data and the occupancy metrics of the agent trajectory data are compared step by step by the trained discriminator network in a sequence of inference steps to generate an inverse reinforcement learning reward.

4. The method according to claim 3, wherein: Training the agent based on the reasoning step-by-step reward includes: training the agent using a nearest policy optimization rule based on the inverse reinforcement learning reward.

5. The method according to claim 3, wherein: The trained discriminator network is trained based on expert trajectory data and agent trajectory data.

6. The method according to claim 1, wherein: The expert trajectory data includes the first to nth expert observations and the first to nth expert actions generated for the first to nth expert observations, wherein n is an integer greater than or equal to 1, and Wherein, generating the agent trajectory data about the reasoning chain includes: The agent takes the 1st expert observation to the i-th expert observation and the 1st expert action to the i-1th expert action as states, and generates the i-th agent action based on the states, where i is an integer from 1 to n.

7. The method according to claim 1, wherein: The expert trajectory data is not manually annotated with rewards step by step in the order of reasoning steps.

8. The method of claim 1, further comprising performing supervised fine-tuning on the agent before training the agent based on the per-inference step reward.

9. The method according to claim 1, wherein: The agent applies to one or more of the following: Online shopping assistant; Web browsing assistant; Scientific laboratory assistant; Home task assistants; and Multi-hop question and answer assistant.

10. The method according to claim 1, wherein: The agent is implemented by a large language model or the agent is the large language model itself.

11. A device for training an intelligent agent, the intelligent agent being associated with a large language model, the device comprising: an expert trajectory data obtaining module, configured to obtain expert trajectory data about a reasoning chain, the reasoning chain comprising a plurality of reasoning steps; an agent trajectory data generation module configured to operate the agent to generate agent trajectory data about the reasoning chain based on the expert trajectory data; A per-reasoning step reward generation module is configured to compare the expert trajectory data and the agent trajectory data step by step according to the order of the reasoning steps to generate a per-reasoning step reward, An agent training module is configured to train the agent based on the reward for each reasoning step.

12. A device for training an intelligent agent, comprising: one or more processors; as well as One or more memories, wherein a computer executable program is stored, and when the computer executable program is executed by the processor, the method according to any one of claims 1 to 10 is performed.

13. A computer program product stored on a computer-readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 10.

14. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

Images