Model processing method and related device

Abstract

The invention discloses a model processing method, which is applied to the technical field of artificial intelligence (AI). In the model processing method, the first model is used as a planner interacting with the environment, and the second model is used as an anti-environment feedback and an anti-device for promoting the improvement of the first model, so that the intelligent agent with anti-and self-improvement capabilities can be constructed based on the first model and the second model. Moreover, the first model and the second model are jointly trained based on reinforcement learning, it can be ensured that the first model serving as a planner and the second model serving as an antiphonic device can promote each other in the joint training process, and it is ensured that an antiphonic result provided by the second model can effectively promote the first model to plan a better action. And the overall performance of the intelligent agent is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), and in particular to a model processing method and related apparatus. BACKGROUND

[0002] For a long time, researchers in the field of AI have been committed to realizing intelligent agents that can independently think and dynamically interact with the environment. These intelligent agents are expected to mimic and surpass human decision-making and execution capabilities in many fields, providing new solutions to complex problems.

[0003] In recent years, with the emergence of large language models (LLMs) with strong understanding capabilities, researchers have begun to use large language models to build advanced intelligent agents. Intelligent agents based on large language models have demonstrated unprecedented capabilities in understanding complex language inputs, making complex plans, and using tools, not only achieving breakthroughs in technology, but also bringing huge commercial value.

[0004] Intelligent agents based on large language models actually use the language understanding and thinking capabilities of large language models themselves to make decisions, so the capabilities of intelligent agents are closely related to the performance of large language models themselves. In some scenarios, when the performance of the large language model used by the intelligent agent is insufficient, the intelligent agent is also difficult to achieve excellent performance. Therefore, how to improve the performance of intelligent agents has become a problem to be solved. SUMMARY

[0005] The present application provides a model processing method that can improve the performance of intelligent agents implemented based on models.

[0006] In a first aspect, a model processing method is provided, which is applied to execute an intelligent agent task. The model processing method includes: first, an execution device obtains first state information, which is used to indicate the historical interaction content between the first model and the environment. For example, when the first model is used to execute a question and answer task, the first state information can be a question that indicates that the user (i.e., the environment) asks the first model.

[0007] Then, the execution device inputs the first state information into the first model, and the first model performs task planning based on the first state information to obtain a first action. For example, in the case where the first model is used to process a question and answer task, the first model can output an answer to the first state information, so that the execution device can return the answer output by the first model to the user (e.g., display the answer output by the first model on a display screen), thereby implementing the first action.

[0008] When the feedback result of the environment obtained by performing the first action does not satisfy the preset requirement, the execution device inputs the first state information, the first action and the feedback result into the second model to obtain a reflection result, and the reflection result is used to indicate that the first model determines the direction of the action based on the interaction process with the environment. That is, the second model is used to reflect the interaction process of the first model with the environment when the interaction effect of the first model with the environment cannot meet the requirement, so as to determine how to improve the interaction effect of the first model with the environment. Therefore, the reflection result output by the second model is specifically used to indicate that the first model determines the direction of the action based on the interaction process with the environment.

[0009] Finally, the second state information is input into the first model to obtain the second action, and the second state information includes the first state information, the first action, the feedback result and the reflection result. That is, in the current processing process of the first model, the input of the first model is all the interaction content of the first model with the environment before and the reflection result of the second model for the interaction content.

[0010] Among them, the first model and the second model are obtained based on joint training of reinforcement learning. That is, the first model and the second model are obtained by joint training of the same batch of training data driven by a reward value.

[0011] In the scheme, by taking the first model as a planner for interacting with the environment and taking the second model as a reflector for reflecting the feedback of the environment and promoting the improvement of the first model, an agent with reflection and self-improvement ability can be constructed based on the first model and the second model. Moreover, the first model and the second model are jointly trained based on reinforcement learning, which can ensure that the first model as a planner and the second model as a reflector can promote each other in the joint training process, and ensure that the reflection result provided by the second model can effectively promote the first model to plan a better action, and improve the overall performance of the agent.

[0012] In a possible implementation, the method further includes: performing joint training on the first model and the second model based on reinforcement learning to obtain an updated first model and an updated second model. The updated first model and the updated second model continue to be used to interact with the environment.

[0013] Among them, in the process of performing joint training on the first model and the second model, the first model is updated based on a first loss value, the second model is updated based on a second loss value, the first loss value is obtained based on a first reward value, the first reward value is obtained based on the feedback result, and the second loss value is obtained based on the difference between a second reward value and the first reward value, and the second reward value is obtained based on the feedback of the environment when the second action is performed.

[0014] That is, when the first model is updated based on the first loss value determined by the first reward value, the training target of the first model is actually to make the first model obtain a greater reward value as much as possible, that is, the action determined by the first model can obtain more positive environmental feedback results. When the second model is updated based on the second loss value, the training target of the second model is actually to make the second model provide better reflection results as much as possible, so that the action determined by the first model based on the reflection results can ultimately obtain better environmental feedback results.

[0015] In this scheme, after the first model and the second model run for a period of time, the first model and the second model are continuously updated based on the data accumulated during the running of the first model and the second model, so that the first model and the second model can continuously learn and update, thereby ensuring that the performance of the first model and the second model can continuously adapt to the changing environment, and ensuring that the first model and the second model always have optimal performance.

[0016] In a possible implementation, the first loss value is specifically obtained based on the first reward value and a first sub-loss value, and the first sub-loss value is a loss function value obtained by training the first model by using imitation learning. The second loss value is specifically obtained based on the difference between the second reward value and the first reward value and a second sub-loss value, and the second sub-loss value is a loss function value obtained by training the second model by using imitation learning.

[0017] In this scheme, by simultaneously using reinforcement learning and imitation learning to construct the loss function for training the first model and the second model, the training process of the first model and the second model can be as stable as possible, and it is ensured that the first model and the second model can learn the knowledge expected by the user, avoid deviation in the learning process of the first model and the second model, and ensure the training effect of the first model and the second model.

[0018] In a possible implementation, the first sub-loss value is obtained based on training the first model by using first training data, and the first training data includes interaction content between the first expert model and the environment. The second sub-loss value is obtained based on training the second model by using second training data, and the second training data includes the reflection results output by the second expert model based on the interaction content between the first expert model and the environment.

[0019] That is, the essence of imitation learning is to make the first model imitate the behavior of the first expert model as much as possible and make the second model imitate the behavior of the second expert model as much as possible, so that the first model learns how to plan the action to be performed based on the interaction content with the environment, and the second model learns how to output the reflection results that can improve the interaction effect between other models and the environment based on the input content.

[0020] In a possible implementation, the first model and the second model are obtained through joint training based on reinforcement learning after training based on imitation learning. That is, in the initial training stage, the first model and the second model are trained based on imitation learning (for example, the first model and the second model are trained by using the first training data and the second training data described above), and then the first model and the second model are jointly trained based on reinforcement learning.

[0021] In this scheme, the two models are trained through imitation learning first, so that the two models can imitate the behavior of the expert model, thereby quickly guiding the two models to learn effective knowledge and improving the training efficiency of the models. Moreover, after the imitation learning of the two models is performed, reinforcement learning is further performed on the two models, thereby further improving the performance of the first model and the second model obtained through training.

[0022] In a possible implementation, the first state information is input into the first model to obtain the first action, specifically including: inputting, by the execution device, the first state information into the first model to obtain component calling instructions output by the first model; and based on the component calling instructions, calling and executing, by the execution device, a target component to obtain the first action. That is, the first model needs to cooperate with other tool components to obtain the final first action to be executed. The target component can be specifically a software module running on the execution device, or a hardware module (for example, a Bluetooth communication module, a loudspeaker, or a microphone) deployed on the execution device.

[0023] In this scheme, by setting that the first model can call various tool components to process tasks, the first model can actually process diversified tasks, thereby ensuring the realizability and practicability of the scheme.

[0024] In a possible implementation, the target component includes at least one of a database query engine, a search engine, or a code execution engine.

[0025] In a possible implementation, the feedback result does not meet the preset requirement, specifically including: the feedback result is negative feedback for the first action. That is, if the feedback result of the environment is negative feedback, it can be considered that the first action executed by the execution device does not meet the expectation of the environment, thereby giving negative feedback for the first action. The negative feedback provided by the environment can have various implementation manners, which are not limited in the present application.

[0026] In a possible implementation, the first model and the second model are both large language models. The first model and the second model can be two large language models with the same structure but different parameters, or the first model and the second model can be two large language models with different structures.

[0027] In one possible implementation, the first model is used to perform at least one of the following tasks: question answering, content recommendation, interactive programming, embodied intelligence, or content generation.

[0028] Secondly, a model training method is provided, comprising: acquiring first state information, which indicates the historical interaction content between a first model and the environment; inputting the first state information into the first model to obtain a first action; when the feedback result of the environment obtained from executing the first action does not meet the preset requirements, inputting the first state information, the first action, and the feedback result into a second model to obtain a reflection result, which indicates that the first model determines the direction of the action based on the interaction process with the environment; inputting second state information into the first model to obtain a second action, which includes the first state information, the first action, the feedback result, and the reflection result; and performing joint training on the first model and the second model based on reinforcement learning to obtain an updated first model and an updated second model; wherein, during the joint training of the first model and the second model, the first model is updated based on a first loss value, the second model is updated based on a second loss value, the first loss value is obtained based on a first reward value, the first reward value is obtained based on the feedback result, the second loss value is obtained based on the difference between the second reward value and the first reward value, and the second reward value is obtained based on the feedback of the environment when the second action is executed.

[0029] In one possible implementation, the first loss value is specifically obtained based on the first reward value and the first sub-loss value, which is a loss function value obtained by training the first model using imitation learning; the second loss value is specifically obtained based on the difference and the second sub-loss value, which is a loss function value obtained by training the second model using imitation learning.

[0030] In one possible implementation, the first sub-loss value is obtained by training the first model based on the first training data, which includes the interaction content between the first expert model and the environment; the second sub-loss value is obtained by training the second model based on the second training data, which includes the reflection results output by the second expert model based on the interaction content between the first expert model and the environment.

[0031] In one possible implementation, the first and second models are obtained by first training based on imitation learning and then jointly training based on reinforcement learning.

[0032] In one possible implementation, inputting the first state information into the first model to obtain the first action includes: inputting the first state information into the first model to obtain the component invocation instruction output by the first model; and invoking and executing the target component based on the component invocation instruction to obtain the first action.

[0033] In one possible implementation, the target component includes at least one of a database query engine, a search engine, or a code execution engine.

[0034] In one possible implementation, the feedback result does not meet the preset requirements, including: the feedback result is negative feedback for the first action.

[0035] In one possible implementation, both the first and second models are large language models.

[0036] In one possible implementation, the first model is used to perform at least one of the following tasks: question answering, content recommendation, interactive programming, embodied intelligence, or content generation.

[0037] Thirdly, a model processing apparatus is provided, comprising: an acquisition module for acquiring first state information, the first state information indicating historical interaction content between a first model and the environment; a processing module for inputting the first state information into the first model to obtain a first action; the processing module further comprising inputting the first state information, the first action, and the feedback result into a second model to obtain a reflection result when the feedback result from the environment obtained by executing the first action does not meet preset requirements, the reflection result indicating to the first model to determine the direction of the action based on the interaction process with the environment; and the processing module further comprising inputting second state information into the first model to obtain a second action, the second state information including the first state information, the first action, the feedback result, and the reflection result; wherein the first model and the second model are obtained based on joint training using reinforcement learning.

[0038] In one possible implementation, the processing module is also used for:

[0039] Joint training of the first and second models is performed based on reinforcement learning to obtain updated first and second models;

[0040] In the process of jointly training the first model and the second model, the first model is updated based on the first loss value, the second model is updated based on the second loss value, the first loss value is obtained based on the first reward value, the first reward value is obtained based on the feedback result, the second loss value is obtained based on the difference between the second reward value and the first reward value, and the second reward value is obtained based on the feedback from the environment when the second action is performed.

[0041] In one possible implementation, the first loss value is specifically obtained based on the first reward value and the first sub-loss value, which is the loss function value obtained by training the first model using imitation learning;

[0042] The second loss value is specifically obtained based on the difference and the second sub-loss value, which is the loss function value obtained by training the second model using imitation learning.

[0043] In one possible implementation, the first sub-loss value is obtained by training the first model based on the first training data, which includes the interaction content between the first expert model and the environment.

[0044] The second sub-loss value is obtained by training the second model based on the second training data, which includes the reflection results output by the second expert model based on the interaction content between the first expert model and the environment.

[0045] In one possible implementation, the first and second models are obtained by first training based on imitation learning and then jointly training based on reinforcement learning.

[0046] In one possible implementation, the processing module is also used for:

[0047] Input the first state information into the first model to obtain the component call instructions output by the first model;

[0048] Based on the component invocation instruction, the target component is invoked and executed to obtain the first action.

[0049] In one possible implementation, the target component includes at least one of a database query engine, a search engine, or a code execution engine.

[0050] In one possible implementation, the feedback result does not meet the preset requirements, including:

[0051] The feedback result was negative feedback regarding the first action.

[0052] In one possible implementation, both the first and second models are large language models.

[0053] In one possible implementation, the first model is used to perform at least one of the following tasks: question answering, content recommendation, interactive programming, embodied intelligence, or content generation.

[0054] Fourthly, a model training apparatus is provided, comprising: an acquisition module for acquiring first state information, the first state information indicating the historical interaction content between a first model and the environment; a processing module for inputting the first state information into the first model to obtain a first action; the processing module is further configured to input the first state information, the first action, and the feedback result into a second model when the feedback result from the environment obtained by executing the first action does not meet preset requirements, to obtain a reflection result, the reflection result indicating to the first model to determine the direction of the action based on the interaction process with the environment; the processing module is further configured to input second state information into the first model to obtain a second action, the second state information including the first state information, the first action, the feedback result, and the reflection result; and to perform joint training on the first model and the second model based on reinforcement learning to obtain an updated first model and an updated second model; wherein, during the joint training of the first model and the second model, the first model is updated based on a first loss value, the second model is updated based on a second loss value, the first loss value is obtained based on a first reward value, the first reward value is obtained based on the feedback result, the second loss value is obtained based on the difference between the second reward value and the first reward value, and the second reward value is obtained based on the feedback from the environment when the second action is executed.

[0055] In one possible implementation, the first loss value is specifically obtained based on the first reward value and the first sub-loss value, which is a loss function value obtained by training the first model using imitation learning; the second loss value is specifically obtained based on the difference and the second sub-loss value, which is a loss function value obtained by training the second model using imitation learning.

[0056] In one possible implementation, the first sub-loss value is obtained by training the first model based on the first training data, which includes the interaction content between the first expert model and the environment; the second sub-loss value is obtained by training the second model based on the second training data, which includes the reflection results output by the second expert model based on the interaction content between the first expert model and the environment.

[0057] In one possible implementation, the first and second models are obtained by first training based on imitation learning and then jointly training based on reinforcement learning.

[0058] In one possible implementation, the processing module is further configured to: input the first state information into the first model to obtain the component invocation instruction output by the first model; and, based on the component invocation instruction, invoke and execute the target component to obtain the first action.

[0059] In one possible implementation, the target component includes at least one of a database query engine, a search engine, or a code execution engine.

[0060] In one possible implementation, the feedback result does not meet the preset requirements, including: the feedback result is negative feedback for the first action.

[0061] In one possible implementation, both the first and second models are large language models.

[0062] In one possible implementation, the first model is used to perform at least one of the following tasks: question answering, content recommendation, interactive programming, embodied intelligence, or content generation.

[0063] A fifth aspect of this application provides a model processing apparatus, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any implementation thereof is implemented. For details regarding the steps in the various possible implementations of the first aspect executed by the processor, please refer to the first aspect; further details will not be repeated here.

[0064] The sixth aspect of this application provides a model training apparatus, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the second aspect or any implementation thereof is implemented. For details regarding the steps in the various possible implementations of the first aspect executed by the processor, please refer to the second aspect; further details will not be repeated here.

[0065] The seventh aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any implementation of the first or second aspect described above.

[0066] The eighth aspect of this application provides a circuit system including a processing circuit configured to perform the method of any implementation of the first or second aspect described above.

[0067] The ninth aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform any implementation of the first or second aspect described above.

[0068] The tenth aspect of this application provides a chip system including a processor for supporting an execution device in implementing the functions involved in any implementation of the first or second aspect described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device. This chip system may be composed of chips or may include chips and other discrete devices.

[0069] The beneficial effects of aspects two through nine mentioned above can be referred to the introduction of aspect one above, and will not be repeated here. Attached Figure Description

[0070] Figure 1 A schematic diagram illustrating the application scenario architecture of a model processing method provided in this application;

[0071] Figure 2 A flowchart illustrating a model processing method provided in this application;

[0072] Figure 3 A flowchart illustrating another model processing method provided in this application;

[0073] Figure 4 A flowchart illustrating a first model calling a target component to obtain a first action, provided in this application;

[0074] Figure 5 A flowchart illustrating the process of obtaining reflective results using a second model provided in this application;

[0075] Figure 6 A schematic diagram illustrating negative feedback to an environment provided in this application;

[0076] Figure 7 A diagram illustrating negative feedback for another environment provided in this application;

[0077] Figure 8 A schematic diagram illustrating a process for performing joint training of a first model and a second model based on reinforcement learning, provided for this application;

[0078] Figure 9A A schematic diagram of the system architecture used in the model training method provided in this application;

[0079] Figure 9B A flowchart illustrating a model training method provided in this application;

[0080] Figure 10 A schematic diagram of the structure of a model processing device provided in this application;

[0081] Figure 11 A schematic diagram of the structure of a computing device provided in this application;

[0082] Figure 12 This application provides a schematic diagram of the structure of a computing device cluster;

[0083] Figure 13 This application provides a schematic diagram of another computing device cluster structure.

[0084] Figure 14This application provides a schematic diagram of the structure of a chip;

[0085] Figure 15 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0086] The embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0087] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0088] First, some technical terms used in this application will be introduced.

[0089] (1) Neural Network

[0090] Neural networks can be composed of neural units, which can refer to units represented by x. sThe arithmetic unit that takes input data as input can output the following:

[0091]

[0092] Where s = 1, 2, ..., n, n is a natural number greater than 1, and Ws is x s The weights are denoted by b, the bias of the neural unit is denoted by f, and the activation functions are denoted by f, which introduce nonlinear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting multiple individual neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be a region composed of several neural units.

[0093] (2) Large Language Model (LLM)

[0094] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0095] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0096] (3) Prompt

[0097] Prompts originated as an input format designed by researchers for downstream tasks. Their purpose is to help pre-trained models "recall" what they "learned" during pre-training, hence the name "cue word." For large language models, a prompt is the user's input, instructing the model on the task to be performed. A prompt can be a simple question, a longer text, or a set of instructions, depending on the user's specific needs. Generally, a prompt is a short text string that provides context and task-related information to help the model better understand the requirements and generate the correct output. For example, in question-answering tasks, a prompt might contain a description of the question or topic to help the large language model generate the correct answer. Furthermore, prompts are often designed by humans to help large language models better understand specific tasks or domains.

[0098] In this way, when a large language model generates content, it first processes the prompt and then outputs content based on its understanding of the prompt. The working principle of the large language model is to predict the probability of the next word appearing based on the preceding context of the user input, thereby generating the following text word by word. Therefore, differences in the user's input prompt directly affect the quality of the large language model's output. In some cases, even a difference of just a few words in the user's input prompt can result in significantly different content generated by the large language model.

[0099] (4) Intelligent Agent

[0100] An intelligent agent is an independent entity capable of thinking and interacting with its environment. A large language model intelligent agent, on the other hand, is an intelligent agent that utilizes the capabilities of a large language model, combined with task planning, decision-making mechanisms, and environmental interaction capabilities, enabling it to perform complex tasks in dynamic environments.

[0101] (5) Imitation Learning (IL)

[0102] Imitation learning refers to agent learning strategies based on expert trajectories, typically providing expert decision-making and control information. Generally, each expert trajectory contains a series of states and actions. By extracting all state-action pairs from the expert trajectories, a dataset can be constructed. In imitation learning tasks, the model uses this constructed dataset to learn the latent relationships between states and actions, where states represent features and actions represent labels. Therefore, the specific goal of imitation learning is to evaluate the most suitable mapping between states and actions so that the model can replicate the expert trajectories as much as possible. These expert trajectories can be sampled from pre-trained expert models or from human demonstrations.

[0103] (6) Reinforcement learning

[0104] Reinforcement learning, a branch of machine learning, focuses on how an agent takes different actions in an environment to maximize cumulative reward. Generally, reinforcement learning consists of an agent, an environment, states, actions, and a reward. The agent is the ontology in reinforcement learning, acting as a learner or decision-maker. The environment encompasses everything outside the agent and is primarily composed of a set of states. A state is data representing the environment, and the state set contains all possible states. Actions are the actions the agent can perform, and the action set contains all possible actions. The reward is the positive / negative feedback signal the agent receives after performing an action, and the reward set contains all feedback information the agent can obtain. Reinforcement learning is essentially learning the mapping from environment states to actions.

[0105] After an agent performs an action, the environment transitions to a new state, providing a reward signal (positive or negative) for this new state. Subsequently, based on the new state and the reward from the environment, the agent executes a new action according to a specific strategy. This process describes how the agent and the environment interact through states, actions, and rewards, with the ultimate goal of maximizing the agent's accumulated rewards.

[0106] Through reinforcement learning, an intelligent agent can understand its own state and the actions it should take to maximize its reward. Since the interaction between an intelligent agent and its environment is similar to that between humans and their environment, reinforcement learning can be considered a general learning framework that can be used to solve problems related to general artificial intelligence. Therefore, reinforcement learning is also known as a machine learning method for general artificial intelligence.

[0107] Furthermore, two crucial concepts in reinforcement learning tasks are exploitation and exploration, sometimes also called exploitation and probing, respectively. In reinforcement learning, exploitation refers to the agent selecting the optimal action from all known state-action pair distributions, based on the principle of maximizing the action's value. In other words, when the agent chooses from known actions, this is called exploitation (or exploitation). Exploration, on the other hand, refers to the agent selecting unknown actions outside of the known state-action pair distributions.

[0108] (7) Loss Function

[0109] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference (usually preceded by initialization, where parameters are pre-configured for each layer). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value, and this process continues until the neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are crucial equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference between the predicted and target values, thus training the neural network becomes a process of minimizing this loss.

[0110] (8) Back propagation (BP) algorithm

[0111] Neural networks can employ backpropagation algorithms to refine the parameters of the initial prediction model during training, thereby minimizing the error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then used to update the parameters of the initial prediction model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process designed to obtain the optimal parameters of the prediction model, such as the weight matrix.

[0112] Specifically, during model training, the backpropagation algorithm is typically used to calculate the gradients of each node in the model. This allows for the adjustment of node weights based on the gradients of each node, thereby minimizing the model's loss function value. The gradient represents the rate of change of a function at a given point. Furthermore, the gradient of each node in the model can be determined by calculating its partial derivatives.

[0113] (9) Gradient descent method

[0114] Gradient descent is a first-order optimization algorithm commonly used in machine learning to recursively approximate a minimum-bias prediction model. To find a local minimum of a function using gradient descent, iterative searches must be performed at points a predetermined step distance away in the opposite direction of the gradient (or approximate gradient) at the current point on the function. Gradient descent is one of the most frequently used methods for solving the prediction model parameters of machine learning algorithms, i.e., unconstrained optimization problems.

[0115] Specifically, when finding the minimum value of the loss function, gradient descent can be used to iteratively solve the problem step by step, obtaining the minimized loss function and the predicted model parameter values. Conversely, if we need to find the maximum value of the loss function, we need to use gradient ascent iteratively.

[0116] (10) Embodied intelligence

[0117] Embodied intelligence is a field of artificial intelligence development that refers to the ability of an intelligent system or machine to interact with its environment in real time through perception and interaction. It can be simply understood as various types of robots performing a wide variety of tasks in real physical environments, thus completing the evolutionary process of artificial intelligence.

[0118] Embodied intelligent robots are the physical form of "embodied intelligence" and are expected to become the ultimate carrier of AI. Their overall architecture consists of a perception layer, an interaction layer, and a movement layer. The most significant characteristic of "embodied intelligence" is its ability to autonomously perceive the physical world from a protagonist's perspective, learn using anthropomorphic thinking, and thus produce the behavioral responses expected by humans, rather than passively waiting for data to be fed to it.

[0119] Generally, intelligent agents based on large language models utilize the language understanding and reasoning capabilities of the large language model itself to make decisions. Therefore, the agent's capabilities are closely related to the performance of the large language model. However, the performance of the large language model is also closely related to the number of parameters in the large language model. The larger the number of parameters, the higher the performance tends to be; conversely, the smaller the number of parameters, the lower the performance tends to be. Furthermore, a larger number of parameters requires more computational power and time to run the large language model. Therefore, in some scenarios, such as when the terminal device deploying the large language model has insufficient computing power or when the response time of the agent's task is critical, the large language model used by the agent is often a small one. In such cases, the agent often struggles to achieve excellent performance.

[0120] In view of this, this application provides a model processing method that, by using a first model as a planner interacting with the environment and a second model as a reflector reflecting on environmental feedback and promoting the improvement of the first model, enables the construction of an intelligent agent with reflective and self-improving capabilities based on the first and second models. Furthermore, the first and second models are jointly trained based on reinforcement learning, ensuring that during joint training, the first model as the planner and the second model as the reflector can mutually promote each other, and guaranteeing that the reflection results provided by the second model can effectively help the first model determine better actions, thus jointly improving the overall performance of the intelligent agent.

[0121] Specifically, the model processing method provided in this application can be applied to execution devices. For example, the execution device may be a server, robot, smartphone, personal computer (PC), laptop, wireless execution device in industrial control, wireless execution device in self-driving, wireless execution device in smart grid, wireless execution device in logistics warehousing, wireless execution device in transportation safety, wireless execution device in smart city, etc.

[0122] The above describes the device used in the model processing method provided in this application. The following describes the application scenarios of the model processing method provided in this application. Specifically, the model processing method provided in this application can be applied to intelligent agent scenarios to complete intelligent agent tasks in various scenarios. These intelligent agent tasks include, but are not limited to, question-answering tasks, content recommendation tasks, interactive programming tasks, embodied intelligence tasks, or content generation tasks. A question-answering task refers to a user posing a question, which the intelligent agent then considers and provides an answer to. A content recommendation task involves inputting historical interactions between the user and existing content (such as products, music, applications, or videos) into the intelligent agent, which then outputs recommended content to the user. An interactive programming task involves the intelligent agent interacting with the user to assist in completing a specific programming task. An embodied intelligence task refers to the intelligent agent, as an independent entity, interacting with the environment in real time through perception and interaction. A content generation task involves inputting given content description information into the intelligent agent, which then generates corresponding content (such as translated content, text content, image content, or video content) based on the content description information.

[0123] In general, an agent task refers to a task in which an agent interacts with its environment. The agent performs corresponding actions by acquiring information about the environment's state (such as the agent's historical interactions with the environment and changes in the environment), and obtains feedback results from the environment after performing the actions. In this way, the agent can update the environment's state information based on the feedback results, thereby executing the next action.

[0124] For example, in a question-answering task, the model used to answer a user's question can be considered an agent. The user's question can be seen as the state of the agent's environment; the agent's answer can be seen as the action performed; and the user's feedback can be seen as the feedback to the environment. The agent needs to determine its actions based on the state of the environment, such as providing answers that satisfy the user as much as possible, thereby maximizing the positive feedback it receives from the environment.

[0125] For example, content recommendation tasks can specifically include social recommendation tasks, product recommendation tasks, or advertising recommendation tasks. Taking product recommendation tasks as an example, the model used to recommend products to users can be regarded as an intelligent agent. The user clicking to browse product information recommended by the server, or the user closing product information recommended by the server, can be regarded as the state of the recommendation environment in which the intelligent agent exists; the intelligent agent recommending a specific product to the user can be regarded as the action performed by the intelligent agent. Since the number of products that the intelligent agent can recommend is massive, the action space to which the intelligent agent can perform actions is continuous, that is, the action space can be regarded as including infinite actions. For the intelligent agent, it needs to determine the actions it performs based on the state of the recommendation environment, such as recommending certain types of products, recommending products within certain price ranges, etc., so as to maximize the positive feedback obtained by the intelligent agent from the recommendation environment, that is, to encourage users to click to browse the product information recommended by the intelligent agent as frequently as possible.

[0126] For example, please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the application scenario architecture of a model processing method provided in this application. For example... Figure 1As shown, the first model and the second model in the model processing method provided in this application constitute an intelligent agent, where the first model and the second model can be, for example, large language models. Based on the state information of the input environment, the intelligent agent uses the first model to understand and process the input, and performs planning and reasoning to generate a preliminary task execution plan. When executing the plan, the intelligent agent can directly feed back the output of the first model to the environment, thereby realizing the execution of actions based on the state information of the environment. Optionally, the intelligent agent can also call other tool components (such as query engines, search engines, or code execution engines) to complete specific operations, such as querying databases, searching for relevant text materials, or executing code, and finally execute the corresponding actions based on the content obtained from calling the tool components. Furthermore, the intelligent agent obtains feedback results from the environment by interacting with the environment, and the feedback results can be stored in a database. Then, the intelligent agent uses the second model to reflect on the stored feedback results from the environment, analyze and summarize the problems and shortcomings in the execution process, and thus propose improvement strategies. The database not only records past experiences and feedback, but also supports updating the model in the intelligent agent, supporting the intelligent agent's continuous self-learning and adaptation. In this way, intelligent agents can continuously improve their task performance and environmental adaptability through cycles of understanding, execution, storing feedback, reflection, and learning.

[0127] Please see Figure 2 and Figure 3 , Figure 2 A flowchart illustrating a model processing method provided in this application; Figure 3 A flowchart illustrating another model processing method provided in this application. (For example...) Figure 2 and Figure 3 As shown, the model processing method provided in this application includes the following steps 201-204.

[0128] Step 201: Obtain first state information, which is used to indicate the historical interaction content between the first model and the environment.

[0129] In this application, the first model and the second model can constitute an agent for interacting with the external environment to complete the agent's tasks. Within the agent, the first model is responsible for interacting with the environment, while the second model is responsible for reflecting on the interaction process between the first model and the environment, thus providing the first model with reflection results to promote better interaction. Optionally, both the first model and the second model can be large language models, meaning both have strong natural language understanding and processing capabilities. Of course, the first model and the second model can also be other types of models, which are not specifically limited here.

[0130] For example, the first model can be at least one of the following: question-answering task, content recommendation task, interactive programming task, embodied intelligence task, or content generation task. During the interaction between the first model and the environment, the execution device can obtain first state information, which indicates the historical interaction content between the first model and the environment. Furthermore, the first state information may also include information such as task prompts and environment descriptions, where the task prompts are information used to indicate the task that the first model needs to complete, and the environment descriptions are used to describe the current environment in which the first model is located.

[0131] Specifically, when the first model interacts with the environment for the first time, the first state information can be the content provided by the environment to the first model. For example, when the first model is used to perform a question-answering task, the first state information can be the question that the user (i.e., the environment) asks the first model. When the first model has interacted with the environment multiple times, the first state information can include the content of the interactions between the first model and the environment. For example, when the first model is used to perform a question-answering task, the first state information can be the question that the user asks the first model and the answer that the first model returns to the user.

[0132] Step 202: Input the first state information into the first model to obtain the first action.

[0133] After acquiring the first state information, the execution device inputs it into the first model. The first model then performs task planning based on this information to determine the action to be performed in response to the first state information. After task planning, the first model can output corresponding content, enabling the execution device to determine the first action to be performed in response to the first state information. For example, in the case of a question-and-answer task, the first model can output an answer to the first state information, allowing the execution device to return the answer to the user (e.g., displaying the answer on a screen), thereby executing the first action.

[0134] In other words, the first state information is used as input to the first model, which processes the first state information to determine the first action that the execution device needs to perform in response to the environment.

[0135] In some scenarios, after the first state information is input into the first model, the first model can directly output the corresponding content, thereby obtaining the first action required by the execution device. That is, the first model independently completes the processing of the first state information and obtains the corresponding first action, without relying on other tool components.

[0136] Optionally, in other scenarios, after the first state information is input into the first model, the first model may need to call other tool components to complete specific operations in the process of processing the first state information in order to obtain the first action required by the execution device.

[0137] For example, the execution device inputs first state information into the first model and obtains a component invocation instruction output by the first model. This component invocation instruction indicates how to invoke the target component and how to execute it. Then, based on the component invocation instruction, the execution device invokes and executes the target component to obtain the first action. That is, the first model needs to cooperate with other tool components to obtain the final first action to be executed. Specifically, the target component can be a software module running on the execution device or a hardware module deployed on the execution device (such as a Bluetooth communication module, speaker, or microphone). Taking a software module running on the execution device as an example, the target component can specifically include at least one of a database query engine, a search engine, or a code execution engine.

[0138] It should be noted that after the execution device invokes and executes the target component, the execution device can obtain the aforementioned first action based on the execution result of the target component. For example, in the case where the first model is used to perform a question-and-answer task, the question included in the first state information is to find a specific document in the database. In this case, when processing the first state information, the first model determines that it needs to invoke the database query engine to search for relevant data in order to return an accurate answer to the user. Therefore, after processing the first state information, the first model will output a database query engine invocation instruction to instruct the database query engine to query the specific document. In this way, after the execution device invokes the database query engine based on the database query engine invocation instruction to query the specific document indicated by the first model, the specific document to be returned to the user can be obtained, thereby obtaining the aforementioned first action (i.e., returning the queried specific document to the user).

[0139] After executing the device call and executing the target component, the executing device may also need to return the execution result of the target component to the first model. The first model then processes the execution result of the target component and outputs the corresponding content to obtain the first action. For example, please refer to [link to relevant documentation]. Figure 4 , Figure 4 This application provides a schematic diagram of a first model that calls a target component to obtain a first action. For example... Figure 4As shown, when the first model is used to perform a question-and-answer task, it determines, upon processing the first state information (i.e., the question asked by the user), that it needs to invoke a search engine to search for relevant text in order to return an accurate answer to the user. At this point, the first model outputs a component invocation instruction (i.e., a search engine invocation instruction) to instruct the search engine to search for specific content. Thus, after the execution device invokes the search engine based on the search engine invocation instruction to search for the specific content indicated by the first model, it returns the search results (i.e., the search results) to the first model. The first model then outputs the answer to be returned to the user based on the search results, thereby achieving the first action described above (i.e., returning the answer output by the first model to the user).

[0140] Step 203: When the feedback result of the environment obtained by performing the first action does not meet the preset requirements, the first state information, the first action and the feedback result are input into the second model to obtain the reflection result. The reflection result is used to instruct the first model to determine the direction of the action based on the interaction process with the environment.

[0141] After obtaining the first action based on the first model, the execution device can execute the first action and obtain feedback from the environment. If the feedback from the environment obtained by the execution device does not meet the preset requirements, the execution device triggers the agent's reflection, that is, inputting the first state information, the first action, and the feedback result into the second model. The second model then reflects on the interaction process between the first model and the environment, thereby obtaining the reflection result output by the second model. That is, the second model is used to reflect on the interaction process between the first model and the environment when the interaction effect between the first model and the environment cannot meet the requirements, in order to determine how to improve the interaction effect between the first model and the environment. Therefore, the reflection result output by the second model is specifically used to instruct the first model to determine the direction of the action based on the interaction process with the environment. In other words, the reflection result can actually be understood as instructing the first model on how to determine the action to be executed based on the interaction process with the environment, that is, guiding the first model from which direction to consider in order to determine the action to be executed, ensuring that the first model can determine the action with better interaction effect.

[0142] For example, please refer to Figure 5 , Figure 5 This is a flowchart illustrating the process of obtaining reflective results using a second model provided in this application. For example... Figure 5As shown, when the first model is used to perform a question-and-answer task, the aforementioned first state information can specifically be the question asked by the user to the first model (or a prompt based on the user's question): "Who is mentioned in the title of Chapter 20 of Journey to the West?". After processing the first state information, the first model obtains a first action, which specifically returns the following answer to the user in the environment: "Tang Sanzang is in trouble at Huangfeng Ridge, and Pigsy rushes ahead halfway up the mountain." After the execution device executes the first action, the feedback result returned by the user in the environment is "Incorrect answer." Therefore, the execution device can determine that the feedback result of the environment does not meet the preset requirements, and thus inputs the aforementioned first state information, first action, and feedback result into the second model. After processing the input, the second model obtains the following reflection result: "The content of the previous answer was the title itself; a character should be selected from the title." Obviously, the reflection result points out the problem that occurred during the interaction between the first model and the environment (i.e., the answer does not match the question asked), and gives the direction that the first model should consider next (i.e., it should select a character from the title).

[0143] In this application, the environmental feedback not meeting the preset requirements specifically refers to negative feedback regarding the first action. That is, if the environmental feedback is negative, it can be considered that the first action performed by the execution device has not met the environment's expectations, thus providing negative feedback for the first action. The negative feedback provided by the environment can be implemented in various ways, and this application does not specifically limit its implementation.

[0144] For example, please see Figure 6 , Figure 6 This is a diagram illustrating negative feedback provided by the environment described in this application. Figure 6 As shown, when the first model is used to perform a question-and-answer task, the question asked by the user in the environment (i.e., the first state information mentioned above) is "Who is mentioned in the title of Chapter 20 of Journey to the West?". The answer returned by the first model in response to the user's question (i.e., the first action mentioned above) is "The title of Chapter 20 of Journey to the West mentions: Tang Sanzang is in trouble at Yellow Wind Ridge, and Pigsy rushes to the front halfway up the mountain." Furthermore, while the execution device displays the answer returned by the first model on the display screen, the execution device can also display feedback buttons near the returned answer (e.g., below the answer), such as a "like" button representing positive feedback and a "dislike" button representing negative feedback. If the user clicks the "dislike" button on the execution device's display screen, it means that the feedback provided by the user is negative, indicating that the answer provided by the first model is not the correct answer.

[0145] For example, please see Figure 7 , Figure 7A diagram illustrating negative feedback for another environment provided in this application. For example... Figure 7 As shown, when the first model is used to perform the question-answering task, the question asked by the user in the environment (i.e., the first state information mentioned above) is "Who is mentioned in the title of Chapter 20 of Journey to the West?". The answer returned by the first model in response to the user's question (i.e., the first action mentioned above) is "The title of Chapter 20 of Journey to the West mentions: 'Tang Sanzang is in trouble at Huangfeng Ridge, and Pigsy rushes to the front halfway up the mountain.'" If the user provides the following verbal feedback for the returned answer, "Incorrect answer," it indicates that the feedback provided by the user is negative, meaning that the answer provided by the first model is not the correct answer.

[0146] In general, the environment can provide negative feedback in different ways in different scenarios, and this application does not make any specific limitations on this.

[0147] Step 204: Input the second state information into the first model to obtain the second action. The second state information includes the first state information, the first action, the feedback result, and the reflection result.

[0148] After obtaining the reflection result output by the second model, second state information can be constructed based on the first state information, the first action, the feedback result from the environment, and the reflection result. This second state information is then input into the first model, which continues to process it. In other words, during this round of processing by the first model, its input consists of all previous interactions between the first model and the environment, as well as the reflection result from the second model regarding those interactions. Ultimately, after processing the second state information, the first model obtains the second action, which in turn causes the execution device to perform the second action.

[0149] like Figure 6 and Figure 7 As shown, after receiving negative feedback, the second model reflects on the content of the first model's previous interaction with the user and inputs the reflection results into the first model. Finally, the first model returns the answer based on the reflection results (i.e., the second action mentioned above): "Sorry, the characters mentioned in the title of the twentieth chapter of Journey to the West should be: Tang Sanzang and Zhu Bajie."

[0150] It should be noted that after the device performs the second action, if the feedback from the environment still does not meet the preset requirements, all interactions between the first model and the environment can be input into the second model, which will then output new reflection results, thereby continuing to promote the improvement of the first model.

[0151] In this application, to ensure the effective cooperation between the first and second models—that is, to ensure that the reflection results provided by the second model can effectively help the first model understand the feedback provided by the environment and plan better actions—the first and second models are obtained through joint training based on reinforcement learning. Specifically, both the first and second models are driven by reward values ​​and are jointly trained using the same batch of training data.

[0152] In other words, by setting the first model and the second model to be jointly trained based on reinforcement learning, it can be ensured that the first model, as the planner, and the second model, as the reflector, can promote each other during the joint training process. Furthermore, the reflection results provided by the second model can effectively promote the first model to plan better actions, thereby jointly improving the overall performance of the agent.

[0153] Optionally, in order to ensure that the performance of the first and second models can continuously adapt to the changing environment, after the first and second models have been running for a period of time, they can be updated based on the data accumulated during their operation, so as to ensure that the first and second models can continuously learn and update.

[0154] For example, the above model processing method further includes: performing joint training on the first model and the second model based on reinforcement learning to obtain an updated first model and an updated second model. The updated first model and the updated second model continue to be used to interact with the environment. That is, after the first model and the second model have been online for a period of time, a batch of trajectory data can be obtained. This batch of trajectory data records the interaction data between the first model and the environment, as well as the reflection result data obtained by the second model based on the interaction data between the first model and the environment. Thus, based on this batch of trajectory data, it is possible to continue performing joint training on the first model and the second model based on reinforcement learning to obtain an updated first model and an updated second model.

[0155] Specifically, during the joint training of the first and second models, the first model is updated based on a first loss value, and the second model is updated based on a second loss value. The first loss value is obtained based on a first reward value, which is based on the feedback from the environment when the first action is performed. The second loss value is obtained based on the difference between the second and first reward values, which is based on the feedback from the environment when the second action is performed.

[0156] For example, please refer to Figure 8 , Figure 8 This application provides a schematic diagram of a process for jointly training a first model and a second model based on reinforcement learning. For example...Figure 8 As shown, for the first model, the feedback result of the environment when the first action is performed is first obtained, and the first reward value is determined based on the feedback result of the environment. Then, the first loss value is determined based on the first reward value, and the first model is updated based on the first loss value.

[0157] For the second model, the feedback results of the environment when the first action is performed and the feedback results of the environment when the second action is performed are first obtained. A first reward value is determined based on the feedback results of the environment when the first action is performed, and a second reward value is determined based on the feedback results of the environment when the second action is performed. Then, a second loss value is determined based on the difference between the second reward value and the first reward value, and the second model is updated based on the second loss value.

[0158] Generally speaking, for the first reward value, the more positive the environmental feedback when performing the first action, the larger the first reward value; the more negative the environmental feedback when performing the first action, the smaller the first reward value. Therefore, when updating the first model based on the first loss value determined by the first reward value, the training objective of the first model is actually to make the first model obtain as large a reward value as possible, that is, to make the actions determined by the first model obtain the most positive environmental feedback.

[0159] Similarly, for the second reward value, the more positive the environmental feedback when performing the second action, the larger the second reward value; the more negative the environmental feedback when performing the second action, the smaller the second reward value. Therefore, determining the second loss value based on the difference between the second and first reward values ​​allows the second loss value to characterize the improvement in environmental feedback that the action determined by the first model can obtain after the second model provides a reflection result. A larger difference between the second and first reward values ​​indicates a greater improvement in the reward value that the first model can obtain after the second model provides a reflection result, meaning a better reflection result provided by the second model. Therefore, when updating the second model based on the second loss value, the training objective of the second model is actually to make the second model provide better reflection results as much as possible, so that the action determined by the first model based on the reflection results can ultimately obtain better environmental feedback results.

[0160] In this scheme, after the first and second models have been running for a period of time, they are updated based on the data accumulated during their operation. This allows the first and second models to continuously learn and update, thereby ensuring that their performance can adapt to the changing environment and guaranteeing that they always have optimal performance.

[0161] Optionally, the first loss value used to update the first model can be obtained based on the first reward value and the first sub-loss value, where the first sub-loss value is a loss function value obtained by training the first model using imitation learning. The second loss value can be obtained based on the difference and the second sub-loss value, where the second sub-loss value is a loss function value obtained by training the second model using imitation learning.

[0162] Understandably, when using reinforcement learning to jointly train the first and second models, the knowledge learned by the first and second models may conflict with the user's expectation that they will actually learn. Therefore, to ensure that the first and second models learn the knowledge expected by the user and to stabilize their training process, imitation learning can be combined with reinforcement learning for training. Thus, for the first model, the first loss value used to update it is obtained based on both reinforcement learning and imitation learning. That is, the first loss value is obtained based on the first reward value and the first sub-loss value, where the first reward value is obtained based on reinforcement learning, and the first sub-loss value is obtained based on imitation learning. Similarly, the second loss value used to update the second model is also obtained based on both reinforcement learning and imitation learning.

[0163] In this scheme, by simultaneously employing reinforcement learning and imitation learning to construct the loss function for training the first and second models, the training process of the first and second models can be stabilized as much as possible, and it can be ensured that the first and second models can learn the knowledge expected by the user, avoiding deviations in the learning process of the first and second models, and ensuring the training effect of the first and second models.

[0164] Optionally, the first sub-loss value is obtained by training the first model on the first training data, which includes the interaction content between the first expert model and the environment. For example, the first model can be trained using supervised learning based on the first training data to obtain the first sub-loss value. The first expert model serves the same purpose as the first model: to interact with the environment and plan the desired action based on this interaction. Furthermore, the first expert model is a pre-trained and high-performance model; for example, its parameter count is greater than that of the first model to ensure high performance. Thus, because the first expert model has high performance, its interaction content with the environment can be considered an expert trajectory, which can be used to guide the training process of the first model. In other words, the essence of imitation learning is to make the first model imitate the behavior of the first expert model as closely as possible, thereby enabling the first model to learn how to plan the desired action based on its interaction with the environment.

[0165] The second sub-loss value is obtained by training the second model on second training data. This second training data includes the reflection results output by the second expert model based on the interactions between the first expert model and the environment. For example, the second model can be trained using supervised learning based on the second training data to obtain the second sub-loss value. The second expert model serves the same purpose as the first model: to output reflection results to improve the model's interaction with the environment. Furthermore, the second expert model is a pre-trained and high-performance model; for example, its parameter count is greater than that of the first model, ensuring high performance. Therefore, because the second expert model has high performance, its behavior of outputting reflection results based on input can be considered an expert trajectory, which can guide the training process of the second model. In other words, the essence of imitation learning is to make the second model imitate the behavior of the second expert model as much as possible, thereby enabling the second model to learn how to output reflection results that improve the interaction between other models and the environment based on the input content.

[0166] The above describes how reinforcement learning and imitation learning are used to continuously update the first and second models after they are deployed. During the training phase before the first and second models are deployed, imitation learning and reinforcement learning can also be used to train them.

[0167] Optionally, the first and second models can be obtained by first training based on imitation learning and then jointly training based on reinforcement learning. That is, in the initial training phase, the first and second models are first trained based on imitation learning (for example, using the first and second training data mentioned above to train the first and second models), and then the first and second models are jointly trained based on reinforcement learning.

[0168] Generally, during the initial training phase of a model, the parameters are usually randomly initialized, resulting in relatively poor model performance. Directly training the model using reinforcement learning may hinder its ability to quickly learn effective knowledge, leading to low training efficiency. Therefore, this approach first trains two models through imitation learning, enabling them to mimic the behavior of an expert model and quickly learn effective knowledge, thus improving training efficiency. Furthermore, after imitation learning, reinforcement learning is then applied to both models to further enhance the performance of the resulting first and second models.

[0169] Specifically, imitation learning often relies on expert demonstrations (i.e., training data based on expert models), meaning the model can only learn behaviors present in the dataset and struggles to surpass the performance of expert models. Secondly, imitation learning lacks the ability to learn from broader reward signals (such as negative feedback), which can lead to suboptimal results when handling complex tasks. Therefore, after performing imitation learning on the model, learning environmental information through trial and error and a wider range of reward signals allows both the first and second models to subsequently outperform expert models, resulting in better performance.

[0170] The above describes the model processing method provided in this application. To facilitate understanding, the following will use specific examples to explain in detail the model training and model processing methods provided in this application from the perspectives of model training and model usage.

[0171] Please see Figure 9A , Figure 9A A schematic diagram of the system architecture used in the model training method provided in this application. (See diagram for example.) Figure 9A As shown, the system architecture includes a first expert model, a second expert model, a first model, and a second model. The first expert model acts as the teacher planner, and the second expert model acts as the teacher reflector. Together, they constitute an expert agent. The first and second expert models are typically large-scale language models responsible for interacting with the environment to generate expert trajectories (i.e., training data used to train the first and second models).

[0172] The first model acts as the student planner, and the second model acts as the student reflector. Together, they constitute a student agent. Typically, the first model (student planner) has fewer parameters than the first expert model (teacher planner), and the second model (student reflector) has fewer parameters than the second expert model (teacher reflector), to ensure lower inference latency and resource consumption. Since the first and second models are insufficient for the agent's tasks during initial training, this application requires them to imitate the behavior of the first and second expert models for initialization (i.e., imitation learning). After initialization, the first and second models exhibit good performance. Therefore, this application further requires them to interact with the environment, thereby using reinforcement learning to train the first and second models and jointly optimize the student agent's planning and reflection abilities.

[0173] Please see Figure 9B , Figure 9B This is a flowchart illustrating a model training method provided in this application. Figure 9BAs shown, the training process of the first model as a student planner and the second model as a student reflector can be divided into two stages: imitation learning and reinforcement learning.

[0174] like Figure 9B As shown in (a), in the first stage, the teacher planner interacts with the environment, and the teacher reflector generates reflective results (i.e., language rewards) based on the feedback from the environment and the teacher planner, thereby promoting better planning by the teacher planner. In this way, by collecting data on the interaction between the teacher planner and the teacher reflector and the environment, a set of expert trajectories can be obtained. After evaluation by an evaluator, this set of expert trajectories can be filtered to obtain high-quality expert trajectories to form an expert dataset. Based on the expert dataset, supervised learning can be performed on the student planner and student reflector, thereby enabling fine-tuning of the student planner and student reflector.

[0175] like Figure 9B As shown in (b), in the second stage, the student planner interacts with the environment and receives planner rewards (i.e., rewards for feedback from the environment). Simultaneously, when the environmental feedback received by the student planner is insufficient, the student reflector reflects on the interaction process between the student planner and the environment, outputting a reflection result (i.e., a verbal reward) to improve the student planner's next output. Furthermore, the reflector reward is defined by the reflection effect (i.e., the difference between the planner reward before and after reflection). Thus, based on planner rewards and reflection rewards, joint training of the student planner and student reflector using reinforcement learning can be achieved.

[0176] Specifically, suppose an agent (such as the teacher agent or student agent mentioned above) consists of a planner π and a reflector μ. The planner π interacts directly with the environment to plan actions, while the reflector μ generates verbal reflections to help the planner π iteratively improve the plan.

[0177] Planner π is similar to a policy model in reinforcement learning, used to determine the policy of a given state s. t Time-generated action a t State s t It can be in text format, consisting of task prompts, environment descriptions, and historical interactions. Action a t Specifically, it can include two distinct components: ideas and actions. Ideas represent the planner π's explicit thought process about the task; actions refer to the planner π's actual interactive responses, such as using tools and performing the task. The action 'a' generated by the planner π... t It can be represented by the following formula 1.

[0178] a t=π(s) t ) Formula 1

[0179] The reflector μ is similar to the reward model in reinforcement learning. When faced with complex tasks, agents based on large language models often interact with the environment multiple times to complete the task. The task may fail due to execution errors or encountering specific constraints. When the agent fails on the k-th attempt, it remembers the unsuccessful trajectory τ. k It will be provided to the reflector μ to produce a reflection result f expressed in natural language. k .

[0180] Reflection results f k As a semantic gradient signal, the planner π is improved without updating the parameters. Here, the reflection result f generated by the reflector μ is... k This can be expressed using the following formula 2.

[0181] f k =μ(τ) k ) Formula 2

[0182] After reflection by the reflector μ, the state information input to the planner π for the next iteration is adjusted based on the initial state of the previous trial and the feedback received. in, The state information of the planner π is input for the (k+1)th iteration. The state information of planner π is input for the kth time.

[0183] In summary, the overall goal is to iteratively optimize the planner π through feedback provided by the reflector μ, aiming to optimize decisions and maximize the cumulative reward of each trial.

[0184] In the imitation learning process, this application first creates an expert dataset using a first expert model as a teacher planner and a second expert model as a teacher reflector. Then, this application uses the expert dataset to fine-tune the first model as a student planner and the second model as a student reflector, enabling the smaller-scale first and second models to improve performance by learning from the demonstrations of the larger-scale first and second expert models, thus quickly adapting to new tasks. Specifically, the imitation learning process can be divided into two parts: expert data collection and imitation learning training.

[0185] 1. Expert data collection.

[0186] Let π be the teacher planner and the teacher reflector. expert and μ expertThe teacher planner and teacher reflector interact extensively with various environments in the training set to generate expert trajectories for fine-tuning. This application implements different rule-based evaluators for the expert trajector trajector and teacher reflector across multiple datasets. Based on the evaluators, high-quality expert trajectories can be selected from the collected trajectories to create a planner training dataset. and reflector training dataset Among them, the planner training dataset This includes content on how the teacher planner interacts with the environment, and a reflector training dataset. This includes the content of the teacher planner's interaction with the environment and the reflection results provided by the teacher reflector for the content of the interaction.

[0187] 2. Imitation learning training.

[0188] During the imitation learning training process, supervised learning is used to fine-tune the student planner π. θ and student reflector μ Φ This allows the student planner and student reflector to initially adapt to agent tasks. Among them, the student planner π... θ The training objective is to enable the student planner π θ Action distribution π(a|s) and action distribution π of the teacher planner expert (a|s) should be as consistent as possible, student reflector μ Φ The training objective is to enable students to reflect their μ function. Φ Action distribution μ(f|τ) and action distribution μ of the teacher reflector expert (a|s) should be as consistent as possible. Specifically, the student planner π θ The training objective can be represented by the following formula 3, student reflector μ Φ The training objective can be represented by the following formula 4.

[0189]

[0190]

[0191] in, This represents the loss function of the student planner during the imitation learning and training process. Let E[] represent the loss function of the student planner during the imitation learning training process, and E[] represent the expectation. The goal of imitation learning is to minimize the loss function of the student planner during the training process. and

[0192] While imitation learning has yielded significant results in training student planners and student reflectors, it still has several limitations. First, imitation learning relies on expert demonstrations, meaning that student planners and reflectors can only learn behaviors present in the dataset, making it difficult to surpass the performance of teacher planners and reflectors. Second, imitation learning lacks the ability to learn from broader reward signals (such as negative feedback), leading to suboptimal results when handling complex tasks. To address these issues, this application uses a joint policy gradient algorithm based on reinforcement learning to simultaneously optimize both the student planner and student reflector. By learning environmental information through trial and error and a wider range of reward signals, the student planner and student reflector can surpass expert demonstrations and achieve superior performance. Specifically, the reinforcement learning process can include the following two steps: reward design and joint policy gradient optimization.

[0193] 1. Reward design.

[0194] The reward function R of the student planner π The reward R provided by the environment τ The reward R τ Based on the dataset definition, such as task completion rate, a non-linear reward function Rμ is specifically designed for the student reflector. This function is defined as the difference in environmental reward between the reflected trial and the previous failed trial. Furthermore, a coefficient is introduced for the student reflector to link the student reflector's reward to the value of the current state, adjusting as the difficulty of increasing the value increases. In summary, in the k-th trial, the reward function of the student planner... The reward function of the student reflector can be expressed using Formula 5. Formula 6 can be used to represent this.

[0195]

[0196]

[0197] in, This represents the reward value obtained based on the feedback from the environment in the k-th trial; α represents the reward value obtained based on the feedback result of the environment in the (k+1)th trial; α represents the coefficient.

[0198] 2. Joint strategy gradient optimization.

[0199] To address the high inference and training costs of large language models and the low sample utilization in online reinforcement learning algorithms, this application proposes an offline joint policy gradient optimization algorithm. Specifically, for the student planner and student reflector, a replay buffer can be constructed to store the historical trajectories of the student planner and student reflector. In each iteration, these historical trajectories are used for offline optimization of the student planner and student reflector based on reinforcement learning. Specifically, the optimization objectives for the student planner and student reflector are shown in Equations 7 and 8.

[0200]

[0201]

[0202] in, The loss function of the student planner during the reinforcement learning training process (i.e., the first loss value obtained based on the first reward value); The loss function of the student reflector during reinforcement learning training (i.e., the second loss value obtained above based on the difference between the second reward value and the first reward value); w π (s,a) and w μ (τ,f) represents the importance sampling weights. Considering that the importance sampling coefficients may introduce significant variance, this problem can be mitigated by applying a cutoff function to restrict the importance sampling terms to the interval {1-∈, 1+∈}.

[0203] Furthermore, the knowledge gained during reinforcement learning may conflict with the knowledge gained during the preceding imitation learning process, causing the model to forget previous knowledge and thus leading to performance degradation. Therefore, based on reinforcement learning, a regularization term based on imitation learning can be introduced to mitigate this effect and stabilize the training process. Finally, the enhanced loss functions of the student planner and student reflector can be expressed by Equations 9 and 10.

[0204]

[0205]

[0206] in, The enhanced loss function for the student planner during reinforcement learning training (i.e., the first loss value obtained above based on the first reward value and the first sub-loss value); λ is the augmented loss function for the student reflector during reinforcement learning training (i.e., the second loss value obtained above based on the difference between the second reward value and the first reward value, and the second sub-loss value); π and λ μ It is a regularized weight that balances the influence of reinforcement learning objectives and imitation learning objectives.

[0207] Taking the intelligent agent task as a question-and-answer task as an example, the process of training the student planner and student reflector in practical applications is as follows.

[0208] 1. Assume the training set contains 10 question-answering tasks. An expert agent, consisting of a teacher planner and a teacher reflector, completes these 10 tasks in n rounds. In each round, the expert agent can search Wikipedia 5 times to gather relevant information to answer the questions. If the answer is incorrect or the search count exceeds five, the task fails. If successful, it doesn't try again; if the task fails, the reflector summarizes the reason for the previous failure. For example, "Last time I got stuck in an infinite loop while searching for a certain keyword; next time I will try a different keyword." The expert agent repeats this process until it successfully completes the task or reaches the maximum number of attempts.

[0209] 2. Assume the expert agent ultimately completes 8 out of 10 tasks, with 3 tasks completed on the first attempt and 5 tasks completed after reflection. An evaluator is used to filter the expert trajector paths obtained from the 10 tasks, and the 8 completed planner trajector paths are used as the planner training dataset. And 5 reflections that help the planner complete the task are used as the training dataset for the reflector.

[0210] 3. Based on supervised learning (i.e., formulas 3 and 4 above), respectively adopt... and Fine-tune the student planner and student reflector.

[0211] 4. The 10 tasks are completed using a finely tuned student planner and a finely tuned student reflector, with a process similar to that of an expert agent, and the obtained trajectories are stored in the replay buffer.

[0212] 5. After collecting a batch of completed task trajectories, use the reward function to calculate the reward for the student planner trajectory and the student reflector trajectory respectively.

[0213] 6. Using the collected trajectories, reinforcement learning (i.e., formulas 9 and 10) is used to continue training the student planner and student reflector in the student agent, thereby obtaining the trained student planner (i.e., the first model) and the trained student reflector (i.e., the second model).

[0214] It should be noted that after training the student planner and student reflector based on the above process, the first and second models are obtained, and thus can be deployed and applied online. Furthermore, after the first and second models are deployed, they can interact with the environment to obtain new trajectories, which can be stored in the database. In this way, at regular intervals, reinforcement learning (i.e., Equations 9 and 10 above) can be used to update the first and second models based on the new trajectories stored in the database, thereby continuously optimizing their performance.

[0215] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0216] Please see Figure 10 , Figure 10 This is a schematic diagram of a model processing device provided in this application. Figure 10 As shown, the model processing device includes: an acquisition module 1001, used to acquire first state information, which indicates the historical interaction content between the first model and the environment; a processing module 1002, used to input the first state information into the first model to obtain a first action; the processing module 1002 is further used to input the first state information, the first action, and the feedback result into a second model when the feedback result obtained from the execution of the first action does not meet the preset requirements, to obtain a reflection result, which instructs the first model to determine the direction of the action based on the interaction process with the environment; the processing module 1002 is further used to input second state information into the first model to obtain a second action, which includes the first state information, the first action, the feedback result, and the reflection result; wherein, the first model and the second model are obtained based on joint training of reinforcement learning.

[0217] In one possible implementation, the processing module 1002 is further configured to:

[0218] Joint training of the first and second models is performed based on reinforcement learning to obtain updated first and second models;

[0219] In the process of jointly training the first model and the second model, the first model is updated based on the first loss value, the second model is updated based on the second loss value, the first loss value is obtained based on the first reward value, the first reward value is obtained based on the feedback result, the second loss value is obtained based on the difference between the second reward value and the first reward value, and the second reward value is obtained based on the feedback from the environment when the second action is performed.

[0220] In one possible implementation, the first loss value is specifically obtained based on the first reward value and the first sub-loss value, which is the loss function value obtained by training the first model using imitation learning;

[0221] The second loss value is specifically obtained based on the difference and the second sub-loss value, which is the loss function value obtained by training the second model using imitation learning.

[0222] In one possible implementation, the first sub-loss value is obtained by training the first model based on the first training data, which includes the interaction content between the first expert model and the environment.

[0223] The second sub-loss value is obtained by training the second model based on the second training data, which includes the reflection results output by the second expert model based on the interaction content between the first expert model and the environment.

[0224] In one possible implementation, the first and second models are obtained by first training based on imitation learning and then jointly training based on reinforcement learning.

[0225] In one possible implementation, the processing module 1002 is further configured to:

[0226] Input the first state information into the first model to obtain the component call instructions output by the first model;

[0227] Based on the component invocation instruction, the target component is invoked and executed to obtain the first action.

[0228] In one possible implementation, the target component includes at least one of a database query engine, a search engine, or a code execution engine.

[0229] In one possible implementation, the feedback result does not meet the preset requirements, including:

[0230] The feedback result was negative feedback regarding the first action.

[0231] In one possible implementation, both the first and second models are large language models.

[0232] In one possible implementation, the first model is used to perform at least one of the following tasks: question answering, content recommendation, interactive programming, embodied intelligence, or content generation.

[0233] Both the acquisition module 1001 and the processing module 1002 can be implemented in software or in hardware. For example, the implementation of the processing module 1002 will be described below. Similarly, the implementation of the acquisition module 1001 can be referenced to that of the processing module 1002.

[0234] As an example of a software functional unit, processing module 1002 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module A may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0235] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0236] As an example of a hardware functional unit, the processing module 1002 may include at least one computing device, such as a server. Alternatively, the processing module 1002 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0237] The processing module 1002 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1002 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1002 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0238] Please see Figure 11 , Figure 11 A schematic diagram of the structure of a computing device provided in this application. Figure 11 The computing device 1100 shown can be used to execute the task processing method provided in this embodiment. For example... Figure 11 As shown, the computing device 1100 includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0239] Bus 1102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 11 The bus 1102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1102 may include a path for transmitting information between various components of the computing device 1100 (e.g., memory 1106, processor 1104, communication interface 1108).

[0240] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0241] The memory 1106 may include volatile memory, such as random access memory (RAM). The processor 1104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0242] The memory 1106 stores executable program code, and the processor 1104 executes the executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the model processing method described above. That is, the memory 1106 stores instructions for executing the model processing method.

[0243] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0244] It should be understood that the computing device 1100 according to this application is used to perform, for example... Figure 2 The model processing method described herein can correspond to the respective subject executing the method according to this application, and for the sake of brevity, it will not be described in detail here.

[0245] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0246] Please see Figure 12 , Figure 12 This is a schematic diagram of a computing device cluster provided in this application. Figure 12 As shown, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for executing model processing methods.

[0247] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the model processing method. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions for executing the model processing method.

[0248] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0249] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 One possible implementation method is shown. Figure 13 This is a schematic diagram of another computing device cluster structure provided in this application. Figure 13 As shown, in the computing device cluster 1300, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 1106 in computing device 1100A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1106 in computing device 1100B stores instructions for executing the functions of the processing module.

[0250] It should be understood that Figure 13 The functions of computing device 1100A shown can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0251] It should be understood that the computing device 1100 or computing device cluster 1300 in this application may correspond to this application. Figure 10 The task processing device in the application can be used to execute the task processing device described herein. Figure 2 The corresponding main body of the model processing method. Furthermore, the above and other operations and / or functions of each module in computing device 1100 or computing device cluster 1300 are respectively for implementing... Figure 2 For the sake of brevity, the corresponding process of the model processing method will not be described in detail here.

[0252] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0253] For details, please refer to Figure 14 , Figure 14 This application provides a schematic diagram of a chip structure, which can be represented as a neural processing unit (NPU), a graphics processing unit (GPU), or a tensor processing unit (TPU). The following description uses an NPU 1400 as an example. The NPU 1400 is mounted as a coprocessor on the host CPU, which allocates tasks. The core of the NPU is the arithmetic circuit 1403, which is controlled by a controller 1404 to retrieve matrix data from memory and perform multiplication operations.

[0254] In some implementations, the arithmetic circuit 1403 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1403 is a two-dimensional pulsating array. The arithmetic circuit 1403 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1403 is a general-purpose matrix processor.

[0255] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1402 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1401 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1408.

[0256] Unified memory 1406 is used to store input and output data. Weight data is directly transferred to weight memory 1402 via Direct Memory Access Controller (DMAC) 1405. Input data is also transferred to unified memory 1406 via DMAC.

[0257] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1409.

[0258] The Bus Interface Unit (BIU) 1410 is used by the instruction fetch memory 1409 to fetch instructions from external memory, and also by the memory access controller 1405 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0259] The DMAC is mainly used to move input data from external memory DDR to unified memory 1406, or to weight data to weight memory 1402, or to input data to input memory 1401.

[0260] The vector computation unit 1407 includes multiple arithmetic processing units that, when needed, further process the output of the computation circuit 1403, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0261] In some implementations, the vector computation unit 1407 can store the processed output vector in the unified memory 1406. For example, the vector computation unit 1407 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1403, such as performing linear interpolation on feature planes extracted from a convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1407 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1403, for example, for use in subsequent layers of the neural network.

[0262] The instruction fetch buffer 1409 connected to the controller 1404 is used to store the instructions used by the controller 1404;

[0263] Unified memory 1406, input memory 1401, weighted memory 1402, and instruction fetch memory 1409 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0264] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0265] It should be understood that this application Figure 14 The chip in this application may correspond to the chip in this application. Figure 10 The task processing device in the application, or deployed on the computing device 1100 or computing device cluster 1300 of this application. Furthermore, this application... Figure 14 The chip in the application may correspond to the one that executes this application. Figure 2 The corresponding main body of the model processing method, and the above and other operations and / or functions of each module in the chip are respectively implemented to achieve... Figure 2 The corresponding process of the model processing method is omitted here for brevity. Furthermore, in this application, the chip structure is not subject to... Figure 14 The chip structure shown may be limited to more or fewer hardware structures to achieve the aforementioned. Figure 2 The function of the method shown.

[0266] Please refer to Figure 15 , Figure 15 This is a schematic diagram of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 2 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0267] Figure 15 A conceptual partial view of an example computer-readable storage medium arranged according to at least some embodiments shown herein is schematically illustrated. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device. In one embodiment, the computer-readable storage medium 1500 is provided using a signal bearer medium 1501. The signal bearer medium 1501 may include one or more program instructions 1502 that, when executed by one or more processors, can provide the above-described instructions for… Figure 2 The described function or part of the function.

[0268] In some examples, the signal carrying medium 1501 may include a computer-readable medium 1503, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0269] In some embodiments, the signal-bearing medium 1501 may comprise a computer-recordable medium 1504, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1501 may comprise a communication medium 1505, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1501 may be transmitted by a wireless communication medium 1505 (e.g., a wireless communication medium conforming to the IEEE 1202.X standard or other transmission protocols).

[0270] One or more program instructions 1502 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1502 conveyed to the computing device via a computer-readable medium 1503, a computer-recordable medium 1504, and / or a communication medium 1505.

[0271] It should be understood that the computer-readable storage medium 1500 in this application can be deployed on Figure 10 The task processing apparatus described herein, or the computing device 1100 or computing device cluster 1300 of this application, is used to implement the task processing apparatus, computing device 1100, or computing device cluster 1300 provided herein by reading one or more program instructions 1502 from the computer-readable storage medium 1500. Figure 2 The model processing method introduced.

[0272] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0273] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0274] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0275] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0276] The above description is merely a specific embodiment of the present invention. Those skilled in the art can conceive of variations or substitutions based on the specific embodiments provided in this application, and all such variations or substitutions should be covered within the protection scope of this application.

Claims

1. A model processing method characterized by comprising: The method comprises: obtaining first state information, the first state information being used to indicate historical interaction content of a first model with an environment; inputting the first state information into the first model to obtain a first action; when a feedback result of the environment obtained by executing the first action does not satisfy a preset requirement, inputting the first state information, the first action and the feedback result into a second model to obtain a reflection result, the reflection result being used to indicate that the first model determines a direction of an action based on an interaction process with the environment; inputting second state information into the first model to obtain a second action, the second state information comprising the first state information, the first action, the feedback result and the reflection result; wherein the first model and the second model are obtained based on joint training of reinforcement learning.

2. The method of claim 1, wherein, The method further comprises: performing joint training of the first model and the second model based on reinforcement learning to obtain an updated first model and an updated second model; wherein, in the process of performing joint training of the first model and the second model, the first model is updated based on a first loss value, the second model is updated based on a second loss value, the first loss value is obtained based on a first reward value, the first reward value is obtained based on the feedback result, the second loss value is obtained based on a difference between the second reward value and the first reward value, and the second reward value is obtained based on feedback of the environment when the second action is executed.

3. The method of claim 2, wherein, The first loss value is specifically obtained based on the first reward value and a first sub-loss value, and the first sub-loss value is a loss function value obtained by training the first model using imitation learning; The second loss value is specifically obtained based on the difference and a second sub-loss value, and the second sub-loss value is a loss function value obtained by training the second model using imitation learning.

4. The method of claim 3, wherein, The first sub-loss value is obtained by training the first model based on first training data, and the first training data comprises interaction content of a first expert model with the environment; The second sub-loss value is obtained by training the second model based on second training data, and the second training data comprises a reflection result output by a second expert model based on the interaction content of the first expert model with the environment.

5. The method according to any one of claims 1 to 4, characterized in that, The first model and the second model are obtained based on joint training of reinforcement learning after being trained based on imitation learning.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: obtaining first state information, the first state information being used to indicate historical interaction content of a first model with an environment; inputting the first state information into the first model to obtain a first action; 7. The method of claim 6, wherein, when a feedback result of the environment obtained by executing the first action does not satisfy a preset requirement, inputting the first state information, the first action and the feedback result into a second model to obtain a reflection result, the reflection result being used to indicate that the first model determines a direction of an action based on an interaction process with the environment; 8. The method according to any one of claims 1 to 7, characterized in that, inputting second state information into the first model to obtain a second action, the second state information comprising the first state information, the first action, the feedback result and the reflection result; wherein the first model and the second model are obtained based on joint training of reinforcement learning.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: performing joint training of the first model and the second model based on reinforcement learning to obtain an updated first model and an updated second model; wherein, in the process of performing joint training of the first model and the second model, the first model is updated based on a first loss value, the second model is updated based on a second loss value, the first loss value is obtained based on a first reward value, the first reward value is obtained based on the feedback result, the second loss value is obtained based on a difference between the second reward value and the first reward value, and the second reward value is obtained based on feedback of the environment when the second action is executed. The first loss value is specifically obtained based on the first reward value and a first sub-loss value, and the first sub-loss value is a loss function value obtained by training the first model using imitation learning; The second loss value is specifically obtained based on the difference and a second sub-loss value, and the second sub-loss value is a loss function value obtained by training the second model using imitation learning. The first sub-loss value is obtained by training the first model based on first training data, and the first training data comprises interaction content of a first expert model with the environment; The second sub-loss value is obtained by training the second model based on second training data, and the second training data comprises a reflection result output by a second expert model based on the interaction content of the first expert model with the environment. The first model and the second model are obtained based on joint training of reinforcement learning after being trained based on imitation learning. The method further comprises: obtaining first state information, the first state information being used to indicate historical interaction content of a first model with an environment; inputting the first state information into the first model to obtain a first action; when a feedback result of the environment obtained by executing the first action does not satisfy a preset requirement, inputting the first state information, the first action and the feedback result into a second model to obtain a reflection result, the reflection result being used to indicate that the first model determines a direction of an action based on an interaction process with the environment; inputting second state information into the first model to obtain a second action, the second state information comprising the first state information, the first action, the feedback result and the reflection result; wherein the first model and the second model are obtained based on joint training of reinforcement learning. The first model and the second model are both large language models.

10. The method according to any one of claims 1 to 9, characterized in that, The first model is used to perform at least one of a question and answer task, a content recommendation task, an interactive programming task, an embodied intelligence task, or a content generation task.

11. A model processing apparatus characterized by comprising: Comprise: An acquisition module is configured to acquire first state information, the first state information being used to indicate historical interaction content of a first model and an environment; A processing module is configured to input the first state information into the first model to obtain a first action; The processing module is further configured to input the first state information, the first action, and a feedback result of performing the first action into a second model to obtain a reflection result when the feedback result of the environment obtained by performing the first action does not satisfy a preset requirement, the reflection result being used to indicate that the first model determines the direction of the action based on the interaction process with the environment; The processing module is further configured to input second state information into the first model to obtain a second action, the second state information comprising the first state information, the first action, the feedback result, and the reflection result; The first model and the second model are obtained based on joint training of reinforcement learning.

12. The apparatus of claim 11, wherein, The processing module is further configured to: Perform joint training of the first model and the second model based on reinforcement learning to obtain an updated first model and an updated second model; In the process of performing joint training of the first model and the second model, the first model is updated based on a first loss value, the second model is updated based on a second loss value, the first loss value is obtained based on a first reward value, the first reward value is obtained based on the feedback result, the second loss value is obtained based on a difference between the second reward value and the first reward value, and the second reward value is obtained based on feedback of the environment when the second action is performed.

13. The apparatus of claim 12, wherein, The first loss value is specifically obtained based on the first reward value and a first sub-loss value, and the first sub-loss value is a loss function value obtained by training the first model using imitation learning; The second loss value is specifically obtained based on the difference and a second sub-loss value, and the second sub-loss value is a loss function value obtained by training the second model using imitation learning.

14. The apparatus of claim 13, wherein, The first sub-loss value is obtained by training the first model based on first training data, and the first training data comprises interaction content of a first expert model and the environment; The second sub-loss value is obtained by training the second model based on second training data, and the second training data comprises a reflection result output by a second expert model based on the interaction content of the first expert model and the environment.

15. The apparatus of any of claims 11-14, wherein, The first model and the second model are obtained based on joint training of reinforcement learning after being trained based on imitation learning.

16. The apparatus of any one of claims 11-15, wherein, The processing module is further configured to: Input the first state information into the first model to obtain component calling instructions output by the first model; Based on the component calling instructions, a target component is called and executed to obtain the first action.

17. The apparatus of claim 16, wherein, The target component comprises at least one of a database query engine, a search engine, or a code execution engine.

18. The apparatus of any of claims 11-17, wherein, The feedback result does not meet preset requirements, including: The feedback result is negative feedback for the first action.

19. The apparatus of any of claims 11-18, wherein, The first model and the second model are both large language models.

20. The apparatus of any of claims 11-19, wherein, The first model is used to perform at least one of a question and answer task, a content recommendation task, an interactive programming task, an embodied intelligence task, or a content generation task.

21. A computing device, comprising: The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 10.

22. A cluster of computing devices, characterized in that, The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 10. The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 10.

23. A computer storage medium, comprising, The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 10.

24. A computer program product, characterised in that, ​

Images