Agent-based collaborative reasoning and action large language model learning method

Abstract

The invention discloses a big language model learning method for collaborative reasoning and action based on agency, which comprises the following steps: defining a task demand, and initializing a universal prompt vector; designing a support data set of small sample scene learning, so as to enable the large language model agent learning to cooperate with the logic of reasoning and action; designing an interface for calling an external tool and acquiring external information by the model agent; selecting a proper tool combination from a preset tool library; through an agent packaging prompt, an external tool and a large language model, the designed support data set serves as a prompt word and is submitted to the large language model for agent so as to carry out small sample learning, and a'reasoning-action 'cooperative normal form is mastered; and applying the trained model agent to a new task instance, evaluating an output effect, correcting a logic error or a fact error through human intervention, and performing adjustment and optimization. The method is visual, easy to design, universal, flexible, efficient, stable, aligned with human thinking logic and highly controllable.

Description

[Technical field]

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large language model learning method based on agent-based collaborative reasoning and action. [Background technology]

[0002] The large language model is a breakthrough in the field of artificial intelligence in recent years. It is a deep learning model that learns rich semantic knowledge and language patterns from large-scale unstructured text data based on self-supervised learning methods. It has achieved unprecedented performance in various natural language processing tasks. Compared with traditional methods that rely on task-specific feature engineering, large language models adopt a more general and robust learning method. They can apply the general language capabilities obtained from pre-training to specific downstream tasks in a transfer learning manner, greatly improving the generalization ability and practicality of natural language processing technology.

[0003] However, existing large language models also expose some key limitations. Large language models perform poorly for tasks that require the use of common sense knowledge and complex reasoning. They have difficulty in making reasonable inferences and logical analysis, and may exhibit inconsistent or problematic behaviors, such as fantasizing, generating defective code, and generating harmful content. Large language models are trained based on data, and they lack human intuition and judgment. Therefore, when faced with new scenarios and new tasks, they often cannot accurately perceive the specific situation and respond appropriately. Large language models do not interact enough with the external environment. Currently, most large language models rely mainly on static text training data and lack in-depth interaction and experience with the external environment. These limitations restrict the use of large language models in broader and more professional application scenarios to a certain extent. In view of this, it is necessary to provide a large language model learning method based on agent-based collaborative reasoning and action to overcome the above defects. [Summary of the invention]

[0004] The purpose of this invention is to provide a large language model learning method based on agent-based collaborative reasoning and action, aiming to solve the problem that most existing large language models mainly rely on static text training data and lack in-depth interaction and experience with the external environment, so that the model can dynamically generate reasoning logic and interact with the external environment, thereby improving the efficiency and reliability of solving complex tasks.

[0005] In order to achieve the above object, the present invention provides a large language model learning method based on agent-based collaborative reasoning and action, comprising:

[0006] Step S10: clarify the task requirements and initialize a general prompt vector as the starting point for learning the large language model; wherein the prompt vector includes a guide, context information, task description, output format indication, restriction conditions, and sample output;

[0007] Step S20: Design a support dataset for small sample scenario learning to enable the large language model agent to learn the logic of collaborative "reasoning" and "action"; wherein each manually annotated sample in the support dataset contains an input question and a detailed record of the "actions", "thinking" and environmental "observations" involved in solving the problem by real humans in the scenario;

[0008] Step S30: Designing an interface for the large language model agent to call external tools and obtain external information;

[0009] Step S40: According to the task requirements, select a suitable tool combination from the preset tool library, coordinate the input and output formats between the tools, and then select an adapted large language model, input the information provided by each tool into the large language model for reasoning and output generation;

[0010] Step S50: encapsulate prompts, external tools and large language models through agents, and hand over the designed support data set as prompts to the large language model agent for small sample learning to master the "reasoning-action" collaborative paradigm;

[0011] Step S60: Apply the trained large language model agent to the new task instance, evaluate the output effect, correct its logical errors or factual errors through human intervention, and perform tuning.

[0012] In a preferred embodiment, the large language model is integrated into the intelligent virtual human dialogue system, which specifically includes the following steps:

[0013] When the user initiates a conversation, the speech recognition module is called to recognize the user's speech input as the corresponding text input content;

[0014] The recognized text input is passed as the input question / task to the large language model agent, which then generates a textual answer through logical analysis and "reasoning-action" collaboration;

[0015] The text output is passed to the text-to-speech module and generates audio speech output close to real human voice;

[0016] The audio speech output is handed over to the 3D graphics rendering module to generate a virtual digital human animation that matches the speech, so as to feed back the text output of the large language model agent to the user in a form close to a real-life conversation.

[0017] In a preferred embodiment, in step S60, if a logical error or a factual error occurs in the agent's logic chain, the "thinking" where the logical error or the factual error first occurs is located according to the "reasoning-action" logic chain output by the agent, and the "thinking" is manually edited to correct the logic or facts after the error, so that the agent can re-output the subsequent content of the "thinking" after the correction.

[0018] In a preferred embodiment, it also includes:

[0019] The framework for building a large language model agent to interact with the environment to solve tasks is as follows:

[0020] For a task x that needs to be solved, at time node t, the large language model agent will analyze the current task environment and progress and obtain an "observation" At the same time, according to the decision strategy Π(a t |c t ) Make an "action" Among them, c(o1, a1, ..., o t-1 , a t-1 , o t ) represents the context of the current task, Represents the mapping from the context in which the model agent is located to the behavioral decisions;

[0021] The model agent's "action" a t Expand the behavior space in, The language space of the model representing the complement, the behavior in the language space Represents the "thinking" of the large language model agent on the current task process; the "thinking" of the large prediction model The current problem context t The information contained in the problem is integrated to update the current problem context This helps the big prediction model make subsequent behavioral decisions Generate more logical decisions in the process;

[0022] Specifically, the original input problem x is decomposed into a series of relatively easy-to-solve sub-problems x t , the model agent solves the current problem context c step by step in a step-by-step manner t problems that need to be solved, and generate new "observations" and "thoughts" to update the context t+1 , where the decision strategy of the model agent Π(a t |c t )Only the current subtask needs to be mapped.

[0023] In a preferred embodiment, when the large prediction model is “observed” t When it is discovered that there is unknown information in the current problem context, the large language model will infer the means that can be used to resolve the unknown information and incorporate the means into the agent's "thinking" Then the agent will take the next "action" a t+1 Invoke corresponding tools to solve corresponding subtasks; wherein the means include: using search engines and encyclopedias to search and acquire knowledge, and obtaining fields to be queried through interaction with databases.

[0024] In a preferred embodiment, in step S50, the support data set D is divided into a training set D train , validation set D val and the test set D test ;D train As auxiliary set A, from D val and D test Random sampling forms a large number of evaluation tasks T;

[0025] Among them, for a sampling task, there are two subsets A S and A Q Composition, corresponding to the support set S and query set Q respectively;

[0026] The small sample learning includes a two-step optimization between the base learner and the meta learner; in the inner loop of the small sample learning, A S Perform fast learning to find the optimal parameters of the basic learner; in the outer loop of small sample learning, through A Q To calculate the gradient of each new task relative to the optimal parameters, the randomly initialized model parameters are then updated to adapt to different tasks, thereby enhancing the generalization ability of the large speech model.

[0027] In a preferred embodiment, if the solution of the task requires obtaining external information, the information source involved is indicated in "Thinking" and the required information is described;

[0028] If the task involves the use of external tools, specify the tools needed to solve the current subtask and the purpose of using the tools in "Thinking", and call the corresponding external tools in "Action";

[0029] In any "thinking" step, if the context of the current task already contains all the necessary information for the large language model agent to answer the original input question, then write the answer to the question in "thinking", and in the next "action", output the answer to the question and terminate the process;

[0030] For tasks that mainly rely on logical reasoning, "thinking" and "acting" are generated alternately so that the task-solving trajectory contains multiple "thinking-acting-observing" steps;

[0031] For decision-making tasks involving a large number of "actions", "thinking" will only appear more sparsely in the most needed positions in the task trajectory relative to logical reasoning tasks, allowing the large language model agent to decide on the asynchronous generation of "thinking" and "actions" on its own.

[0032] In a preferred embodiment, in step S60, tuning is performed by means of prompt fine-tuning, and the prompt fine-tuning includes hard prompts; wherein the hard prompts are manually preset discrete tags, which guide the output of the large language model by modifying pre-defined semantic and grammatical rules.

[0033] In a preferred embodiment, in step S60, the tuning is performed by means of prompt fine-tuning, wherein the prompt fine-tuning includes soft prompts; the specific implementation steps of the soft prompts are as follows:

[0034] Adjust the general cue vector representation, add some new cue information according to the specific needs of the target task, and then concatenate these new cue vectors with the general cue vector to form a composite cue vector; the concatenation methods include: simple concatenation, weighted concatenation, hierarchical concatenation, conditional concatenation, and encoding-decoding concatenation;

[0035] In the small sample learning stage, some trainable soft prompts are introduced, including task-related words and phrases, to help the large language model understand and generate language expressions in specific fields, describe reasoning steps, and action strategies, thereby guiding the large language model to make more reasonable reasoning and actions; then adjust the trainable soft prompts based on the feedback information to gradually optimize the model behavior.

[0036] In a preferred embodiment, in step S60, the optimization is performed by fine-tuning the model, and the specific implementation steps are as follows:

[0037] Designing a training set for fine-tuning a large language model; wherein the task solution trajectory of each sample in the training set is designed in the same manner as in step S20, and the number of samples exceeds the number of samples in small sample learning;

[0038] The training set is handed over to the large language model for training and fine-tuning, and some parameters of the model are updated so that the model can better grasp the thinking paradigm of collaborative "reasoning" and "action".

[0039] The agent-based collaborative reasoning and action-based large language model learning method provided by the present invention has the following beneficial effects compared with the existing large language model:

[0040] (1) Intuitive and easy to design and implement: The prompt design of the framework is very straightforward and intuitive. For each task solution trajectory in a sample, during the annotation process, it is only necessary to record the human "thinking" in natural language based on the intuition and problem-solving logic at each step to form a complete logical chain.

[0041] (2) General and flexible: Due to the flexible thinking space of the large model agent and the "reasoning-action" logic that is applicable to solving almost all problems and tasks, the framework can be easily migrated to various tasks with different action spaces and reasoning requirements, including but not limited to question answering, implementation verification, mathematical problems, numerical analysis, word games, etc.;

[0042] (3) Efficient and stable: This framework only requires a small amount of scenario samples to learn, which can enable the large language model agent to demonstrate strong generalization capabilities on new task instances. Its accuracy in solving tasks and answering questions is much higher than that of the baseline large language model agent that does not use this framework.

[0043] (4) Aligned with human thinking logic and highly controllable: The framework ensures that the large language model agent can output a highly interpretable decision sequence and reasoning process, allowing users to easily check the accuracy of reasoning and facts in each step. In addition, users of the framework can also control the agent's behavior in real time and correct errors by editing the "thinking" output by the agent.

Brief Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A flowchart of a large language model learning method based on agent-based collaborative reasoning and action provided by the present invention;

[0046] Figure 2 A logical diagram of the collaborative "reasoning" and "action" of the large language model agent provided by the present invention;

[0047] Figure 3 The core idea of ​​small sample learning provided by the present invention;

[0048] Figure 4 The workflow diagram of the intelligent virtual digital human dialogue system with a large language model agent proposed by the present invention;

[0049] Figure 5A framework diagram of the large language model learning device based on agent-based collaborative reasoning and action provided by the present invention. [Specific implementation method]

[0050] In order to make the purpose, technical solution and beneficial technical effect of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the specific implementation methods described in this specification are only for explaining the present invention, not for limiting the present invention.

[0051] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0052] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0053] Embodiment 1

[0054] In an embodiment of the present invention, a large language model learning method based on agent-based collaborative reasoning and action is provided, which is used to propose a general paradigm for large language models to collaborate on "reasoning" and "action", aiming to efficiently and accurately solve various language reasoning and decision-making tasks. The framework prompts large language model agents to generate task-related language reasoning paths and actions in an interleaved manner, so that the model can perform dynamic reasoning to create, maintain and adjust high-level action plans (reasoning to plan actions), and can also interact with the external environment (such as search engines, Wikipedia) to incorporate additional information into the reasoning process (action to assist reasoning).

[0055] like Figure 1 and Figure 2 As shown, the agent-based collaborative reasoning and action large language model learning method includes steps S10-S60.

[0056] In this embodiment, the method first further includes the step of building a framework for a large language model agent to interact with the environment to solve the task, specifically steps S101-S102.

[0057] Step S101: For a task x to be solved urgently, at time node t, the large language model agent will analyze the current task environment and progress and obtain an "observation" At the same time, according to the decision strategy Π(a t |c t) Make an "action" Among them, c(o1,a1,...,o t-1 ,a t-1 ,o t ) represents the context of the current task, Represents the mapping from the context in which the model agent is located to the behavioral decisions.

[0058] In general, training a model to learn a policy Π is a complex and difficult process because the mapping from the context in which the model agent is located to the behavioral decisions is is highly complex and implicit, and it usually requires a lot of computation to implement.

[0059] In order to enable the large language model to draw logically coherent and reliable action decisions from the task context, the following improvements are made to the model agent: Step S102, the action a of the model agent is t Expand the behavior space in, The language space of the model that represents the supplement. Behavior in the language space Represents the "thinking" of the large language model agent's reasoning about the current task process. "Thinking" of the large prediction model Describes a series of subtasks x that the model agent currently needs to solve in order to solve the original input task X t , and the real "action" a t a t Different, the model's "thinking" It will not affect the external environment of the model. In other words, "thinking" without "action" will not produce new "observations". t . Model "Thinking" The essence of the problem is to understand the current problem context c t The information contained in the problem is integrated to update the current problem context This helps the big prediction model make subsequent behavioral decisions Make more logical decisions in the process.

[0060] Specifically, in this way, the original input problem x is decomposed into a series of relatively easy-to-solve sub-problems x t , the model agent solves the current problem context c step by step in a step-by-step manner t problems that need to be solved, and generate new "observations" and "thoughts" to update the context t+1 In this process, the decision-making strategy of the model agent Π(a t |c t ) only needs to be done for the current subtask xt Due to the simpler and more straightforward nature of each sub-problem, the agent’s decision-making is less likely to be wrong or illusory, thus ensuring the logic and reliability of the overall problem solving.

[0061] At the same time, the reasoning action framework can effectively help large language model agents interact with the external environment or use external tools to obtain external information that can support decision-making. t When the current problem context contains unknown information, the model will infer the means that can be used to solve the unknown information (such as using search engines, Wikipedia for search and knowledge acquisition, or obtaining the fields to be queried through interaction with the database) and incorporate it into the agent's "thinking" Then, the agent will take the next action a t+1 The corresponding tool is called to solve the corresponding subtask.

[0062] It should be noted that large language models trained solely on large-scale training data are often difficult to quickly adapt to new task scenarios and have limited generalization capabilities. Therefore, we introduced the idea of ​​small sample learning in this framework to improve the learning and generalization capabilities of the model.

[0063] Combination Figure 3 As shown in Figure 2, the core of small sample learning is to use a small number of samples to simulate the learning method of humans, and quickly adapt and migrate to new tasks through meta-learning. Specifically, small sample learning divides the support dataset D into a training set D train , validation set D val and the test set D test ;D train As auxiliary set A, from D val and D test A large number of evaluation tasks T are formed by random sampling; for a sampling task T, there are two subsets A S and A Q They correspond to the support set S and the query set Q respectively.

[0064] Few-shot learning involves two-step optimization between the base learner and the meta-learner. In the inner loop of few-shot learning, A S Perform fast learning to find the optimal parameters of the basic learner; in the outer loop of small sample learning, through A Q To calculate the gradient of each new task relative to the optimal parameters, the randomly initialized model parameters are then updated to adapt to different tasks, thereby enhancing the generalization ability of the large speech model.

[0065] Applying the idea of ​​small sample learning to the collaborative reasoning and action of large language models can help the model better learn from a small amount of contextual information and generalize to new task scenarios. In the process of interaction between the model and the external environment, small sample learning can also support the model to quickly adapt to new information sources and effectively incorporate them into the decision-making process of reasoning and action, constantly enriching and improving its own knowledge and action strategies.

[0066] Step S10: Clarify the task requirements and initialize a general prompt vector as the starting point for learning the large language model; the prompt vector includes the guide words, context information, task description, output format instructions, restrictions, and sample output. This prompt vector is used as the starting point for learning the large language model, and will be further optimized through small sample learning and fine-tuning.

[0067] Step S20: Design a supporting dataset for small sample scenario learning to enable the large language model agent to learn the logic of collaborative "reasoning" and "action". Each manually labeled sample in the supporting dataset D contains an input question and a detailed record of the "actions", "thinking" and environmental "observations" involved in solving the problem by real humans in the scenario.

[0068] It should be noted that in "Thinking", the original input task is decomposed into a series of subtasks according to the logic of human problem solving; the current "observation" and task context are analyzed to understand the current task progress and point out the sub-problem that needs to be solved most. In "Thinking", we will insert common sense that helps solve the current sub-problem to prevent the model from taking actions that go against common sense or having illusions that deviate from the facts. In "Action", take corresponding actions based on the current "thinking" to deal with the sub-tasks that need to be solved urgently. In "Observation", describe the changes brought about by the previous "action" to the task context environment and summarize its impact on the progress of task completion.

[0069] Step S30: Design an interface for the large language model agent to call external tools and obtain external information.

[0070] Among them, external tools include but are not limited to a certain encyclopedia query interface, search engine call interface, database interaction interface and Python programming interface. For example: design an interface for the agent to use Wikipedia for knowledge acquisition to query and obtain the necessary external information required to solve the current task; for topics involving timeliness, design an interface for the agent to call the search engine to obtain the latest information; design an interface for the agent to interact with the database to query field information related to the input problem; design an interface for the agent to use Python programming to solve problems related to numerical calculation and data processing through programming.

[0071] Step S40: According to the task requirements, select a suitable tool combination from the preset tool library, coordinate the input and output formats between the tools, and then select an adaptive large language model, and input the information provided by each tool into the large language model for reasoning and output generation.

[0072] Furthermore, when the large prediction model is used to “observe” O t When it is discovered that there is unknown information in the current problem context, the large language model will infer the means that can be used to resolve the unknown information and incorporate the means into the agent's "thinking" Then the agent will take the next "action" a t+1 The corresponding tools are called to solve the corresponding subtasks; the means include: using search engines and encyclopedias for searching and knowledge acquisition, and obtaining the fields to be queried through interaction with the database.

[0073] Step S50: By encapsulating prompts, external tools and large language models through agents, the designed support data set is handed over to the large language model agent as prompts for small sample learning to master the "reasoning-action" collaborative paradigm. At the same time, the model will also learn how to flexibly use necessary external tools to solve corresponding subtasks.

[0074] If the task requires external information, the information source involved should be specified in "Thinking" and the required information should be described; if the task involves the use of external tools, the tool required to solve the current subtask and the purpose of using the tool should be specified in "Thinking", and the corresponding external tool should be called in "Action";

[0075] In any "thinking" step, if the context of the current task already contains all the necessary information for the large language model agent to answer the original input question, then write the answer to the question in "thinking", and in the next "action", output the answer to the question and terminate the process;

[0076] For tasks that mainly rely on logical reasoning, "thinking" and "acting" are generated alternately so that the task-solving trajectory contains multiple "thinking-acting-observing" steps;

[0077] For decision-making tasks involving a large number of "actions", "thinking" will only appear more sparsely in the most needed positions in the task trajectory relative to logical reasoning tasks, allowing the large language model agent to decide on the asynchronous generation of "thinking" and "actions" on its own.

[0078] Step S60: Apply the trained large language model agent to the new task instance, evaluate the output effect, correct its logical errors or factual errors through human intervention, and perform tuning.

[0079] Specifically, the large language model agent is applied to new task instances. The agent will imitate the logic of collaborative "reasoning" and "action" in the learning process and flexibly use external tools to solve new tasks / problems.

[0080] Among them, if there is a logical error or factual error in the agent's logic chain, according to the "reasoning-action" logic chain output by the agent, the "thinking" with the earliest logical error or factual error is located, and the recipient manually edits the "thinking" to correct the logic or facts after the error, so that the agent can re-output the subsequent content of the "thinking" after the correction. That is, the user can browse the "reasoning-action" logic chain output by the agent, locate the "thinking" with the earliest logical or factual error, and manually edit the "thinking" to correct its error.

[0081] Therefore, this workflow describes how to use small sample learning to let the large language model agent learn this general paradigm of collaborative "reasoning" and "action". Since the large language model has a strong ability to learn situations, for a specific problem domain, only 5-6 task solution trajectory samples need to be manually annotated to form a supporting data set and handed over to the model for learning, so that the model can master this thinking paradigm and show excellent generalization ability on new unknown samples.

[0082] In an embodiment of the present invention, the framework supports fine-tuning to further improve the performance of the model, one is prompt fine-tuning, and the other is model fine-tuning. Prompt fine-tuning includes two forms: hard prompts and soft prompts.

[0083] Hard prompts are manually preset discrete tags that guide the output of large language models by modifying pre-defined semantic and grammatical rules. This method has strong interpretability and can better control the behavior of the model, but it also requires more manual knowledge and costs.

[0084] Soft prompts are a self-learning method that guides the large language model to produce better expected output through gradient optimization. The specific implementation steps are as follows:

[0085] Adjust the general cue vector representation, add some new cue information according to the specific needs of the target task, and then concatenate these new cue vectors with the general cue vector to form a composite cue vector; the concatenation methods include: simple concatenation, weighted concatenation, hierarchical concatenation, conditional concatenation, and encoding-decoding concatenation;

[0086] In the small sample learning stage, some trainable soft prompts are introduced, including task-related words and phrases, to help the large language model understand and generate language expressions in specific fields, describe reasoning steps, and action strategies, thereby guiding the large language model to make more reasonable reasoning and actions; then adjust the trainable soft prompts based on the feedback information to gradually optimize the model behavior.

[0087] Therefore, soft-cue fine-tuning can adapt the pre-trained model to new downstream tasks without adjusting the entire model. Compared with traditional fine-tuning models, cued fine-tuning has the advantages of efficiency, flexibility and interpretability.

[0088] The specific implementation steps of the model fine-tuning method are as follows:

[0089] Design a training set for fine-tuning a large language model; wherein the task solution trajectory of each sample in the training set is designed in the same manner as in step S20, and the number of samples exceeds the number of samples in small sample learning, that is, the training set needs to contain a large number of samples (e.g., 3000);

[0090] The training set is handed over to the large language model for training and fine-tuning, and some parameters of the model are updated to enable the model to better grasp the thinking paradigm of collaborative "reasoning" and "action". Compared with small sample learning, fine-tuning the model with a large number of samples can allow the model to be exposed to more diverse and wider task samples, and more comprehensively learn the use of this thinking paradigm in various types of task cases, thereby deepening the model's understanding of this framework and allowing the model to have better generalization performance when facing new task samples.

[0091] Combination Figure 4 As shown, the present invention can also integrate the large language model into the intelligent virtual human dialogue system, which specifically includes the following steps S201-S204.

[0092] Step S201: When a user initiates a conversation, a speech recognition module is called to recognize the user's speech input as corresponding text input content;

[0093] Step S202: The recognized text input content is passed to the large language model agent as an input question / task, and the large language model then generates a text answer through logical analysis and "reasoning-action" collaboration;

[0094] Step S203: Submitting the text output to a text-to-speech module and generating a natural, smooth, and close-to-real human voice audio output;

[0095] Step S204: the audio speech output is handed over to the 3D graphics rendering module to generate a virtual digital human animation matching the speech, so as to feed back the text output of the large language model agent to the user in a form close to real-life conversation.

[0096] Embodiment 2

[0097] The present invention provides a large language model learning device 100 based on collaborative reasoning and action of an agent, which is used to propose a general paradigm for allowing a large language model to collaborate in "reasoning" and "action", aiming to efficiently and accurately solve various language reasoning and decision-making tasks. It should be noted that the implementation principle and specific implementation method of the large language model learning device 100 based on collaborative reasoning and action of an agent can refer to the above-mentioned large language model learning method based on collaborative reasoning and action of an agent, so it will not be repeated below.

[0098] like Figure 5 As shown, the large language model learning device 100 based on agent-based collaborative reasoning and action includes:

[0099] The prompt vector initialization module 10 is used to clarify the task requirements and initialize a general prompt vector as the starting point for learning the large language model; wherein the prompt vector includes a guide, context information, task description, output format indication, restriction conditions, and sample output;

[0100] A support data set design module 20 is used to design a support data set for small sample scenario learning, so as to enable the large language model agent to learn the logic of collaborative "reasoning" and "action". Each manually annotated sample in the support data set includes an input problem and a detailed record of the "actions", "thinking" and environmental "observations" involved in solving the problem by real humans in the scenario.

[0101] Tool and interface design module 30, used to design an interface for a large language model agent to call external tools and obtain external information;

[0102] The tool combination coordination module 40 is used to select a suitable tool combination from a preset tool library according to task requirements, coordinate the input and output formats between the tools, and then select an adapted large language model, input the information provided by each tool into the large language model for reasoning and output generation;

[0103] The model small sample learning module 50 is used to encapsulate prompts, external tools and large language models through agents, and deliver the designed support data set as prompts to the large language model agent for small sample learning to master the "reasoning-action" collaborative paradigm;

[0104] The output evaluation and tuning module 60 is used to apply the trained large language model agent to new task instances, evaluate the output effect, correct its logical errors or factual errors through human intervention, and perform tuning.

[0105] Embodiment 3

[0106] The present invention provides a terminal, which includes a memory, a processor, and a computer program stored in the memory. When the computer program is executed by the processor, the various steps of the large language model learning method based on agent-based collaborative reasoning and action as described in any one of the above-mentioned embodiments are implemented.

[0107] Embodiment 4

[0108] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various steps of the large language model learning method based on agent-based collaborative reasoning and action as described in any of the above-mentioned embodiments.

[0109] Embodiment 5

[0110] The present invention provides a computer program product, including a computer program or instructions, which, when processed and executed, implement the various steps of the large language model learning method based on agent-based collaborative reasoning and action as described in any of the above-mentioned embodiments.

[0111] In summary, the agent-based collaborative reasoning and action-based large language model learning method provided by the present invention has the following beneficial effects compared with the existing large language model:

[0112] (1) Intuitive and easy to design and implement: The prompt design of the framework is very straightforward and intuitive. For each task solution trajectory in a sample, during the annotation process, it is only necessary to record the human "thinking" in natural language based on the intuition and problem-solving logic at each step to form a complete logical chain.

[0113] (2) General and flexible: Due to the flexible thinking space of the large model agent and the "reasoning-action" logic that is applicable to solving almost all problems and tasks, the framework can be easily migrated to various tasks with different action spaces and reasoning requirements, including but not limited to question answering, implementation verification, mathematical problems, numerical analysis, word games, etc.;

[0114] (3) Efficient and stable: This framework only requires a small amount of scenario samples to learn, which can enable the large language model agent to demonstrate strong generalization capabilities on new task instances. Its accuracy in solving tasks and answering questions is much higher than that of the baseline large language model agent that does not use this framework.

[0115] (4) Aligned with human thinking logic and highly controllable: The framework ensures that the large language model agent can output a highly interpretable decision sequence and reasoning process, allowing users to easily check the accuracy of reasoning and facts in each step. In addition, users of the framework can also control the agent's behavior in real time and correct errors by editing the "thinking" output by the agent.

[0116] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0119] In the embodiments provided by the present invention, it should be understood that the disclosed systems or devices / terminal equipment and methods can be implemented in other ways. For example, the system or device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0122] The present invention is not limited to what is described in the specification and implementation modes, and therefore additional advantages and modifications can be easily realized by those skilled in the art. Therefore, without departing from the spirit and scope of the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details, representative devices, and illustrative examples shown and described herein.

Claims

1. A method for learning a large language model based on agent-based collaborative reasoning and action, characterized in that: include: Step S10: clarify the task requirements and initialize a general prompt vector as the starting point for learning the large language model; wherein the prompt vector includes a guide, context information, task description, output format indication, restriction conditions, and sample output; Step S20: Design a support dataset for small sample scenario learning to enable the large language model agent to learn the logic of collaborative "reasoning" and "action"; wherein each manually labeled sample in the support dataset contains an input question and a detailed record of the "actions", "thinking" and environmental "observations" involved in solving the problem by real humans in the scenario; Step S30: Designing an interface for the large language model agent to call external tools and obtain external information; Step S40: According to the task requirements, select a suitable tool combination from the preset tool library, coordinate the input and output formats between the tools, and then select an adapted large language model, input the information provided by each tool into the large language model for reasoning and output generation; Step S50: encapsulate prompts, external tools and large language models through agents, and hand over the designed support data set as prompts to the large language model agent for small sample learning to master the "reasoning-action" collaborative paradigm; Step S60: Apply the trained large language model agent to the new task instance, evaluate the output effect, correct its logical errors or factual errors through human intervention, and perform tuning.

2. The agent-based collaborative reasoning and action-based large language model learning method as claimed in claim 1, characterized in that: Integrating the large language model into the intelligent virtual human dialogue system specifically includes the following steps: When the user initiates a conversation, the speech recognition module is called to recognize the user's speech input as the corresponding text input content; The recognized text input is passed as the input question / task to the large language model agent, which then generates a textual answer through logical analysis and "reasoning-action" collaboration; The text output is passed to the text-to-speech module and generates audio speech output close to real human voice; The audio speech output is handed over to the 3D graphics rendering module to generate a virtual digital human animation that matches the speech, so as to feed back the text output of the large language model agent to the user in a form close to a real-life conversation.

3. The agent-based collaborative reasoning and action large language model learning method as claimed in claim 1, characterized in that: In step S60, if a logical error or a factual error occurs in the agent's logic chain, the "thinking" where the logical error or the factual error first occurs is located according to the "reasoning-action" logic chain output by the agent, and the "thinking" is manually edited to correct the logic or facts after the error, so that the agent can re-output the subsequent content of the "thinking" after the correction.

4. The agent-based collaborative reasoning and action large language model learning method as claimed in claim 1, characterized in that: Also includes: The framework for building a large language model agent to interact with the environment to solve tasks is as follows: For a task x that needs to be solved urgently, at time node t, the large language model agent will analyze the current task environment and progress and obtain an "observation" At the same time, according to the decision strategy π(a t |c t ) Make an "action" Among them, c(o1, a1, ..., o t-1 , a t-1 , o t ) represents the context of the current task, Represents the mapping from the context in which the model agent is located to the behavioral decisions; The model agent's "action" a t Expand the behavior space in, The language space of the model representing the complement, the behavior in the language space Represents the "thinking" of the large language model agent on the current task process; the "thinking" of the large prediction model The current problem context t The information contained in the problem is integrated to update the current problem context This helps the big prediction model make subsequent behavioral decisions Generate more logical decisions in the process; Specifically, the original input problem x is decomposed into a series of relatively easy-to-solve sub-problems x t , the model agent solves the current problem context c step by step in a step-by-step manner t problems that need to be solved, and generate new "observations" and "thoughts" to update the context t+1 , where the decision strategy of the model agent Π(a t |c t )Only the current subtask needs to be mapped.

5. The agent-based collaborative reasoning and action large language model learning method as claimed in claim 4, characterized in that: When the big prediction model is "observed" t When it is discovered that there is unknown information in the current problem context, the large language model will infer the means that can be used to resolve the unknown information and incorporate the means into the agent's "thinking" Then the agent will take the next "action" a t+1 Invoke corresponding tools to solve corresponding subtasks; wherein the means include: using search engines and encyclopedias to search and acquire knowledge, and obtaining fields to be queried through interaction with databases.

6. The agent-based collaborative reasoning and action-based large language model learning method as claimed in claim 5, characterized in that: In step S50, the support data set D is divided into a training set D train , validation set D val and the test set D test ;D train As auxiliary set A, from D val and D test Random sampling forms a large number of evaluation tasks T; Among them, for a sampling task, there are two subsets A S and A Q Composition, corresponding to the support set S and query set Q respectively; The small sample learning includes a two-step optimization between the base learner and the meta learner; in the inner loop of the small sample learning, A S Perform fast learning to find the optimal parameters of the basic learner; in the outer loop of small sample learning, through A Q To calculate the gradient of each new task relative to the optimal parameters, the randomly initialized model parameters are then updated to adapt to different tasks, thereby enhancing the generalization ability of the large speech model.

7. The agent-based collaborative reasoning and action large language model learning method as claimed in claim 6, characterized in that: If solving the task requires obtaining external information, then indicate the information source involved in "Thinking" and describe the information needed; If the task involves the use of external tools, specify the tools needed to solve the current subtask and the purpose of using the tools in "Thinking", and call the corresponding external tools in "Action"; In any "thinking" step, if the context of the current task already contains all the necessary information for the large language model agent to answer the original input question, then write the answer to the question in "thinking", and in the next "action", output the answer to the question and terminate the process; For tasks that mainly rely on logical reasoning, "thinking" and "acting" are generated alternately so that the task-solving trajectory contains multiple "thinking-acting-observing" steps; For decision-making tasks involving a large number of "actions", "thinking" will only appear more sparsely in the most needed positions in the task trajectory relative to logical reasoning tasks, allowing the large language model agent to decide on the asynchronous generation of "thinking" and "actions" by itself.

8. The agent-based collaborative reasoning and action large language model learning method as claimed in claim 7, characterized in that: In step S60, tuning is performed by means of prompt fine-tuning, wherein the prompt fine-tuning includes hard prompts; wherein the hard prompts are manually preset discrete tags, which guide the output of the large language model by modifying pre-defined semantic and grammatical rules.

9. The agent-based collaborative reasoning and action large language model learning method as claimed in claim 7, characterized in that: In the step S60, the tuning is performed by means of prompt fine-tuning, wherein the prompt fine-tuning includes soft prompting; the specific implementation steps of the soft prompting are as follows: Adjust the general cue vector representation, add some new cue information according to the specific needs of the target task, and then concatenate these new cue vectors with the general cue vector to form a composite cue vector; the concatenation methods include: simple concatenation, weighted concatenation, hierarchical concatenation, conditional concatenation, and encoding-decoding concatenation; In the small sample learning stage, some trainable soft prompts are introduced, including task-related words and phrases, to help the large language model understand and generate language expressions in specific fields, describe reasoning steps, and action strategies, thereby guiding the large language model to make more reasonable reasoning and actions; then adjust the trainable soft prompts based on the feedback information to gradually optimize the model behavior.

10. The agent-based collaborative reasoning and action large language model learning method according to claim 7, characterized in that: In step S60, the optimization is performed by fine-tuning the model, and the specific implementation steps are as follows: Designing a training set for fine-tuning a large language model; wherein the task solution trajectory of each sample in the training set is designed in the same manner as in step S20, and the number of samples exceeds the number of samples in small sample learning; The training set is handed over to the large language model for training and fine-tuning, and some parameters of the model are updated to enable the model to better grasp the thinking paradigm of collaborative "reasoning" and "action".

Images