Method for finely adjusting intelligent agent and computing equipment

Abstract

A method for fine tuning an agent and a computing device obtain a reasoning result output by a reasoning agent for a target task, the reasoning result at least comprising a reasoning conclusion and a logic path, the logic path shows a logic support relationship among rounds of reasoning executed in a reasoning process executed by the reasoning agent for the target task; determining a reward score corresponding to the target task according to the continuity of the logic path; and according to the reward score, parameter adjustment is performed on a large language model in the reasoning agent through a reinforcement learning method, so that the logic continuity of the reasoning agent in facing a complex task can be improved, and the credibility of an output result of the reasoning agent is improved.

Description

Technical Field

[0001] The embodiments in this specification belong to the field of data processing technology, and in particular relate to a method for fine-tuning an intelligent agent and a computing device. Background Technology

[0002] Deep Research Agents (DR Agents) are intelligent agent systems capable of autonomously completing the entire process of "hypothesis formulation → evidence collection → iterative verification → conclusion output" in long-term, multi-step, cross-source information retrieval, reasoning, and writing tasks, much like human researchers. The concept of Deep Research Agents transforms large language models from auxiliary tools in the research process into end-to-end problem-solving experts. Users only need to describe complex data analysis tasks (such as analyzing the current development status of an industry or the market prospects of an emerging technology) to the Deep Research Agent to directly obtain a complete task report, without requiring additional user intervention in the analysis process.

[0003] However, currently, deep research agents suffer from problems such as mismatch between conclusions and evidence, and even fabricated conclusions, when faced with complex tasks requiring long-term reasoning. This significantly reduces the credibility of the deep research agent's output.

[0004] Therefore, this embodiment provides a technical solution for fine-tuning intelligent agents to at least partially solve the above-mentioned problems. Summary of the Invention

[0005] This specification provides an embodiment of a method and computing device for fine-tuning an intelligent agent, including:

[0006] This specification provides a method for fine-tuning an intelligent agent, the method comprising:

[0007] Obtain the reasoning result output by the reasoning agent for the target task, wherein the reasoning result includes at least the reasoning conclusion and the logical path, and the logical path shows the logical support relationship between each round of reasoning performed by the reasoning agent for the target task.

[0008] Based on the coherence of the logical path, determine the reward score corresponding to the target task;

[0009] Based on the reward score, the parameters of the large language model in the reasoning agent are adjusted using reinforcement learning methods.

[0010] A second aspect of this specification provides a computer-readable storage medium having a computer program stored thereon that, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0011] A third aspect of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in the first aspect.

[0012] The scheme for fine-tuning an agent provided in this embodiment can improve the logical coherence of the reasoning agent when facing complex tasks and enhance the credibility of the output results of the reasoning agent. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the reasoning process of a reasoning agent in one embodiment of this specification;

[0015] Figure 2 This is a schematic diagram of a node construction method using one round as an example in one embodiment of this specification;

[0016] Figure 3 This is a flowchart illustrating a method for fine-tuning an intelligent agent according to one embodiment of this specification. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0018] First, a brief introduction to the technical terms used in this manual:

[0019] Intelligent Agent: An intelligent agent is a software system capable of operating autonomously with minimal human intervention. It possesses the following core capabilities: making action decisions based on the environment, autonomously executing tasks based on these decisions, and iteratively making decisions and executing tasks based on feedback from the execution results until a preset goal is achieved. Currently, the main implementation of intelligent agents involves using a large language model as the core for receiving user tasks and making action decisions. An external intelligent agent toolkit is used to execute the action decisions made by the large language model. Specifically, the large language model in the intelligent agent can be pre-trained and combined with prompting engineering and retrieval enhancement generation schemes. This large language model can autonomously arrange task processing flows (e.g., by inputting task processing flows as prompt words into the large language model) and output call statements for the intelligent agent toolkit (e.g., by attaching the functional introduction, interaction methods, and other relevant information of the intelligent agent toolkit to the large language model in the form of a database). This intelligent agent toolkit may include several functional tools required for processing related tasks (e.g., if an intelligent agent is designed for reasoning tasks, then the functional tools may include retrieval tools, browsing tools, filtering tools, etc.). The large language model can transmit call statements to the corresponding function tools through a pre-defined interface, and receive the execution results from the function tools through the corresponding interface. The intelligent agent involved in this embodiment is a reasoning intelligent agent.

[0020] Graph structure data: A graph structure is a non-linear data structure commonly used to represent relationships between objects, consisting of nodes and edges between them. Graph structure data describes the graph structure and can record the nodes and their connections (edges) using a preset data format. Therefore, obtaining this graph structure data is equivalent to determining the corresponding node relationship graph. In this embodiment, the reasoning actions of the reasoning agent and their corresponding execution results are set as nodes, and the logical support relationships between nodes are set as edges, thus obtaining the logical path represented in the form of graph structure data.

[0021] As mentioned above, existing deep research agents suffer from lost logical connections and logical confusion when performing long-term reasoning. To completely solve this problem, this embodiment improves the training process of the reasoning agent (which is the deep research agent provided in this embodiment) to enhance the logical rigor of the reasoning process.

[0022] To make the fine-tuning process explained later easier to understand, the following section first introduces the process by which the reasoning agent processes the target task.

[0023] In this embodiment, the reasoning agent consists of a base model and a reasoning toolkit connected to the base model. The base model is a large language model, and the reasoning toolkit includes at least various function tools required for information retrieval and processing. Referring to the previous introduction of the agent concept, the base model can undergo prompting engineering and be fine-tuned accordingly. It has the ability to perform multiple rounds of reasoning iteratively, decide the reasoning actions to be executed in each round, and generate calling statements that can be recognized by the reasoning toolkit.

[0024] Figure 1 This diagram illustrates the reasoning process of a reasoning agent in one embodiment of this specification. First, the description text of the target task is input into the base model, signifying that the reasoning agent has received the target task. After receiving the description text, the base model iteratively performs multiple rounds of reasoning based on it. In each round of reasoning, the base model uses information acquired in previous rounds as input and generates the corresponding output for the current round. This output can be broadly categorized into two types: calls to the reasoning tool library and sub-conclusions.

[0025] When the base model generates a call statement, the call statement is transmitted to the inference tool library through a preset interface. The inference tool library then calls the corresponding function tool to execute the call statement. Adapting to the application scenario of the inference agent (outputting corresponding inference conclusions based on the user-input target task), in some implementations, the function tools in the inference tool library may include search tools (for retrieving data from the network and / or local database based on corresponding keywords), browsing tools (for obtaining retrieved data), filtering tools (for filtering the obtained data), and aggregation tools (for integrating the obtained data), etc. Correspondingly, the base model can output call statements for the aforementioned various function tools. In this embodiment, the continuous process of the base model generating a call statement and transmitting it to the inference tool library, and the inference tool library executing the call statement, is called the evidence acquisition action performed by the inference agent; the data returned to the base model by the inference tool library after receiving the call statement is called evidence information. For subsequent rounds of the evidence acquisition action, the evidence information will be used as part of the model input of the base model.

[0026] On the other hand, the process of the base model generating sub-conclusions is called the conclusion derivation action performed by the reasoning agent. In subsequent rounds of the conclusion derivation action, the sub-conclusions will be used as part of the model input of the base model.

[0027] In summary, the evidence acquisition action and the conclusion deduction action are the reasoning actions that the reasoning agent can execute, and the sub-conclusions and evidence information are the execution results corresponding to the reasoning actions.

[0028] Furthermore, in some implementations, the reasoning action in the first round of reasoning also includes task decomposition. When faced with a target task that is too vague or semantically rich, the basic model can decompose the target task based on its description text, outputting several sub-problems derived from the task breakdown. These sub-problems output in the first round then provide guidance for the reasoning actions in subsequent rounds. For example, if the description text of the target task is "Go to Beijing for 5 days in January 2026 with a budget of 10,000 yuan, how should I arrange it?", the basic model can decompose the target task into three sub-problems: "time feasibility analysis," "budget analysis," and "itinerary design." In subsequent reasoning rounds, these three sub-problems are reasoned about separately to obtain sub-conclusions, which are then summarized to obtain the reasoning conclusion for the complete target task.

[0029] It should also be noted that, unlike the previous rounds, in the final round, the reasoning agent will consistently perform the conclusion derivation action. The sub-conclusion generated in this final round is the complete reasoning conclusion for the target task. In addition to performing the conclusion derivation action, in the final round, the basic model in the reasoning agent also generates corresponding logical paths based on the reasoning actions and execution results of each round, to explicitly demonstrate the logical relationship between each reasoning action and execution result in the reasoning process for the target task.

[0030] The format and content requirements for the logical path can be pre-input into the basic model as prompts, so that the basic model outputs logical paths that conform to the preset format and content. This logical path can be represented in natural language (e.g., a logical path can be expressed as "The reasoning conclusion is the result of the execution of reasoning action k, supported by sub-conclusions A and B; sub-conclusion A is the result of the execution of reasoning action J, supported by evidence X…"), or as graph-structured data as described above; this specification does not impose any restrictions on this representation.

[0031] Specifically, after completing the conclusion derivation in the final round, the total number of rounds of reasoning, the reasoning actions performed in each round, and the execution result after each round's reasoning actions are determined. Accordingly, the reasoning actions and execution results of each round are treated as nodes. By determining the edges connecting the nodes based on the logical support relationships between them, the logical path (represented in the form of graph structure data) corresponding to the complete reasoning process can be determined.

[0032] It's important to note that while each round executes a reasoning action, not every result of that action can be modeled as a node. For example, regarding the evidence acquisition action, there might be a situation where the current round performs an evidence acquisition action, but because previous rounds also performed similar actions with similar parameters, the current round may not have acquired any new evidence. Therefore, this execution result, which doesn't include new evidence, cannot be modeled as a node.

[0033] In other words, in this embodiment, the execution results of each node should have differences in content.

[0034] On the other hand, since the results of the conclusion deduction actions executed in different rounds always differ in content, each execution of the conclusion deduction action will produce a corresponding execution result (sub-conclusion) that can be modeled as a node. That is, there is a one-to-one correspondence between the conclusion deduction action and the sub-conclusion. Therefore, when modeling the logical path, only the evidence acquisition action, the evidence, and the sub-conclusion can be modeled as nodes, without considering the conclusion deduction action.

[0035] Specifically, in the logical path represented by graph structure data, the following can be recorded: the node type (node ​​type may include acquisition nodes representing evidence acquisition actions, evidence nodes representing evidence information, and conclusion nodes representing sub-conclusions), the text content of each node (for acquisition nodes, the text content may be the call statement generated by the basic model; for evidence nodes, the text content may be the sub-conclusion content generated by the basic model; for evidence nodes, the text content may be a reference fragment with source tagging), the round corresponding to each node, and the connection relationship between each node (edges between nodes).

[0036] Figure 2 This diagram illustrates a node construction method using one round as an example in one embodiment of this specification. Figure 2 As shown, for the current round, the output result of the current round can be determined. This output result may be a call statement to the inference tool library or a sub-conclusion. If the output result is a call statement, the acquisition node corresponding to the current round can be modeled. On the one hand, based on the call statement of the current round (for example, if the call statement of the current round is a call statement to the filtering tool, then obviously the acquisition node is connected to the evidence node of the previous round), the connection relationship between the acquisition node and each node corresponding to the previous round is determined. On the other hand, based on the evidence information returned by the inference tool library in the current round and the evidence nodes corresponding to each previous round, it is determined whether the evidence information of the current round contains new content, and thus whether to model the evidence node corresponding to the current round.

[0037] On the other hand, if the output result is a sub-conclusion, the conclusion node corresponding to the current round can be modeled, and the connection relationship of the conclusion node of the current round can be determined based on the model input of the basic model of the current round and the text content of the sub-conclusion.

[0038] Therefore, by modeling each round according to the above process, a logical path can be constructed from the nodes corresponding to each round and the connection relationships between the nodes.

[0039] In the case where the reasoning action includes task decomposition action, the node type may also include question nodes representing sub-problems (multiple sub-problems obtained from one task decomposition action can be modeled as multiple sub-problem nodes respectively), and the text content of the sub-problem node can be the sub-problem content generated by the basic model.

[0040] The following is a brief explanation of the method for determining the edges between nodes in a logical path.

[0041] On the one hand, the execution results of the same round are generated by the corresponding reasoning actions, and nodes corresponding to the same round naturally have logical support relationships. On the other hand, the evidence acquisition action, in addition to simply acquiring evidence, can also include further processing of the evidence information from previous rounds (e.g., using filtering tools, aggregation tools, etc.) to obtain the evidence information corresponding to the current round; the conclusion deduction action specifically includes reasoning based on the evidence or sub-conclusions of previous rounds to obtain the sub-conclusions corresponding to the current round. Thus, the aforementioned evidence acquisition and conclusion deduction actions can also enable nodes in different rounds to have corresponding logical support relationships. Based on the above-mentioned various logical support relationships, directed edges can be used to represent logical support (e.g., a directed edge from node A to node B indicates that node A supports node B), and the edges between nodes can be determined.

[0042] It should be noted that, Figure 2 This is merely a visual representation of the construction process of this logical path. In the actual reasoning process, it is only necessary to use the model input and output results of each round of the basic model in the reasoning process for the target task as the input of the basic model in the final round (in conjunction with the preset prompt words) so that the output of the basic model can be represented as a logical path of graph structure data.

[0043] Therefore, after completing the final round of reasoning, in addition to the conventional reasoning conclusion, the reasoning agent in this embodiment can also output an additional logical path. This logical path intuitively and concisely demonstrates the reasoning process of the reasoning agent. On the one hand, it allows users to more conveniently check the reasoning process and understand the reasoning logic of the reasoning agent; on the other hand, it also provides a corresponding basis for the subsequent training process.

[0044] It should be noted that if the basic model is only prompted with engineering constraints to output the corresponding logical path, the reasoning process of the basic model is not optimized because the actual parameters of the basic model remain unchanged. That is, logical errors may still exist in the reasoning process of the basic model. Therefore, this embodiment provides a method for fine-tuning the agent, adjusting the parameters of the basic model in the reasoning agent based on the logical path.

[0045] Figure 3 The diagram illustrates a flowchart of a method for fine-tuning an intelligent agent according to one embodiment of this specification. This method can be executed using any computing device or server with computing capabilities, and this specification does not impose any limitations on it. The method includes:

[0046] Step S301: Obtain the reasoning result output by the reasoning agent for the target task, wherein the reasoning result includes at least the reasoning conclusion and the logical path, and the logical path shows the logical support relationship between each round of reasoning performed by the reasoning agent for the target task.

[0047] The method for obtaining the reasoning results can be found in [reference]. Figure 1 as well as Figure 1 The corresponding descriptions will not be repeated here.

[0048] Step S303: Determine the reward score corresponding to the target task based on the coherence of the logical path.

[0049] As mentioned above, existing technologies suffer from problems such as mismatch between conclusions and evidence, and fabricated conclusions. These problems also exist for the basic model without parameter adjustments. The logical path generated in step S301 models these problems more intuitively as issues of coherence between rounds of reasoning. For example, if a round lacks logical support from any other round, then clearly, the reasoning in that round is fabricated or has no effect on other rounds. Therefore, the coherence of this logical path reflects the logical rigor of the reasoning agent's process for the target task. However, compared to the more abstract concept of "logical rigor," this embodiment provides a more explicit model of the logical support relationships between rounds of reasoning, reducing the difficulty of adjusting the parameters of the basic model based on this logical path.

[0050] Since this logical path is generated based on the reasoning process of the base model and does not have a "standard answer" that can be used as a label, this embodiment uses reinforcement learning to adjust the parameters of the base model. Therefore, in step S303, the reward score corresponding to the target task must first be determined based on this logical path.

[0051] In some implementations, for logical paths represented in the form of graph-structured data, a pre-defined coherence scoring rule can be established. This coherence scoring rule can then be written as a scoring function that scores based on the attributes of each node (node ​​type and connection relationship). In step S303, the logical path is input into this scoring function, and the coherence score of the logical path (i.e., the target task corresponding to the logical path) can be determined using the scoring function. This allows for a relatively simple determination of the coherence score of the logical path. For example, the coherence scoring rule could be that for a conclusion node, the more nodes pointing to the conclusion node, the higher the coherence score corresponding to the conclusion node; for an acquisition node, if the acquisition node does not point to any node, the coherence score corresponding to the acquisition node is negative.

[0052] However, using scoring functions to score the coherence of logical paths cannot incorporate semantic understanding of the text content corresponding to each node during the scoring process. It can only mechanically score the connection relationship between nodes, making it difficult to evaluate complex logic.

[0053] Therefore, a coherence score for the logical path can be determined using a large language model. In some implementations, this embodiment also involves an evaluation model, which is a large language model; the logical path can be input into the evaluation model to determine the coherence score of the logical path.

[0054] Specifically, the logical path and preset scoring prompts can be input into the evaluation model to determine the coherence score of the logical path. The scoring prompts indicate several scoring dimensions, including at least the evidence coverage dimension, reasoning consistency dimension, evidence matching dimension, and reasoning efficiency dimension. The evidence coverage dimension measures the connection between conclusion nodes and evidence nodes; the reasoning consistency dimension measures the logical consistency between interconnected conclusion nodes; the evidence matching dimension measures the semantic matching degree between conclusion nodes and connected evidence nodes; and the reasoning efficiency dimension measures the connection between acquisition nodes and evidence nodes.

[0055] The dimensions of evidence coverage measure whether each conclusion node is supported by evidence nodes (reflected by edges directly pointing from evidence nodes to that conclusion node) and whether each conclusion node is supported by multiple evidence nodes. The dimension of reasoning consistency measures whether the text content of interconnected (directly connected) conclusion nodes is logically consistent. The dimension of evidence matching measures the semantic matching degree between the text content of interconnected conclusion nodes and evidence nodes, and whether the text content of the conclusion node corresponding to the final round is covered or supported by the text content of connected evidence nodes or conclusion nodes. The dimension of reasoning efficiency measures whether each acquisition node has supporting evidence nodes.

[0056] This specification does not impose restrictions on the specific scoring rules for each scoring dimension. For example, for any scoring dimension, the value space of the sub-scorings of that scoring dimension can be set as a discrete space according to the corresponding measurement standard, with either satisfaction or non-satisfaction (e.g., 1 can represent satisfaction, and -1 represents non-satisfaction); alternatively, the degree of satisfaction of the measurement standard can be determined by the large language model, and the value space of the sub-scorings of that scoring dimension can be set as a continuous space (e.g., the value space can be set as [-1, 1], where closer to 1 indicates a higher degree of satisfaction, and closer to 0 indicates a lower degree of satisfaction). Preferably, the value space corresponding to the sub-scorings of each scoring dimension can be the same.

[0057] After determining the sub-ratings corresponding to each rating dimension, the sub-ratings of each rating dimension can be merged according to the preset weights of each rating dimension to determine the coherence rating corresponding to the logical path.

[0058] In some implementations, the coherence score corresponding to the logical path can be directly used as the reward score for the target task, and then the subsequent step S305 can be executed.

[0059] In other implementations, the reasoning conclusion and the corresponding conclusion prompt words can be input into the evaluation model to determine the evaluation model's score for the reasoning conclusion. Then, based on preset weights, the coherence score and the result score are merged to determine the reward score corresponding to the target task.

[0060] The specific method for constructing the conclusion prompt is not limited in this specification. For example, if the target task is a reasoning task with a standard answer (such as a calculation task, a true / false judgment task, etc.), the conclusion prompt can be built based on the standard answer corresponding to the target task and the exact-match (EM) rule; if the target task is a reasoning task with an answer template (such as a reading comprehension task, an event extraction task, etc.), the conclusion prompt can be built based on the template answer corresponding to the target task and the fuzzy matching rule (such as the F1 score); if the target task is a reasoning task without a clear answer (such as a trip recommendation task, an industry analysis task, etc.), the conclusion prompt can be built based on human preference.

[0061] Therefore, this reward score integrates coherence score and outcome score, which allows subsequent parameter adjustments to the basic model based on the reward frequency score to improve the logical coherence of the basic model without sacrificing its basic reasoning ability.

[0062] Step S305: Based on the reward score, adjust the parameters of the large language model in the reasoning agent using a reinforcement learning method.

[0063] After determining the reward score, reinforcement learning methods can be used to adjust the parameters of the base model. As described earlier regarding the reasoning process of the inference agent, for a given target task, the agent performs multiple rounds of iterative reasoning. Correspondingly, the base model also undergoes multiple rounds of generation, producing the output (call statement or sub-conclusion) for each round. Since an output of the large language model consists of several tokens, each token is iteratively generated based on the preceding tokens in the current round's output. Therefore, the generation process of each token in an output constitutes a (state-action) trajectory required for reinforcement learning. Consequently, multiple rounds of generation result in multiple trajectories. When fine-tuning the large language model using reinforcement learning, it is necessary to determine the reward corresponding to each trajectory (the reward is further determined based on the reward score), and then independently adjust the parameters of the large language model for each trajectory.

[0064] However, the reward score determined in step S303 is a reward score for the target task, that is, a comprehensive score for multiple trajectories (i.e., multiple rounds). Therefore, this embodiment needs to decompose the reward score to determine the reward score corresponding to each trajectory in order to adjust the parameters of the basic model.

[0065] In some implementations, the reward score is decomposed into each round according to the preset weights corresponding to each round, and then the parameters of the basic model are adjusted by reinforcement learning methods based on the sub-reward scores corresponding to each round.

[0066] It should be noted that, since this embodiment has already proposed improvements to the method for fine-tuning the agent in terms of both reward scoring and logical path, even without further improvements to the reward scoring decomposition method, the logical coherence of the reasoning agent can still be improved.

[0067] However, the aforementioned method for determining sub-reward scores has a significant problem that greatly affects parameter adjustment. In multi-round inference processes, there are always rounds where the output is not helpful to the overall result, and may even have a negative impact. If the reward score is positive, and the reward score is decomposed according to the preset weights corresponding to each round, then the sub-reward score assigned to each round, regardless of its size, will be positive. Thus, the strategy for generating negative rounds is actually enhanced during parameter adjustment, making it difficult to achieve good parameter adjustment results using the method of decomposing reward scores based on fixed weights for each round.

[0068] In some implementations, the decomposition of reward scores can also be achieved using this logical path (represented in the form of graph structure data)—the reward score can be decomposed into sub-reward scores corresponding to each node based on the connection relationships between the nodes in the logical path.

[0069] Specifically, the reward score can first be considered as the reward score corresponding to the conclusion node of the final round. Then, iterative decomposition is performed in multiple rounds. The decomposition steps in any round include: for any node in the current round that has not been decomposed and has sub-reward scores, the sub-reward scores of that node are distributed to the nodes supporting that node according to preset weights (i.e., for node A, if there is an edge from node B to node A, then the reward score of node A is given to node B). This process is repeated multiple times until the sub-reward scores of each node remain unchanged, thus determining the sub-reward scores corresponding to each node in the logical path.

[0070] As mentioned above, each node can record the round it corresponds to, thus determining the sub-reward score for each round.

[0071] The method described above, which decomposes the reward score to each node using logical paths, ensures that the output of the base model in each round obtains a sub-reward score that accurately reflects the generation effect of the current round. This solves the problem in existing technologies where the reward score does not match the output of each round. By using this sub-reward score to adjust the parameters of the base model, the generation strategy corresponding to rounds with good generation effects can be precisely enhanced, while the generation strategy corresponding to rounds with poor generation effects can be corrected, thereby improving the efficiency and effectiveness of parameter adjustment of the base model.

[0072] After determining the sub-reward score corresponding to each round, for any given round, the actual reward can be determined based on the sub-reward score corresponding to that round and the preset cost constraint. The actual reward is then substituted into a pre-constructed expected reward function, and the gradient of the expected reward function is estimated using Monte Carlo simulation sampling. Here, the expected reward function represents the expected value of the reward obtained under different model parameters. The parameters of the large language model in the reasoning agent are adjusted according to the determined gradient.

[0073] In some implementations, this cost constraint can be constructed using Lagrange multipliers and determined based on the number of times the evidence acquisition action corresponding to the objective task is executed during the inference process, as well as a preset threshold for the number of executions. This allows the base model to learn the ability to cautiously call tool libraries, minimizing the generation of redundant call statements.

[0074] The expected reward function can be constructed based on various common reinforcement learning methods, such as Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), and REINFORCE-with-baseline, etc. This specification does not impose any restrictions on it.

[0075] like Figure 3 The method shown is a fine-tuning of an agent, which can improve the logical coherence of the reasoning agent when facing complex tasks and enhance the credibility of the agent's output.

[0076] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them outstanding generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0077] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0078] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0079] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0080] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0081] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0082] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0083] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0084] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0088] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0089] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0090] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0092] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0093] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for fine-tuning an intelligent agent, the method comprising: Obtain the reasoning result output by the reasoning agent for the target task, wherein the reasoning result includes at least the reasoning conclusion and the logical path, and the logical path shows the logical support relationship between each round of reasoning performed by the reasoning agent for the target task. Based on the coherence of the logical path, determine the reward score corresponding to the target task; Based on the reward score, the parameters of the large language model in the reasoning agent are adjusted using reinforcement learning methods.

2. The method as described in claim 1, wherein obtaining the reasoning result output by the reasoning agent for the target task specifically includes: The target task is input into the reasoning agent, so that the reasoning agent iteratively performs multiple rounds of reasoning according to the target task and determines the logical path according to the multiple rounds of reasoning. Each round of reasoning includes performing reasoning actions and obtaining corresponding reasoning results. The reasoning actions include at least evidence acquisition actions and conclusion deduction actions. The execution result corresponding to the evidence acquisition action is evidence information, the execution result corresponding to the conclusion deduction action is a sub-conclusion, and the reasoning conclusion is the sub-conclusion corresponding to the final round.

3. The method as described in claim 2, wherein, The logical path is represented as graph structure data, which shows several nodes and directed edges between the nodes. The nodes include acquisition nodes representing evidence acquisition actions, evidence nodes representing evidence information, and conclusion nodes representing sub-conclusions. The edges between the nodes show the logical support relationship between the corresponding nodes.

4. The method of claim 3, wherein the method relates to an evaluation model, and the evaluation model is a large language model; Based on the coherence of the logical path, the reward score corresponding to the target task is determined specifically as follows: The logical path is input into the evaluation model, and the evaluation model is used to determine the coherence score of the logical path. Based on the coherence score, the reward score corresponding to the target task is determined.

5. The method as described in claim 4, wherein the logical path is input into the evaluation model, and the evaluation model is used to determine the coherence score of the logical path, specifically including: The logical path and preset scoring prompts are input into the evaluation model to determine the coherence score of the logical path. The scoring prompts indicate several scoring dimensions, including at least the evidence coverage dimension, reasoning consistency dimension, evidence matching dimension, and reasoning efficiency dimension. The evidence coverage dimension measures the connection between conclusion nodes and evidence nodes, the reasoning consistency dimension measures the logical consistency between connected conclusion nodes, the evidence matching dimension measures the semantic matching degree between conclusion nodes and connected evidence nodes, and the reasoning efficiency dimension measures the connection between acquisition nodes and evidence nodes.

6. The method of claim 4, wherein, The reasoning action corresponding to the first round of reasoning also includes: task decomposition action; the execution result of the task decomposition action is several sub-problems obtained by splitting the target task. The node also includes a problem node representing a subproblem.

7. The method as described in claim 1, wherein determining the reward score corresponding to the target task based on the coherence score specifically includes: The reasoning conclusion is input into the evaluation model, and the evaluation model is used to determine the score of the reasoning conclusion. Based on the coherence score and the result score, the reward score corresponding to the target task is determined.

8. The method as described in claim 3, wherein, based on the reward score, parameters of the large language model in the reasoning agent are adjusted using a reinforcement learning method, specifically including: Based on the connection relationship between each node in the logical path, the reward score is decomposed into a sub-reward score corresponding to each node; Based on the sub-reward score corresponding to each node, determine the sub-reward score corresponding to each round; Based on the sub-reward scores corresponding to each round, the parameters of the large language model in the reasoning agent are adjusted using reinforcement learning methods.

9. The method as described in claim 8, wherein, based on the reward score, parameters of the large language model in the reasoning agent are adjusted using a reinforcement learning method, specifically including: The actual return is determined based on the reward score and the preset cost constraint, wherein the cost constraint is determined based on the number of times the evidence acquisition action is executed during the reasoning process and the preset execution number threshold. The actual reward is fed into a pre-constructed expected reward function, and the gradient of the expected reward function is estimated using Monte Carlo simulation sampling, where the expected reward function represents the expected value of the reward obtained under different model parameters; The parameters of the large language model in the reasoning agent are adjusted based on the determined gradient.

10. A computing device comprising a memory and a processor, the memory storing executable code, wherein the processor, when executing the executable code, implements the method as claimed in any one of claims 1-9.

Images