Agent decision trajectory generation method and device based on uncertainty guidance

Abstract

The application provides an agent decision trajectory generation method and device based on uncertainty guidance, wherein the method comprises the following steps: constructing a task input set; sampling and enhancing a generated candidate input in an iterative process; generating a plurality of candidate decision trajectories by sampling a plurality of times for each candidate input by using a to-be-distilled agent; performing uncertainty evaluation from three dimensions of semantic divergence, tool call sequence divergence and output result divergence, and screening high-uncertainty inputs; obtaining a reference decision trajectory for the high-uncertainty inputs, calculating an alignment score of the candidate trajectory and the reference trajectory, and converting the alignment score into a soft preference distribution by using a softmax function; performing joint training based on a supervised learning loss of the reference trajectory and a distillation loss of the soft preference distribution; repeating iteration until a termination condition is met, and obtaining a distilled agent model; thereby, training resources can be focused on high-uncertainty key decision scenarios, and the decision consistency and reliability of the agent in a multi-step task can be improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer technology, and in particular to a method for generating decision trajectories of intelligent agents based on uncertainty guidance, a device for generating decision trajectories of intelligent agents based on uncertainty guidance, a computer-readable storage medium, and a computer device. Background Technology

[0002] With the continuous development of large language model technology, intelligent agents built based on language models have shown broad application potential in multiple fields. These agents can perform semantic understanding and planning of input tasks, and invoke external tools or system interfaces during task execution, gradually completing the target task through multiple rounds of interaction. Compared to traditional single-round generative models, language model agents, relying on multi-step decision-making mechanisms and tool collaboration capabilities, can accomplish more complex and practically valuable tasks, and have been gradually applied in scenarios such as task automation, decision support, and interactive assistance.

[0003] In practical systems, language model agents typically employ a "planning-execution-feedback" workflow. Specifically, the agent first generates intermediate decisions or operational instructions based on task input, then invokes external tools to obtain environmental feedback, and continuously generates subsequent decisions based on this feedback, thus forming a multi-step, continuous execution process. During this process, the agent not only outputs the final task result but also generates a decision trajectory containing intermediate decision text, tool invocation names and parameter configurations, and tool execution feedback. Because the decision trajectory unfolds gradually and its steps are interdependent, a deviation in an intermediate decision can easily accumulate and amplify in subsequent steps, affecting the overall stability and reliability of task execution. Furthermore, the degree of uncertainty varies at different decision points during the generation process; some critical decision steps are more prone to inconsistencies, mismatched tool selections, or unreasonable parameter settings, leading to tool invocation errors and further inducing unreliable output risks related to tool invocation.

[0004] To address the aforementioned issues, existing technologies typically employ rule constraints, execution verification, and training optimization to improve the behavior of language model agents. For example, pre-defined rules limit the agent's operational scope, or the generated tool call results and parameter configurations are verified and corrected during execution to reduce obvious errors. Furthermore, methods incorporate human feedback or labeled data for supervised fine-tuning or preference alignment of the model, thereby improving the agent's decision-making accuracy and controllability in specific task scenarios. These methods have improved the performance of language model agents in certain tasks to some extent and have been applied in relevant systems.

[0005] However, in complex task scenarios where multi-step decision-making and tool invocation are highly coupled, existing technologies still have certain limitations. On the one hand, rule-based or post-processing methods typically focus on constraining the output of a single step, making it difficult to continuously guide the consistency of the overall decision-making trajectory throughout multiple rounds of execution, and they are prone to insufficient adaptability when the task environment changes. On the other hand, existing solutions generally lack effective identification and modeling mechanisms for high-uncertainty decision-making scenarios and key decision-making locations, making it difficult to concentrate training and optimization resources on the key links most prone to bias, resulting in low optimization efficiency. In addition, training methods that rely on a large amount of manual feedback or labeled data are usually costly and difficult to efficiently learn stable and consistent multi-step decision-making strategies under limited supervision.

[0006] In summary, it is necessary to propose a new technical solution that can effectively learn and align the multi-step decision trajectory of an agent by combining the uncertainty characteristics of the decision-making process. This can reduce training and supervision costs, improve the consistency and reliability of the agent's decisions in complex task execution, and reduce the risk of tool invocation errors and the resulting unreliable outputs. Summary of the Invention

[0007] This invention aims to address the problem of decision bias and inconsistency in execution that easily occur in the multi-step decision-making and tool invocation process of language model agents in the prior art. It proposes an agent decision trajectory generation method based on uncertainty guidance. By performing targeted learning and alignment of multi-step decision trajectories, the agent to be trained can form more stable tool selection and parameter settings in key decision-making stages, reduce the risk of error accumulation and unreliable output in the multi-step execution process, and thus improve the accuracy and controllability of the generated decision trajectory.

[0008] A second objective of this invention is to provide a computer-readable storage medium.

[0009] The third objective of this invention is to provide a computer device.

[0010] The fourth objective of this invention is to propose an intelligent agent decision trajectory generation device based on uncertainty guidance.

[0011] To achieve the above objectives, a first aspect of the present invention proposes a method for generating agent decision trajectories based on uncertainty guidance, characterized by the following steps: obtaining a task input set and constructing an initialized agent model to be distilled; sampling from the task input set to obtain a candidate task input set; sampling each task input in the candidate task input set multiple times using the agent model to be distilled to obtain corresponding K candidate decision trajectories, wherein each candidate decision trajectory includes intermediate inference text, tool call sequence, tool execution result, and final output result; and performing uncertainty evaluation on the K candidate decision trajectories of each task input based on semantic divergence, tool call sequence divergence, and output result divergence to obtain the uncertainty of each task input. The process involves: assigning a corresponding uncertainty score; sorting each task input based on the uncertainty score; obtaining a set of high-uncertainty inputs based on the sorting results; acquiring a reference decision trajectory generated by the teacher model for each high-uncertainty task input in the set of high-uncertainty inputs; aligning and scoring the K candidate decision trajectories corresponding to each high-uncertainty task input with the reference decision trajectory to obtain a corresponding soft preference distribution; obtaining a joint loss function based on the soft preference distribution; iteratively training the agent model to be distilled based on the joint loss function until a preset convergence condition is met to obtain a trained decision trajectory generation model; and using the trained decision trajectory generation model to perform multi-step decision-making and tool invocation on the actual task input to generate a decision trajectory.

[0012] The uncertainty-guided intelligent agent decision trajectory generation method of this application can guide the intelligent agent to focus on learning key decision trajectories that are prone to deviation under limited supervision, thereby reducing the risk of error propagation in multi-step task execution and improving the rationality of tool calls and the compliance of parameter settings.

[0013] In addition, the uncertainty-guided intelligent agent decision trajectory generation method proposed in the above embodiments of the present invention may also have the following additional technical features: Optionally, sampling from the task input set to obtain a candidate task input set includes: sampling from the task input set to obtain candidate inputs; performing redundancy removal processing on the candidate inputs to obtain a seed input set; and performing enhancement processing on the seed input set to obtain a diversified enhanced candidate task input set.

[0014] Optionally, uncertainty assessment is performed on the K candidate decision trajectories for each task input based on semantic divergence, tool call sequence divergence, and output result divergence to obtain an uncertainty score corresponding to each task input. This includes: vectorizing each candidate decision trajectory, calculating the cosine distance between decision trajectories and averaging them to obtain semantic divergence; extracting the tool call sequence for each candidate decision trajectory, using normalized edit distance to measure the differences between different candidate decision trajectories at the tool call order and parameter level to obtain tool divergence; normalizing and grouping the final output results of each candidate decision trajectory, calculating the entropy of the output distribution and normalizing it to obtain output result divergence. The semantic divergence, tool call sequence divergence, and output result divergence are weighted and summed to obtain the uncertainty score corresponding to each task input.

[0015] Optionally, a set of high-uncertainty inputs can be obtained based on the sorting results, including: selecting task inputs whose uncertainty scores are in the top N or whose scores exceed a preset threshold based on the sorting results, so as to obtain a set of high-uncertainty inputs.

[0016] Optionally, the K candidate decision trajectories corresponding to each high-uncertainty task input are aligned and scored with the reference decision trajectory to obtain the corresponding soft preference distribution. This includes: comparing the K candidate decision trajectories with the reference decision trajectory, calculating the differences between each candidate decision trajectory at the semantic level and the tool invocation level to obtain the comprehensive alignment score corresponding to each candidate decision trajectory; and feeding the comprehensive alignment score into a softmax function with a temperature parameter to obtain the corresponding soft preference distribution.

[0017] Optionally, the soft preference distribution can be obtained using the following formula:

[0018] Where x represents a task input from a set of high-uncertainty inputs, i represents the index of a candidate decision trajectory, and K represents the total number of candidate decision trajectories corresponding to each task input x. This represents the overall alignment score between the i-th candidate decision trajectory and the reference decision trajectory. Indicates the temperature coefficient. This represents the preference probability value assigned to the i-th candidate decision trajectory under task input x.

[0019] Optionally, the joint loss function includes a supervised learning loss based on the reference decision trajectory and a trajectory-level distillation loss based on the soft preference distribution.

[0020] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium storing an uncertainty-guided agent decision trajectory generation program thereon, which, when executed by a processor, implements the uncertainty-guided agent decision trajectory generation method as described above.

[0021] To achieve the above objectives, a third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for generating intelligent agent decision trajectories based on uncertainty guidance.

[0022] To achieve the above objectives, a fourth aspect of the present invention proposes an agent decision trajectory generation device based on uncertainty guidance, comprising: a construction module for acquiring a task input set and constructing an initialized agent model to be distilled; a sampling module for sampling from the task input set to obtain a candidate task input set; a candidate decision trajectory generation module for sampling each task input in the candidate task input set multiple times using the agent model to be distilled to obtain corresponding K candidate decision trajectories, wherein each candidate decision trajectory includes intermediate inference text, tool call sequence, tool execution result, and final output result; and an uncertainty evaluation module for performing uncertainty evaluation on the K candidate decision trajectories of each task input based on semantic divergence, tool call sequence divergence, and output result divergence to obtain the uncertainty corresponding to each task input. The system comprises: a deterministic scoring module; a ranking module for ranking each task input according to the uncertainty score and obtaining a set of high-uncertainty inputs based on the ranking results; an alignment scoring module for obtaining a reference decision trajectory generated by a teacher model for each high-uncertainty task input in the set of high-uncertainty inputs, and aligning and scoring the K candidate decision trajectories corresponding to each high-uncertainty task input with the reference decision trajectory to obtain the corresponding soft preference distribution; an iterative training module for obtaining a joint loss function based on the soft preference distribution, and iteratively training the agent model to be distilled according to the joint loss function until a preset convergence condition is met to obtain a trained decision trajectory generation model; and a decision trajectory generation module for performing multi-step decision-making and tool invocation on the actual task input using the trained decision trajectory generation model to generate a decision trajectory. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the uncertainty-guided intelligent agent decision trajectory generation method proposed in this invention. Figure 2This is a flowchart illustrating a method for generating an agent's decision trajectory based on uncertainty guidance according to an embodiment of the present invention. Figure 3 This is a block diagram of an uncertainty-guided intelligent agent decision trajectory generation device according to an embodiment of the present invention. Detailed Implementation

[0024] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0025] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0026] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0027] Figure 1 This is a flowchart illustrating a method for generating decision trajectories for intelligent agents based on uncertainty guidance. Figure 1 As shown, the method for generating an agent's decision trajectory based on uncertainty guidance according to an embodiment of the present invention includes the following steps: S101, Obtain the task input set and construct the initialized agent model to be distilled.

[0028] As an example, task input is generated in the form of natural language instructions, structured task descriptions, or a combination of both; the generated task input is filtered based on basic constraints to remove input samples that do not meet the tool interface constraints, have illegal parameter formats, or cannot be executed, so as to obtain a task input set.

[0029] As an example, the model of the agent to be distilled is initialized and trained. The training method can be trajectory format constraint, tool call legality constraint, or lightweight reinforcement learning, etc. The purpose is to ensure that the decision trajectory generated in the subsequent multi-trajectory sampling and uncertainty calculation process can be correctly parsed.

[0030] S102, sample from the task input set to obtain the candidate task input set.

[0031] As one embodiment, sampling from the task input set to obtain a candidate task input set includes: sampling from the task input set to obtain candidate inputs; performing redundancy removal processing on the candidate inputs to obtain a seed input set; and performing enhancement processing on the seed input set to obtain a diversified enhanced candidate task input set.

[0032] S103, sample each task input in the candidate task input set multiple times using the agent model to be distilled to obtain K corresponding candidate decision trajectories, where each candidate decision trajectory includes intermediate inference text, tool call sequence, tool execution result and final output result.

[0033] In other words, for each task input, the agent to be distilled is used to perform K independent samplings to generate K candidate decision trajectories.

[0034] S104. Based on semantic divergence, tool call sequence divergence, and output result divergence, perform uncertainty assessment on the K candidate decision trajectories for each task input to obtain the uncertainty score corresponding to each task input.

[0035] As an example, uncertainty assessment is performed on K candidate decision trajectories for each task input based on semantic divergence, tool call sequence divergence, and output result divergence to obtain an uncertainty score corresponding to each task input. This includes: vectorizing each candidate decision trajectory, calculating the cosine distance between decision trajectories and averaging them to obtain the semantic divergence; extracting the tool call sequence for each candidate decision trajectory, using normalized edit distance to measure the differences between different candidate decision trajectories at the tool call order and parameter levels to obtain the tool divergence; normalizing and grouping the final output results of each candidate decision trajectory, calculating the entropy of the output distribution and normalizing it to obtain the output result divergence; and weighted summing the semantic divergence, tool call sequence divergence, and output result divergence to obtain the uncertainty score corresponding to each task input.

[0036] S105, sort each task input according to the uncertainty score, and obtain the set of high uncertainty inputs based on the sorting results.

[0037] As an example, obtaining a set of high-uncertainty inputs based on the ranking results includes: selecting task inputs whose uncertainty scores are in the top N or whose scores exceed a preset threshold based on the ranking results, so as to obtain a set of high-uncertainty inputs.

[0038] S106. For each high uncertainty task input in the high uncertainty input set, obtain the reference decision trajectory generated by the teacher model, and align and score the K candidate decision trajectories corresponding to each high uncertainty task input with the reference decision trajectory to obtain the corresponding soft preference distribution.

[0039] As an example, for each high-uncertainty task input, the corresponding K candidate decision trajectories are aligned and scored with the reference decision trajectory to obtain the corresponding soft preference distribution. This includes: comparing the K candidate decision trajectories with the reference decision trajectory, calculating the differences between each candidate decision trajectory at the semantic level and the tool invocation level to obtain the comprehensive alignment score corresponding to each candidate decision trajectory; and feeding the comprehensive alignment score into a softmax function with a temperature parameter to obtain the corresponding soft preference distribution.

[0040] As an example, the soft preference distribution is obtained according to the following formula:

[0041] Where x represents a task input from a set of high-uncertainty inputs, i represents the index of a candidate decision trajectory, and K represents the total number of candidate decision trajectories corresponding to each task input x. This represents the overall alignment score between the i-th candidate decision trajectory and the reference decision trajectory. Indicates the temperature coefficient. This represents the preference probability value assigned to the i-th candidate decision trajectory under task input x.

[0042] In other words, for each task input in the set of high-uncertainty inputs, a reference decision trajectory generated by a reference agent or an existing reliable policy is obtained. The reference agent can be a larger, more capable teacher model (such as GPT-4, Claude, etc.), an existing agent that has been sufficiently trained and performs stably on the target task, or a reliable policy based on human annotation. Each task input corresponds to one reference decision trajectory.

[0043] In the semantic comparison, each candidate decision trajectory and the reference decision trajectory are vectorized to obtain corresponding semantic vectors. Then, the cosine similarity (or cosine distance) between the two vectors is calculated to measure the closeness of the two trajectories in overall semantic content. The closer the semantics, the higher the alignment score of the candidate trajectory at this level. In the tool invocation comparison, the tool invocation sequence (including tool name and parameter configuration) of each candidate decision trajectory and the tool invocation sequence of the reference decision trajectory are extracted. Normalized edit distance (Levenshtein distance) is used to measure the differences between the two at the tool invocation order and parameter level. The smaller the edit distance, the higher the alignment at the tool invocation level. Finally, the semantic comparison results and the tool invocation comparison results are weighted and combined to obtain a comprehensive alignment score for each candidate decision trajectory relative to the reference decision trajectory. The comprehensive alignment score is a scalar value; a higher value indicates that the candidate decision trajectory is closer to the reference decision trajectory and has a higher degree of alignment.

[0044] S107. Obtain the joint loss function based on the soft preference distribution, and iteratively train the agent model to be distilled according to the joint loss function until the preset convergence condition is met to obtain the trained decision trajectory generation model.

[0045] As an example, the joint loss function includes a supervised learning loss based on a reference decision trajectory and a trajectory-level distillation loss based on a soft preference distribution.

[0046] In other words, the supervised learning loss and trajectory-level distillation loss are weighted and combined using weight parameters to obtain a joint loss function. The model parameters of the agent to be distilled are then updated based on the joint loss function to obtain the agent to be distilled after one round of distillation training. The steps of task input selection, diversification enhancement, multi-trajectory sampling, uncertainty assessment, preference construction, and joint training are repeated to continuously iterate and optimize the model until the preset training rounds, model convergence conditions, or resource budget constraints are met, resulting in a distilled agent decision trajectory generation model.

[0047] S108 uses a trained decision trajectory generation model to perform multi-step decision-making and tool calls on the actual task input to generate a decision trajectory.

[0048] In other words, the agent model, which employs distillation, receives task input and executes a multi-step decision-making process of "planning-execution-feedback": first, it generates intermediate decisions or operational instructions; then, it calls external tools to obtain environmental feedback; and finally, it continuously generates subsequent decisions based on the feedback information, forming a multi-step, continuous execution process. Because the model has undergone targeted distillation and alignment training in high-uncertainty critical decision-making scenarios, it is able to make more stable tool selections and parameter settings.

[0049] To better understand the above technical methods, such as Figure 2 As shown, the following is a detailed explanation using a specific example, including the following steps: Step 1: Construct the initial task input set Based on the task description of the target application scenario and the interface specifications of the callable tools, construct an initial task input set. Task inputs are used to trigger the agent to perform multi-step decisions and tool invocation operations. The form of task inputs can be natural language instructions, structured task descriptions, or a combination of both. When constructing the initial task input set, the generated task inputs are filtered for basic constraints to eliminate input samples that do not meet tool interface constraints, have invalid parameter formats, or cannot be executed, thereby ensuring the executability of task inputs in subsequent processes.

[0050] Step 2: Initialize the agent to be distilled Select the language model agent to be distilled As training subjects. Before proceeding with uncertainty assessment and trajectory distillation, the agent to be distilled... Initial training is performed to ensure the system can stably output structured decision trajectories and correctly generate tool call formats. Initial training can be based on trajectory format constraints, tool call validity constraints, or lightweight reinforcement learning methods. Its purpose is to ensure that the decision trajectories generated during subsequent multi-trajectory sampling and uncertainty calculations can be correctly parsed.

[0051] Step 3: Enter the iterative trajectory distillation process After initialization, the iterative trajectory distillation process begins. In each iteration, the initial task input set is used... Select a portion of the task input as candidate input This is used for subsequent uncertainty assessment and trajectory distillation training.

[0052] Step 4: Generate a diverse set of candidate task inputs In each iteration, for candidate inputs Redundancy removal is performed to obtain a representative set of seed inputs. Subsequently, the seed input is enhanced, including but not limited to rewriting, recombination, or combination, to generate a diverse set of candidate task inputs. This ensures the coverage and effectiveness of subsequent uncertainty assessments.

[0053] Step 5: Multi-trajectory sampling to generate candidate decision trajectories For candidate input set For each input x in the process, the agent to be distilled is used. Multiple samplings are performed to generate K candidate decision trajectories, denoted as... Each candidate decision trajectory It includes intermediate inference text, tool call sequence, tool execution results, and final output results.

[0054] (1) Step 6: Screening high-value inputs based on uncertainty assessment Uncertainty assessment is performed on multiple candidate decision trajectories for the same input x. The uncertainty assessment consists of three parts: semantic divergence, tool call sequence divergence, and output divergence.

[0055] (2) First, semantic divergence measurement. For each candidate decision trajectory... Vectorization And calculate the cosine distance between the trajectories. The semantic divergence degree is obtained by averaging the distances across all trajectory pairs. .

[0056] (3) Secondly, tool call sequence divergence measurement. Extract the tool call sequence for each candidate trajectory. Normalized edit distance is used to measure the differences in tool call order and parameters among different trajectories, thus obtaining the tool divergence degree. The pairwise distances of the tool sequence differences are represented as follows: .

[0057] (4) Third, output divergence measurement. The final output of the candidate decision trajectories is normalized and grouped to... Indicates the first The output results are grouped in K The proportion of each candidate decision trajectory is used to calculate the entropy of the output distribution. , and according to Normalize.

[0058] Based on the above three types of divergence, the input is defined as follows: The uncertainty score U(x).

[0059] Based on uncertainty score Input set of candidate tasks Sort the task inputs and select the ones with the highest scores. Task inputs with a bit or uncertainty score exceeding a preset threshold constitute a high-uncertainty input set. This allows the subsequent trajectory distillation process to focus on key input scenarios where decision-making disagreements are significant.

[0060] Step 7: Obtain the reference decision trajectory and construct the soft preference distribution For highly uncertain input sets Each task input in , obtain from the reference agent Or a reference decision trajectory generated by an existing reliable strategy. Candidate decision trajectories Reference Decision Trajectory The differences between the two methods are compared, and the differences at the semantic level and the tool invocation level are calculated separately. A trajectory alignment score is then constructed. Furthermore, by using temperature parameters... The softmax function transforms the alignment scores into a soft preference distribution. .

[0061] (5) Step 8: Joint trajectory distillation training and model update After obtaining the soft preference distribution, the agent to be distilled... Perform model updates. Training objectives include supervised learning loss based on reference decision trajectories. And trajectory-level distillation loss based on soft preference distribution. Let the model's prediction distribution on the candidate trajectory set be... The joint loss function is defined as follows: ,in, These are the loss weight parameters.

[0062] (6) Step 9: Iterate until the termination condition is met. Repeat steps 3 through 8 until the preset number of training rounds is reached, the model convergence condition is met, or the resource budget constraint is satisfied, ultimately yielding the distilled agent model. .

[0063] Through the above specific implementation methods, the present invention can guide an agent to focus on learning key decision trajectories in high uncertainty decision-making scenarios under limited demonstration trajectory conditions, thereby improving the consistency and reliability of decision-making in multi-step task execution and reducing the risk of tool invocation errors and the resulting unreliable output.

[0064] In addition, the present invention also proposes a computer-readable storage medium storing an uncertainty-guided agent decision trajectory generation program thereon, which, when executed by a processor, implements the uncertainty-guided agent decision trajectory generation method described above.

[0065] In addition, this invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described method for generating intelligent agent decision trajectories based on uncertainty guidance.

[0066] Figure 3 This is a block diagram of an uncertainty-guided intelligent agent decision trajectory generation device according to an embodiment of the present invention. Figure 3 As shown, the uncertainty-guided intelligent agent decision trajectory generation device includes: a construction module 10, a sampling module 20, a candidate decision trajectory generation module 30, an uncertainty evaluation module 40, a sorting module 50, an alignment scoring module 60, an iterative training module 70, and a decision trajectory generation module 80. The system comprises the following modules: Construction module 10 acquires the task input set and constructs an initialized agent model to be distilled; Sampling module 20 samples from the task input set to obtain a candidate task input set; Candidate decision trajectory generation module 30 samples each task input in the candidate task input set multiple times using the agent model to be distilled to obtain K corresponding candidate decision trajectories, each including intermediate inference text, tool call sequence, tool execution result, and final output result; Uncertainty assessment module 40 assesses the uncertainty of the K candidate decision trajectories for each task input based on semantic divergence, tool call sequence divergence, and output result divergence to obtain an uncertainty score for each task input; and Ranking module 50 sorts the candidate decision trajectories based on their uncertainty scores. Qualitative scoring sorts each task input and obtains a set of high-uncertainty inputs based on the sorting results. Alignment scoring module 60 obtains a reference decision trajectory generated by the teacher model for each high-uncertainty task input in the set of high-uncertainty inputs, and aligns and scores the K candidate decision trajectories corresponding to each high-uncertainty task input with the reference decision trajectory to obtain the corresponding soft preference distribution. Iterative training module 70 obtains a joint loss function based on the soft preference distribution, and iteratively trains the agent model to be distilled based on the joint loss function until a preset convergence condition is met to obtain a trained decision trajectory generation model. Decision trajectory generation module 80 uses the trained decision trajectory generation model to perform multi-step decision-making and tool calls on the actual task input to generate a decision trajectory.

[0067] It should be noted that the foregoing explanation of the embodiment of the uncertainty-guided intelligent agent decision trajectory generation method also applies to the uncertainty-guided intelligent agent decision trajectory generation device of this embodiment, and will not be repeated here.

[0068] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0073] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0074] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0075] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0076] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0077] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0078] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0079] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for generating decision trajectories for intelligent agents based on uncertainty guidance, characterized in that, Includes the following steps: Obtain the task input set and construct the initialized agent model to be distilled; Samples are taken from the task input set to obtain a candidate task input set; For each task input in the candidate task input set, the agent model to be distilled is sampled multiple times to obtain K corresponding candidate decision trajectories, wherein each candidate decision trajectory includes intermediate inference text, tool call sequence, tool execution result and final output result; Uncertainty assessment is performed on the K candidate decision trajectories for each task input based on semantic divergence, tool call sequence divergence, and output result divergence to obtain the uncertainty score corresponding to each task input; Each task input is sorted according to the uncertainty score, and a set of high uncertainty inputs is obtained based on the sorting results; For each high uncertainty task input in the set of high uncertainty inputs, a reference decision trajectory generated by the teacher model is obtained, and the K candidate decision trajectories corresponding to each high uncertainty task input are aligned and scored with the reference decision trajectory to obtain the corresponding soft preference distribution. The joint loss function is obtained based on the soft preference distribution. The model of the agent to be distilled is iteratively trained based on the joint loss function until the preset convergence condition is met to obtain the trained decision trajectory generation model. The trained decision trajectory generation model is used to perform multi-step decision-making and tool calls on the actual task input to generate a decision trajectory.

2. The method for generating intelligent agent decision trajectories based on uncertainty guidance as described in claim 1, characterized in that, Sampling is performed from the task input set to obtain a candidate task input set, including: Candidate inputs are obtained by sampling from the set of task inputs; The candidate inputs are deredundant to obtain a seed input set; The seed input set is enhanced to obtain a diversified enhanced candidate task input set.

3. The method for generating intelligent agent decision trajectories based on uncertainty guidance as described in claim 1, characterized in that, Uncertainty is evaluated on the K candidate decision trajectories for each task input based on semantic divergence, tool call sequence divergence, and output result divergence to obtain an uncertainty score for each task input, including: Each candidate decision trajectory is vectorized, and the cosine distance between decision trajectories is calculated and averaged to obtain the semantic divergence degree. The tool call sequence of each candidate decision trajectory is extracted, and the normalized edit distance is used to measure the differences between different candidate decision trajectories in terms of tool call order and parameters to obtain the tool divergence degree. The final output of each candidate decision trajectory is normalized and grouped, the entropy of the output distribution is calculated and normalized to obtain the output divergence degree. The semantic divergence, tool call sequence divergence, and output result divergence are weighted and summed to obtain the uncertainty score corresponding to each task input.

4. The method for generating an agent's decision trajectory based on uncertainty guidance as described in claim 1, characterized in that, The high-uncertainty input set is obtained based on the sorting results, including: Based on the ranking results, select task inputs with uncertainty scores in the top N or scores exceeding a preset threshold to obtain a set of high uncertainty inputs.

5. The method for generating an agent's decision trajectory based on uncertainty guidance as described in claim 1, characterized in that, For each high-uncertainty task input, the corresponding K candidate decision trajectories are aligned and scored with the reference decision trajectory to obtain the corresponding soft preference distribution, including: The K candidate decision trajectories are compared with the reference decision trajectory, and the differences between each candidate decision trajectory at the semantic level and the tool invocation level are calculated to obtain the comprehensive alignment score corresponding to each candidate decision trajectory. The comprehensive alignment score is fed into a softmax function with a temperature parameter to obtain the corresponding soft preference distribution.

6. The method for generating an agent's decision trajectory based on uncertainty guidance as described in claim 5, characterized in that, The soft preference distribution can be obtained using the following formula: Where x represents a task input from a set of high-uncertainty inputs, i represents the index of a candidate decision trajectory, and K represents the total number of candidate decision trajectories corresponding to each task input x. This represents the overall alignment score between the i-th candidate decision trajectory and the reference decision trajectory. Indicates the temperature coefficient. This represents the preference probability value assigned to the i-th candidate decision trajectory under task input x.

7. The method for generating an agent's decision trajectory based on uncertainty guidance as described in claim 1, characterized in that, The joint loss function includes supervised learning loss based on the reference decision trajectory and trajectory-level distillation loss based on the soft preference distribution.

8. A computer-readable storage medium, characterized in that, It stores an uncertainty-guided agent decision trajectory generation program, which, when executed by a processor, implements the uncertainty-guided agent decision trajectory generation method as described in any one of claims 1-7.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating intelligent agent decision trajectories based on uncertainty guidance as described in any one of claims 1-7.

10. A device for generating decision-making trajectories for intelligent agents based on uncertainty guidance, characterized in that, include: The building module is used to obtain the task input set and to build the initialized agent model to be distilled; A sampling module is used to sample from the task input set to obtain a candidate task input set; The candidate decision trajectory generation module is used to sample each task input in the candidate task input set multiple times using the agent model to be distilled, so as to obtain K corresponding candidate decision trajectories. Each candidate decision trajectory includes intermediate inference text, tool call sequence, tool execution result and final output result. The uncertainty assessment module is used to assess the uncertainty of K candidate decision trajectories for each task input based on semantic divergence, tool call sequence divergence, and output result divergence, so as to obtain the uncertainty score corresponding to each task input. The sorting module is used to sort each task input according to the uncertainty score, and obtain a set of high uncertainty inputs based on the sorting results; The alignment scoring module is used to obtain the reference decision trajectory generated by the teacher model for each high uncertainty task input in the high uncertainty input set, and to align and score the K candidate decision trajectories corresponding to each high uncertainty task input with the reference decision trajectory to obtain the corresponding soft preference distribution. The iterative training module is used to obtain a joint loss function based on the soft preference distribution, and to iteratively train the agent model to be distilled based on the joint loss function until a preset convergence condition is met to obtain a trained decision trajectory generation model. The decision trajectory generation module is used to perform multi-step decision-making and tool invocation on the actual task input using the trained decision trajectory generation model in order to generate a decision trajectory.

Images