Integrated skill agent training method and electronic device

Abstract

The application discloses an integrated skill agent training method and electronic equipment. The method comprises the following steps: constructing a structured skill action space containing a plurality of atomic skill actions; performing a Monte Carlo tree search for each training sample to construct an inference trajectory tree, and obtaining a comprehensive score of a trajectory in the inference trajectory tree; screening an initial preferred trajectory and an initial poor trajectory based on the comprehensive score of the trajectory in the inference trajectory tree, and constructing an initial preference pair dataset; supervising fine-tuning of a strategy model based on the initial preferred trajectory, and obtaining a fine-tuned strategy model; performing multi-dimensional quality evaluation on the initial preferred trajectory and the initial poor trajectory, adjusting the optimization weight of each preference pair in the initial preference pair dataset based on the evaluation result, directly optimizing the fine-tuned strategy model by using the preference pair dataset with the adjusted weight, and obtaining a target strategy model. The application can improve the reasoning accuracy and skill arrangement efficiency of an agent in a multi-language multi-hop document question and answer.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning technology, and specifically to a method and electronic device for training intelligent agents with integrated skills. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) and Vision-Language Models (VLMs), agent technology has become an important paradigm for solving complex tasks. An agent is an autonomous system capable of perceiving environmental states, making policy-based decisions, and executing actions to achieve specific goals. In the field of Document Question Answering (DocQA), especially in multi-language multi-hop document QA scenarios, agents need to handle structurally heterogeneous, lengthy, and multi-language unstructured documents. To aggregate information from sparse evidence scattered across multiple pages and different layout elements (such as text blocks, tables, and charts), agents are typically endowed with a series of executable "skills." These skills include reasoning skills (such as query optimization and logic review) and utility skills (such as page selection, content parsing, element location, and region zooming). Agents dynamically orchestrate these skills to form a complete reasoning trajectory from question input to answer output. How to train an agent to efficiently and accurately orchestrate these skills and maintain logical consistency and correctness in long-range reasoning is a core challenge in the current technology field.

[0003] Existing agent training methods have mainly evolved along two paths: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). SFT learns behavioral patterns by having the model mimic high-quality human demonstration data, but its performance is limited by the coverage of the demonstration data and it is difficult to discover better policies through self-exploration. Therefore, reinforcement learning methods that introduce feedback mechanisms have become a key means to improve the decision-making ability of agents. Among them, Reinforcement Learning from Human Feedback (RLHF) is the most mainstream technical framework in recent years. The core idea of ​​RLHF is to use human preference ranking of the model's generated results to construct reward signals, thereby guiding the optimization of the policy model.

[0004] The standard implementation of RLHF typically includes two key stages: reward model training and policy optimization. First, it collects preference ranking data from human annotators on different model responses to the same prompt, training an independent reward model (RM). This model aims to fit the human preference distribution, outputting a scalar reward value for any given state-action pair or complete trajectory. Second, it uses the Proximal Policy Optimization (PPO) algorithm to update the policy model with the goal of maximizing the cumulative reward. PPO is a policy gradient-based reinforcement learning algorithm whose core lies in introducing a clipping mechanism to limit the difference between the old and new policies, preventing training collapse due to excessively large policy update steps. However, despite its theoretical stability advantages, PPO often exhibits extremely high instability during training in practical applications with large language models. This instability primarily stems from its high sensitivity to hyperparameters, such as the learning rate, clip range, value loss coefficient, and KL divergence penalty coefficient. Even small changes in these hyperparameters can lead to gradient explosion or vanishing gradients, making model convergence difficult. Furthermore, the PPO algorithm requires real-time sampling of a large number of trajectories and computation of the advantage function during training. This process involves multiple forward and backward propagations, incurring significant computational overhead and placing extremely high demands on hardware resources.

[0005] More importantly, traditional RLHF (Reference-Based Reasoning Highlighting) often employs sparse rewards based on the correctness of the final result in long-range reasoning tasks such as document question answering. Sparse rewards based on the correctness of the final result mean that only after the agent completes the entire reasoning trajectory and outputs the final answer, a binary (0 or 1) or continuous scalar reward value is given based on whether the answer matches the ground truth, without providing any explicit feedback signals in intermediate reasoning steps. In long trajectories of multi-hop reasoning, the agent may need to perform dozens of skill calls to reach the endpoint. If credit assignment is based solely on the correctness of the final answer, the model will face serious attribution difficulties: when the final answer is incorrect, the model cannot determine whether the failure was caused by an early page selection error, a mid-term element extraction bias, or a later logical deduction error; conversely, when the final answer is correct by chance, the model cannot distinguish which intermediate steps are truly effective and which are redundant or even erroneous. This mechanism, lacking fine-grained process supervision, is prone to causing the model to learn false correlations. That is, it retains the wrong path that leads to the correct answer, or discards the valid path that leads to the correct answer but has flaws in the middle. This causes the error to accumulate in long sequences, which severely limits the agent's generalization ability and robustness in complex tasks.

[0006] To overcome the complexity of training explicit reward models and the instability of the PPO algorithm, researchers have begun exploring preference optimization methods that do not require independent reward models and have attempted to introduce search algorithms to improve inference quality. Among these, Monte Carlo Tree Search (MCTS), a classic heuristic search algorithm, has been introduced into the field of large-scale model inference due to its outstanding performance in game theory and combinatorial optimization. To better understand the role of MCTS in agent training, it can be likened to a continuously growing "decision tree." In this tree, each node represents a "state" of the agent during the inference process, i.e., the set of all inference steps and contextual information generated so far; the edges connecting the nodes represent the "actions" taken by the agent, which in traditional text inference are usually generating the next sentence or thought chain fragment. The root node corresponds to the initial question, while the leaf nodes correspond to the state where inference has ended and the final answer has been given. MCTS searches for optimal paths in a vast possibility space through four stages of iterative iteration:

[0007] Selection: Starting from the root node, search downwards along the branches of the tree that have already been visited and have high evaluation value until a node with unexplored possibilities is found. This is similar to prioritizing the most promising paths in a maze.

[0008] Expansion: At the found node, attempt to generate new subsequent steps (i.e., new child nodes), thereby extending the tree outward.

[0009] Simulation / Rollout: Starting with the newly expanded node, quickly and randomly "test run" the remaining steps to see if the correct answer can be obtained. This step is to quickly estimate the potential of this new path.

[0010] Backpropagation: The simulation results (success or failure) and the value assessments of nodes along the path are propagated back to all ancestor nodes on the path, updating their statistics (such as number of visits, average score). This makes the value estimation of each node in the tree increasingly accurate.

[0011] Currently, the representative existing technology combining MCTS with self-training is ReST-MCTS* (Reinforced Self-Training via Monte Carlo Tree Search with Process Reward Guidance). ReST-MCTS* aims to address the scarcity of process supervision data in LLM self-training. Its core lies in automatically inferring the process reward for each step using tree search, thereby avoiding reliance on manually labeled process data. Its basic implementation process is as follows:

[0012] First, the system initializes a Policy Model and a Value Model. The Policy Model is responsible for generating inference steps, while the Value Model acts as a Process Reward Model (PRM), evaluating the quality of any part of the inference state. Simply put, the PRM is like a "referee"; it doesn't look at the final answer, but rather at whether each step in the inference process is reasonable.

[0013] Next, the MCTS search phase begins. For each problem to be solved, MCTS constructs a search tree. The root node of the tree represents the initial state of the problem. The search process involves the four iterative steps described above. Unlike standard MCTS, ReST-MCTS* introduces deep involvement of Policy Response Management (PRM) in the expansion and backtracking phases. Specifically, in the expansion phase, after the policy model generates several possible next actions, PRM immediately scores each newly generated intermediate step. This score reflects the probability that the step logically leads to the correct answer. This score is used as an important basis for the initial value estimation of new nodes, thus guiding MCTS to prioritize exploring branches that PRM deems "more reasonable," rather than blindly exploring randomly. In the backtracking phase, in addition to traditional win / loss feedback, step-level value labels provided by PRM are also used to update the value statistics of nodes.

[0014] After the search is completed, ReST-MCTS* uses the rich data collected in the search tree to perform self-evolution in two aspects:

[0015] On one hand, complete inference trajectories with correct final answers and high PRM scores for each node in the path are selected. These trajectories are considered high-quality demonstration data and used for supervised fine-tuning (SFT) of the policy model. This process is similar to supervised learning using high-quality demonstration data, aiming to make the policy model fit the distribution of the preferred trajectories, thereby increasing its probability of generating high-confidence inference steps.

[0016] On the other hand, the step-level value labels inferred from the tree structure (i.e., the comprehensive value estimate obtained after multiple searches and simulations of each intermediate node, representing the probability that the step leads to the correct answer) are used as training targets to train or update the value model (PRM). This makes the PRM, as the "judge," increasingly professional, enabling it to evaluate the quality of any intermediate step more accurately.

[0017] Through this closed-loop iteration of "strategy model generates data -> MCTS searches and evaluates -> updates value model -> value model guides data search and filtering -> updates strategy model", ReST-MCTS* achieves mutual promotion and self-evolution of strategy model and value model, improving the model's reasoning ability without the need for manual annotation of the right and wrong of each step.

[0018] While ReST-MCTS* has achieved remarkable results in logically rigorous tasks such as mathematical reasoning and code generation, its technical implementation mechanism reveals two fundamental technical flaws when directly applied to the specific domain of multilingual, multi-hop document question answering:

[0019] First, the disconnect between action space definition and document interaction skills leads to a lack of adaptive skill orchestration capabilities.

[0020] ReST-MCTS defines "actions" in MCTS as generating the next text fragment or thought chain step (TextualStep). However, in multilingual, multi-hop document question answering, the core capability of the agent lies in the structured interaction with the document environment, i.e., invoking atomic skills such as "page selection," "element location," and "region zooming." ReST-MCTS lacks native support for this structured skill action space, implicitly embedding skill invocation within free text generation, resulting in an excessively large search space lacking structural constraints. Under this mechanism, the model struggles to learn when to invoke specific tool skills to obtain precise evidence, instead tending to "guess" answers by generating large amounts of descriptive text. This search, lacking an explicit skill orchestration mechanism, often results in inefficient and redundant skill combinations in the generated trajectories, failing to adapt to the structured features of complex documents and hindering the reuse and transfer of skills across different tasks and multilingual scenarios.

[0021] Second, the evaluation dimension is too narrow and lacks fine-grained constraints on process quality, making it impossible to construct high-quality preference pairs to support refined optimization.

[0022] The value model (PRM) upon which ReST-MCTS relies primarily evaluates the probability of each step leading to the correct answer, i.e., a result-oriented, single-scalar value estimation. However, in complex document question-answering reasoning, a high-quality trajectory not only requires a correct final result but also must meet multi-dimensional process quality standards, including linguistic consistency in a multilingual environment, logical rigor of reasoning steps, and effectiveness of skill orchestration. ReST-MCTS's single-scalar PRM cannot distinguish between trajectories that are "logically correct but skill-redundant" and those that are "logically correct and skill-concise," nor can it effectively identify noisy data where "multilingual expressions are inappropriate but the result happens to be correct." Due to the lack of such multi-dimensional, fine-grained evaluation, ReST-MCTS* can only perform supervised fine-tuning (SFT) based on simple result correctness and value thresholds to filter data, and cannot construct "preferred-inferior" preference pairs that reflect differences in intermediate reasoning quality. This makes it difficult to effectively combine this method with advanced preference alignment algorithms such as Direct Preference Optimization (DPO), and to leverage DPO's advantages in correcting intermediate reasoning errors and improving training stability, thus limiting the agent's robustness and performance ceiling in long-range reasoning.

[0023] In summary, existing technologies for handling multilingual, multi-hop document question answering tasks suffer from two main bottlenecks: insufficient skill orchestration capabilities and a lack of fine-grained process supervision. Specifically, how to overcome the limitations of text-based action space to achieve adaptive orchestration of structured skills, and how to overcome the limitations of single-scalar evaluation to establish a multi-dimensional evaluation system encompassing language, logic, and skills to construct high-quality preference pairs and support collaborative optimization between SFT and DPO, remain pressing technical challenges. Therefore, a novel agent training method is urgently needed. This method defines a structured skill action space, utilizes MCTS to deeply explore skill combination trajectories, and automatically constructs high-quality preference pairs through multi-dimensional fine-grained evaluation to achieve performance breakthroughs for agents in complex document question answering scenarios. Summary of the Invention

[0024] The present invention aims to solve the above-mentioned technical problems and provide an intelligent agent training method and electronic device with integrated skills. By constructing a structured skill action space and combining Monte Carlo tree search and multi-dimensional weighted preference optimization, efficient, accurate and interpretable intelligent agent training can be achieved.

[0025] To address the aforementioned technical problems, the first aspect of this invention provides a method for training an intelligent agent with integrated skills, comprising:

[0026] Construct a structured skill action space containing multiple atomic skill actions;

[0027] For each training sample, a Monte Carlo tree search is performed to construct an inference trajectory tree, and a comprehensive score of the trajectory in the inference trajectory tree is obtained; wherein, the expansion phase of the Monte Carlo tree search is based on sampling candidate actions in the structured skill action space;

[0028] Based on the comprehensive scores of the trajectories in the inference trajectory tree, the initial preferred trajectories and the initial unpreferred trajectories are selected to construct the initial preference pair dataset;

[0029] The policy model is supervised and fine-tuned based on the initial preferred trajectory to obtain the fine-tuned policy model.

[0030] The initial preferred trajectory and the initial undesired trajectory are subjected to multi-dimensional quality evaluation. Based on the evaluation results, the optimization weights of each preference pair in the initial preference pair dataset are adjusted. The adjusted preference pair dataset is then used to perform direct preference optimization on the fine-tuned policy model to obtain the target policy model.

[0031] In some embodiments, the structured skill action space includes a set of reasoning skills and a set of tool skills, both of which are composed of the atomic skill actions;

[0032] The set of reasoning skills includes query optimization skills for optimizing query content and review and correction skills for correcting intermediate reasoning steps;

[0033] The set of tools and skills includes page selection skills for retrieving relevant document pages, page parsing skills for extracting structured elements from document layouts, element selection skills for identifying key evidence, and region magnification skills for zooming in on local areas.

[0034] In some embodiments, the multi-dimensional quality assessment includes language consistency assessment, logical rigor assessment, and skill arrangement effectiveness assessment;

[0035] The step of adjusting the optimization weights of each preference pair in the initial preference pair dataset based on the evaluation results includes:

[0036] Calculate the sub-scores of the initial preferred trajectory and the initial undesired trajectory in three dimensions: language consistency, logical rigor, and skill arrangement effectiveness;

[0037] The sub-scores are weighted and aggregated to obtain a multi-dimensional total evaluation score for each trajectory;

[0038] The optimization weights for the corresponding preference pairs are determined based on the difference in the total multi-dimensional evaluation scores of the initial preferred trajectory and the initial undesired trajectory.

[0039] In some embodiments, the weighted aggregation of the sub-scores to obtain a multi-dimensional total evaluation score for each trajectory includes:

[0040] Obtain preset weights for language consistency, logical rigor, and skill arrangement effectiveness;

[0041] The language consistency sub-score is multiplied by the language consistency weight to obtain the first weighted score;

[0042] The second weighted score is obtained by multiplying the logical rigor score by the logical rigor weight.

[0043] The third weighted score is obtained by multiplying the sub-score for skill arrangement effectiveness by the skill arrangement effectiveness weight.

[0044] The first weighted score, the second weighted score, and the third weighted score are summed to obtain the total multi-dimensional evaluation score.

[0045] In some embodiments, the step of directly optimizing the fine-tuned policy model using the weighted preferences on the dataset includes:

[0046] Construct a direct preference optimization loss function, in which the optimization weights are included as coefficients to amplify or reduce the contribution of a specific preference to the update of model parameters;

[0047] Based on the direct preference optimization loss function, the parameters of the fine-tuned policy model are updated to maximize the log probability difference between the preferred trajectory and the undesired trajectory.

[0048] In some embodiments, the Monte Carlo tree search further includes a selection phase, a simulation phase, and a backtracking phase;

[0049] In the selection phase, starting from the root node, child nodes are recursively selected until leaf nodes are selected based on the historical average value of the nodes.

[0050] In the simulation phase, starting from the newly created child node, subsequent atomic skill actions are sampled until the termination state, and a simulation reward based on the correctness of the termination state is obtained.

[0051] In the backtracking phase, the simulated reward is propagated backward along the selected path, accumulated to the visit count of each node on the path, and the historical average value of each node is updated.

[0052] The overall score is determined based on the historical average value of the leaf nodes or path nodes after the backtracking phase update.

[0053] In some embodiments, the sampling of candidate actions based on the structured skill action space includes:

[0054] Obtain the status information of the current node, which includes the currently executed atomic skill action sequence and the corresponding document context;

[0055] The state information is input into the strategy model, and the execution probability of each atomic skill action in the structured skill action space is output.

[0056] Based on the execution probability, N candidate atomic skill actions are sampled, where N is an integer greater than 1;

[0057] For each candidate atomic skill action, a corresponding child node is generated, and the candidate atomic skill action is used as an edge connecting the current node and the corresponding child node.

[0058] In some embodiments, the step of selecting initial preferred trajectories and initial undesirable trajectories based on the comprehensive scores of the trajectories in the inference trajectory tree includes:

[0059] Calculate the cumulative value score for each complete trajectory in the inference trajectory tree;

[0060] Sort all complete trajectories from highest to lowest according to the cumulative value score;

[0061] The complete trajectory with the top-ranked preset proportion is selected as the initial preferred trajectory;

[0062] The complete trajectory after ranking is selected as the initial inferior trajectory according to a preset ratio.

[0063] In some embodiments, the method further includes:

[0064] The target strategy model is deployed on the server or client side to receive document question-and-answer requests input by users, dynamically arrange atomic skill actions based on the target strategy model, generate reasoning trajectories, and output answers.

[0065] A second aspect of the present invention provides an electronic device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the agent training method for integrated skills as described in any of the preceding claims.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] 1. Improved accuracy and interpretability of reasoning: By constructing a structured skill action space, the infinite text generation space is mapped to a finite discrete action space. Combined with Monte Carlo tree search, optimal skill combination paths can be systematically explored. Compared to traditional end-to-end generation, the reasoning trajectory generated by this method has a clear semantic structure, which not only improves the accuracy of the final answer but also makes the reasoning process transparent and traceable.

[0068] 2. Fine-grained process supervision was achieved: A multi-dimensional quality assessment mechanism (language consistency, logical rigor, and skill arrangement effectiveness) was introduced, overcoming the limitations of traditional methods that rely solely on the binary label of the final answer. By dynamically adjusting the optimization weights of preference pairs, the model can distinguish between trajectories that are "accidentally correct but logically flawed" and those that are "logically rigorous and efficient," thereby learning more robust reasoning strategies.

[0069] 3. Enhanced multilingual generalization ability: The language consistency module in the multi-dimensional evaluation is specifically designed for multilingual scenarios, effectively constraining the semantic fidelity of the model when switching between different languages. Experiments show that this method achieves significant performance improvements on multiple languages, including low-resource languages, solving the problem of poor generalization ability of existing models in cross-language document question answering.

[0070] 4. Improved training efficiency and data utilization: Preference pairs are constructed using trajectories automatically generated by MCTS, eliminating the need for expensive manually labeled data. Simultaneously, the weighted direct preference optimization mechanism allows the model to focus on high-quality, differentiated samples, accelerating the convergence process and achieving high performance with minimal resources—that is, surpassing the performance of large-scale general-purpose models even on a model with a smaller parameter scale. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a schematic diagram of the overall process of the intelligent agent training method for integrated skills provided in the embodiments of the present invention;

[0073] Figure 2 This is a schematic diagram of the composition structure of the structured skill action space provided in an embodiment of the present invention;

[0074] Figure 3 This is a schematic diagram of the Monte Carlo tree search process based on the structured skill action space provided in an embodiment of the present invention;

[0075] Figure 4 This is a detailed processing flowchart of the extended stage provided in an embodiment of the present invention;

[0076] Figure 5 This is a schematic diagram of the filtering process for constructing a preference-based dataset provided in an embodiment of the present invention;

[0077] Figure 6 This is a schematic diagram of the architecture of the multi-dimensional quality assessment system provided in this embodiment of the invention;

[0078] Figure 7 This is a schematic diagram of the weighted direct preference optimization process provided in an embodiment of the present invention;

[0079] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention;

[0080] Figure 9 This is a schematic diagram of the architecture of the intelligent agent reasoning system provided in an embodiment of the present invention;

[0081] Figure 10 This is a schematic diagram of the internal processing logic of the core inference engine provided in the embodiments of the present invention. Detailed Implementation

[0082] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0083] This invention provides a method for training an agent with integrated skills. This method aims to solve the technical problems of agents struggling to adaptively arrange structured skills and lacking fine-grained process supervision in multilingual, multi-hop document question-answering scenarios. Figure 1 A schematic diagram of the overall process of this training method is shown. Figure 1 As shown, this method mainly includes the following steps: constructing a structured skill action space, performing Monte Carlo tree search to construct an inference trajectory tree, selecting initial preference pairs in the dataset, supervising and fine-tuning the policy model, performing multi-dimensional quality assessment and weight adjustment, and directly optimizing preferences to obtain the target policy model. The specific implementation of each step will be described in detail below with reference to the accompanying figures.

[0084] In step S110, a structured skill action space containing multiple atomic skill actions is constructed. In multilingual, multi-hop document question answering tasks, the agent needs to interact with a complex document environment. This interaction is not simple text generation but relies on a series of executable atomic operations. To standardize the agent's behavior space and reduce search complexity, this embodiment predefines a structured skill action space. This space consists of a finite number of atomic skill actions, each corresponding to a specific, indivisible operation instruction.

[0085] Figure 2 A schematic diagram illustrating the composition of a structured skill action space is shown. For example... Figure 2 As shown, the structured skill action space 200 can be divided into two main subsets: the reasoning skill set 201 and the tool skill set 202. The reasoning skill set 201 mainly involves the internal thought processing of the agent, used to optimize reasoning paths and correct logical errors; the tool skill set 202 mainly involves the interaction between the agent and the external document environment, used to acquire and process evidence information.

[0086] Specifically, the reasoning skill set 201 includes query optimization skill 201a and review and correction skill 201b. Query optimization skill 201a is used to regenerate or rewrite query statements when the current search results are unsatisfactory or the question is vague, in order to improve the accuracy of subsequent searches. For example, when the initial query "2023 profit" returns too many irrelevant results, the agent calls this skill to optimize the query to "net profit in the fourth quarter of 2023", thereby narrowing the search scope. Review and correction skill 201b is used to self-reflect on and correct intermediate conclusions generated during the reasoning process. For example, when the agent finds a logical contradiction between previous and subsequent steps, it calls this skill to re-evaluate the validity of the previous step and generate a corrected reasoning step.

[0087] The tool skills set 202 includes page selection skill 202a, page parsing skill 202b, element selection skill 202c, and region zooming skill 202d. Page selection skill 202a is used to locate the most relevant page to the current query from a multi-page document. In multi-hop question answering, evidence is often scattered across different pages; this skill can filter out the top-K candidate pages from dozens or even hundreds of pages of documents based on semantic similarity scoring. Page parsing skill 202b is used to extract structured information from selected pages. Since documents may contain various layout elements such as text, tables, and charts, this skill can identify page layouts and convert unstructured visual information into machine-readable structured data, such as JSON-formatted tabular data or lists of text blocks. Element selection skill 202c is used to locate key evidence in the parsed structured data. For example, in a large financial statement, this skill can accurately locate the cell value at the intersection of the "Revenue" row and the "2023" column. Region zooming skill 202d is used to process detailed information in high-resolution documents. When crucial evidence is located in an image or small font area, it is difficult to identify at normal resolution. This skill can perform local magnification and super-resolution reconstruction of a specified area to improve the accuracy of optical character recognition (OCR) or visual understanding.

[0088] By constructing the structured skill action space described above, the agent's decision-making process is mapped from an infinite text generation space to a finite discrete action space. This mapping not only reduces the dimensionality of the search space but also endows the agent with explicit operational semantics. In practical deployment, each atomic skill action can be encapsulated as an independent function module or API interface. The agent only needs to output the action identifier and necessary parameters to drive the underlying execution engine to complete the corresponding operation. This decoupled design allows skill modules to be updated and optimized independently without affecting the structure of the overall policy model.

[0089] In step S120, Monte Carlo Tree Search (MCTS) is performed for each training sample to construct an inference trajectory tree, obtaining a comprehensive score for the trajectories in the inference trajectory tree. Traditional MCTS algorithms are typically applied to game theory or text-based step-length inference tasks, where the action space is often tokens in the vocabulary or natural language sentences. However, in this embodiment, the action space of MCTS is strictly limited to the structured skill action space constructed in step S110. This change ensures that the search process is no longer blindly generating text, but rather finding optimal execution sequences within a limited skill combination space.

[0090] Figure 3 A schematic diagram of the Monte Carlo tree search process based on a structured skill action space is shown. (For example...) Figure 3As shown, the inference trajectory tree 300 consists of multiple nodes 301 and edges 302 connecting the nodes. The root node 301a represents the initial state, i.e., the user-input question and the initial document context. Each non-root node 301 represents an intermediate state, which is composed of all executed atomic skill action sequences on the path from the root node to the current node and their resulting document contexts. Each edge 302 represents an atomic skill action selected from the structured skill action space.

[0091] The MCTS process comprises four iterative phases: Selection, Expansion, Simulation, and Backpropagation. Among these, the Expansion phase is the core improvement in this embodiment, which is based on sampling candidate actions from the structured skill action space.

[0092] During the selection phase, starting from the root node 301a, child nodes are recursively selected using a tree policy until an expandable leaf node 301b is reached. A commonly used tree policy is the Upper Confidence Bound applied to Trees (UCT) algorithm, whose formula is as follows:

[0093] (1)

[0094] in, Indicates the current child node, Indicates the parent node, Represents a node The cumulative reward value, Represents a node Number of visits This indicates the total number of times the parent node has been visited. The exploration coefficient is used to balance exploitation and exploration. This formula ensures that the search process both favors nodes with high historical average value and does not ignore nodes with fewer visits and unknown potential value.

[0095] During the expansion phase, when leaf node 301b is reached, the system needs to sample candidate actions from the structured skill action space to create new child nodes. Figure 4 The detailed processing flow of the expansion phase is shown. For example... Figure 4As shown, the current node's state information 401 is first obtained. This information includes the currently executed atomic skill action sequence (such as "page selection -> page parsing") and the current document context (such as the extracted table data). Next, the state information 401 is input into the policy model 402. The policy model 402 is a pre-trained large language model whose output layer has been modified to predict the probability of the next token instead of the probability of the next token, thus predicting the execution probability distribution 403 of each atomic skill action in the structured skill action space.

[0096] Assuming the structured skill action space includes Each atomic skill action, the strategy model 402 outputs a dimension of... probability vector ,in Indicates execution of the first The probability of each atomic skill action. To improve search efficiency, not all possible actions are typically expanded; instead, samples are taken based on the probability distribution. The system identifies high-probability candidate atomic skill actions. For example, Top-K sampling or nucleus sampling strategies can be employed. For each selected candidate atomic skill action, a corresponding child node is created, and the action is used as an edge connecting the parent and child nodes. Simultaneously, the atomic skill action is executed, the environment state is updated, a new document context is obtained, and this context is stored in the child node.

[0097] This strategy-model-guided expansion approach allows MCTS to quickly focus on promising branches within a vast skill set space, avoiding the inefficiencies of random search. More importantly, because actions are structured skills, the state changes of each child node are deterministic and interpretable, providing a clear foundation for subsequent value assessment.

[0098] During the simulation phase, starting from the newly expanded child nodes, subsequent actions are quickly sampled until the termination state using the default policy to estimate the potential value of that node. The default policy can be a stochastic policy or a lightweight, fast policy model. The simulation process generates a complete inference trajectory and awards simulation rewards based on the correctness of the final answer. If the final answer matches the standard answer, a positive reward (e.g., +1) is given; otherwise, a negative reward or zero reward is given. Additionally, process rewards can be introduced, such as a small reward for each successful use of an effective skill, to encourage efficient skill orchestration.

[0099] During the backtracking phase, the reward values ​​obtained in the simulation phase are propagated back to the root node along the selected path. For each node on the path, its visit count is updated. and cumulative reward value The specific update rules are as follows: , ,in This is a simulated reward. After multiple iterations, the historical average value of each node is... It gradually converges to the true expected return in this state.

[0100] After completing a predetermined number of MCTS iterations (e.g., 1000 searches per question), the inference trajectory tree is constructed. At this point, multiple complete inference trajectories can be extracted from the tree. The overall score for each trajectory can be determined based on the historical average value of its leaf nodes or nodes along the path. For example, the average value of all nodes on the trajectory can be used as the overall score, or only the value estimates of the leaf nodes can be used. The overall score reflects the likelihood that the trajectory was evaluated as leading to the correct answer during the search process.

[0101] Through the aforementioned MCTS process, the system not only obtains the final answer but also accumulates a large amount of intermediate inference data, including which skill combinations are effective, which are ineffective, and which skill is more likely to succeed in a specific state. This data forms the basis for subsequent model optimization. It is worth noting that, because the action space is structured, the generated trajectories have a clear semantic structure, facilitating subsequent automated analysis and evaluation.

[0102] In step S130, based on the comprehensive scores of the trajectories in the inference trajectory tree, initial preferred trajectories and initial unpreferred trajectories are selected to construct the initial preference pair dataset. The quality of trajectories generated by MCTS search varies greatly; some trajectories are logically sound, skill-efficient, and provide correct answers, while others are redundant, lengthy, misuse skills, and provide incorrect answers. To construct high-quality training data, trajectories need to be selected based on their comprehensive scores.

[0103] Figure 5 This illustrates a filtering process for constructing a dataset based on preferences. Figure 5 As shown, the system first calculates the cumulative value score 501 for each complete trajectory in the inference trajectory tree. The cumulative value score can be the average value of all nodes on the trajectory, or a weighted sum, with the weight decreasing with depth to emphasize the importance of early decisions. Next, all complete trajectories are sorted from highest to lowest cumulative value score 502.

[0104] Two thresholds or percentages can be set, such as the top 20% and the bottom 20%. The top 20% of trajectories are selected as the initial preferred trajectories 503. These trajectories typically have high overall scores, meaning they are frequently accessed during the search process and have high evaluation value, likely containing correct reasoning logic and effective skill arrangement. The bottom 20% of trajectories are selected as the initial inferior trajectories 504. These trajectories have low overall scores and may contain logical errors, skill abuse, or incorrect final answers.

[0105] An initial preferred trajectory and an initial undesired trajectory are paired to form a preference pair. Each preference pair contains a positive sample (preferred trajectory) and a negative sample (undesired trajectory), along with a corresponding prompt. This results in an initial preference pair dataset of 505. This dataset reflects the model's relative preference in the original search space, indicating that the model should be more inclined to generate preferred trajectories than undesired ones.

[0106] However, preference pairs selected solely based on the overall MCTS score have limitations. MCTS value estimation, primarily based on the correctness of the final answer and heuristic evaluations during the search process, may fail to capture subtle flaws in the trajectory's language expression, logical coherence, or skill efficiency. For example, two trajectories might both provide the correct answer, but one might use redundant skill calls (such as repeatedly parsing the same page), while the other is concise and efficient. MCTS might assign similarly high scores because both ultimately answer correctly, making it impossible to distinguish between superior and inferior trajectories. To address this issue, this embodiment introduces a subsequent multi-dimensional quality assessment step to refine the initial preference pairs.

[0107] In step S140, the policy model is subjected to supervised fine-tuning (SFT) based on the initial optimized trajectory to obtain the fine-tuned policy model. SFT is a warm-up stage before reinforcement learning, which aims to allow the policy model to initially grasp the behavioral patterns of high-quality trajectories.

[0108] Specifically, state-action pairs are extracted from the initial optimized trajectory. For each step in the trajectory, the state represents the current context information, and the action represents the atomic skill action executed at that step and its parameters. These state-action pairs are then used to construct supervised training data. The training objective is to minimize the cross-entropy loss between the policy model's predicted action distribution and the actual executed actions.

[0109] (2)

[0110] in, The parameter is The strategy model For a moment state, For a moment The actual atomic skill actions performed. Let be the trajectory length. By minimizing this loss function, the policy model learns the probability distribution of performing high-value skills in a given state.

[0111] The purpose of the SFT stage is to allow the model to "imitate" excellent solutions. After SFT, the policy model is more inclined to choose skill combinations that have proven effective in MCTS when generating skill actions. This provides a better starting point for the subsequent DPO stage, avoiding the instability caused by reinforcement learning from scratch. It should be noted that SFT only utilizes the positive information of the preferred trajectory and not the negative information of the undesired trajectory. Therefore, the model may still retain some subtle bad habits, which need to be corrected in subsequent steps.

[0112] In step S150, a multi-dimensional quality assessment is performed on the initial preferred trajectory and the initial unpreferred trajectory. Based on the assessment results, the optimization weights of each preference pair in the dataset are adjusted. This step is the key difference between this invention and traditional reinforcement learning alignment methods. Traditional Direct Preference Optimization (DPO) typically assumes that all positive samples have equally high quality and all negative samples have equally low quality, or it is trained based solely on simple binary labels (win / loss). However, in complex scenarios such as multilingual, multi-hop document question answering, the quality of a trajectory is a continuous and multi-dimensional spectrum. Even if a trajectory's final answer is correct, if there are logical jumps, confusing language expressions, or redundant skill calls in its intermediate reasoning process, its "purity" as a positive sample will be greatly reduced. Conversely, a trajectory with an incorrect answer but rigorous reasoning logic and reasonable skill calls contains a different level of negative signal intensity than a completely fabricated trajectory.

[0113] Figure 6 A schematic diagram of the architecture of a multi-dimensional quality assessment system is shown. For example... Figure 6 As shown, the multi-dimensional quality assessment system 600 includes three parallel assessment modules: a language consistency assessment module 601, a logical rigor assessment module 602, and a skill arrangement effectiveness assessment module 603. These three modules independently score the input training trajectory 604, and finally, a weighted aggregation module 605 generates a multi-dimensional assessment total score 606 for the trajectory.

[0114] The language consistency evaluation module 601 is primarily used to evaluate the semantic consistency and alignment of multilingual text. In a multilingual environment, the intermediate reasoning steps generated by the agent may mix multiple languages ​​(e.g., Chinese questions, English documents, Indonesian reasoning), or contain grammatical errors, unclear references, etc. This module can use a pre-trained multilingual natural language understanding model (such as mBERT or XLM-R) as the encoder to calculate the semantic similarity between adjacent steps in the trajectory, as well as the consistency of the entire trajectory with the original question in terms of language style. Specifically, the trajectory can be divided into several sentence or paragraph units, and the conditional probability score of each unit with the context can be calculated. If the language expression of a certain step is abrupt or breaks with the preceding and following text, the score of that step is low. The language consistency score of the entire trajectory is calculated. It can be defined as the geometric mean of the scores of all steps, to penalize any single low-quality step:

[0115] (3)

[0116] in, Indicates time The generated text content, Indicates the previous text history, Indicates the current document context. This represents the conditional probability given by the language model. This score reflects the fluency and coherence of the trajectory at the linguistic level.

[0117] The Logical Rigor Assessment Module 602 evaluates the logical coherence and sufficiency of evidence in reasoning steps. In multi-hop question answering, each step of reasoning should be supported by clear evidence, and there should be causal or progressive relationships between the steps. This module can analyze the trajectory by constructing a logical dependency graph. Specifically, it extracts the claim and evidence cited in each step of reasoning and uses a Natural Language Inference (NLI) model to determine whether the evidence implies the claim. If the claim in a certain step cannot be derived from the evidence cited or the conclusion of the previous step, it is judged as a logical break. Logical Rigor Score The average confidence level of the logical implication relation across all steps can be calculated:

[0118] (4)

[0119] in, Indicates time The cited evidence fragments, Indicates time The conclusion drawn from the reasoning Represents a natural language reasoning model. This represents the confidence level of the implied probability in the model's output. The higher the score, the more rigorous the reasoning process of the trajectory and the lower the risk of hallucination.

[0120] The Skill Orchestration Effectiveness Assessment Module 603 is used to evaluate the rationality of the selection and combination strategies of atomic skill actions. Different skill combinations lead to different execution efficiencies and resource consumption. For example, for simple fact queries, directly calling the "element selection" skill may be more efficient than first "page selection" and then "page parsing"; while for complex chart analysis, the "region zoom" skill may be necessary. This module evaluates the skill sequences in the trajectory based on a preset skill efficiency rule base and historical execution statistics. Specifically, a skill redundancy index can be defined to detect whether there are repeated calls to the same skill to process the same data, or whether there are unnecessary intermediate transformation steps. Skill Orchestration Effectiveness Score It can be represented as:

[0121] (5)

[0122] in, Skill redundancy is defined as the proportion of repeated or invalid skill calls to the total number of calls. This represents the normalized execution cost, including the number of API calls and processing time. and This is an adjustment factor used to balance the impact of redundancy and cost. This score encourages agents to adopt streamlined and efficient skill orchestration strategies.

[0123] After obtaining the sub-scores in the three dimensions, the weighted aggregation module 605 performs a linear weighting on them to obtain the multi-dimensional evaluation total score for each trajectory. :

[0124] (6)

[0125] in, , , These are weights for language consistency, logical rigor, and skill arrangement effectiveness, respectively, and satisfy the following conditions: These weights can be adjusted based on the specific application scenario. For example, in a financial Q&A scenario where accuracy is extremely important, the weights can be increased. The value; and in real-time interactive scenarios where response speed is sensitive, it can be improved. The value of .

[0126] After obtaining the multi-dimensional evaluation score for each trajectory, the system adjusts the optimization weights of each preference pair in the initial preference pair dataset based on this score. In the initial preference pair dataset, each pair contains an initial preferred trajectory. (winner) and an initial inferior selection trajectory (Loser). Traditional DPO methods typically assign the same weight to all preference pairs. This embodiment dynamically adjusts the weights based on the quality difference between the two trajectories. Specifically, it defines the preference intensity difference. for:

[0127] (7)

[0128] Optimize weights It can be set to Monotonically increasing functions, for example:

[0129] (8)

[0130] in, For the Sigmoid function, This is a temperature coefficient used to control the distribution range of the weights. When... A larger value indicates that the preferred trajectory is significantly better than the inferior trajectory in multiple dimensions. This preference provides a strong supervisory signal and is therefore assigned a higher weight. ;when When the weights are small, it indicates that the two samples are of similar quality, the supervision signal is weak, and they are assigned lower weights. This mechanism makes the model pay more attention to samples with significant quality differences and strong teaching significance during training, thereby accelerating convergence and increasing the upper limit of the final model's performance.

[0131] In step S160, the adjusted preferences are used to perform Direct Preference Optimization (DPO) on the fine-tuned policy model to obtain the target policy model. DPO is a preference optimization algorithm that does not require explicit training of the reward model. It optimizes the preference data directly on the policy model by reconstructing the loss function.

[0132] Figure 7 The processing flow of weighted direct preference optimization is shown. For example... Figure 7 As shown, the inputs include a fine-tuned policy model 701, a reference model 702 (usually the initial model before SFT or a copy of the model after SFT), and optimization weights. The preferences of dataset 703.

[0133] The standard DPO loss function aims to maximize the log probability difference between the preferred trajectory and the inferior trajectory. This embodiment introduces optimization weights based on this. Construct a weighted direct preference optimization loss function :

[0134] (9)

[0135] in, This represents the expectation operation. Indicates from dataset Sampling prompts, preferred trajectory, and undesired trajectory. This represents the target strategy model to be optimized. Represents the reference model. These are hyperparameters used to control the degree of deviation from the reference model. This is the Sigmoid function.

[0136] The core idea of ​​formula (9) is: if the model Assigning a preferred trajectory The probability relative to the reference model Significantly increased, while also assigning a poor selection trajectory. If the probability decreases significantly, the term within the parentheses is positive and large, resulting in a smaller loss. This is achieved by multiplying by the weight. Those preferences for high-quality differences contribute more to the gradient, thus dominating the direction of parameter updates.

[0137] During training, the system reads preference pairs in batches and calculates the model for each trajectory. and The logarithmic probability is used to calculate the loss in formula (9), and the model parameters are updated through backpropagation. To prevent over-optimization of the model from leading to catastrophic forgetting or distribution shift, limitations are typically imposed. The value is within a small range (e.g., 0.1-0.5), and is constrained by KL divergence. and The distance.

[0138] After several rounds of iterative training, training stops when the loss function converges or reaches the preset number of training steps, resulting in the final target policy model. This model not only inherits the basic skill execution capabilities learned in the SFT stage, but also learns how to distinguish between superior and inferior performance under multi-dimensional criteria through the DPO stage, thus exhibiting higher linguistic consistency, logical rigor, and skill orchestration efficiency when generating inference trajectories.

[0139] The embodiments consisting of steps S110 to S160 can achieve the following technical effects:

[0140] First, by constructing a structured skill action space and binding the MCTS extension stage to this space, the problem of excessively large and difficult-to-converge search spaces in traditional text-generative search methods is solved. The discreteness and determinism of atomic skill actions make the structure of the search tree clearer, enabling the agent to learn reusable skill combination patterns rather than rote memorization of specific text sequences. This demonstrates stronger generalization ability when faced with unseen document structures or problems.

[0141] Second, a multi-dimensional quality assessment mechanism is introduced to address the shortcomings of relying solely on the correctness of the final answer for supervision. Actual testing revealed that approximately 30% of the trajectories selected based solely on answer correctness contained logical inconsistencies or grammatical errors. These problems are amplified in long-range reasoning, leading to final errors. Fine-grained assessment across three dimensions—language, logic, and skill—effectively eliminates these "pseudo-high-quality" trajectories, ensuring the high purity of the training data.

[0142] Third, by dynamically adjusting the DPO optimization weights based on multi-dimensional evaluation scores, fine-grained preference alignment is achieved. Compared to uniformly weighted DPO, weighted DPO can more effectively utilize the supervision signals of high-quality samples while suppressing noise interference from low-quality, disparate samples. Experiments show that, with the same amount of data and number of training steps, weighted DPO can improve the accuracy of the model on complex multi-hop question-answering tasks by approximately 5-8 percentage points, and significantly improve the fluency and interpretability of the reasoning process.

[0143] Fourth, the two-stage training strategy of SFT and weighted DPO balances learning efficiency and alignment effectiveness. SFT provides good initialization parameters, avoiding the instability of DPO in the early stages of training; weighted DPO further refines the model's behavioral preferences based on SFT, making it more in line with the multi-dimensional evaluation criteria of human experts. This combined strategy significantly improves the robustness and professionalism of the model in complex scenarios while ensuring its basic capabilities.

[0144] In one different embodiment, the specific implementation of multi-dimensional evaluation can be modified. For example, in addition to using a pre-trained model for scoring, a rule-based checker can be introduced. For the effectiveness of skill orchestration, some hard rules can be preset, such as "prohibiting the consecutive invocation of the same page selection skill" or "the page must be selected before parsing." Trajectories that violate these hard rules are directly given the lowest score or removed from the preferred trajectory pool. This hybrid evaluation method combines the flexibility of neural networks with the determinism of rule systems, further ensuring the baseline quality of the training data.

[0145] In another different embodiment, there can be several variations in how the optimization weights are calculated. Besides using the sigmoid function mapping, piecewise linear functions or exponential functions can also be employed. For example, a threshold can be set. ,when When, the weight is set to 1; when When the weight is set to 0 (i.e., the preference pair is discarded because the preferred trajectory is actually of lower quality than the undesirable trajectory, which may be due to labeling errors or search anomalies); when the weight is between 0 and... Linear interpolation between the two values. This hard thresholding method is simpler to calculate and can completely filter out invalid preference pairs, making it suitable for scenarios with extremely high data quality requirements and limited computing resources.

[0146] Furthermore, a parallel search mechanism can be introduced into the MCTS search process. Since the expansion and simulation of each node are independent, multiple MCTS instances can be executed in parallel using a distributed computing cluster, and then the generated trajectory trees can be merged. This can significantly shorten the time required to build large-scale preference pair datasets, especially when dealing with massive training samples. Parallelization is a key means to achieve practical engineering implementation.

[0147] In the second embodiment, an intelligent agent reasoning system based on the above training method is provided. This system aims to apply the target policy model trained in the first embodiment to real-world multilingual, multi-hop document question-answering scenarios. Unlike the first embodiment, which focuses on offline training, this embodiment emphasizes dynamic skill orchestration and state management during online reasoning, embodying the deployment and application process described in claim 9.

[0148] Figure 9 A schematic diagram of the architecture of this agent reasoning system is shown. Figure 9 As shown, the system mainly includes a user interaction interface 901, a request preprocessing module 902, a core inference engine 903, a skill execution environment 904, and a result postprocessing module 905.

[0149] User interaction interface 901 is responsible for receiving natural language query requests from users. Considering multilingual scenarios, this interface supports input in multiple languages. Request preprocessing module 902 performs standardization processing on the raw query, including removing noisy characters, identifying key entities, and performing preliminary intent classification. This structured information is then encapsulated into an initial state. This information is then passed to the core inference engine 903.

[0150] The core inference engine 903 is the central hub of this system, and it deploys the target policy model trained in the first embodiment. The engine's workflow is a typical perception-decision-execution loop. At each time step... The engine receives the current state. This state includes the historical dialogue context, the sequence of skills executed, and intermediate results returned from the skill execution environment 904.

[0151] Figure 10 The internal processing logic of the core inference engine is shown. For example... Figure 10 As shown, the state encoder 1001 converts unstructured text states and structured document data into a unified vector representation. Based on this vector representation, the policy network 1002 outputs the probability distribution of each atomic skill in the structured skill action space. The action selector 1003 samples or greedily selects the next atomic skill action based on the probability distribution. It is important to note that at this point, the model no longer performs Monte Carlo tree search, but instead directly uses the trained policy model to perform single-step or multi-step autoregressive generation to achieve real-time response.

[0152] Unlike traditional end-to-end generative models, Figure 9 The core reasoning engine 903 does not directly generate the final answer, but instead generates a series of atomic skill actions. For example, for a revenue comparison problem, the engine might first generate a "page selection" action. This action is then sent to the skill execution environment 904.

[0153] The Skill Execution Environment 904 can be a sandboxed execution layer containing the concrete implementation code of all atomic skills. It isolates the model from direct interaction with external resources. The Skill Execution Environment 904 includes a document retrieval submodule 904a, a document parsing submodule 904b, a data extraction submodule 904c, and a visual enhancement submodule 904d. When a "page selection" action is received, the document retrieval submodule 904a retrieves relevant document fragments from the vector database; when a "page parsing" action is received, the document parsing submodule 904b loads the specified page and outputs structured JSON data; when an "element selection" action is received, the data extraction submodule 904c locates specific fields in the structured data; when a "region zoom-in" action is received, the visual enhancement submodule 904d calls an OCR service to perform high-precision recognition of the specified coordinate region.

[0154] After execution, the skill execution environment 904 feeds back the execution result to the core inference engine 903 to update the current state. The engine continues to make decisions based on the new state until it generates an "end" action or reaches the maximum number of inference steps. This decoupled architecture allows the skill execution environment 904 to be upgraded independently of the model, such as replacing it with a more advanced OCR engine without retraining the model.

[0155] The post-processing module 905 receives the final inference trajectory and answer generated by the core inference engine 903. It formats and organizes the answer, for example, by integrating data from multiple sources into a comparison table and generating explanatory text in natural language. Finally, the results are presented to the user through the user interface 901.

[0156] In one different implementation, to adapt to different deployment environments, the system can adopt a cloud-edge collaborative architecture. A complete, heavyweight model is maintained in the cloud for handling complex multi-hop inference tasks; a distilled, lightweight policy model is deployed at the edge, performing only single-step skill prediction. Upon receiving a user request, the lightweight model quickly generates the most probable skill sequence. If the confidence level is higher than a threshold, it is executed directly; otherwise, the current state is uploaded to the cloud, where the heavyweight model intervenes for deep inference. This mechanism ensures data privacy and reduces latency while also providing the capability to handle complex problems.

[0157] Furthermore, the structured skill action space can be tailored for specific vertical domains. For example, in non-image-intensive text question answering, the "region magnification" skill can be removed, retaining only the core high-frequency skills. This not only reduces the dimensionality of the model's output layer and lowers computational cost, but also simplifies the dependency libraries of the skill execution environment, making it easier to deploy on resource-constrained devices. This demonstrates the flexibility of constructing the structured skill action space in claim 1, meaning that the space can be defined and adjusted according to actual needs.

[0158] In the specific implementation of step S150, the multi-dimensional quality assessment is not only a simple scoring of the trajectory, but also a key bridge connecting MCTS search and DPO optimization. Figure 6 The internal data flow of a multi-dimensional quality assessment system is shown. For example... Figure 6 As shown, when the initial preferred trajectory and initial inferior selection trajectory When they enter the evaluation system 600, they are first fed into three independent evaluation modules in parallel.

[0159] The language consistency assessment module 601 employs a cross-language alignment-based loss function to quantify scores when processing multilingual mixed text. Specifically, for each text segment in the trajectory, its cosine similarity to the context in the multilingual embedding space is calculated. If an unexpected language switch occurs at any step (e.g., a sudden switch from Chinese to undefined gibberish or a non-target language), its similarity score will significantly decrease. The module's output... This not only reflects grammatical correctness but also demonstrates the fidelity of semantics when translating between different languages. This is crucial for multilingual document question answering, as incorrect language switching often indicates a misunderstanding of the document content by the model.

[0160] The logical rigor assessment module 602 delves into the internal structure of reasoning. It doesn't just check the final answer, but constructs a temporary logical dependency graph. In this graph, nodes represent claims in the reasoning steps, and edges represent supporting evidence relationships. The module uses a pre-trained Natural Language Reasoning (NLI) model to verify the validity of each edge. If a claim lacks supporting evidence or a circular argument exists, the trajectory is considered invalid. Scores will be penalized. This fine-grained logic check can effectively identify "pseudo-optimal" trajectories that happen to be correct but whose reasoning process is full of illusions, preventing the model from learning incorrect reasoning shortcuts.

[0161] The skill orchestration effectiveness evaluation module 603 focuses on execution efficiency. In actual engineering, it has been found that the model is prone to falling into the trap of "skill abuse," such as repeatedly calling page parsing skills to process the same page, or calling region zoom skills when zooming is unnecessary. Module 603 calculates redundancy by comparing the skill sequences in the trajectory with a preset efficiency rule base. At the same time, based on historical statistical data, the average time taken for each skill call is estimated as the cost. The coefficients in formula (5) and It's not fixed, but can be dynamically adjusted according to the training stage. In the early stages of training, the focus is on encouraging accuracy in skill usage. and The initial values ​​are set relatively small; in the later stages of training, the focus is on optimizing inference efficiency, and these two coefficients are appropriately increased to encourage the model to learn a more concise skill arrangement.

[0162] After obtaining the sub-scores for each dimension, the weighted aggregation module 605 executes the calculation of formula (6). Here, the weights... These are hyperparameters, but in a variation of this embodiment, they can also be dynamically generated based on the task type. For example, for fact-based question answering, improving... (Logical) weighting; for summary-type question answering, increase The weighting of (language). This dynamic weighting mechanism makes the evaluation system more adaptable.

[0163] Based on the total evaluation score The system executes formulas (7) and (8) to calculate the optimization weights. Here, a temperature coefficient is introduced. Its function is to control the "focus" of training. When When the value is large, only preference pairs with extremely large quality differences will receive high weights, and the model will focus on learning the most obvious differences in quality; when When the value is small, more preference pairs are included in the training, helping the model explore a wider solution space. In actual training, a curriculum learning strategy can be used, gradually increasing the value as the number of training rounds increases. This value allows the model to transition from extensive learning to fine-tuning.

[0164] In step S160, the dataset is used to perform direct preference optimization (DPO) on the fine-tuned policy model using the preferences with adjusted weights. Figure 7 The data flow of this process is clearly illustrated. The fine-tuned strategy model 701 is used as the current model to be optimized. Reference model 702 serves as a fixed benchmark. Both receive input from preference pair dataset 703. .

[0165] The weighted loss function shown in formula (9) is one of the core innovations of this embodiment. The traditional DPO loss function treats all samples equally, while this invention introduces... This allows for "prioritization" of high-quality samples. During backpropagation, the gradient... The size is directly affected by This means that for positive samples that significantly outperform negative samples in terms of language, logic, and skill orchestration, the model parameters will be updated more dramatically, thus converging to the better policy distribution more quickly.

[0166] To prevent overfitting, this embodiment also introduces an early stopping mechanism during training. The weighted DPO loss on the validation set is monitored, and training is stopped and the optimal model parameters are saved when the loss no longer decreases or even begins to increase. Furthermore, the reference model 702 typically performs a soft update every certain number of steps. ,in A small update coefficient (e.g., 0.05) is used. This dynamically updated reference model can prevent the model from deviating too far from the initial distribution and maintain the diversity of the generated data.

[0167] The target policy model obtained through the above training process possesses the ability to accurately orchestrate structured skills in complex multilingual environments. It not only generates correct answers but also provides clear, logically rigorous, and efficient reasoning paths. This solves the technical problems mentioned in the background section, such as existing agents easily losing their way in multi-hop reasoning, exhibiting chaotic skill invocation, and lacking interpretability.

[0168] Figure 8 A schematic diagram of the hardware structure of an electronic device used to implement the above method is shown. For example... Figure 8As shown, the electronic device 800 includes a processor 801, a memory 802, a communication interface 803, and a bus 804. The processor 801 can be a central processing unit (CPU), a graphics processing unit (GPU), or an application-specific integrated circuit (ASIC), used to execute the various steps in the aforementioned intelligent agent training method for integrated skills. The memory 802 stores a computer program, which, when executed by the processor 801, implements the aforementioned method flow. The communication interface 803 is used for data interaction with external devices or networks, such as receiving training data and uploading trained models. The bus 804 connects the various components, enabling the transmission of data and instructions.

[0169] The technical effects of the present invention will be described in detail below with reference to experimental data.

[0170] 1. Specific implementation details of multi-dimensional evaluation

[0171] In step S150, the multi-dimensional quality assessment module 600 is implemented using a "self-evaluation" mechanism. Specifically, the evaluator is not an independent external model, but directly uses the strategy model. It itself serves as an evaluator (Judge Model).

[0172] The inputs to the evaluation process include preset evaluation prompts (prompt_eval) and the original question. and current status Strategy Model Based on these inputs, output the scoring probabilities in three dimensions: language consistency. Logical rigor and the effectiveness of skill arrangement .

[0173] Weighted aggregate score of these three dimensions Calculated using the following formula:

[0174] (12)

[0175] Among them, weight , , It is a non-negative number and satisfies the normalization condition. In a preferred example of this embodiment, the recommendation weight is set as follows: , , This weighting emphasizes the central role of logical rigor in multi-hop reasoning while also considering the fluency of language expression and the efficiency of skill execution. Since the evaluator shares parameters with the policy model, the model's evaluation capability improves synchronously with optimization in the SFT and DPO stages, forming a self-evolving closed loop.

[0176] 2. Interaction Mechanism between MCTS and Policy Model

[0177] During the Monte Carlo tree search in step S120, the policy model Intervention is carried out at multiple stages, specifically including:

[0178] (1) Action selection prior: In the expansion and simulation phases, the policy model is based on the interaction history. The probability distribution for predicting the next skill move is given by the formula: This ensures that the search process focuses on high-probability skill combinations. .

[0179] (2) Value estimation: In the expansion phase, the policy model estimates the current state. Conduct self-assessment to generate value estimates The value estimate is rewarded based on the accuracy of the result. Together they constitute state rewards :

[0180] (13)

[0181] in, Indicate the correctness of the answer indicating the termination state. This represents the self-evaluation score of the intermediate state. This hybrid reward mechanism allows MCTS to focus not only on the correctness of the final answer, but also on the quality of the reasoning process.

[0182] (3) Backtracking update: During the backtracking phase, the action value is updated using formulas (9)-(11). State value and number of visits This will guide the subsequent selection phase.

[0183] 3. Preferences for filtering the quality of the build

[0184] In step S130, when constructing the initial preference pair dataset, in addition to filtering Top-K and Bottom-K trajectories based on MCTS reward scores, a strict quality filtering mechanism based on a threshold is also introduced. Specifically, a high-performance large language model (such as GPT-4o) is used as the evaluator. The candidate data undergoes answerability and fidelity checks. Only data with an evaluation score exceeding a preset threshold is considered responsive. And the answer The cited evidence The data is only included in the training set when it is fully supported. This mechanism effectively removes noisy data and ensures high-quality preference pairs.

[0185] To verify the performance of the present invention in a multilingual scenario, comparative experiments were conducted on 10 languages. These 10 languages ​​include: English (EN), Chinese (ZH), Cantonese (YUE), Indonesian (ID), Malay (MS), Filipino (TL), Tamil (TA), Thai (TH), Vietnamese (VI), and Burmese (MY).

[0186] The experimental baseline was chosen from DocAgent (based on Gemini-2.5-Pro), the most powerful general-purpose document question-answering agent currently available. The proposed method (DocSkill) is based on the Qwen3-VL-8B model and employs a two-stage SFT+D training strategy. In the SFT stage, LoRA is used for efficient parameter fine-tuning, and the rank size... KL regularization strength parameters in the DPO phase Each sample undergoes 100 MCTS iterations.

[0187] Table 1 shows a comparison of Exact Match (EM) and F1 scores for each method across 10 languages:

[0188] method EN ZH YUE ID MS TL TA TH VI MY Average EM / F1 DocAgent (Gemini-2.5-Pro) 33.2 / 40.5 31.5 / 37.8 28.0 / 35.0 34.0 / 41.5 31.5 / 38.2 29.8 / 36.5 22.0 / 28.5 28.0 / 34.5 29.5 / 36.5 17.5 / 24.0 30.5 / 35.5 DocSkill(SFT) 41.5 / 50.1 38.8 / 47.0 35.2 / 42.8 42.1 / 51.0 40.2 / 48.8 36.8 / 45.2 30.8 / 38.9 34.2 / 43.0 36.4 / 45.2 26.8 / 34.8 44.2 / 46.7 DocSkill(SFT+DPO) 42.1 / 50.8 39.3 / 47.6 35.7 / 43.2 42.8 / 51.6 40.7 / 49.3 37.2 / 45.7 31.3 / 39.4 34.8 / 43.6 36.9 / 45.8 27.2 / 35.4 44.7 / 47.1

[0189] As can be seen from Table 1:

[0190] 1. Significantly Improved Overall Performance: Compared to the strongest baseline DocAgent, the proposed method (DocSkill SFT+DPO) achieves a 14.2% improvement in average EM metric (from 30.5% to 44.7%) and an 11.6% improvement in average F1 metric (from 35.5% to 47.1%). This demonstrates that, through structured skill space and preference learning, a smaller model (8 parameters) can outperform a large-scale general model (Gemini-2.5-Pro) in a specific domain.

[0191] 2. Low-resource languages ​​benefit more: The improvements of this invention are particularly significant in low-resource languages ​​such as Burmese (MY) and Tamil (TA), with EM improvements of 9.7% and 9.3%, respectively. This is attributed to explicit modeling of language consistency in the multidimensional evaluation and effective pruning of erroneous reasoning paths by the MCTS search.

[0192] 3. Contribution of the DPO stage: Compared with the stage using only SFT, the introduction of weighted DPO further improved the average EM by 3.6% and F1 score by 2.4%. This demonstrates that preference optimization based on multi-dimensional evaluation weights can effectively correct the bad habits remaining in the SFT stage and further improve the inference quality and skill orchestration efficiency of the model.

[0193] In summary, the experimental data fully demonstrates the effectiveness and advancement of the technical solution of this invention in multilingual, multi-hop document question answering tasks. Through the construction of a structured skill action space, deep coupling of MCTS search and policy model, and weighted DPO optimization based on multi-dimensional evaluation, this invention significantly improves the inference accuracy, robustness, and generalization ability of the agent.

[0194] Those skilled in the art will understand that the above experimental data are only used to illustrate the technical effects of the present invention and do not constitute a limitation on the scope of protection of the present invention. In practical applications, specific performance indicators may fluctuate due to differences in hardware environment, dataset distribution, and hyperparameter settings, but the trend of technical improvement brought about by the core technical means proposed in this invention is certain.

[0195] Based on the description of the above embodiments and experimental data, the technical solution of the present invention has generated a significant synergistic effect by organically combining structured skill action space, Monte Carlo tree search exploration and multi-dimensional quality assessment, and effectively overcomes several technical bottlenecks in the prior art.

[0196] In the foregoing embodiments of this invention, skill definition, search algorithm, and preference optimization are not simply stacked together, but rather a synergistic effect is achieved through tight logical coupling:

[0197] First, the structured skill action space provides a controllable solution space for MCTS search. Traditional text generation search spaces are enormous and discrete, making MCTS difficult to converge. By limiting the action space to finite atomic skills (such as page selection and element extraction), S1 transforms the continuous text generation problem into a discrete combinatorial optimization problem. This not only significantly reduces the branching factor of MCTS, making deep search possible with limited computational resources, but also endows each search node with a clear semantic meaning, laying the foundation for subsequent quality evaluation.

[0198] Secondly, MCTS exploration provides rich process data for multi-dimensional assessment. Traditional supervised fine-tuning relies solely on the final answer, lacking supervision of the reasoning process. MCTS, through simulation and backtracking, generates a large number of complete trajectories containing intermediate states, skill sequences, and final results. These trajectories include not only successful paths but also instructive negative samples of failure. This rich process data allows the multi-dimensional assessment module to deeply analyze the logical coherence and skill effectiveness of each step, rather than simply focusing on the correctness of the result.

[0199] Finally, multi-dimensional quality assessment provides fine-grained supervision signals for DPO optimization. Traditional DPO is usually based on binary labels or simple reward models, which are easily affected by noise. S3 introduces fine-grained assessments across three dimensions: language, logic, and skill, and dynamically adjusts the weights of preference pairs, enabling the model to distinguish between "accidentally correct incorrect reasoning" and "logically sound correct reasoning" during the optimization process. This refined feedback mechanism, in turn, guides MCTS to more accurate value estimation, forming a virtuous cycle.

[0200] This invention specifically addresses four major technical bottlenecks in the field of multilingual, multi-hop document question answering:

[0201] (1) Problem of chaotic skill orchestration: Existing methods often allow the model to generate text instructions freely, resulting in arbitrary, redundant, or even conflicting skill calls. This invention defines a clear skill interface through S1 and combines it with search optimization in S2, enabling the model to learn to select a better skill combination in a specific state. Experimental data show that the improvement in skill orchestration effectiveness score is directly correlated with the improvement in inference efficiency, reducing unnecessary API calls and computational overhead.

[0202] (2) Error accumulation problem in long-range reasoning: In multi-hop question answering, errors in early steps can cause subsequent reasoning to completely deviate from the correct path. The MCTS in S2 can identify and prune those paths where logical breaks occur early through a backtracking mechanism. At the same time, the logical rigor assessment module in S3 can detect the evidence support in intermediate steps to prevent the model from "making things up". This process-level error correction mechanism significantly improves the robustness of long-range reasoning.

[0203] (3) Lack of fine-grained process supervision: Traditional methods struggle to obtain quality labels for each step of the inference process. This invention utilizes a policy model self-evaluation and multi-dimensional weighting mechanism to automatically generate high-quality process-level supervision signals. Fine-grained preference alignment can be achieved without manually labeling the quality of each step, significantly reducing data construction costs.

[0204] (4) Insufficient multilingual generalization ability: Existing models perform poorly on low-resource languages, mainly due to a lack of high-quality multilingual inference data. This invention explicitly constrains the semantic fidelity of the model during multilingual switching through the language consistency assessment in S3. Experimental results show that the improvement of this method on low-resource languages ​​such as Burmese and Tamil is greater than that on high-resource languages, demonstrating its advantage in cross-language generalization.

[0205] In addition to the expected performance improvement, the above embodiments of the present invention have also achieved some additional technical effects:

[0206] First, it achieves state-of-the-art (SOTA) performance without manually labeled preference data. Traditional preference learning heavily relies on expensive manually labeled data (such as the comparison data in RLHF). This invention constructs preference pairs entirely based on automatically generated MCTS trajectories and multi-dimensional evaluation scores, requiring no human intervention. However, experimental data shows that DocSkill outperforms baseline models trained on large-scale human feedback (such as DocAgent) in both average EM and F1 scores across 10 languages. This indicates that fine-grained, multi-dimensional supervisory signals generated by the algorithm may be more instructive than sparse manually labeled preference signals in specific vertical domains.

[0207] Second, the "small model outperforms large model" effect. The Qwen3-VL-8B model used in this invention has a much smaller parameter count than the baseline model Gemini-2.5-Pro. Typically, small models struggle to compete with large models on complex inference tasks. However, through integrating a skill space and reinforcement learning optimization, the small model demonstrates extremely strong domain-specific adaptability. This suggests that by optimizing the inference architecture and supervision mechanisms, the insufficient model parameter size can be compensated for, providing a new technical path for deploying high-performance agents at low cost.

[0208] Third, the self-evolving evaluation capability. Since the evaluator shares parameters with the policy model, its judgment ability as an evaluator improves synchronously as the model is optimized in the SFT and DPO stages. This enhanced "self-reflection" capability results in increasingly higher quality preference pairs generated later, further accelerating model convergence. This self-reinforcing mechanism is particularly evident in the later stages of training, with a smoother decline in the loss curve and a higher final performance ceiling.

[0209] In summary, through innovative architecture design and algorithm integration, this invention not only improves various indicators but also demonstrates unique technical advantages in data efficiency, model size, and generalization ability, possessing extremely high practical value and promising prospects for widespread application.

[0210] It should be understood that the method steps in the foregoing embodiments can be implemented by program instructions controlling related hardware, or by dedicated circuits, programmable logic devices, or a combination thereof. Correspondingly, the systems, devices, modules, units, or components in the foregoing embodiments can be implemented in software, hardware, or a combination of both. The division of modules, units, or components is merely a logical division for the purpose of illustrating the technical solution; in actual implementation, they can be combined, split, or integrated as needed.

[0211] In one embodiment, the electronic device may include a processor, a memory, and a communication interface, wherein the memory is used to store program instructions, and the processor is used to call and execute the program instructions to implement all or part of the steps in the foregoing method embodiments. The electronic device may be a server, a terminal device, an edge computing node, a cloud computing device, or other device with data processing capabilities.

[0212] In one embodiment, this application may also be implemented in the form of a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement all or part of the steps in the foregoing method embodiments. The computer-readable storage medium may be a read-only memory, random access memory, flash memory, hard disk, solid-state drive, optical disk, or other non-transitory storage medium.

[0213] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0214] Furthermore, the terms "including," "comprising," and "having" used in the specification are all non-exclusive inclusions; the terms "first," "second," etc., are only used to distinguish technical features and do not indicate limitations on order, quantity, or importance. The execution order of each step in the method embodiments is also not absolutely limited. Without departing from the technical concept of this application, the steps can be adjusted in order, executed in parallel, combined, or split for execution.

Claims

1. A method for training an agent with integrated skills, comprising: Construct a structured skill action space containing multiple atomic skill actions; For each training sample, a Monte Carlo tree search is performed to construct an inference trajectory tree, and a comprehensive score of the trajectory in the inference trajectory tree is obtained; wherein, the expansion phase of the Monte Carlo tree search is based on sampling candidate actions in the structured skill action space; Based on the comprehensive scores of the trajectories in the inference trajectory tree, the initial preferred trajectories and the initial unpreferred trajectories are selected to construct the initial preference pair dataset; The policy model is supervised and fine-tuned based on the initial preferred trajectory to obtain the fine-tuned policy model. The initial preferred trajectory and the initial undesired trajectory are subjected to multi-dimensional quality evaluation. Based on the evaluation results, the optimization weights of each preference pair in the initial preference pair dataset are adjusted. The adjusted preference pair dataset is then used to perform direct preference optimization on the fine-tuned policy model to obtain the target policy model.

2. The agent training method for integrated skills as described in claim 1, wherein, The structured skill action space includes a set of reasoning skills and a set of tool skills, both of which are composed of atomic skill actions. The set of reasoning skills includes query optimization skills for optimizing query content and review and correction skills for correcting intermediate reasoning steps; The set of tools and skills includes page selection skills for retrieving relevant document pages, page parsing skills for extracting structured elements from document layouts, element selection skills for identifying key evidence, and region magnification skills for zooming in on local areas.

3. The agent training method for integrated skills as described in claim 1, wherein, The multi-dimensional quality assessment includes language consistency assessment, logical rigor assessment, and skill arrangement effectiveness assessment. The step of adjusting the optimization weights of each preference pair in the initial preference pair dataset based on the evaluation results includes: Calculate the sub-scores of the initial preferred trajectory and the initial undesired trajectory in three dimensions: language consistency, logical rigor, and skill arrangement effectiveness; The sub-scores are weighted and aggregated to obtain a multi-dimensional total evaluation score for each trajectory; The optimization weights for the corresponding preference pairs are determined based on the difference in the total multi-dimensional evaluation scores of the initial preferred trajectory and the initial undesired trajectory.

4. The agent training method for integrated skills as described in claim 3, wherein, The weighted aggregation of the sub-scores yields a multi-dimensional total evaluation score for each trajectory, including: Obtain preset weights for language consistency, logical rigor, and skill arrangement effectiveness; The language consistency sub-score is multiplied by the language consistency weight to obtain the first weighted score; The second weighted score is obtained by multiplying the logical rigor score by the logical rigor weight. The third weighted score is obtained by multiplying the sub-score for skill arrangement effectiveness by the skill arrangement effectiveness weight. The first weighted score, the second weighted score, and the third weighted score are summed to obtain the total multi-dimensional evaluation score.

5. The agent training method for integrated skills as described in claim 1, wherein, The step of directly optimizing the fine-tuned policy model using the adjusted preference dataset includes: Construct a direct preference optimization loss function, in which the optimization weights are included as coefficients to amplify or reduce the contribution of a specific preference to the update of model parameters; Based on the direct preference optimization loss function, the parameters of the fine-tuned policy model are updated to maximize the log probability difference between the preferred trajectory and the undesired trajectory.

6. The agent training method for integrated skills as described in claim 1, wherein, The Monte Carlo tree search also includes a selection phase, a simulation phase, and a backtracking phase; In the selection phase, starting from the root node, child nodes are recursively selected until leaf nodes are selected based on the historical average value of the nodes. In the simulation phase, starting from the newly created child node, subsequent atomic skill actions are sampled until the termination state, and a simulation reward based on the correctness of the termination state is obtained. In the backtracking phase, the simulated reward is propagated backward along the selected path, accumulated to the visit count of each node on the path, and the historical average value of each node is updated. The overall score is determined based on the historical average value of the leaf nodes or path nodes after the backtracking phase update.

7. The agent training method for integrated skills as described in claim 1, wherein, The candidate actions sampled based on the structured skill action space include: Obtain the status information of the current node, which includes the currently executed atomic skill action sequence and the corresponding document context; The state information is input into the strategy model, and the execution probability of each atomic skill action in the structured skill action space is output. Based on the execution probability, N candidate atomic skill actions are sampled, where N is an integer greater than 1; For each candidate atomic skill action, a corresponding child node is generated, and the candidate atomic skill action is used as an edge connecting the current node and the corresponding child node.

8. The agent training method for integrated skills as described in claim 1, wherein, The process of selecting initial preferred trajectories and initial undesirable trajectories based on the comprehensive scores of the trajectories in the inference trajectory tree includes: Calculate the cumulative value score for each complete trajectory in the inference trajectory tree; Sort all complete trajectories from highest to lowest according to the cumulative value score; The complete trajectory with the top-ranked preset proportion is selected as the initial preferred trajectory; The complete trajectory after ranking is selected as the initial inferior trajectory according to a preset ratio.

9. The agent training method for integrated skills as described in any one of claims 1 to 8, further comprising: The target strategy model is deployed on the server or client side to receive document question-and-answer requests input by users, dynamically arrange atomic skill actions based on the target strategy model, generate reasoning trajectories, and output answers.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the agent training method for integrated skills as described in any one of claims 1 to 8.

Images