Intelligent agent distillation method and apparatus, electronic device, storage medium, program product

Abstract

Embodiments of the present application provide an agent distillation method and device, electronic equipment, storage medium and program product. The method comprises: obtaining a trajectory text generated by a teacher agent performing an interactive task, mapping the trajectory text into a token sequence, and marking reasoning tokens and action tokens in the token sequence. For the prediction position of each token, the prediction distribution of a student model and a teacher model is obtained in a complete token space. If the token is a reasoning token, a loss component is calculated after probability normalization of the prediction distribution in a reasoning token space. If the token is an action token, a loss component is calculated after probability normalization of the prediction distribution in an action token space. The reasoning token space and the action token space are subsets of the complete token space and are mutually exclusive. The total loss is obtained by aggregating the loss components, and the student model parameters are iteratively optimized. Embodiments of the present application can decouple reasoning and action gradient interference and reduce the computational complexity of the training process.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to an intelligent agent distillation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Large model-driven agents perform well in interactive tasks, but they have a large number of parameters, high inference latency, and high deployment costs. Knowledge distillation is a common method to transfer the capabilities of large models to smaller models. It compresses the model size while preserving performance by having the student model learn the output distribution of the teacher model. In existing technologies, the distillation of student agents involves a large amount of computation, requiring significant computing resources, and the distillation effect cannot be guaranteed, resulting in poor learning efficiency for the student agents. Summary of the Invention

[0003] This application provides an agent distillation method, apparatus, electronic device, storage medium, and program product to alleviate or solve the technical problems of large computational load and poor distillation effect in the agent distillation process in the prior art.

[0004] In a first aspect, embodiments of this application provide an agent distillation method, comprising: Obtain the trajectory text generated by the teacher intelligent agent performing interactive tasks; the trajectory text is natural language text, and the teacher intelligent agent is constructed based on a teacher model; The trajectory text is mapped to a complete lexical space to obtain a lexical sequence, and the segment type to which each lexical in the lexical sequence belongs is marked; the segment type includes reasoning segments and action segments; For the predicted position of each word in the word sequence, obtain the student prediction distribution output by the student model and the teacher prediction distribution output by the teacher model in the complete word space; For each lexical unit, if the lexical unit is a reasoning lexical unit marked as belonging to the reasoning segment, then the student prediction distribution and the teacher prediction distribution are probability normalized respectively in the reasoning lexical unit space, and the loss component of the reasoning lexical unit is calculated based on the result of probability normalization. If the lexical unit is an action lexical unit marked as belonging to the action segment, then the student prediction distribution and the teacher prediction distribution are probability normalized respectively in the action lexical unit space, and the loss component of the action lexical unit is calculated based on the result of probability normalization. The reasoning lexical unit space and the action lexical unit space are subsets of the complete lexical unit space and are mutually disjoint. The loss components of each inference term and each action term are summarized to calculate the total loss. The parameters of the student model are then iteratively optimized based on the total loss to obtain the student agent after distillation training.

[0005] In some embodiments of this application, the step of probability normalizing the student prediction distribution and the teacher prediction distribution in the inference lexical space includes: In the student prediction distribution, the probability values ​​of the predicted words belonging to the inference word space are selected, and the selected probability values ​​are normalized to obtain the result of normalizing the probability of the student prediction distribution in the inference word space. The probability values ​​of predicted words belonging to the inference word space are selected from the teacher prediction distribution, and the selected probability values ​​are normalized to obtain the result of normalizing the probability of the teacher prediction distribution within the inference word space.

[0006] In some embodiments of this application, the step of marking the inference lexical units belonging to the inference segment and the action lexical units belonging to the action segment in the lexical unit sequence includes: Determine the segment type corresponding to each lexical in the lexical sequence; Based on the segment type corresponding to each word, a mask sequence of the same length as the word sequence is generated, wherein each mask in the mask sequence is used to mark the segment type of the corresponding word.

[0007] In some embodiments of this application, each mask in the mask sequence includes multiple binary sub-masks, which include inference sub-masks, action sub-masks, and observation sub-masks; each binary sub-mask takes a value of 0 or 1 to characterize whether the lexical at the corresponding position belongs to the segment type corresponding to the binary sub-mask, and at most one of the multiple binary sub-masks at the same position takes a value of 1.

[0008] In some embodiments of this application, the fragment type of the trajectory text further includes observation fragments, and the method further includes: Mark the observed lexical units in the lexical sequence that belong to the observed segment; For each observed term, the corresponding loss term is set to 0.

[0009] In some embodiments of this application, the trajectory text may be multiple, and the method may further include: Calculate the complexity score of each trajectory text, whereby the complexity score reflects the complexity of the interactive task corresponding to the trajectory text; Sort the multiple trajectory texts in ascending order of their complexity scores; The student model is trained using the sorting results.

[0010] Secondly, embodiments of this application provide an intelligent agent distillation apparatus, comprising: The generation module is used to obtain the trajectory text generated by the teacher agent performing interactive tasks; the trajectory text is natural language text, and the teacher agent is constructed based on a teacher model. A tagging module is used to map the trajectory text to a complete lexical space to obtain a lexical sequence, and to tag the segment type to which each lexical in the lexical sequence belongs; the segment type includes reasoning segments and action segments; The acquisition module is used to acquire the student prediction distribution output by the student model and the teacher prediction distribution output by the teacher model within the complete lexical space, based on the predicted position of each lexical in the lexical sequence. The normalization module is used to perform probability normalization on the student prediction distribution and the teacher prediction distribution in the inference word space for each word element. If the word element is a reasoning word element marked as belonging to the inference segment, the module calculates the loss component of the inference word element based on the probability normalization result. If the word element is an action word element marked as belonging to the action segment, the module calculates the loss component of the action word element based on the probability normalization result. The inference word space and the action word space are subsets of the complete word space and are mutually disjoint. The distillation module is used to summarize the loss components of each inference word and the loss components of each action word, calculate the total loss, and iteratively optimize the parameters of the student model based on the total loss to obtain the student agent after distillation training.

[0011] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0012] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0013] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0014] Based on any of the above technical solutions, this application has at least the following beneficial effects or advantages: This embodiment calculates the loss components by performing probability normalization on inference terms and action terms in their respective independent term spaces. The inference-side loss is determined solely by the relative probability relationships within inference terms, and the action-side loss is determined solely by the relative probability relationships within action terms. The gradient signals of the two types of losses do not interfere with each other, avoiding the problem of gradient dominance and masking of action term gradients due to a large number of inference terms. The student model can simultaneously and evenly learn the reasoning and action decision-making abilities of the teacher model. The distilled student agent shows simultaneous improvement in reasoning quality and action accuracy in interactive tasks, resulting in a higher task success rate and execution efficiency closer to that of the teacher agent. Furthermore, during the training process of the neural network model, the normalization of the prediction distribution for inference terms omits the normalization of the probability of action terms, and vice versa, saving computational resources during training.

[0015] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0016] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0017] Figure 1 A flowchart of an agent distillation method provided in an embodiment of this application is shown; Figure 2 Another flowchart of an agent distillation method provided in an embodiment of this application is shown; Figure 3 The diagram shows the gradient decoupling effect of an agent distillation method provided in this application embodiment; Figure 4 This illustration shows a comparative diagram of the performance of student agents at different model sizes distilled by the agent distillation method provided in this application embodiment; Figure 5 This paper shows a schematic diagram of the structure of an intelligent agent distillation device provided in an embodiment of this application; Figure 6 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0019] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following related technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. It should be noted that the application scenarios or application examples provided in this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions.

[0020] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] See Figure 1 The flowchart shown is for an intelligent agent distillation method, which specifically includes steps 101 to 105.

[0022] Step 101: Obtain the trajectory text generated by the teacher agent performing the interactive task; Step 102: Map the trajectory text to the complete lexical space to obtain the lexical sequence, and mark the segment type to which each lexical in the lexical sequence belongs; Step 103: For the predicted position of each word in the word sequence, obtain the student prediction distribution output by the student model and the teacher prediction distribution output by the teacher model in the complete word space; Step 104: For each word element, if the word element is a reasoning word element marked as belonging to a reasoning segment, then the student prediction distribution and teacher prediction distribution are probability normalized separately in the reasoning word element space, and the loss component of the reasoning word element is calculated based on the probability normalization result. If the word element is an action word element marked as belonging to an action segment, then the student prediction distribution and teacher prediction distribution are probability normalized separately in the action word element space, and the loss component of the action word element is calculated based on the probability normalization result. The reasoning word element space and the action word element space are subsets of the complete word element space and are mutually disjoint. Step 105: Summarize the loss components of each inference term and each action term, calculate the total loss, and iteratively optimize the parameters of the student model based on the total loss to obtain the student agent after distillation training.

[0023] The teacher and student models can be large language models (LLMs) or multimodal large models capable of processing natural language text. Both are neural networks with word prediction capabilities. The teacher and student models are the main weights of the neural network, including model parameters and feedforward computation units. For example, a decoder-based large language model, such as a decoder-only Transformer network, can be used. The teacher model is a more powerful model with a large number of parameters, and its weights are frozen during training. The student model can use a smaller, similarly structured network, updating all or part of its network parameters through distillation iterations to achieve capability transfer. The teacher agent uses the teacher model as its computational kernel. The neural network parameters used by the teacher agent are the same as those of the teacher model, but it may optionally include supporting program entities that constitute the interactive task. These supporting program entities include, but are not limited to, task prompt templates, environmental interaction interfaces, and action parsing logic. The student model is a distilled and optimized neural network. The student agent reuses the same supporting program logic as the teacher agent, using the student model as its computational kernel. The neural network parameters of the student agent are equivalent to those of the student model.

[0024] The trajectory text is generated by the teacher agent performing interactive tasks round by round in the interactive environment. The trajectory text is natural language text data, conforming to the rules of natural language. For example, interactive tasks include, but are not limited to, natural language question-and-answer tasks, tool-calling tasks, webpage navigation tasks, code execution tasks, entity information retrieval tasks, and multi-round environmental game tasks. Among them, the natural language question-and-answer task refers to receiving natural language text questions input by the user, and the agent combining external information to complete the question answer. The task input consists of various natural language questions posed by the user, and the task output is the reasoned answer content, which can be used in knowledge consultation scenarios. The tool-calling task refers to the agent calling external program interfaces to complete the goal according to the task objective. Machine instructions serve as the task interaction content. The agent outputs tool-calling instructions and receives the execution results returned by the tools, such as calling a calculator to perform numerical calculations or calling a query interface to obtain real-time data. The webpage navigation task refers to the agent simulating webpage browsing interaction, outputting interactive actions such as page clicks, page turning, and inputting search terms, receiving text content returned by the webpage, and gradually collecting page information through multiple rounds of interaction to complete the predetermined goal. Code execution tasks refer to the agent receiving code writing requirements, outputting executable program code as action content, delivering it to the code execution environment for execution, receiving program output or error information, and iteratively correcting the code until the task requirements are met. Entity information retrieval tasks refer to the agent searching for information on specified people, locations, objects, etc., obtaining fragmented data returned by the environment through multiple rounds of retrieval actions, and integrating and reasoning to obtain complete information related to the entity. Multi-round environment game tasks refer to the agent engaging in continuous adversarial interaction with the simulation environment, receiving state feedback from the environment, outputting game action instructions, and achieving the task result of winning the game or achieving the goal through multiple rounds of reasoning and action loops. It can be understood that the student agent obtained after distillation training can also be used to perform the same tasks as the teacher agent, possessing the same supporting program logic as the teacher agent.

[0025] In each round, given observations and task cues, the teacher agent generates reasoning content (reason) and action output (action), forming a trajectory text τ=(reason, action) for course sampling. The entire task may include multiple rounds of interaction; correspondingly, the trajectory text generated by the teacher agent is a sequence of alternating reasoning and actions, represented as: .

[0026] in, Represents the trajectory text, where r i ∈R is a reasoning token, such as a CoT (Chain-of-Thought) step, a j ∈A is an action token (such as a tool call or an answer). Let 'reason' represent the reasoning segment, 'a' represent the action segment, and 'k' and 'm' represent the number of trajectory segments and action segments, respectively.

[0027] The trajectory text used in this application embodiment includes reasoning fragments and action fragments. In some implementations, the teacher model adopts a ReAct (Reasoning + Acting, reasoning + action, reasoning-action alternation paradigm) style. Optionally, the fragment type may also include observation fragments. Optionally, the teacher agent sequentially produces reasoning content and action content in each round, and the environment then returns observation information. In cases where the interactive task is executed in multiple rounds, the trajectory text may include multiple rounds of a loop of "reasoning content, action content, and observation information."

[0028] It is understandable that in the trajectory text of each round within the loop, various types of content can include different numbers of fragments. For example, in a certain round of interaction, the reasoning content includes multiple reasoning fragments, but the action content only outputs one action fragment, or the reasoning content includes one reasoning fragment, but the action content includes multiple action fragments. This application embodiment does not limit this. The reasoning fragment is a sequence of lexical terms that the agent completes logical deduction based on existing observation information. It is only used for internal thought organization and will not trigger external execution actions. For example, the text content corresponding to an example reasoning fragment is: [Reasoning] To check today's temperature, I need to call a weather query tool. I should first extract the city name in the user's question and then assemble the query parameters. The action fragment is an executable instruction sequence of lexical terms generated after the agent completes the deduction. After being output, it can drive the operation of external tools or environments. For example, the text content corresponding to an example action fragment is: [Action] Call the weather interface, city Jinzhou, to get real-time temperature data.

[0029] The word sequence can be obtained by segmenting the trajectory text using a word segmentation tool, preserving the original temporal order of the trajectory text. Tagging operations can be implemented by parsing special identifiers in the trajectory or by matching segment boundaries using regular expressions. During forward propagation, the entire word sequence is input completely into the student and teacher models respectively, without splitting, and the prediction distribution is calculated for each prediction position along the original word order. The prediction distribution at each prediction position produces a distribution vector covering the complete word space. The distribution vector represents the original prediction score of each word in the complete prediction space at that prediction position. The complete word space refers to the entire set of candidate words built into the model; both the student and teacher models output based on the complete word space.

[0030] After the teacher and student models output prediction distributions for each prediction position, branching processing is performed based on the lexical tagging results. If the prediction position is an inference lexical, probability normalization is performed in the inference lexical space; if it is an action lexical, probability normalization is performed in the action lexical space. The respective loss components are calculated separately, and finally, all loss components are summed to obtain the total loss, which is used for backpropagation to update the parameters. The inference lexical space and the action lexical space are two subsets obtained from the complete lexical space, and there are no overlapping lexical components between them.

[0031] In this embodiment, when calculating the loss using probability normalization, the complete lexical space is decoupled into two non-overlapping subsets: the inference lexical space and the action lexical space. The predicted probability distribution is normalized in each of the two lexical spaces, and the loss is calculated accordingly. This eliminates gradient coupling interference between inference and action semantics, conforming to the rules of natural language. The two types of optimization signals act independently on their respective subspaces, avoiding interference and masking of the action lexical gradient by the inference lexical gradient when the inference segment of a long thinking process is larger than the output action segment. This reduces training iteration oscillations, improves the convergence success rate of neural network training, reduces computational consumption caused by invalid iterations, shortens the total training time, and improves the distillation training effect. Furthermore, during the training process of the neural network model, since the normalization of the predicted distribution of inference lexicals is omitted from the normalization of the probability of action lexicals, and the normalization of the predicted distribution of action lexicals is omitted from the normalization of the probability of inference lexicals, computational power can be saved during the training process.

[0032] In some embodiments of this application, probability normalization is performed on the student prediction distribution and the teacher prediction distribution within the inference lexical space, including: In the student prediction distribution, the probability values ​​of the predicted words belonging to the inference word space are selected, and the selected probability values ​​are normalized to obtain the result of normalizing the probability of the student prediction distribution in the inference word space. In the teacher prediction distribution, the probability values ​​of the prediction words belonging to the inference word space are selected, and the selected probability values ​​are normalized to obtain the result of normalizing the teacher prediction distribution probability in the inference word space.

[0033] The selection of predicted lexical probability values ​​belonging to the inference lexical space involves extracting the component corresponding to the inference lexical by index from the predicted distribution vector output from the complete lexical space, discarding the remaining components. The extracted probability value set does not yet satisfy the normalization constraint because the original predicted distribution was calculated in the complete lexical space, and the sum of the components after truncation is not equal to 1. Therefore, it is necessary to re-normalize the probability values ​​within the subset so that the sum of all probability values ​​in the inference lexical space is 1. Normalization can be performed using exponential normalization or linear scaling. The student model and teacher model independently perform the above truncation and normalization operations to obtain the student prediction distribution and teacher prediction distribution in the inference lexical space, respectively. These two normalized distributions are used to calculate the distribution difference loss at that position.

[0034] This application embodiment severs the normalization coupling between inference probability and action probability by independently renormalizing within a subset of inference lexical units, so that the loss on the inference side is determined only by the relative probability within the inference lexical units and is not affected by fluctuations in the probability value of action lexical units.

[0035] The objective of this application is to distill a high-capacity ReAct-style teacher agent into a smaller student model while maintaining structured decision-making behavior.

[0036] In a specific example, given a teacher agent π_T(τ), the goal is to train a compact student agent π_θ(τ) such that:

[0037] refer to Figure 2 The diagram shows a specific example flow of the intelligent agent distillation method according to an embodiment of this application, including: sampling interaction trajectories → inputting trajectories into the teacher model → splitting inference segments and action segments → inputting trajectories into the student model → aligning and calculating the loss.

[0038] To achieve sequence-to-sequence modeling, each trajectory text is linearized into a flat form with segment markers. :

[0039] Then it is segmented into words:

[0040] This represents the word segmentation mapping. For each token x_t, a segment label s_t∈{Reason,Action} is assigned, indicating the segment to which it belongs, t∈(1,T), where T is the number of tokens. These segment labels are used to compute the segment-aware loss during training.

[0041] The following symbols are used in the description of the embodiments of this application: Representing structured reasoning-action trajectories, ' represents the linearized form with explicit fragment tokenization, x = Tokenize( ′) represents the sequence of tokens processed by the model. Accordingly, _θ always operates on the input x after word segmentation, while and ′ is used only for segmentation and mask construction.

[0042] The trajectory text τ generated by the teacher agent is decomposed into two types of disjoint segments:

[0043] Where τ (r) τ represents a reasoning segment. (a) Indicates a segment of action. This represents the segmentation function, and the segmentation method can be executed through lightweight rule parsing based on a cross-task consistent prompt template. The segmented trajectory is then tokenized into a sequence x.

[0044] While the above formula assumes a single inference-action pair, the embodiments of this application are also applicable to multiple inference / action fragments, and can be extended to multi-step ReAct trajectory text. K represents the total number of rounds / steps of the reasoning-action-observation triplet in the entire multi-step ReAct trajectory.

[0045] In this embodiment, the loss can be calculated using the Kuhlbek-Leibler (KL) divergence to represent the student's conditional distribution. (·| Distribution of teachers (·| Alignment.

[0046] Inference loss can be expressed as:

[0047] Losses can be expressed as:

[0048] The total loss can be aggregated from these terms to obtain:

[0049] In the above formula, T represents the teacher model, S represents the student model, and t represents the word sequence position index. This represents the inference mask function, where 1 represents a segment of inference and 0 represents the segment of inference. This represents the action mask function; 1 indicates an action segment, and 0 indicates otherwise. and It is a scalar weight that balances the two inference losses and action losses. The inference loss is used to align the inference steps of the teacher model and the student model, that is, to encourage the student model to align its thought process with that of the teacher model. The action loss is used to enable the student model to reproduce the action decisions of the teacher model. It can be set... = =1, weighted in the final loss to achieve reasoning supervision and action supervision.

[0050] In this embodiment, the KL loss is not calculated across the entire vocabulary for individual token-level KL. Token-level KL in related technologies is calculated in the joint space. ∪ (Inference domain Action domain Normalization is performed on the normalization domain, which couples the gradients of high-frequency inference tokens with those of rare but critical action tokens. In contrast, embodiments of this application apply KL to disjoint normalization domains (inference and action domains) separately, thereby altering the normalization space and gradient direction. This decomposition changes the optimization geometry and prevents cross-fragment interference, even when... and When they are equal, this also constitutes a fundamental difference from token-level KL in related technologies.

[0051] In some embodiments of this application, the segment type to which each lexical in the lexical sequence belongs includes: Determine the segment type corresponding to each word in the word sequence; Based on the segment type corresponding to each word, a mask sequence of the same length as the word sequence is generated. Each mask in the mask sequence is used to mark the segment type of the corresponding word.

[0052] The segment type corresponding to each lexical unit can be determined based on the pre-set segment start markers in the trajectory text. When the teacher agent generates the trajectory text, it can insert inference markers at the beginning of inference segments, action markers at the beginning of action segments, and observation markers at the beginning of observation segments. By recognizing these markers, the segment type of subsequent lexical units can be determined until the next marker is encountered. Optionally, segment boundaries can also be delineated through syntactic analysis or template matching. After determining the segment type of each lexical unit, masks are generated sequentially according to the lexical unit sequence, forming a mask sequence. The length of the mask sequence is exactly the same as the length of the lexical unit sequence, and each mask in the mask sequence corresponds one-to-one with a lexical unit position. Each mask is used to store the segment type information of the lexical unit at that position, which is used to quickly determine which computational branch to take at the current position during the loss calculation stage, without repeatedly parsing the text structure.

[0053] In this embodiment, the segment type of each word is pre-labeled by a mask sequence of the same length as the word sequence. During the loss calculation stage, branch routing can be completed directly based on the mask corresponding to each word, thereby improving computational efficiency.

[0054] In some embodiments of this application, each mask in the mask sequence includes multiple binary sub-masks, and the multiple binary sub-masks include inference sub-masks, action sub-masks, and observation sub-masks; each binary sub-mask takes a value of 0 or 1 to characterize whether the lexical at the corresponding position belongs to the segment type corresponding to the binary sub-mask, and at most one of the multiple binary sub-masks at the same position takes a value of 1.

[0055] Each bit mask consists of multiple binary sub-masks arranged side-by-side, each corresponding to a segment type. The inference sub-mask corresponds to the inference segment, the action sub-mask to the action segment, and the observation sub-mask to the observation segment. Each binary sub-mask takes only one of two values: a value of 1 indicates that the current word belongs to the segment type corresponding to that sub-mask, and a value of 0 indicates that it does not. Since a word can only belong to one segment type, at most one of the multiple binary sub-masks at the same position will have a value of 1, while the rest will have values ​​of 0, preventing multiple sub-masks from simultaneously having a value of 1. In some implementations, this multi-channel binary tag structure can achieve parallel computation, where the loss of each type can be multiplied by its corresponding sub-mask and then summed to achieve batch filtering without needing to judge each word individually. The number of sub-masks can be flexibly adjusted according to the number of segment types included in the trajectory text, and is not limited to three.

[0056] The embodiments of this application adopt a multi-channel binary sub-mask structure, with each type of marker being independent of the others, which facilitates batch filtering of loss items by type. At the same time, mutual exclusion constraints ensure that there is no overlap or ambiguity in the segment division.

[0057] In some embodiments of this application, the fragment type of the trajectory text further includes observation fragments, and the method further includes: The observed lexical units in the lexical sequence that belong to the observed segment; For each observed term, the corresponding loss term is set to 0.

[0058] The observed segments originate from feedback information from the interactive environment to the agent's actions, and are not content generated autonomously by the agent. Observed words occupy consecutive positions in the word sequence, and during forward propagation, the predicted positions corresponding to these observed words will also output the prediction distribution in the complete word space; the model will not skip the prediction calculations for these positions. However, since the observed content is determined by the environment and not controlled by the agent, distillation supervision is not applied to the observation positions. Setting the loss component corresponding to the observed words to 0 is equivalent to the observation positions not participating in the calculation of the total loss; during backpropagation, the gradient of the observation positions is 0 and will not contribute to the model parameter updates. Observed words still participate in the prediction calculations for subsequent positions as contextual information, providing environmental state basis for subsequent reasoning and actions.

[0059] This application embodiment supervises only the reasoning and action content generated autonomously by the agent, eliminating interference from environmental feedback information, and enabling the distillation target to focus on the transfer of the agent's own decision-making ability, thereby improving training effectiveness.

[0060] The following example, using multi-step ReAct trajectory text, illustrates the process of generating a mask sequence.

[0061] Multi-step ReAct trajectory text consists of alternating reasoning segments [REASON], action segments [ACT], and observation segments [OBS] in a multi-step ReAct scenario:

[0062] Where r (i) a (i) and o (i) Let i represent the reasoning trajectory, the action performed, and the subsequent observation, respectively.

[0063] right Linearization yields:

[0064] After word segmentation, the word sequence is obtained:

[0065] The mask sequence is constructed below. Lexical sequence Each token_t in the array is assigned a binary mask, including: Inference Submask Action Submask Fragment mask .

[0066] Enforce constraints on each token such that:

[0067] Ensure that each lexical unit belongs to a non-overlapping and functionally disjoint segment. Submasks for reasoning and action ( , ) is used for loss supervision, while the observation submask This provides environmental feedback when the model generates the predicted distribution.

[0068] In this embodiment, only the reasoning and action tokens contribute to the distillation loss. The observation token ( (t)=1) is excluded from the distillation loss because the encoding of the observed lexical units comes from deterministic feedback from the environment rather than agent behavior. This prevents the student model from overfitting to static observed text and focuses the learning on the quality of imitation of reasoning and action decisions.

[0069] This example illustrates a two-round reasoning scenario (K=2), with the trajectory text as follows: [Deduction] Let me check the table first. [Action] Search [tray] [Observation] You see a tray.

[0070] [Reasoning] Now I'm going to pick it up. [Action] Pick up the [tray] [Observation] The tray is already in stock.

[0071] Only inference and action tokens receive gradient updates, fragment-level supervised semantics, and are aligned with multi-round ReAct agent behavior.

[0072] Although the total loss is additive in form, embodiments of this application explicitly decompose the learning signal into structurally disjoint segments: reasoning and action, and apply a segment-specific loss to each segment, preserving the semantics of the multi-stage agent behavior.

[0073] CoT-Policy Alignment Loss (L CoT Supervised reasoning within fragments promotes coherent multi-step reasoning aligned with teachers' thought patterns. Action consistency loss (L...) Act It is only applied to action segments, forcing accurate reproduction of basic decisions.

[0074] Each token uses a binary mask { , These masks are assigned to exactly one functional segment, and the gradient flow is gated. The masks enforce semantic separation during training, ensuring that the student model learns independently for high-level inference and low-level execution, respectively. Embodiments of this application avoid loss interference across segments, better modeling causal dependencies (inference → action).

[0075] Provide an example of training without calculating loss segment by segment in related techniques: Task instruction: "Find the tray" Teacher agent: [Reasoning] "Let me check if there's a tray on the table..." → [Action] Search for [tray] Student agent: [Reasoning] "It might be on the shelf; I should go check there." → [Action] Search [tray] This shows that although the student model performed the correct actions, its reasoning deviated from the teacher model's thought process.

[0076] This application embodiment calculates the loss by separating it into two disjoint lexical spaces, using L... CoT By penalizing semantic bias within the inference process using the KL divergence between the teacher and student token distributions, gradients for aligned multi-step inference are generated. Act Rewards correct predictions in [actions], allowing action alignment even if the reasoning differs.

[0077] In some embodiments of this application, the trajectory text is multiple, and the method further includes: Calculate the complexity score for each trajectory text. The complexity score reflects the complexity of the interactive task corresponding to the trajectory text. Sort multiple trajectory texts in order of complexity score from low to high; The student model is trained using the ranking results.

[0078] Complexity scores quantify the difficulty of the task corresponding to each trajectory and can be calculated in various ways. They can be calculated based on the number of interaction rounds within the trajectory (more rounds, higher score); or based on the total length of the inference fragment (longer inference, higher score); or a weighted average of factors such as the number of tool calls and the size of the task state space. After calculating the complexity score for each trajectory, all trajectories are arranged in ascending order of score, with simpler tasks listed first and more complex tasks last. The course training is conducted in stages. The initial training phase uses only low-complexity trajectory samples. Once the model achieves a certain performance on simple tasks, higher-complexity trajectory samples are gradually introduced, ultimately using trajectories across the entire complexity range for training. The pace of complexity increase during training can be dynamically adjusted based on validation set performance.

[0079] The embodiments of this application organize training samples in order of increasing difficulty, which reduces the initial learning difficulty of the model, helps the model to converge stably, gradually accumulates the ability to perform complex tasks, and improves the final distillation effect.

[0080] To further improve learning efficiency and stability, a course learning approach based on trajectory complexity scores is adopted:

[0081] in and represent the lengths of the reasoning and action segments respectively, reflects teacher uncertainty. α, β, and γ are used to identify the corresponding weights. During training, multiple trajectory texts are sorted according to C(τ), and model training starts from simple trajectory text examples and gradually transitions to more complex examples.

[0082] To better understand the advantages of the embodiments of the present application, the process of calculating loss in the embodiments of the present application is specifically described below.

[0083] In the related art, the token-level KL strategy couples heterogeneous gradients to a single update direction, generating a conflict angle θ between reasoning and action in the parameter space (see Figure 3 ). The embodiments of the present application rephrase this process as orthogonal gradient projection: reasoning tokens and action tokens are in their respective subspaces ( , ), they are normalized separately and then geometrically recombined. This projection eliminates cross-segment interference and generates segment-specific updates, changing the overall optimization geometry.

[0084] Let and represent the token domains of reasoning and action, satisfying ∩ = .

[0085] Standard token-level KL minimization in the related art:

[0086] θ represents taking the gradient with respect to the student model parameter θ, L_token represents the loss function of traditional token-level distillation, KL represents the KL divergence operation, p_T( |x{<t}) represents the conditional probability distribution output by the teacher model based on the context token sequence x_{<t}, p_S( |x_{<t}) represents the conditional probability distribution output by the student model based on the same context token sequence x_{<t}. This formula characterizes the gradient of the loss with respect to the student model parameters under the traditional token-level distillation method. The calculation of this gradient is completed in the complete joint token space combining reasoning and action. Tokens of different semantic types share the probability distribution weight, which will cause gradient signals to couple with each other.

[0087] Normalization over the entire vocabulary ∪ causes semantically incompatible tokens to share probability mass. This coupling biases the gradient towards high-frequency reasoning tokens and suppresses rare but task-critical action tokens, making it impossible to decouple reasoning and action.

[0088] This application addresses the problem by restricting KL calculations to disjoint subspaces:

[0089] _θ represents the gradient of the student model parameters θ, L_SAD is the total loss corresponding to the structured agent distillation in this application, m_r(t) is the inference binary mask at position t, m_a(t) is the action binary mask at position t, KL_Vr represents the KL divergence operation performed in the inference lexical subspace V_r, and KL_Va represents the KL divergence operation performed in the action lexical subspace V_r. The KL divergence operation is performed internally, where p_T and p_S are the probability distributions of corresponding positions in the teacher model and student model, respectively. This formula represents the gradient calculation method of this application, which selects based on the segment type mask of the word position. The gradient of the inference word position is calculated only using the gradient obtained from the inference subspace, and the gradient of the action word position is calculated only using the gradient obtained from the action subspace. The two types of subspaces are independent and do not intersect.

[0090] Gradient projection of the inference and action subspaces:

[0091] _θ represents the gradient of the student model parameters θ, L_SAD is the total loss corresponding to the structured agent distillation in this application, and Π_reason represents the projection operator that performs gradient projection onto the inference subspace. _θL_CoT represents the gradient of the inference loss with respect to the model parameters, and Π_action represents the projection operator that performs gradient projection onto the action subspace. _θL_Act represents the gradient of the action loss with respect to the model parameters. This formula is equivalent to the gradient solution process of this application from the perspective of gradient projection. It projects the gradient of the inference loss to the inference subspace and the gradient of the action loss to the action subspace. After the projection is completed, the gradients are added together, thereby achieving the decoupling of the inference supervision signal and the action supervision signal.

[0092] This application's embodiments, by projecting into different spaces, change the normalization domain and gradient direction, eliminating cross-fragment interference and generating fragment-specific updates. Therefore, the difference from token-level KL strategies in related technologies is geometric, not surface-level; this application's embodiments introduce structure-aware gradient decomposition instead of simply applying KL divergence to a smaller subset of tokens.

[0093] refer to Figure 3The diagram illustrates the gradient optimization geometry for different methods, with the horizontal axis representing the inference subspace and the vertical axis representing the action subspace. Gray arrows indicate coupled gradients obtained from global KL distillation in related techniques, where gradient signals from inference and action interfere with each other. Using the method of this application, blue arrows represent the gradients obtained by projecting the inference loss onto the inference subspace, orange arrows represent the gradients obtained by projecting the action loss onto the action subspace, and green dashed arrows represent the model parameter update amounts obtained after merging the subspace projections. This decouples the inference and action signals, avoiding cross-segment gradient interference. Figure 3 In the middle, ∠θ represents the angle between the inference gradient vector and the action gradient vector. The larger the angle, the more severe the mutual cancellation between the inference supervision signal and the action supervision signal, resulting in training oscillations and slower convergence.

[0094] Figure 4 To compare the performance of student agents at different model sizes, the horizontal axis represents the number of parameters in the student model, which are 120M, 340M, and 760M respectively.

[0095] Figure 4 The top left shows the task success rate. As the number of model parameters increases, the performance of the method in this application (red diamond) is always higher than the word-level baseline (green dot), and it is constantly approaching the performance of the teacher agent.

[0096] Figure 4 The upper right corner displays the inference lexical length. The lower the value of this indicator, the better. The inference lexical length of the method in this application embodiment is closer to that of the teacher agent and will not generate redundant or excessively long inferences.

[0097] Figure 4 The lower left corner displays the CoT matching rate, which measures the degree of matching between student reasoning content and teacher reasoning content. The matching rate of the scheme in this application embodiment is higher than that of the word-level distillation scheme in related technologies.

[0098] Figure 4 The bottom right corner represents the interaction round latency; a lower value is better, as it indicates the number of interaction steps required to complete the task. The method in this application's embodiment requires fewer interaction steps to complete the task than the teacher model, resulting in higher execution efficiency.

[0099] based on Figure 4 In conclusion, under the same model parameter conditions, the method provided in this application embodiment is superior to the word-level distillation scheme provided by related technologies in terms of task success rate, inference alignment degree and execution efficiency. The larger the model, the closer it is to the effect of the teacher agent.

[0100] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide an intelligent agent distillation apparatus, see [link to relevant documentation]. Figure 5 The device includes: The generation module is used to obtain the trajectory text generated by the teacher agent when performing interactive tasks; the trajectory text is natural language text, and the teacher agent is built based on the teacher model. The tagging module is used to map the trajectory text to the complete lexical space to obtain a lexical sequence and to tag the segment type to which each lexical in the lexical sequence belongs; the segment types include reasoning segments and action segments; The acquisition module is used to obtain the student prediction distribution output by the student model and the teacher prediction distribution output by the teacher model within the complete word space, based on the predicted position of each word in the word sequence. The normalization module is used to perform probability normalization on student and teacher prediction distributions within the inference word space for each word element. If the word element is a reasoning word element marked as belonging to an inference segment, the loss component of the inference word element is calculated based on the probability normalization result. If the word element is an action word element marked as belonging to an action segment, the probability normalization on student and teacher prediction distributions within the action word space is performed based on the probability normalization result. The inference word space and the action word space are subsets of the complete word space and are mutually exclusive. The distillation module is used to summarize the loss components of each inference term and each action term, calculate the total loss, and iteratively optimize the parameters of the student model based on the total loss to obtain the student agent after distillation training.

[0101] This device is a set of functional modules for implementing the aforementioned agent distillation method. The generation module receives the trajectory text produced by the teacher agent and converts it into a word sequence through word segmentation. The labeling module identifies the segment type of each word in the word sequence and generates corresponding label information or mask sequences. The acquisition module inputs the complete word sequence into the student and teacher models, performs forward propagation calculations, and obtains the prediction distribution of each prediction position in the complete word space. The normalization module performs branch processing according to word type, performs probability normalization in the corresponding subspace, and calculates the loss component for each position. The distillation module summarizes all loss components to obtain the total loss and performs backpropagation and parameter updates based on the total loss to complete the distillation training of the student model. Each module can be implemented through software code, dedicated hardware circuits, or a combination of both.

[0102] It is understood that the functions of each module in each device of the embodiments of this application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0103] Figure 6 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 6As shown, the electronic device includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the computer program, it implements the method described in the above embodiments. The number of memories 601 and processors 602 can be one or more. In a specific implementation, the electronic device may also include a communication interface 603 for communicating with external devices and exchanging data.

[0104] In practical implementation, if the memory 601, processor 602, and communication interface 603 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0105] Optionally, in a specific implementation, if the memory 601, processor 602 and communication interface 603 are integrated on a single chip, the memory 601, processor 602 and communication interface 603 can communicate with each other through an internal interface.

[0106] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0107] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0108] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0109] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0110] It should be understood that the aforementioned processor can be a CPU (Central Processing Unit), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0111] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0112] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0113] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0114] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0115] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0116] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0117] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0119] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for distilling an intelligent agent, characterized in that, include: Obtain the trajectory text generated by the teacher intelligent agent performing interactive tasks; the trajectory text is natural language text, and the teacher intelligent agent is constructed based on a teacher model; The trajectory text is mapped to a complete lexical space to obtain a lexical sequence, and the segment type to which each lexical in the lexical sequence belongs is marked; the segment type includes reasoning segments and action segments; For the predicted position of each word in the word sequence, obtain the student prediction distribution output by the student model and the teacher prediction distribution output by the teacher model within the complete word space; For each lexical unit, if the lexical unit is a reasoning lexical unit marked as belonging to the reasoning segment, then the student prediction distribution and the teacher prediction distribution are probability normalized respectively in the reasoning lexical unit space, and the loss component of the reasoning lexical unit is calculated based on the result of probability normalization. If the lexical unit is an action lexical unit marked as belonging to the action segment, then the student prediction distribution and the teacher prediction distribution are probability normalized respectively in the action lexical unit space, and the loss component of the action lexical unit is calculated based on the result of probability normalization. The reasoning lexical unit space and the action lexical unit space are subsets of the complete lexical unit space and are mutually disjoint. The loss components of each inference term and each action term are summarized to calculate the total loss. The parameters of the student model are then iteratively optimized based on the total loss to obtain the student agent after distillation training.

2. The method according to claim 1, characterized in that, The step of normalizing the probability of the student prediction distribution and the teacher prediction distribution within the inference lexical space includes: In the student prediction distribution, the probability values ​​of the predicted words belonging to the inference word space are selected, and the selected probability values ​​are normalized to obtain the result of normalizing the probability of the student prediction distribution in the inference word space. The probability values ​​of predicted words belonging to the inference word space are selected from the teacher prediction distribution, and the selected probability values ​​are normalized to obtain the result of normalizing the probability of the teacher prediction distribution within the inference word space.

3. The method according to claim 1, characterized in that, The segment type to which each lexical in the lexical sequence belongs includes: Determine the segment type corresponding to each lexical in the lexical sequence; Based on the segment type corresponding to each word, a mask sequence of the same length as the word sequence is generated, wherein each mask in the mask sequence is used to mark the segment type of the corresponding word.

4. The method according to claim 3, characterized in that, Each mask in the mask sequence includes multiple binary sub-masks, which include inference sub-masks, action sub-masks, and observation sub-masks. Each binary sub-mask has a value of 0 or 1, which is used to indicate whether the lexical at the corresponding position belongs to the segment type corresponding to the binary sub-mask, and at most one of the multiple binary sub-masks at the same position has a value of 1.

5. The method according to claim 1, characterized in that, The trajectory text fragment type also includes observation fragments, and the method further includes: Mark the observed lexical units in the lexical sequence that belong to the observed segment; For each observed term, the corresponding loss term is set to 0.

6. The method according to claim 1, characterized in that, The trajectory text can be multiple, and the method further includes: Calculate the complexity score of each trajectory text, whereby the complexity score reflects the complexity of the interactive task corresponding to the trajectory text; Sort the multiple trajectory texts in ascending order of their complexity scores; The student model is trained using the sorting results.

7. A smart agent distillation apparatus, characterized in that, include: The generation module is used to obtain the trajectory text generated by the teacher agent performing interactive tasks; the trajectory text is natural language text, and the teacher agent is constructed based on a teacher model. A tagging module is used to map the trajectory text to a complete lexical space to obtain a lexical sequence, and to tag the segment type to which each lexical in the lexical sequence belongs; the segment type includes reasoning segments and action segments; The acquisition module is used to acquire the student prediction distribution output by the student model and the teacher prediction distribution output by the teacher model within the complete lexical space, based on the predicted position of each lexical in the lexical sequence. The normalization module is used to perform probability normalization on the student prediction distribution and the teacher prediction distribution in the inference word space for each word element. If the word element is a reasoning word element marked as belonging to the inference segment, the module calculates the loss component of the inference word element based on the probability normalization result. If the word element is an action word element marked as belonging to the action segment, the module calculates the loss component of the action word element based on the probability normalization result. The inference word space and the action word space are subsets of the complete word space and are mutually disjoint. The distillation module is used to summarize the loss components of each inference word and the loss components of each action word, calculate the total loss, and iteratively optimize the parameters of the student model based on the total loss to obtain the student agent after distillation training.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images