Training sample construction, model training method and device

Abstract

The present disclosure provides a training sample construction method and device, and relates to the technical field of computers, in particular to the technical field of deep learning. The specific implementation scheme is: receiving a session request sent by an agent and belonging to the same inference track, and generating an input token sequence based on the text carried by the session request; sending the input token sequence to an inference engine; receiving an output token sequence and the logarithmic probability of each token returned by the inference engine based on a large language model; and generating a training sample corresponding to the inference track based on the input token sequence, the output token sequence, and the logarithmic probability of each token. This method provides accurate training samples for reinforcement learning training without modifying the agent business code.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the field of deep learning technology, and in particular to a method and apparatus for constructing training samples and training models. Background Technology

[0002] The mainstream large-scale model RL (Reinforcement Learning) training framework has a unified architecture, relying on general scheduling, training, and sampling components to complete PPO (Proximal Policy Optimization) / GRPO (Generalized Reinforcement Learning with Preference Optimization) updates.

[0003] Three technical solutions are proposed for multi-round sampling of intelligent agents: multi-round loop built into the framework, token sequence maintained autonomously by the business side, and text trajectory collection decoupled through OpenAI compatible services. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for constructing training samples and training models.

[0005] In a first aspect, embodiments of this disclosure provide a training sample construction method, the method comprising: receiving a session request sent by an agent belonging to the same inference trajectory, and generating an input word sequence based on the text carried in the session request; sending the input word sequence to an inference engine; receiving an output word sequence and the log probability of each word returned by the inference engine based on a large language model; and generating training samples corresponding to the inference trajectory based on the input word sequence, the output word sequence, and the log probability of each word.

[0006] Secondly, embodiments of this disclosure provide a model training method, which includes: dividing multiple sample groups into multiple task units based on the task type corresponding to the sample groups, wherein each sample group includes multiple training samples corresponding to multiple inference trajectories of the same inference task; the training samples are training samples obtained by the method described in any implementation of the first aspect above; for each task unit, allocating the multiple sample groups included in the task unit to multiple graphics processing units (GPUs); and performing model training using the multiple GPUs and the allocated sample groups.

[0007] Thirdly, embodiments of this disclosure provide a training sample construction apparatus, which includes: a first receiving module, a sequence sending module, a second receiving module, and a sample generation module. The first receiving module is configured to receive a session request sent by an agent belonging to the same inference trajectory, and generate an input word sequence based on the text carried in the session request. The sequence sending module is configured to send the input word sequence to an inference engine. The second receiving module is configured to receive the output word sequence and the log probability of each word returned by the inference engine based on a large language model. The sample generation module is configured to generate training samples corresponding to the inference trajectory based on the input word sequence, the output word sequence, and the log probability of each word.

[0008] Fourthly, embodiments of this disclosure provide a model training apparatus, comprising: a first allocation module, a second allocation module, and a model training module. The first allocation module is configured to divide multiple sample groups into multiple task units based on the task type corresponding to the sample groups. Each sample group includes multiple training samples corresponding to multiple inference trajectories of the same inference task. The training samples are training samples obtained by the method described in any implementation of the first aspect above. The second allocation module is configured to allocate the multiple sample groups included in each task unit to multiple graphics processing units (GPUs). The model training module is configured to perform model training using the multiple GPUs and the allocated sample groups.

[0009] Fifthly, embodiments of this disclosure provide an electronic device including one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first or second aspect.

[0010] In a sixth aspect, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the method as described in either the first or second aspect.

[0011] In a seventh aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] Figure 1This is an exemplary system architecture diagram to which this disclosure can be applied; Figure 2a This is a flowchart of an embodiment of the training sample construction method according to the present disclosure; Figure 2b This is a schematic diagram of an embodiment of the training sample construction method according to the present disclosure; Figure 2c This is an overall architecture diagram of the training sample construction method based on this disclosure; Figure 2d This is a flowchart of yet another embodiment of the training sample construction method according to the present disclosure; Figure 2e This is a timing diagram of one embodiment of the training sample construction method according to the present disclosure; Figure 3a This is a flowchart of yet another embodiment of the model training method according to the present disclosure; Figure 3b This is a schematic diagram of an embodiment of the model training method according to the present disclosure; Figure 4a This is a flowchart of an application scenario based on this disclosure; Figure 4b This is a flowchart based on another application scenario of this disclosure; Figure 5 This is a schematic diagram of one embodiment of a training sample construction apparatus according to the present disclosure; Figure 6 This is a schematic diagram of one embodiment of a model training apparatus according to the present disclosure; Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0016] Figure 1 An exemplary system architecture 100 is shown, in which an embodiment of the training sample construction method of this disclosure can be applied.

[0017] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0018] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0019] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0020] Server 105 can be a server that provides various services, such as receiving session requests sent by the agent that belong to the same inference trajectory, generating an input word sequence based on the text carried in the session request; sending the input word sequence to the inference engine; receiving the output word sequence and the log probability of each word returned by the inference engine based on the large language model; and generating training samples corresponding to the inference trajectory based on the input word sequence, the output word sequence, and the log probability of each word.

[0021] It should be noted that server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide training sample construction services), or as a single software program or software module. No specific limitations are made here.

[0022] It should be noted that the training sample construction method provided in the embodiments of this disclosure can be executed by terminal devices 101, 102, and 103, or by server 105 and terminal devices 101, 102, and 103 in cooperation with each other. Accordingly, all parts (e.g., units, sub-units, modules, and sub-modules) of the training sample construction apparatus can be entirely disposed in terminal devices 101, 102, and 103, or can be disposed in server 105 and terminal devices 101, 102, and 103 respectively.

[0023] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0024] Figure 2a A flow 200 illustrating an embodiment of a training sample construction method is provided. This training sample construction method may specifically include the following steps: Step 201a: Receive a session request sent by the agent that belongs to the same inference trajectory, and generate an input word sequence based on the text carried in the session request.

[0025] In this embodiment, since the agent only uses standard OpenAI (Open Artificial Intelligence) chat / completions semantics, the executing entity (e.g., Figure 1 The terminal devices 101, 102, 103 or server 105 can construct an intermediate layer to capture one or more session requests (i.e., messages) submitted by the agent that belong to the same inference trajectory (large language model or large model) in the direction of the agent's request.

[0026] The session request may include information such as system prompts, user questions, model responses, and tool return results.

[0027] For each session request, the text carried by the session request can be segmented and transformed to generate a sequence of input tokens (i.e., tokens), namely input_ids.

[0028] Here, an inference trajectory is used to indicate the complete interaction record formed by the agent through one or more rounds of large model inference and optional tool call interactions triggered by the same task prompt; an inference trajectory may contain one or more attempts, each of which contains one or more large model session requests.

[0029] Step 202a: Send the input word sequence to the inference engine.

[0030] In this embodiment, the execution entity can construct an inference request body that matches the native interface ( / generate) of the inference engine based on the input lexical sequence and the activation information, and send the inference request body to the inference engine.

[0031] Here, the enable information is used to instruct the inference engine to return the output word sequence and the log probabilities of each word.

[0032] Step 203a: Receive the output word sequence and the log probability of each word returned by the inference engine based on the large language model.

[0033] In this embodiment, after receiving the input word sequence and the activation information, the inference engine can call the large language model to process the input word sequence, generate the output word sequence and the log probability of each word, and return the output word sequence and the log probability of each word based on the activation information.

[0034] Here, the output word sequence and the log probability of each word can be stored in a preset field, such as the logprobs field, for transmission.

[0035] In the response direction, the executing entity can use the intermediate layer to receive the output word sequence and the log probability of each word returned by the inference engine based on the large language model, which are contained in preset fields.

[0036] Here, the logarithmic probability of a word is the logarithmic probability value output by the large model when sampling and generating the corresponding word, representing the confidence level of the model in generating that word.

[0037] Step 204a: Based on the input word sequence, the output word sequence, and the log probability of each word, generate training samples corresponding to the inference trajectory.

[0038] In this embodiment, the execution entity can use the intermediate layer to generate and store training samples (also called trajectory samples) corresponding to the inference trajectory based on the input lexical sequence, the output lexical sequence, and the log probability of each lexical.

[0039] One inference trajectory corresponds to one training sample, and different inference trajectories correspond to different training samples.

[0040] The above embodiments of this disclosure receive a session request sent by the agent belonging to the same inference trajectory, and generate an input word sequence based on the text carried in the session request; send the input word sequence to the inference engine; receive the output word sequence and the log probability of each word returned by the inference engine based on the large language model; and generate training samples corresponding to the inference trajectory based on the input word sequence, the output word sequence, and the log probability of each word. This ensures that the "token sequence replayed during training" and the "token sequence fed into the model during sampling" are the same data, and the text only exists in the layer "shown to the agent". This eliminates the "inconsistency between training and inference word segmentation" problem from the root upstream. The agent only sends and receives standard OpenAI messages, that is, it provides accurate training samples for reinforcement learning training without modifying the agent's business code.

[0041] In some alternative approaches, the method further includes: generating response text based on the output lexical sequence and the log probabilities of each lexical; and sending the response text to the agent.

[0042] In this implementation, the executing agent can assemble the output word sequence and the log probabilities of each word into a response text and return it to the intelligent agent.

[0043] For example, the response text can be stored in a specified field of the response body, such as choices[0].message, and returned as an agent in the form of a response body, such as the standard chat.completion response body.

[0044] Here, choices[0].message is used to carry the text content of the large model output and / or tool call instructions. In multi-candidate sampling, choices is an array; in normal single-round inference, the 0th choice[0] is used.

[0045] Here, the response body may also include the finish_reason for the end of the large model inference process of this round of session request, usage statistics, and the custom extended field metadata (which can be used to store the model weight version corresponding to this round of session request), etc.

[0046] This implementation generates response text based on the output lexical sequence and the logarithmic probability of each lexical; the response text is then sent to the agent, so that the text only exists in the "for the agent to see" layer and never participates in the construction of training data; while the agent is completely unaware of the underlying / generate interface.

[0047] In some alternative approaches, generating an input word sequence based on the text carried by the session request includes: in response to determining that the session request is the first session request of the inference trajectory to which it belongs, performing word segmentation on the text carried by the session request to obtain an input word sequence; in response to determining that the session request is not the first session request of the inference trajectory to which it belongs, determining the new content in the text carried by the session request relative to the text corresponding to the prefix word sequence, performing word segmentation on the new content to obtain a new word sequence, and concatenating the new word sequence after the prefix word sequence to obtain the input word sequence.

[0048] In this implementation, the executing entity can determine whether the session request is the first session request of the inference trajectory to which it belongs.

[0049] If so, the text carried in the session request is segmented and converted to obtain the input word sequence.

[0050] If not, i.e. the session request is not the first session request in the inference trajectory to which it belongs, then determine the new content of the text carried by the session request relative to the text corresponding to the prefix word sequence, perform word segmentation on the new content to obtain the new word sequence, and concatenate the new word sequence after the prefix word sequence to obtain the input word sequence.

[0051] Among them, the prefix lexical sequence is determined based on the input lexical sequence and output lexical sequence corresponding to the historical session requests of the reasoning trajectory.

[0052] For example, if the session request is the first session request in the inference trajectory to which it belongs, the prefix word sequence is empty. The text carried by the session request is segmented and converted to obtain the input word sequence, such as token1. After inference by the large model, the output word sequence is token2 and token3. token1, token2 and token3 are stored in sequence to obtain the prefix word sequence. If the next session request is received, and this session request is not the first session request, the new content of the text carried by the session request relative to the text corresponding to the prefix word sequence is determined. The new content is segmented and converted to generate a new word sequence, such as token4. token4 is concatenated to the prefix word sequence to obtain the input word sequence, namely token1, token2, token3 and token4.

[0053] Here, before determining the new content of the text carried in the speech request relative to the text corresponding to the prefix word sequence, a prefix check can be performed, that is, to determine whether the prefix matches. If it matches, then determine the new content of the text carried in the speech request relative to the text corresponding to the prefix word sequence.

[0054] In addition, before performing word segmentation on the new content, the repeated fixed system prompt prefixes can be removed first, and the removed new content can be segmented to generate a new word sequence.

[0055] This implementation method, on the one hand, eliminates the need for repeated word segmentation of the entire historical context in subsequent round requests, significantly reducing the word segmentation computation during multi-round agent interaction, shortening inference preprocessing time, and improving overall inference throughput performance. On the other hand, it concatenates the historical prefix word sequence to ensure continuous alignment of all word subscripts under the entire trajectory, providing a unified benchmark for subsequent trajectory flat array maintenance, loss mask assignment, and word-by-word logarithmic probability storage, preventing sequence misalignment issues caused by multiple full word segmentations of the context, ensuring that the final reinforcement learning training samples are standardized and usable, and adapting to business scenarios of multi-round trajectory incremental continuation and breakpoint continuation training.

[0056] In some alternative approaches, training samples may include a first flat array, a second flat array, a third flat array, and a fourth flat array; that is, training samples may be stored in the form of a first flat array, a second flat array, a third flat array, and a fourth flat array.

[0057] In this implementation, the first flat array, the second flat array, the third flat array, and the fourth flat array have the same length and share a set of position indices. The same position index points to the array element in each flat array that corresponds to the same word.

[0058] The first flat array is used to store the new word sequence and output word sequence of each session request in the order of execution of each session request belonging to the same inference trajectory.

[0059] The second flat array is used to store the loss mask, where the position index corresponding to each word in the output word sequence is marked as participating in the loss calculation, and the position index (or subscript) corresponding to each word in the non-output word sequence is marked as not participating in the loss calculation.

[0060] The semantics of the loss mask (corresponding to the k-th position in the response space) can be represented by the following formula:

[0061] Here, 1 indicates that the token was generated by the trained model in this sampling and does not belong to the old weight prefix; 0 indicates tool return, frame reconstruction prompt / summary tokens, and off-policy prefixes for partial trajectory continuation. The overall physics function is that only "actions actually sampled by the current policy (model inference)" constitute the action space of reinforcement learning and are included in the policy gradient; environmental feedback (tool output) and tokens not generated by this policy must be excluded, otherwise the model will be trained to "predict environmental output". In addition, it can be added that when cropping or slicing trajectories, the existing mask value must be retained, and re-inference according to token type is not allowed (because the information of "old weight prefix" cannot be recovered from the token itself).

[0062] It should be noted that the loss mask annotation method can also be replaced by the following methods: the middleware records the metadata of "index range of tokens generated in this round" in each round, and the training side generates a mask array in batches according to the range (equivalent, just with the annotation delayed); or an "source type" enumeration (model / tool / system / summary) is attached to each round of messages, and the training side maps it to a mask according to the enumeration.

[0063] The third flat array is used to store the corresponding log probabilities at the position indices of each word in the output word sequence, with the remaining positions filled with placeholder values.

[0064] The fourth flat array is used to store the corresponding model weight version at the position index of the word.

[0065] Here, the executing entity can also maintain a Triplet corresponding to a session request or a round of session requests. The Triplet can only store four indices: token_start (token start index) / token_end (token end index) and logprob_start (log probability start index) / logprob_end (log probability end index), which point to the flat array of training samples corresponding to the trajectory, without copying the token.

[0066] Among them, token_start / token_end records the storage range of the first flat array of the trajectory corresponding to the session request interaction (the sequence of newly added tokens and the sequence of output tokens in this round, i.e. all newly added tokens in this round of interaction); logprob_start / logprob_end records the storage range of the log probability of each token in the sequence of output tokens corresponding to the session request interaction in the third flat array of the trajectory corresponding to the session request.

[0067] A single inference trajectory can correspond to one or more attempt attempts (for the same inference trajectory trajectory_id, if an error occurs in the middle, tokens are exhausted, or a continuation is triggered, a new inference trajectory will not be created, but a new attempt will be added instead). One attempt can correspond to one or more session requests. The execution entity can maintain a segment to maintain the local prefix token sequence state within this segment. A segment consists of a batch of consecutive rounds (multiple Triplets) under the same attempt. The prefix matching assumption is satisfied within this segment, and incremental tokenization can be performed safely.

[0068] When the agent performs context compression (such as a summary reset) and the new messages no longer have prefixes of the historical tokens (i.e., they do not match the text corresponding to the prefix term sequence), the current segment is closed and a new segment is opened. The existence of the segment ensures that the prefix assumption of "incremental tokenization" still holds in the compression scenario.

[0069] The storage layer is organized as a dict[trajectory_id -> list[Trajectory]], where each element in the list represents an attempt. During updates, an exact match is performed by attempt_id (to avoid outdated background threads overwriting later-updated attempts; that is, instead of directly overwriting the entire trajectory, the target attempt is found in the list by attempt_id, and only that single element is modified, preventing older threads from overwriting the data of newer attempts). When reading data, if attempt_id is not specified, the latest generated attempt instance in the list is returned.

[0070] Each attempt instance maintains an independent state machine, with states including PENDING, GENERATING, and COMPLETED; and abnormal states including FAILED and ABORTED.

[0071] One inference trajectory corresponds to the first, second, third, and fourth flat arrays. One attempt can have multiple segments, and one segment contains several triplets.

[0072] An example of storing training samples for Segment and Triplet, based on the first, second, third, and fourth flat arrays, is as follows: Figure 2bAs shown, `tokens` is a flat array of the entire sequence, divided into Segment 0 (continuous incremental context) 201b and Segment 1 (newly opened after context compression) 202b. Segment 0 sequentially contains prompt, response0, tool output, response1, tool output, summary / reconstruction prompt, response2, tool output, and response3. The response space is aligned with the three rows of arrays below (i.e., the interval position), excluding the prompt segment. In `loss_masks`, 1 represents inclusion of loss, with the response area marked as 1, the tool output area marked as 0, and the summary / reconstruction prompt area marked as 0. `rollout_log_probs` stores the actual logprob during sampling, with the response area filled with actual values, such as -0.12, -0.87, -0.33, etc., and the tool output and summary / reconstruction prompt areas set to 0. `rollout_weight_versions` generates weight versions position by position, Segment... 0 is v=12 (old weight prefix, the entire mask is set to 0 after rerun), Segment 1 is v=13; token_rewards is filled with 0 in most positions, and final_reward R is written only at the end; Triplet represents a round of large model interaction, only the index range is saved, the token is not copied, and it is divided into Triplet0, Triplet1, Triplet2, and Triplet3; Summary Reset means that the new message is no longer a historical prefix and the old Segment is closed.

[0073] In addition to using segmentation to handle "new messages are no longer historical prefixes", the following methods can also be used: after compression, the entire segment is fully tokenized again and a new trajectory segment is started (at the cost of one full encoding); or the compression is required to only allow "truncating from the head + retaining the suffix", so that the incrementality can still be maintained by using the suffix index.

[0074] This approach leverages the first, second, third, and fourth flat arrays, along with the storage index structure of Segments and Triplets, to achieve three key technical benefits: First, by appending to the flat arrays and storing only the indices in the triplets, the overall storage complexity is O(N), avoiding the O(N²) copy overhead. Second, during the training phase, the indices stored in the triplets can be used to restore the continuous flat arrays into a round-by-round interactive view for round-by-round reward assignment and metric statistics. Third, in trajectory continuation scenarios, the indices can be used to locate the prefix segments generated by the old weights, and the loss mask corresponding to those segments can be invalidated, preventing them from participating in gradient updates.

[0075] In some alternative approaches, the training samples also include a fifth flat array.

[0076] In this implementation, once all session requests for the inference trajectory have been inferred by the large model, the trajectory data of that inference trajectory (i.e., the data in the first, second, third, and fourth flat arrays) can be read from the trajectory storage structure. The trajectory data is then distributed to the reward function bound to the current task type for unified scoring and calculation, and structured reward information is output. The structured reward information includes the global final reward final_reward, the round-by-round interaction reward completion_rewards, the trajectory validity identifier is_valid, and the extended additional information extra_info.

[0077] Based on the above reward information, a dedicated fifth flat array (lexical reward array) can be constructed. The fifth flat array has the same length as the first, second, third, and fourth flat arrays and shares the same set of position indices, so as to align the data of each dimension one by one at the lexical granularity.

[0078] The reward assignment process is divided into two scenarios: global reward assignment and round-by-round reward assignment. By default, only the global final reward `final_reward` is assigned to the last position index of the fifth flat array, and the other positions are initially empty or invalid. If the reward calculation result carries round-by-round interaction rewards `completion_rewards`, then the round-by-round reward assignment logic is started. It iterates through all data triplets in the current attempt, and based on the global position index range recorded by each data triplet, it matches and writes the round-by-round rewards corresponding to each round of large model interaction to the corresponding position in the fifth flat array. At the same time, it verifies that the number of round-by-round rewards is consistent with the number of data triplets to ensure that the round-by-round rewards are accurately aligned with the round-by-round interaction rounds, avoiding the problems of reward mismatch, omission, and over-match.

[0079] award The assignment process can be represented by the following formula:

[0080] in, Indicates the length of the response space; This represents the final reward for the entire trajectory; Indicates the first Wheel (the first) The round-by-round reward of each triplet is assigned to the logprob index range of that triplet; before assignment, it is verified that the number of round-by-round rewards is consistent with the number of triplets.

[0081] After completing the assignment and verification of the lexical reward array, the current running status can be updated to COMPLETED, marking that the sampling process of the training sample corresponding to the inference trajectory has completed a normal closed loop. Then, the processed training data, containing five sets of aligned flat arrays, is delivered to the training sample queue grouped by data source for subsequent large language model reinforcement learning training; simultaneously, the temporary data for this trajectory is deleted from the local trajectory storage structure, releasing memory storage resources. Finally, the agent's release session interface is called to unbind the sticky route corresponding to the current session request, releasing the inference instance resources, completing the full closed loop of sampling, scoring, storage, and resource reclamation for a single inference trajectory.

[0082] This implementation adds a fifth flat array to store reward data, maintaining index alignment with the other four flat arrays, thus achieving precise association between words and corresponding reward data; it also unifies and standardizes the trajectory storage structure, reduces data conversion costs, and ensures the integrity of training samples.

[0083] In some optional ways, the input lexical sequence is sent to the inference engine, including: determining the number of available lexical units based on the total lexical budget corresponding to the inference trajectory and the number of lexical units consumed by the historical session requests corresponding to the inference trajectory; and sending the input lexical sequence and the number of available lexical units to the inference engine.

[0084] In this implementation, during each round of inference (calling the large model to infer the session request) of the training samples corresponding to the inference trajectory, the system performs fine-grained lexical budget control.

[0085] The executing entity can determine the number of available tokens based on the difference between the total token budget corresponding to the inference trajectory and the number of tokens consumed in the historical session requests corresponding to the inference trajectory. Alternatively, it can further consider the statistical interference of the tokens output by the first round of the model, and rely on the first data triplet of the inference trajectory to accurately calculate the initial prompt length using the formula initial_prompt_len = t0.token_end-(t0.logprob_end - t0.logprob_start) (the token interval t0.token_end of the first round triplet minus its response length t0.logprob_end - t0.logprob_star, which gives the initial prompt length initial_prompt_len). Then, it can use the formula remaining = max_response_len_per_trajectory - (number of consumed tokens - initial_prompt_len), where max_response_len_per_trajectory represents the total token budget and remaining represents the number of available tokens.

[0086] Alternatively, it can also be represented by the following formula:

[0087] in, This represents the right endpoint of the token range for the triplet requested in the first session; This represents the left and right endpoints of the logprob interval of the first round triplet, and the difference between them is the length of the first round response, which is the length of the output word sequence. This indicates the token length of the initial prompt (total length in the first round minus the length generated in the first round). This indicates the length of the current inference trajectory tokens array; therefore This represents the number of response tokens consumed (including those generated in each round and those output by the tool). This represents the max_response_len_per_trajectory configured for the data source, which is the response space budget for a single inference trajectory. This represents the max_tokens in the agent's request body; It returns the "context exhausted" semantics. This indicates the number of available lexical units after clamping constraints.

[0088] Based on the number of available lexical units, a clamping constraint is applied to the maximum number of generated lexical units in this round, limiting the scale of generated lexical units. If the number of available lexical units is less than or equal to 0, the system terminates the current round of inference and returns a context exhaustion response. This scenario is not considered a hard failure and does not require triggering abnormal retry and termination processes. Instead, it is handled by the upper layer according to partial response rules, retaining all valid trajectory samples and improving the effective sampling rate.

[0089] This implementation dynamically determines the number of available lexical units in the current round and distributes them to the inference engine, thereby achieving precise management of lexical unit resources, intercepting invalid inference requests, and saving computing power.

[0090] In some alternative approaches, the input lexical sequence is sent to the inference engine, including: in response to determining that the session request is a non-first session request of the inference trajectory, determining the inference instance of the inference engine corresponding to the historical session requests of the inference trajectory; and sending the input lexical sequence to the inference instance.

[0091] In this implementation, inference instances can be matched to the sessions corresponding to the inference trajectories, and sticky route bindings can be established.

[0092] For each session request, it can be determined whether the session request is not the first session request in the relevant inference trajectory. If so, it means that the session request already has a historically sticky inference instance. Determine the reasoning instance of the reasoning engine corresponding to the historical session requests that determine the reasoning trajectory.

[0093] Furthermore, it can be determined whether the inference instance is in a healthy state and whether the operating load does not exceed a preset threshold. If so, the inference instance is directly reused, that is, the input word sequence is sent to the inference instance, and the KV context cached on the reuse instance side is inherited to improve inference efficiency.

[0094] If a session request lacks a historically bound inference instance, or if the original bound instance is abnormal or overloaded, the inference instance with the fewest currently pending requests (the fewest session requests being processed) is selected from all healthy and available inference instances as the target instance. A new sticky binding relationship is then established for the inference trajectory to which the current session request belongs. After selecting the inference instance, the input lexical sequence `input_ids` is encapsulated into an inference request body and forwarded to the target inference instance's native ` / generate` interface to execute the large model inference generation task.

[0095] Here, the ` / generate` interface can also be replaced by the following interfaces: using the OpenAI completions interface (instead of `chat / completions`) that supports `prompt_token_ids` and `logprobs`; or using the engine's gRPC (gRPC Remote Procedure Call framework) / in-process Python API; or using the equivalent token interface of the self-developed engine. The key constraint is only "the input is a token id, and the output includes token-per-logprob".

[0096] The method of obtaining per-token logprob can be replaced by the following: if the lower-level inference engine cannot return logprob, a logprob-only forward pass can be added to the same instance (the log probability of the forward pass is obtained by teacher forcing the sampled token sequence), at the cost of an extra forward pass; or the training side can recalculate it immediately before the weights are updated (equivalent only in the scenario of full synchronization and zero off-policy).

[0097] This implementation enables KV cache reuse, reduces redundant context calculations, lowers inference overhead, and improves inference speed; it also maintains the continuity of trajectory context state, ensuring data alignment for subsequent trajectory sample structured storage.

[0098] In some alternative approaches, the method further includes: in response to detecting a tool invocation command sent by the agent based on the response text, determining whether there is a sandbox bound to the inference trajectory corresponding to the response text; and in response to the existence of such a sandbox, executing the tool invocation command using the sandbox bound to the inference trajectory.

[0099] In this implementation, the system adopts a control mechanism of trajectory binding, on-demand reuse, and unified recycling for intelligent agent tool calls and sandbox resources.

[0100] When it is detected that the agent responds to the text generation tool call command based on the model, it first determines whether the inference trajectory corresponding to the current tool call command is bound to a sandbox instance; if a sandbox exists, the sandbox dedicated to that trajectory is directly reused to execute the tool call command, ensuring that the tool running environment is consistent under a single trajectory.

[0101] Sandboxes can take many forms, such as Kubernetes Pods, Docker, lightweight virtual machines (Firecracker / gVisor), WASM runtimes, raw process + namespace isolation, or remote sandbox services.

[0102] The sandbox adopts an agent-based on-demand creation mechanism, triggering the creation logic only when the tool is first called. Before creation, the reusable sandbox identifier is verified first to avoid the resource consumption of repeated creation.

[0103] The destruction and recycling of sandbox resources are not managed by the agent's `clear()` interface, but are uniformly handled by the intermediate layer in the final `finally` block (i.e., after the sampling process of the training samples corresponding to the entire inference trajectory ends, such as normal completion, early termination, or abnormal failure). Idle sandboxes are returned to the resource pool for unified scheduling. For scenarios involving resuming (attempting) inference trajectories, the aborted inference trajectory retains the bound sandbox instance. During subsequent training resumption, the accumulated file system, processes, and other historical states within the sandbox can be directly reused without repeated environment initialization, ensuring the continuity of interaction and tool operation.

[0104] The agent's clear() interface is only used to idempotently clean up local resources such as its own HTTP client, releasing only agent-level resources and not interfering with the creation, reuse, and recycling lifecycle of the sandbox.

[0105] This implementation method is designed for scenarios where inference trajectories can be continued (tried). It can directly reuse the historical states of the file system, processes, etc. accumulated in the sandbox without repeating the environment initialization, thus ensuring the continuity of interaction and tool operation.

[0106] In a specific example, the agent base class (which is the unified top-level abstract parent class for agents, defining the minimum, unified interface specification that all (business) agents must adhere to, serving as the unified abstract top-level interface for all business agents) only needs to implement two asynchronous methods: the `run_trajectory(trajectory)` method, which completes the entire trajectory and returns a reward, and the `clear()` method, which releases resources with idempotency. The agent can send a POST session request to the injected route address through the standard OpenAI / Litellm client. The overall inference trajectory execution process is uniformly managed by the orchestrator. For a single attempt on a single trajectory, the orchestrator performs dependency injection for the agent instance. The injected parameters include the `router_url` carrying `trajectory_id` and `attempt_id`, the data source `completion_params` (i.e., the task type), the total lexical budget `max_response_len_per_trajectory` for a single inference trajectory, the temperature, and optional `sandbox_env_client`, `reward_fn` reward function, and various custom configuration fields. This injection method allows the agent to acquire reasoning, sandboxing, and scoring capabilities without being aware of it, while also enabling the framework to accurately bind the trajectory_id and attempt_id attribution information for each request without requiring the agent to pass additional parameters.

[0107] During the request execution phase, the middleware parses the router_url prefix, concatenates it to generate a unique session identifier request_id = trajectory_id#attempt_id, and serializes session requests with the same request_id through a coroutine-level mutex lock to avoid context crossover caused by concurrent writes, while maintaining a counter for in-transit (session) requests. Based on `trajectory_id` and `attempt_id`, the corresponding `Trajectory` object is read, and multiple sets of flat arrays (the first, second, third, fourth, and fifth flat arrays corresponding to the inference trajectory) and segment and triplet index information are obtained. The `build_turn_input()` parser constructs the current round of `input_ids` (i.e., the input word sequence) in three scenarios: In the first round, the complete messages are fully encoded using the chat template (i.e., word segmentation) and a new segment is created; In subsequent rounds (not the first round), the system prompt prefix is ​​stripped during incremental encoding, and only newly added messages are encoded and appended to the flat array; If context compression causes a prefix mismatch, the current segment is closed and a new segment is opened to adapt to Summary Reset semantics. Simultaneously, the system completes parameter mapping, converting OpenAI standard parameters into native sampling_params of the large model, and constructs a native / generate request body (i.e., inference request body) containing input_ids, rid, return_logprob, and sampling_params. Among them, return_logprob is forced to be true (i.e., enabling the output word sequence and the log probability return function of each word) to ensure that token-by-token training is feasible, and the skip_special_tokens parameter is configured according to the model family to ensure that special tokens for thought chains and tool calls can be parsed normally.

[0108] Here, the session identifier can be carried using the URL (Uniform Resource Locator) path prefix / {trajectory_id} / {attempt_id}; the session identifier can also be carried in the following ways: placed in a custom HTTP header (such as X-Trajectory-Id), placed in an extended field of the request body, placed in the namespace of the model field (such as model="default@traj#attempt"), or implicitly carried using a separate port / separate base_url for each trajectory.

[0109] Furthermore, fine-grained response token budget management can be implemented. The initial prompt length is calculated using the first round of triplet: `initial_prompt_len = t0.token_end - (t0.logprob_end - t0.logprob_start)`. The remaining token quota (i.e., the number of available tokens) is then calculated using `remaining = max_response_len_per_trajectory - (current total token count - initial_prompt_len)`. The `remaining` parameter is used to clamp the `max_new_tokens` parameter for this round. If `remaining` ≤ 0, the system returns a context exhaustion semantic response, which is processed by the upper layer according to partial response logic and is not considered a hard failure. After parameter validation, priority is given to reusing sticky-bound, healthy, and load-compliant inference instances for the current `trajectory_id`, reusing the KV cache to accelerate inference. If no valid bound instance exists, a new sticky binding is established with the instance among healthy instances that has the fewest pending requests, and the constructed request body is forwarded to the instance's native ` / generate` interface.

[0110] After inference is completed, the return result of the inference engine is parsed by build_assistant_message(). The response_ids sequence and the token logprobs (i.e., the log probability of each token in the output token sequence) are extracted from meta_info.output_token_logprobs and their lengths are verified to be consistent. The model weight version of this round of inference (i.e., this round of session request) is read from meta_info.weight_version. At the same time, the thought chain block and tool call structure are parsed according to the model family to generate a standard agent message containing dialogue content, thought chain / thinking content reasoning_content, and tool call structure tool_calls. Then, `update_trajectory()` is called to append a new triplet to the Trajectory, uniformly maintaining four core flat arrays: the `tokens` array accumulates the incremental prompt token (i.e., newly added tokens) and response token (i.e., output tokens) for this round; the `loss_masks` array only sets valid identifiers for tokens generated by the large model, and invalidates them for tool-returned content, framework reconstruction prompts, and old weight prefix tokens for continued running; the `rollout_log_probs` array writes the actual logprob at the model generation position, and sets it to 0 in other positions; the policy offset metric `rollout_weight_versions` records the weight version range for each round. For the ThinkChain model, the system relies on the start and end token IDs of `think` cached by the word segmentation manager to simultaneously prune historical `think` blocks at the token level and text level to ensure context alignment for subsequent rounds of input.

[0111] Here, the policy offset metric can be expressed by the following formula:

[0112] in, This indicates the model weight version number used when the trajectory starts and ends sampling (i.e., the first and last session requests corresponding to the trajectory, with session requests processed serially); It represents how many parameter updates the trajectory has spanned, i.e. its degree of off-policy; it is used to count the proportion of partial trajectories and the maximum span, and serves as the basis for triggering the "zero mask for old prefix".

[0113] The above session request processing flow is executed cyclically until the inference trajectory for a single trajectory terminates. After `run_trajectory` completes execution (all session requests for inference trajectories have been inferred), the orchestrator retrieves the trajectory data (data in the flat array), performs scoring through the bound `reward_fn`, and outputs a reward result containing `final_reward`, `completion_rewards`, `is_valid`, and `extra_info`. A fifth flat array `token_rewards`, aligned with the other four arrays of equal length, is constructed. By default, `final_reward` is only written to the end of the array. If `completion_rewards` exists, rewards are assigned to each triplet round one by one, and the number of rounds is checked for matching. After processing, the trajectory status is set to `COMPLETED`, pushed to the data source group training queue, local trajectory storage is cleared, and the proxy interface is called to unbind the session stickiness. If any step of the process fails, the trajectory status is set to `FAILED`, triggering session termination and a new `attempt` retry. If the retries are exhausted, an invalid sample with `is_valid=false` is generated.

[0114] The system is configured with a strict and orderly training step termination mechanism, which sequentially executes operations such as terminating all instance requests, canceling active coroutines, and waiting for asynchronous tasks and in-transit requests to be written to disk. Trajectories that are not COMPLETED or ABORTED are uniformly marked as ABORTED to prevent dirty data from being written.

[0115] The sandbox resources are managed entirely by the framework throughout their lifecycle. Existing sandbox identifiers are created and reused on demand during the first tool call, and are not controlled by the agent's `clear()` method. Instead, they are returned to the sandbox pool by the orchestrator's `finally` block. Aborted pending execution paths retain the sandbox file system and process state for reuse in subsequent `attempt` runs. The agent's `clear()` method only idempotently cleans up its own HTTP client and other local resources, completely decoupling them from sandbox, inference paths, and inference session resources.

[0116] The aforementioned process is responsible for dependency injection, session serial lock, key-value cache reuse, flat array maintenance, loss_mask generation, token budget control, sandbox lifecycle, and trajectory final state management. All underlying core capabilities are implemented by the middle layer. The agent passively receives the capabilities injected by the middle layer without needing to be aware of the underlying implementation. The middle layer uniformly schedules all agents that inherit from the base class, achieving non-intrusive and universal empowerment.

[0117] An overall architecture diagram based on the above specific example is as follows: Figure 2cAs shown, the architecture comprises a model trainer 201c, a sampling orchestrator 202c, an agent (multi-round) 203c, a sandbox 204c, and an SGLang Cluster 205c. The Trainer (which issues prompts, consumes final-state trajectory, triggers step boundary synchronization, initiates generate calls, and receives traj group returns) includes RolloutSampler (issues prompts and consumes final-state trajectory), RolloutManager (SGLang engine, weight synchronization), TrainManager (Megatron worker, train), and TeacherManager (Megatron teacher, optimal policy distillation). The AgentFlow internally includes TokenizerManager (responsible for tokenization and chat template), TrajectoryStore (the unique data source during sampling), and _run_trajectory (single / multi-round scheduling, retries, emit, and calls traj_queue).The `add()` function sends the inference trajectory to the `TrajQueue` (which serves as the output queue for the steady-state trajectory) and a background thread for the Router reverse proxy. Internally, the Router integrates `ParserMiddleware` and `TrajectoryParser`. `ParserMiddleware` intercepts ` / v1 / chat / completions`, rewrites it as ` / generate`, converts messages to `input_ids`, configures `return_logprob=True`, and carries the `rid` parameter. This enables messages to be converted to tokens for inference, and the tokens are then encapsulated back into OpenAI format for feedback. `TrajectoryParser` includes `ToolCallParser` and `ReasoningParser`, performing tokenization, logprob alignment, loss_mask alignment, and writing back to the `TrajectoryStore`. `AgentFlow` calls the Agent via `multi_turn`. The Agent follows a multi-turn rule: one trajectory corresponds to one Agent instance, executing a `ReAct` loop, calling the large model, and executing tools. The tools run within the Sandbox, including `traj0`, `traj1`, and so on up to `traj`. N, along with AgentLoop, supports tasks such as mini-swe, gsm8k, and bcp. The Agent module interacts with AgentFlow through / v1 / chat / completions, sending rewards back to AgentFlow and calling exec shell to operate the Sandbox. The Sandbox is based on a Kubernetes Pod or Docker and contains multiple isolated instances of Sandbox N. AgentFlow's Router sends a token in ( / generate interface, input_ids) to the SGLang Cluster. The SGLang Cluster contains inference instances from worker 1, worker 2 up to worker N. After inference is complete, it sends a token out, carrying information such as token_ids, logprobs, and routed_experts to the Router.

[0118] Overall component function description: ParserMiddleware intercepts OpenAI requests, converts messages into tokens for inference, encapsulates tokens and sends them back to OpenAI messages; TrajectoryParser completes tokenization, tool invocation, inference parsing and logprob alignment; _run_trajectory is responsible for single and multi-round scheduling and retries, and sends the final trajectory to TrajQueue.

[0119] Regarding intercepting OpenAI requests, this can be achieved through a "reverse proxy + middleware chain," or through the following methods: intercepting chat calls within the agent process using client SDK (Software Development Kit) instrumentation / Monkey Patch; intercepting using the Sidecar process, Service Mesh filters, eBPF / network layer; or making the rewriting logic a custom endpoint plugin for the inference engine. These methods only change "at which layer to intercept," and the underlying mechanism remains unchanged: "bidirectional rewriting of text input / token output + forced per-token logprob + unified framework mask."

[0120] Among them, the sampling orchestrator AgentFlow is used to receive a batch of trajectories to be sampled from the training controller, create a coroutine task for each trajectory, determine the single-round / multi-round mode according to the data source, instantiate an agent instance for each trajectory and inject dependencies in the multi-round mode, and manage retries, aborts, reward write-back and final state dequeue.

[0121] The reverse proxy router is used for reverse proxying, exposing / {trajectory_id} / {attempt_id} / v1 / chat / completions to the outside world; it implements sticky + minimum load routing; and provides control interfaces such as session release, session termination, global termination, and waiting for disk write-in.

[0122] The ParserMiddleware is used for request-side operations: messages → input_ids and rewriting them into native / generate requests; and response-side operations: output_token_logprobs → response_ids + logprobs, persisting them to the database and reconstructing the standard chat response. It includes per-trajectory concurrency locking, response token budget control, and in-transit request counting. The TrajectoryParser is used for tokenization, trajectory construction and persistence, reasoning and tool call parsing, and token / logprob / loss_mask alignment. The parser is configurable by model family.

[0123] The unified tokenizerManager is used to encapsulate chat template applications, encode, and decode into asynchronous interfaces (thread pool / process pool); and centrally cache metadata such as system_prompt_len, think start and end token ID, and model family identifier.

[0124] TrajectoryStore serves as the unique source of true data during the sampling period, with trajectory_id → [attempt0, attempt1, ...]; updates are performed by matching attempt_id to avoid old threads overwriting new attempts.

[0125] The DataProcessor, which groups the trajectory by source, is used to execute a two-level sharding algorithm. It divides the trajectory group into DP (Data Parallelism) shards and performs a double ordered rearrangement (mini_batch, ds_index). After flattening the trajectory, it puts it into a shared object storage.

[0126] Here, DP slices are assigned to a single GPU sample group / trajectory group set.

[0127] A flowchart based on the above specific example is as follows: Figure 2dAs shown, the process includes receiving an inference trajectory (trajectory201d) (containing ds_index) to be used to generate training samples. First, `_prepare_attempt_template()` (202d) is executed. Partially, the `loss_mask` of the old response segment is set to 0, and the lengths of `loss_masks`, `weight_versions`, and `log_probs` are checked for consistency. If inconsistent, an error is reported. Then, the `GENERATING` state is set and written to `TrajectoryStore` (203d). Next, it checks if the data source has `agent.name` configured (204d), calling `get_rollout_model()`. If not configured (single round), `_execute_single_turn()` (205d) is called, without building an agent, directly requesting the Router. When the context is exhausted, it is handled with a partial response. If configured (multiple rounds), the agent is instantiated and dependencies are injected (206d). `router_url = root address / {trajectory_id} / {attempt_id}`}completion_params·max_response_len·temperature, then call await agent.run_trajectory()207d, which contains the standard chat call flow, that is, the session request processing flow included in the inference trajectory, parse the output and execute the tool in the sandbox, concatenate the next round of messages, loop until the final answer is given, the step limit is reached or the budget is exhausted, and finally execute the agent.`clear()` (idempotent, does not delete sandbox); then check if the attempt was successful (208d). If it failed, set `FAILED` (209d), call `POST / abort_session` to abort the session's requests on the bound worker, increment `attempt_id` by 1, start a new session, create a new agent instance, and retry. After 210d of retries, `emit is_valid=false` and discard the sample. If successful, execute `_post_process_reward()` (211d), call the `reward_fn` bound to the source to score, construct `token_rewards`, write `final_reward` at the end, assign values ​​according to the Triplet interval and verify the quantity, set `COMPLETED`, send to `TraQueue` and delete from `Store`, and execute `DELETE / release_session`. Unbind the sticky binding, and store the final trajectory in TraQueue (grouped by source) 212d, carrying tokens, loss_masks, log_probs, weight_versions, token_rewards, and ds_index.

[0128] The process terminates at the step boundary, in the following order: Cutting upstream, executing POST / abort_all_workers, clearing sticky mappings, canceling tasks, canceling all active asyncio tasks, waiting for enqueueing, waiting for the trajectory-managed future to complete, waiting for disk write, executing POST / wait_inflight_requests_complete to avoid dirty writes, marking the final state, and setting both COMPLETED and ABORTED to ABORTED and emitting; there is also the S16 sandbox preservation logic. The sandbox is managed by the orchestrator and is not destroyed after termination, preserving the file / process state and trajectory feedforward for reuse in the next run; when the next training step continues, the old weights are fed forward, the entire loss_mask is set to 0, and the process returns to the entry point (_prepare_attempt_template()).

[0129] In this process, alignment, masking, scoring, retrying, aborting, and the sandbox lifecycle are all handled by the middle layer, while the agent only executes standard chat calls and execution tools.

[0130] A timing diagram based on the above specific example is as follows: Figure 2eAs shown, this sequence diagram indicates the information transfer between the Agent, ParserMiddleware, Router, and Worker instances. First, the Agent initiates a POST request to the ParserMiddleware (using the standard OpenAI message carrying messages, temperature, and max_tokens) 201e. Internally, the ParserMiddleware sequentially parses the session identifier and locks it 202e. Strong mutual exclusion within the same session and incrementing the in-transit request count by 1 are implemented. Then, via get_trajectory() 203e, the Trajectory object for the attempt is read from the TrajectoryStore. Finally, via build_turn_input() 204e, the first round performs full encoding of the chat template (opening a new segment). Subsequent rounds incrementally encode new messages after stripping the system prefix. If the prefix does not match, the message is reconstructed based on the digest and a new segment is opened. The unique tokenizer source is the TokenizerManager. The response budget is then tightened 205e, and R = L_max - (tokens - P0), max_new_tokens = min (m_res, R), then send the native token request body POST / generate207e, i.e., the inference request body, to the proxy router via token in206e. It includes input_ids (plain text, bypassing server-side tokenization), trajectory_id: attempt_id (for fine-tuning abort), return_logprobs: true (forced encoding, not affected by the caller), sampling_params (max_new_tokens, top_p...). The router stickily selects instance 208e (reusing KV cache + minimum load fallback + forwarding) and then hands it over to the inference instance Worker for execution. The Worker returns token out209e, carrying meta_info, output_token, logprobs, weight_version, and returns 210e to ParserMiddleware as is.After receiving the feedback result, ParserMiddleware executes build_assistant_message()211e sequentially, extracting response_ids / logprobs, retrieving weight_version, parsing the think block and tool calls. The text is only used to send back to the agent and does not participate in training data construction. Through update_trajectory()212e, it appends the Triplet index range, writes tokens / loss_masks / rollout_log_probs / rollout_weight_versions, writes back to TrajectoryStore, executes _to_openai_response()213e, reconstructs the standard chat.completion, and returns choices[0].message / finish_reason / usage to the agent.

[0131] The proxy router and the inference instance worker side include exception isolation. When the data is not 2xx, not JSON, or has finish_reason=abort, it is passed to the caller as is, but not written to the TrajectoryStore to avoid dirty data. At the same time, the alignment invariant is maintained: the token sequence fed into the inference engine during sampling is equal to the token sequence replayed during training. The text never participates in the construction of training data. The training side relies on re-segmentation to eliminate the "training-inference segmentation inconsistency" problem from the root upstream. The agent only sends and receives standard OpenAI messages and is unaware of input_ids, logprob, mask, session stickiness, and termination.

[0132] Figure 3a A flowchart 300 illustrating an embodiment of a model training method is shown. This model training method may specifically include the following steps: Step 301a: Based on the task type corresponding to the sample group, divide the multiple sample groups into multiple task units.

[0133] In this embodiment, during the training phase, the executing entity can divide each sample group into multiple task units based on the task type corresponding to the sample group, i.e., the data source, such as different business data sources like law and mathematics. Each task unit aggregates all sample groups under a single task type. Different task units correspond to different data sources.

[0134] Each sample group includes multiple training samples corresponding to multiple inference trajectories of the same inference task.

[0135] Here, the training samples are as follows: Figure 2a The training samples obtained by the methods described in the corresponding embodiments will not be repeated here.

[0136] Step 302a: For each task unit, the multiple sample groups included in the task unit are assigned to multiple graphics processing units (GPUs).

[0137] In this embodiment, the execution entity can distribute sample groups from each task unit to multiple GPUs (Graphics Processing Units) in batches.

[0138] Here, a GPU is assigned at least one sample group of the task units as described above.

[0139] Step 303a: Model training is performed using multiple GPUs and the assigned sample groups.

[0140] In this embodiment, each GPU in the multiple GPUs can rely on the allocated sample group to complete gradient calculation and model iterative training in parallel, realizing multi-task, multi-GPU distributed parallel training.

[0141] The above embodiments of this disclosure divide multiple sample groups into multiple task units based on the task type corresponding to the sample groups; for each task unit, the multiple sample groups included in the task unit are allocated to multiple graphics processing units (GPUs) for model training. Model training is performed using multiple GPUs and the allocated sample groups, which avoids distributing sample groups from the same data source to a single GPU. This helps to evenly distribute sample groups from the same source across different GPUs and improves the stability of distributed training.

[0142] In some alternative approaches, for each task unit, the multiple sample groups included in the task unit are distributed to multiple graphics processing units (GPUs), including: for each task unit, the multiple sample groups included in the task unit are evenly distributed to multiple graphics processing units (GPUs).

[0143] In this implementation, for each task unit, the executing entity can evenly distribute the multiple sample groups included in the task unit to multiple GPUs used for model training, that is, each GPU is assigned the same number of sample groups.

[0144] Under the condition of ensuring average distribution, the sum of the number of tokens of all training samples in the sample group can be counted, and a preset algorithm, such as a greedy algorithm or a heuristic algorithm, can be used to make the total number of tokens of each GPU as close to balanced as possible, that is, to achieve the load balancing of the total word length borne by each GPU.

[0145] This implementation achieves a balanced distribution of the same sample group across different GPUs, further improving the stability of distributed training.

[0146] In some optional methods, the sample groups allocated to each task unit of the GPU are divided equally according to the total number of training subsets and distributed to each training subset; each training subset is arranged according to the training execution order; within the same training subset, each sample group is arranged sequentially according to the preset task type arrangement rules, and sample groups of the same task type are arranged consecutively.

[0147] In this implementation, the sample groups (sets) allocated to each task unit of the GPU can be divided equally according to the total number of training subsets, i.e., mini_batch, and then allocated to each training subset.

[0148] Here, during the equal distribution process, a pre-set algorithm, such as a greedy algorithm or a heuristic algorithm, can be used to make the total number of tokens in each training subset as close to balanced as possible.

[0149] Each training subset contains evenly distributed sample groups corresponding to various task units. Training subsets on the same GPU can be arranged according to the training execution order.

[0150] Within the same training subset, each sample group can be arranged according to the preset task type arrangement rules, such as mathematics and law, in sequence. Sample groups of the same task type are concentrated and arranged continuously, while sample groups of different task types have clear boundaries and are not mixed with each other.

[0151] For example, first, traverse the training subsets in ascending order (training samples can be read in ascending order of order) and then, under the same training subset number, concatenate the corresponding sample groups in ascending order of data source number (which can be set according to actual needs) to generate a globally fixed and ordered sample list.

[0152] In addition, while ensuring a constant ratio of "per Rank per Mini-Batch", it can be further tightened to "per Micro-Batch" (by performing another layering before dynamic packing), at the cost of reduced packing flexibility.

[0153] It should be noted that, to ensure "constant allocation ratio + balanced length", the allocation process of each task unit can be modeled as an integer programming / minimum cost flow problem to obtain the optimal solution (the cost is the solution time, suitable for small batches). In practical applications, a preset number of allocation ratios can be allowed per Mini-Batch, such as ±1 groups, which can be achieved using "maximum flow / greedy algorithm + tolerance correction" to obtain better length balance (serving as a degradation path when the divisibility constraint is not satisfied). The target quantity for length balance can also be replaced by the number of tokens, the estimated attention computation cost, historical measured time, etc., to more closely approximate the actual computational cost.

[0154] In a specific example, such as Figure 3bAs shown, the example sets up three data sources: Data Source 0: Mathematics, Data Source 1: Retrieval, and Data Source 2: SWE. The ratio of the number of groups in the three data sources is Mathematics:Retrieval:SWE = 8:4:4 = 2:1:1, identified by black, slash, and white respectively. dp_size D = 2, num_mini_batch = 2, and the width of the color block is proportional to the length L_g of the group's tokens. First, a global batch of 16 trajectory groups 301b is constructed, which are trajectory sample groups. Each group consists of N trajectories from the same prompt and are homogeneous within the group. Each group carries a data source label ds_index. Homogeneity within the group is verified during grouping, which is a sufficient condition for a constant ratio; if not satisfied, an error is directly reported. Then, the data sources are layered 302b, which determines multiple task units. Each layer corresponds to one task unit. ds_groups [ds_index] = [group,…] is constructed, and traversed in ascending order by label to ensure determinism. Layer 0 is Mathematics with a total of 8... The tokens are divided into groups according to L_g, with equal number of groups and length for balanced multi-path division. K=2, with 4 groups per path. Layer 1 is for retrieval, with 4 groups in total, K=2, with 2 groups per path. Layer 2 is for SWE, with 4 groups in total, K=2, with 2 groups per path. Long groups and short groups are paired to balance the total token supply.

[0155] Next, the first layer sharding 303b is executed. After equal number of groups and length balancing within the layer, the data is distributed into D DP Ranks (Data Parallelism Ranks). The number of groups in each Rank for each source is equal to the total number of groups in that source / D, regardless of the length distribution. In the example, DP Rank 0 and DP Rank 1 are each assigned 4 math groups + 2 retrieval groups + 2 SWE groups, with the group ratio maintained at 4:2:2 = 2:1:1. The total number of tokens is ΣL / 2 for both. The key is the hard constraint of equal_size and the constant ratio, as well as the balance within the layer and the balance of length. Long SWE group s0 and short SWE group s3 are paired into the same Rank.

[0156] DP Rank is used to indicate the computing power number corresponding to a single GPU.

[0157] Then, the second layer of sharding 304b is performed. Within the Rank, it is further divided into num_mini_batch parts according to the source layer and equal group length. The mini_batch is then sorted in ascending order and ds_index in ascending order to obtain the final ordered list of DP Rank 0 and the final ordered list of DP Rank 1. Each of DP Rank 0 and DP Rank 1 generates two Mini-Batches, namely Mini-Batch0 and Mini-Batch1, and the ratio of each Mini-Batch is maintained at 2:1:1.

[0158] Among them, the continuous arrangement of homologous samples facilitates the normalization of the advantages of source segmentation. The teacher model processes the statistical indicators and source routes according to the source. The training side can achieve a stable ratio by consuming in a fixed step size order, without the need to make any ratio correction on the training side.

[0159] Furthermore, a flat distribution can be implemented, with each DP segment aggregated into a fielded batch structure of tokens, loss_masks, rollout_log_probs, rollout_weight_versions, token_rewards, segments, triplets, metadata, and ds_indices. This structure is placed in Ray shared object storage to achieve zero-copy distribution, only distributing object references. When training Actors, tensor views are reconstructed according to segment / triplet index ranges, and ds_indices are integrated into the loss calculation and teacher routing process with each batch.

[0160] Existing technologies use global random shuffling, concat+shuffle equal-weight sampling of the dataset. This results in several issues: Rank 0 Mini-Batch 0 has zero retrievals and zero SWEs (Search Results for Weaknesses), while Rank 1 Mini-Batch 0 has all SWEs, leading to an explosion in token quantity. Consequences include: source advantage normalization statistics built on incomplete samples; parameter updates dominated by a single task causing inter-task interference; catastrophic forgetting and training oscillations; and the inability of multi-teacher online distillation to complete source routing within a single Mini-Batch, with different ranks waiting for each other, creating a dynamic bottleneck. In contrast, existing "length-balanced splitting" methods are insensitive to data source labels, and only one can be chosen between length balance and constant proportion. This example first layers the data, then balances it within each layer, making proportion a hard constraint (equal number of sets) and length balance an intra-layer objective, allowing both objectives to be achieved simultaneously.

[0161] This implementation allows the training process to read and segment training subsets sequentially with a fixed step size. Each training subset is composed of sample groups from the same batch corresponding to each data source, strictly ensuring that the sample proportion of each data source in each batch remains consistent with the overall global proportion. Simultaneously, sample groups from the same data source are arranged continuously within a single training subset, with clear and unmixed data source boundaries. This facilitates the implementation of data source-based advantage normalization, indicator statistics, and teacher routing adjustments, effectively improving the stability and controllability of multi-data source mixed training.

[0162] In some alternative approaches, model training using multiple GPUs and allocated sample sets includes: for each GPU, flattening the sample set allocated to that GPU to generate a one-dimensional array; storing the one-dimensional array in shared memory and passing a pointer to the shared memory to the training process so that the training process can read the one-dimensional array based on the pointer and train the large language model.

[0163] In this implementation, for each GPU, the execution entity can flatten and expand the sample group allocated to that GPU to generate a one-dimensional array; store the one-dimensional array in shared memory, and pass a pointer to the shared memory, i.e., an object reference, to the training process.

[0164] The training process, or training execution unit, after obtaining the object reference, can restore the flattened one-dimensional array into the corresponding tensor view based on the index range of the segment and triplet, for use in model training.

[0165] This implementation transmits the data source index field synchronously to the training side along with the training batch, enabling subsequent processes such as loss calculation, training metric statistics, and teacher routing control to be executed precisely according to the data source dimension. At the same time, relying on the zero-copy shared memory transmission mode, it significantly improves the data loading and transmission efficiency of large-scale multi-data source distributed training.

[0166] In some alternative approaches, model training is performed using multiple GPUs and allocated sample sets, including: during training, each training sample is automatically routed to a teacher model bound to the task type of that training sample for online distillation.

[0167] In this implementation, the executing entity can construct a one-to-one mapping relationship between the data source index and the teacher model index, forming a fixed data source-teacher mapping table.

[0168] Different teacher models are associated with different task types.

[0169] During the forward computation of teacher distribution in the model, the training side can group the samples according to the data source index carried by the training subset. Using teacher routing, for samples grouped by different data sources, the corresponding exclusive teacher model weights are switched one by one through tensor backup and weight swapping. After the group completes independent forward computation, the output results of each teacher model are merged according to the original arrangement number of the samples to complete the multi-teacher distillation computation process in a single training step.

[0170] Among them, TeacherWorker is used to route to the corresponding teacher model according to the data source index during online distillation and switch teacher weights within the same batch.

[0171] Here, teacher routing can be replaced by: having each teacher reside in a set of independent Workers, and routing samples to the corresponding Workers according to the source (eliminating the need for weight swapping in and out, at the cost of increased GPU memory usage); or making the teacher model an independent inference service, requesting different service addresses according to the source.

[0172] This implementation method enables online distillation by matching samples from different data sources with corresponding dedicated teacher models within the same training iteration. Specifically, it achieves precise adaptation training by matching mathematical data source samples with mathematical teacher models and code data source samples with code teacher models. This approach can be effectively implemented by relying on the core characteristics of the aforementioned training subset arrangement mechanism. It leverages the regularity of complete data source samples and continuous arrangement of homogeneous samples within each training subset to ensure accurate grouping of multiple data sources, orderly switching of teacher weights, and efficient and stable distillation calculations. This enables deep adaptation of multi-teacher heterogeneous distillation and multi-task hybrid training.

[0173] In a specific example, the set of sample groups (traj_groups) collected in a single training step (each sample group consists of multiple training samples for the same inference task, and all training samples within the group have the same data source label ds_index, which has been verified at the sampling exit queue) is taken as input; the data parallelism dp_size (i.e., the total number of GPUs) and the number of Mini-Batches num_mini_batch (i.e., the total number of training subsets) are taken as input, and the output is dp_size slices, each of which is an ordered list of sample groups, which is suitable for distributed large language model training scenarios with multiple data sources and multiple teacher models.

[0174] In this example, the data sources are configured independently, with an array of data source configuration items named `data_sources`. Each data source can be independently configured with dataset paths and field mappings, binding agents and their specific parameters, binding reward functions and their specific parameters, online distillation specific teacher model names, and the number of inference trajectories corresponding to a single inference task (or prompt). Number of single-step inference tasks The system includes response lexical budget (e.g., total lexical budget) and various sampling parameters. Fields not configured separately automatically inherit the common default configuration, and the system adaptively determines whether to enable multi-round training mode based on whether the agent.name field is configured in the data source.

[0175] It can perform self-consistency constraint verification of training subset size.

[0176] Specifically, a mandatory training subset size check is performed before training begins to ensure a strictly constant ratio of multiple data sources. This is achieved by defining the total number of globally divisible training subsets using the following formula, and by mandating that all data sources satisfy the following constraints:

[0177] Where S represents the number of data sources; This represents the number of prompts sampled at each training step for the i-th data source, i.e., the number of inference tasks; This represents the number of trajectories sampled for each prompt from the i-th data source, i.e., the number of training samples, or the size of the sample group or trajectory group; This represents the number of trajectories or training samples in the training subset (the smallest unit of gradient accumulation); D represents the data parallelism dp_size, i.e., the total number of GPUs. Physically, the numerator is the total number of trajectories or training samples in a global batch, divided by the Mini-Batch size to obtain the gradient accumulation steps; it requires that the prompt number of each source is divisible by "Rank number × accumulation steps", meaning that the sample set of each source falls into D without remainder. "num_mini_batch" cells.

[0178] This verification condition is a sufficient condition for the constant allocation ratio of data sources for each computing power node and each training subset. It can ensure that the number of data source groups is evenly distributed to all shard units (i.e., GPUs) without any remainder. If the verification fails, an error will be directly reported, thus eliminating the allocation deviation caused by approximate allocation.

[0179] Furthermore, data can be hierarchically divided into buckets (one-time hierarchical division) to define multiple task units.

[0180] Based on the data source index ds_index, the input traj_groups are bucketed, and the data structure ds_groups[ds_index] = [group, ...] is constructed. All sample groups from the same data source are grouped into independent task units, achieving preliminary isolation and stratification of multi-source data.

[0181] It can ensure load balancing of sharded units within the same data source.

[0182] Specifically, the data source levels are traversed in ascending order of ds_index, i.e., each task unit is traversed to ensure the determinism of sample partitioning. For each data source level, the total word length L_g of each group is calculated, which is the sum of the number of words in all training samples within the group (i.e., the data in the flat array corresponding to the inference trajectory is determined). An equal-size length balanced partitioning strategy with equal_size=True is adopted. Relying on a greedy algorithm, all sample groups in the current data source are evenly divided into dp_size computing power partitions to ensure that the number of groups in each partition is completely equal and the total word length is as balanced as possible. Finally, the r-th group of data is assigned to the r-th Rank partition (corresponding to a single GPU).

[0183] This approach differs from the traditional global distribution scheme that is unaware of data sources. It relies on a fixed group allocation rule to ensure that the number of groups in each data source at each Rank is always equal to the ratio of the total number of groups in that source to dp_size. It is not affected by the distribution of sample sequence lengths, thus achieving a dual guarantee of constant allocation and load balancing.

[0184] Furthermore, it is possible to perform secondary hierarchical arrangement and double ordered rearrangement within the Rank.

[0185] For each Rank, perform fine-grained secondary normalization on the sample groups. First, perform secondary stratification based on ds_index. Then, for each layer of homogeneous sample groups, use an equal-size + length balancing strategy to divide them into num_mini_batch subsets (i.e., the set of sample groups divided into each training subset) to generate structured data partitions_by_ds[ds_index][mb_idx].

[0186] Subsequently, all subsets are concatenated following a dual rule: ascending order of the outer `mb_idx` (training subset index) and ascending order of the inner `ds_index`, forming the final ordered sample group list. During training, the training subsets of this ordered sample group list are read at fixed steps, ensuring that the `mb_idx`-th training subset accurately contains the subsets with the same index corresponding to each data source. This guarantees that the proportion of each data source within a batch is consistent with the global distribution. Simultaneously, samples from the same source are continuously arranged within a single training subset, providing structural support for subsequent source-based normalization, indicator statistics, and teacher routing.

[0187] Here, the consistency of the proportions can be characterized by the following formula:

[0188] in, Indicates the first The set of sample groups from the source; Indicates the first share; Indicates the number of divisions (in one stratification). In the second layer ); Indicates the first Total length of tokens in each sample group; constraints The "equal number of groups" constraint is the source of the constant allocation ratio; the objective function is to minimize the range of the total number of tokens in each group, which is used to eliminate the shortcomings of dynamic programming; the physical meaning is to make the computational cost (number of tokens) of each group as close as possible under the hard constraint that the number of groups in each group must be equal.

[0189] Furthermore, sample flattening and zero-copy distribution via shared memory can be implemented.

[0190] Specifically, the sample groups within each DP slice can be expanded into a one-dimensional array and uniformly aggregated into a standardized batch data structure containing tokens, loss masks, rollout log probs, rollout weight versions, token rewards, segments, triplets, metadata, and ds_indices. The aggregated data is stored in Ray object storage, achieving zero-copy transfer via shared memory. Only object references are distributed to the corresponding Rank (corresponding to a single GPU) training Actor, i.e., the training process. The training Actor restores the flattened array to a standard tensor view based on the segment and triplet index ranges.

[0191] This approach can synchronously transmit the data source index array ds_indices to the training side along with the training subset, supporting fine-grained calculation and control by source throughout the entire process, while avoiding the overhead of serializing and copying large amounts of word data, thus improving the throughput efficiency of distributed training data.

[0192] This example allows for dynamic sampling scheduling based on data source quotas.

[0193] Specifically, during the sampling phase, a triplet quota parameter of ds_targets[i], ds_accepted[i], and ds_running[i] is maintained for each data source, corresponding to the target number of samples, the number of accepted samples, and the number of samples in transit for the data source, respectively. The scheduler triggers new sample generation for data source i only when the data source has not reached its quota limit, i.e., when ds_accepted[i] + ds_running[i] < ds_targets[i]. For samples rejected by the reward filter, only the corresponding data source is sampled again. Excess samples are discarded or cached according to the data source dimension. At the same time, evaluation samples are independently labeled and routed separately, without occupying the training sampling quota, effectively solving the performance degradation problem of fast tasks crowding out slow tasks in multi-source mixed training, and ensuring the fairness of multi-source sampling training.

[0194] Furthermore, dequeueing can be adaptively grouped and aggregated based on data source.

[0195] Specifically, the sampling exit queue aggregates training samples based on the corresponding inference task identifier prompt_id, and determines whether the training samples are grouped by combining the sample group size N_i configured independently for each data source. After grouping, the consistency of ds_index of all training samples within the group is forcibly verified; if the data source identifiers are inconsistent, an error is directly reported. This mechanism supports different group sizes configured for different data sources, while ensuring the homogeneity of all trajectory groups from the source, thus solidifying the operational foundation of the pre-layered and segmented approach.

[0196] This example allows for the return of source samples that can be continued across training steps.

[0197] After the training step abort boundary is triggered, the training samples that have been fully grouped can be returned to the corresponding data source buffer according to ds_index. The buffer prioritizes reading cached samples and supports pluggable replay strategies such as FIFO. Incomplete sample groups (also known as trajectory groups) that have not been fully assembled are retained and reused as a whole. When the system saves and restores checkpoints, it simultaneously persists the list of unused prompt words and reconstructs the trajectory group structure to ensure that the constant multi-data source allocation mechanism continues to take effect in scenarios where training steps are continued.

[0198] Furthermore, online distillation can be performed using a multi-teacher model that routes data sources.

[0199] Specifically, each data source can be independently configured with a dedicated `teacher_name`. During the initialization phase of the Teacher Worker (responsible for executing the training process of the teacher model's forward inference), a one-to-one mapping relationship between the data source index and the teacher model index is established: `ds_to_teacher_idx[ds_index] = teacher_idx`. When the model performs forward calculation of the teacher distribution, the training samples are grouped by source according to the `ds_indices` carried in the batch (the data source index array passed to the training side along with the training subset). The corresponding dedicated teacher model weights are switched one by one through tensor backup and weight swapping. After the group completes independent forward calculation, the multi-teacher output results are merged according to the original sample sequence number. This mechanism can complete accurate online distillation of multiple data sources and multiple teachers within a single training iteration, realizing the adaptation of different data sources to the corresponding dedicated teacher model training. Its feasibility depends on the well-organized structure of the training subsets guaranteed above, that is, each source in each training subset has training samples and they are arranged continuously.

[0200] In another specific example, taking the training of a general agent using a combination of three data sources as an example, the complete application process is as follows: Configuration declaration (snippet illustration) data_sources: - dataset: { prompt_data_path: <mathematical data>, input_key: question, label_key: answer} agent: { name: gsm8k} # Mathematical multi-turn agent reward: { name: gsm8k} teacher_name: math_expert # This source uses math teacher (online distillation) num_trajectories_per_prompt: 8 # = 8 num_prompts_per_step: 32 # = 32 max_response_len_per_trajectory: 20480 - dataset: { prompt_data_path: <retrieval question-answer data>, input_key: prompt, label_key: answer} agent: { name: bcp, retrieval_service_url: <retrieval service>, search_topk:3, max_turns: 15} reward: { name: bcp} num_trajectories_per_prompt: 8 # = 8 num_prompts_per_step: 16 # = 16 - dataset: { prompt_data_path: <SWE data>, input_key: prompt, label_key: label, metadata_key: metadata} agent: { name: mini-swe, step_limit: 50, request_timeout: 1200} reward: { name: r2e_gym} # Run unit tests to determine if the repair is successful num_trajectories_per_prompt: 8 # = 8 num_prompts_per_step: 16 # = 16 agentflow: sandbox: { type: k8s} # SWE sources require a sandbox Verify according to the formula: "num_mini_batch"=(32+16+16)×8 / 64=8; when D=2, D "num_mini_batch"=16, while 32 mod 16=0 and 16 mod 16=0, so the verification passes.

[0201] The sampling phase is executed in parallel by source. The mathematical source involves multi-round dialogues between agents at the address injected by the framework, involving "problem-solving → self-checking → correction," without requiring a sandbox. The retrieval source generates a retrieval query each round, calls an external retrieval service to retrieve the top-k documents, and uses these documents as messages for the next round (the framework automatically sets the mask of these document tokens to 0). The SWE source allocates a K8s Pod sandbox for each trajectory (the target repository is already checked out within the image, and the working directory is the repository root by default, which can be overwritten by sample metadata). Within the sandbox, agents execute shell commands to read files, modify code, and run tests, up to 50 steps. The reward function runs unit tests within the sandbox to determine the success of the repair. All three sources share the same inference service cluster. The agent performs sticky routing by session, ensuring that all 50 requests for the same SWE trajectory fall on the same instance, reusing the KV cache. Source-based quota scheduling ensures that slow tasks like SWE are not squeezed out of batches by fast tasks like mathematical tasks.

[0202] During the training phase, all trajectory groups generated at each step are distributed after being divided into two layers: in each Mini-Batch of each Rank, the ratio of the number of groups of Math:Retrieval:SWE is always 32:16:16 = 2:1:1, and the total number of tokens in each slice is approximately equal; during online distillation, the Teacher Worker switches the Math source samples to the Math Teacher forward according to ds_indices.

[0203] When the training step boundary is reached, an ordered termination is performed: the unfinished SWE trajectory is marked as terminated but its sandbox and the generated trajectory prefix are retained. When the next step is continued, the entire old weight prefix mask is set to 0 (to avoid using the old policy action for on-policy updates). The code modifications and environment state completed in the sandbox are directly reused.

[0204] See also Figure 4a , Figure 4a This is a flowchart of an application scenario based on the above embodiments.

[0205] like Figure 4aAs shown, a dataset 401a is first prepared, which includes Data Source 0: Mathematics (configuration description: stored in jsonl / parquet file format, with field key names as input_key / label_key), Data Source 1: Retrieval Question Answering (deterministic epoch counting, length filtering), Data Source 2: SWE, namely Software Engineering (metadata_key, repo_path, etc.); then the DataSource component fetches prompts and replicates them to form group 402a, specifically fetching prompts from each source and each of the prompts is replicated for a number of copies to form a group, while generating a trajectory identifier carrying a source tag, and the relevant identifier is written into ds_index; then the process proceeds to the RolloutSampler component to perform dynamic sampling scheduling by source quota 403a, maintaining the triplet quota ds_targets, ds_accepted, ds_running for each source, setting the trigger condition as accepted+running < targets, and routing separately only for the candidate generation samples, so as to prevent tasks from squeezing out candidate tasks; after that, the process enters the sampling orchestration layer AgentFlow to perform sampling orchestration processing 404a, which sequentially includes instantiating agents by source (supporting modifications for gsm8k / bcp / mini-swe, standard chat calls, business code, etc.), a sandbox pool (based on K8s Pod / Docker, with the lifecycle managed by the framework, and the state retained after termination for troubleshooting), a session token-level proxy Router+ParserMiddleware (session = trajectory_id+dataset_id, sticky routing reuses KV cache, fine-tuning termination, requests include messages+input_ids, distinguishing full-word / gradient-free groups, responses return output_tokens+response_logprobs, loss_mask is uniformly labeled, model generation = 1, tool output / reconstructed prompt / fallback = 0), TrajectoryStore (as the sole ground truth data during the sampling period, storing tokens / loss_masks / rollout_log_probs / rollout_weight_versions, supporting Segment / Triplet index access, zero token copy), scoring by source reward function (gsm8k / bcp / r2_gym, constructing token_rewards), TrajQueue (grouped by source, by reference size The process involves grouping data, verifying consistency of ds_index within each group, and filtering rewards (e.g., discarding groups with identical rewards but no gradients, replenishing the filtered quotas according to the source, and feeding back to the RolloutSampler component without changing the global allocation). This component outputs traj group and token input_ids, which are then sent to the inference cluster SGLang Workers (worker1~worker N) corresponding to the inference engine for inference engine processing (405a). The inference cluster outputs token ids and logprobs and sends them back. The data then enters the DataProcessor component in the two-layer sharding (406a), which includes the first processing step: grouping by data source, equal multiples within groups, length balancing to D ranks, and further sharding and rearranging within each rank. The second processing step involves flattening and shared memory (aggregating into a field-based batch structure, zero-copy ray.put, including ds_indices). The processed data is then sent to the Trainer component for training (407a), including DP Rank 0~N (search reference region reconstruction, tensor view, DP Rank 1 dynamic batch size binning, DP...). The training process includes: MAX gap, routing by source teacher (ds_to_teacher_idx, grouping by ds_indices, teacher copy weight redirection), loss and update (normalization by source advantage, reuse of rollout_log_probs, calculation of importance ratio KL), and output metrics by source (batch statistics for train / eval). After training, weight synchronization is performed (weight_version +1), and the updated weights are sent back to the inference cluster to enter the next sampling cycle. The entire link has a complete transmission path for ds_index: trajectory identifiers are written to ds_id during generation → sampling scheduling maintains quotas by source → agent / reward function / token budget / sampling parameters are selected by source → TrajQueue verifies same-source status within the group by source group size → two-layer sharding uses ds_index as the sharding key → sharding is performed with ds_indices. Fields enter the training side → the training side normalizes by source advantage, performs statistical indices by source, and routes the teacher model by source → after step boundary termination, incomplete groups are returned by source. Only one source label is maintained throughout the entire link, and there is no step of "which task the training side carries the sample to". This ensures that the ratio of each Rank to each Mini-Batch is stable, reproducible and verifiable. Partial trajectories are returned by source: the complete batch ds_index is returned to each source buffer (FIFO can be inserted), incomplete groups are retained as a whole, and the checkpoint persists the unused prompt list.

[0206] See also Figure 4b , Figure 4b This is another flowchart of an application scenario based on the above embodiments.

[0207] like Figure 4b As shown, the inference tasks of the three types of data sources (scenario A corresponds to data source 0: mathematical reasoning 401b; scenario B corresponds to data source 1: retrieval enhancement question answering 402b; scenario C corresponds to data source 2: software engineering (SWE) 403b) maintain the original form of the inference task (dialogue / external service call / in-container code repair), without modifying each other, but sharing the same sampling orchestration AgentFlow 404b, the same inference cluster 405b and the same training update 406b.

[0208] In this scenario, data source 0 is mathematical reasoning. It can be configured with ds_index=0, agent=gsm8k-reward (a dedicated agent for mathematical reasoning), and group size N0=8. It includes a mathematical agent (pure multi-turn dialogue, no tools, no external tokens, short sequences, approximately 2K tokens, single-turn mode is also possible) and self-reflection / self-checking (the answer from the previous round is used as the input for the next round). Rewards are determined by exact answer matching / numerical equivalence. `final_reward` is written to `token_rewards`. There are no round-by-round rewards. Typical characteristics include short trajectories, short processing time, high success rate, and easy quota "preemption." The framework configures `max_response_len`, `per_trajectory_actor_num`, `temperature / top_p` independent quotas `ds_targets`, and independent counting for this source, preventing it from "running fast" and occupying more global batches.

[0209] Scenario B corresponds to data source 1: retrieval-enhanced question answering. The configuration includes ds_index=1, agent=retrieval-agent (a dedicated agent for retrieval-question answering), and group size N1=8. It includes a retrieval agent (ReAct: think → retrieve → read document → rethink, medium-length multi-round retrieval) and external retrieval services, such as API (Application Programming Interface) / vector libraries, web scraping, out-of-process operations, and network latency. Rewards are based on answer correctness and verifiable citations. The retrieved document token has a loss_mask=0 (content not used in the training environment). Typical characteristics include a medium number of rounds, variable context length, requiring fine-tuning and retrying per session. The framework is configured separately for this source, with a maximum retry limit on failures, and attempt_id+1 for a new session (old requests are aborted, session is reset). Context truncation can trigger compressed summarization and a new segment, ensuring token-level alignment is not compromised.

[0210] Scenario C corresponds to data source 2: Software Engineering (SWE), configured with ds_index=2, agent=mini-swe-agent (software engineering-specific agent), reward=r2e_gym, group size N2=4, including SWE agents (reading files / modifying code, running tests / submitting patches, dozens of rounds, approximately 30K tokens), a K8s Pod sandbox (bash, file system, lifecycle managed by the framework, Docker is also possible) and a code repository (repo_path, unit tests, metadata passed in). The reward is the unit test pass rate / patch applicability (tested in the sandbox, environment token:loss_mask=0 in bash echo, error stack, test output, etc.). Typical characteristics are extremely long trajectories, long processing time, and frequent crossing of training step boundaries, making it most prone to "preemptive" task deadlock. The framework configures a separate ultra-large token budget, independent step limit, sandbox type and image, and independent quota ds_targets for this source. [2] The quota triplet guarantees that even if the entire process cannot be completed in one step, it will only be used to replenish the source and will not be borrowed for over-task purposes.

[0211] Scenarios A, B, and C all access the same sampling orchestration layer, AgentFlow, via standard chat calls (shared by all three task types, with no task-specific branch code). This layer includes a session token / token-level agent Router + ParserMiddleware (session = trajectory_id + dataset_id, token is rewritten bidirectionally in input_ids, sticky routing reuses KV cache, fine-tuning and abort, and all three task types follow the same agent link), TrajectoryStore (unified trajectory representation, storing tokens / loss_masks / rollout_log_probs / weight_versions / token_rewards, supporting Segment / Triplet index access, and heterogeneous tasks are grouped into the same training samples), and source-based quota scheduling + source-based grouping (maintaining ds_targets / ds_accepted / ds_running, and grouping by source). Grouping and verifying for same source within the group, reward filtering and replenishment by source to prevent task deadlock), sandbox pool and registry extension (@register_agent / @register_reward / @register_sandbox automatic discovery, new tasks only require two classes: agent + reward function, sandboxes are created on demand, selected by source, and lifecycle is bound to trajectory state machine, new scenarios do not change the framework code); the sampling orchestration layer outputs input_ids / token_ids / logprobs are sent to the same inference cluster (SGLang Workers, worker1 to worker N, sharing KV cache and batch processing).

[0212] The inference cluster also incorporates training step boundary logic (abrupt but retaining the sandbox and context, trajectory roll-forward, resuming in the next step, crucial for long trajectory SWE scenarios, ensuring the sandbox is not destroyed (file / process state is preserved), setting the old weight feedforward loss_mask to 0 for the entire segment, and retaining groups that are not aligned to the next step); after the inference results are processed by trajectory grouping by source + two-layer sharding, they enter the same training update phase, achieving a constant ratio of M0:M1:M2 for each DP Rank and each Mini-Batch. This phase includes source advantage normalization (statistics from each source do not contaminate each other), source teacher routing (optional ds_to_teacher_idx, online distillation for multiple teachers), one reverse + one parameter update (three types of capabilities are optimized simultaneously), and source output indicators (observable by source in train / eval); after training is completed, the weights are synchronized, and after the update, the next sampling cycle begins.

[0213] The scenarios are scalable: The three scenarios mentioned above are just examples. When new scenarios such as "tool calling assistant, GUI operation, data analysis, and multimodal question answering" are introduced, only an agent class and a reward function need to be added and a data source unit needs to be added to the configuration. The agent rewriting, trajectory representation, quota scheduling, two-layer sharding, teacher routing and indicator statistics on the framework side do not need to be changed. Moreover, the new scenarios can be mixed with existing scenarios in the same batch for training with a constant ratio.

[0214] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a training sample construction apparatus, which is similar to... Figure 2a The method embodiments shown correspond to those described.

[0215] like Figure 5 As shown, the training sample construction device 500 of this embodiment includes: a first receiving module 501, a sequence sending module 502, a second receiving module 503, and a sample generation module 504.

[0216] The first receiving module 501 can be configured to receive a session request sent by the intelligent agent that belongs to the same inference trajectory, and generate an input word sequence based on the text carried in the session request.

[0217] The sequence sending module 502 can be configured to send the input word sequence to the inference engine.

[0218] The second receiving module 503 can be configured to receive the output word sequence and the log probability of each word returned by the inference engine based on the large language model.

[0219] The sample generation module 504 can be configured to generate training samples corresponding to the inference trajectory based on the input word sequence, the output word sequence, and the log probability of each word.

[0220] In some optional embodiments of this example, the first receiving module may further include a first submodule, a second submodule, and a third submodule. The first submodule may be configured to, in response to determining that the session request is the first session request of the to which the inference trajectory belongs, perform word segmentation on the text carried by the session request to obtain an input word sequence. The second submodule may be configured to, in response to determining that the session request is not the first session request of the to which the inference trajectory belongs, determine the new content in the text carried by the session request relative to the text corresponding to the prefix word sequence, perform word segmentation on the new content to obtain a new word sequence. The third submodule may be configured to concatenate the new word sequence after the prefix word sequence to obtain the input word sequence.

[0221] In some optional embodiments of this example, the training samples include a first flat array, a second flat array, a third flat array, and a fourth flat array.

[0222] In some alternative embodiments of this example, the training samples may also include a fifth flat array.

[0223] In some optional embodiments of this example, the sequence sending module includes a determining submodule and a sending submodule. The determining submodule can be configured to determine the number of available lexical units based on the total lexical budget corresponding to the inference trajectory and the number of lexical units consumed by the historical session requests corresponding to the inference trajectory. The sending submodule can be configured to send the input lexical sequence and the number of available lexical units to the inference engine.

[0224] In some optional embodiments of this example, the sequence sending module includes an acknowledgment submodule and a push submodule. The acknowledgment submodule can be configured to determine the inference instance of the inference engine corresponding to the historical session requests of the inference trajectory in response to determining that the session request is not the first session request of the inference trajectory. The push submodule can be configured to send the input lexical sequence to the inference instance.

[0225] In some optional embodiments of this embodiment, the device further includes a text generation module and a text return module. The text generation module can be configured to generate response text based on the output lexical sequence and the logarithmic probability of each lexical; the text return module can be configured to send the response text to the intelligent agent.

[0226] In some optional embodiments of this example, the device further includes a judgment module and a reuse module. The judgment module can be configured to determine whether a sandbox bound to the inference trajectory corresponding to the response text exists in response to detecting a tool call instruction sent by the agent based on the response text. The reuse module can be configured to execute the tool call instruction using the sandbox bound to the inference trajectory in response to its existence.

[0227] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a model training device, which is similar to... Figure 3a The method embodiments shown correspond to those described.

[0228] like Figure 6 As shown, the model training device 600 in this embodiment includes: a first allocation module 601, a second allocation module 602, and a model training module 603.

[0229] The first allocation module 601 can be configured to divide multiple sample groups into multiple task units based on the task type corresponding to the sample groups. Each sample group includes multiple training samples corresponding to multiple inference trajectories of the same inference task. The training samples are as follows: Figure 2a The training samples obtained by the method described in the corresponding embodiment.

[0230] The second allocation module 602 can be configured to allocate multiple sample groups included in each task unit to multiple graphics processing units (GPUs) used for model training.

[0231] The model training module 603 can be configured to train the model using multiple GPUs and allocated sample sets.

[0232] In some alternative embodiments of this example, the second allocation module may be further configured to, for each task unit, evenly distribute the multiple sample groups included in the task unit to multiple GPUs used for model training.

[0233] In some optional ways of this embodiment, the sample groups of each task unit assigned to the GPU are divided equally according to the total number of training subsets and distributed to each training subset respectively; each training subset is arranged according to the training execution order; within the same training subset, each sample group is arranged sequentially according to the preset task type arrangement rules, and sample groups of the same task type are arranged consecutively.

[0234] In some alternative embodiments of this example, the model training module may be configured to flatten and expand the sample group allocated to each GPU to generate a one-dimensional array; store the one-dimensional array in shared memory; and pass a pointer to the shared memory to the training process so that the training process can read the one-dimensional array based on the pointer and train the model.

[0235] In some alternative embodiments of this example, the model training module may be configured such that during the training process, each training sample is automatically routed to a teacher model bound to the task type of that training sample for online distillation.

[0236] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0237] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0238] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0239] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0240] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0241] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as methods for constructing training samples. For example, in some embodiments, the methods for constructing training samples may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods for constructing training samples described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform methods for constructing training samples by any other suitable means (e.g., by means of firmware).

[0242] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0243] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0244] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0245] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0246] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0247] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service ecosystem to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services. Servers can also be categorized as distributed system servers or servers incorporating blockchain technology.

[0248] According to the technical solution of this disclosure, accurate training samples are provided for reinforcement learning training without modifying the agent's business code.

[0249] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0250] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for constructing training samples, comprising: Receive a session request sent by the intelligent agent that belongs to the same inference trajectory, and generate an input word sequence based on the text carried by the session request; The input word sequence is sent to the inference engine; Receive the output word sequence and the log probability of each word returned by the inference engine based on the large language model; Based on the input lexical sequence, the output lexical sequence, and the logarithmic probability of each lexical, training samples corresponding to the inference trajectory are generated.

2. The method according to claim 1, wherein, The step of generating an input lexical sequence based on the text carried in the session request includes: In response to determining that the session request is the first session request of the inference trajectory to which it belongs, the text carried by the session request is segmented and converted to obtain the input word sequence; In response to determining that the session request is not the first session request of the inference trajectory to which it belongs, the text carried by the session request is determined to have new content relative to the text corresponding to the prefix word sequence, and the new content is segmented and converted to obtain the new word sequence. The newly added lexical sequence is concatenated to the prefix lexical sequence to obtain the input lexical sequence, wherein the prefix lexical sequence is determined based on the input lexical sequence and output lexical sequence corresponding to the historical session requests of the reasoning trajectory.

3. The method according to claim 2, wherein, The training samples include a first flat array, a second flat array, a third flat array, and a fourth flat array. The first flat array, the second flat array, the third flat array, and the fourth flat array have the same length and share a set of position indices. The same position index points to the array element in each flat array that corresponds to the same word. The first flat array is used to store the new word sequence and the output word sequence of the session request in the order of execution of the session request; The second flat array is used to store the loss mask, wherein the position index corresponding to each word in the output word sequence is marked as participating in the loss calculation, and the position index corresponding to each word in the non-output word sequence is marked as not participating in the loss calculation; The third flat array is used to store the corresponding log probability at the position index of each word in the output word sequence; The fourth flat array is used to store the corresponding model weight version at the position index of the word.

4. The method according to claim 3, wherein, The training samples also include a fifth flat array, which is used to store the evaluation results obtained by evaluating the data in the first flat array, the second flat array, the third flat array, and the fourth flat array using a reward function.

5. The method according to claim 1, wherein, Sending the input lexical sequence to the inference engine includes: The number of available lexes is determined based on the total lexical budget corresponding to the inference trajectory and the number of lexes consumed by the historical session requests corresponding to the inference trajectory. The input lexical sequence and the number of available lexical units are sent to the inference engine.

6. The method according to claim 1, wherein, Sending the input lexical sequence to the inference engine includes: In response to determining that the session request is not the first session request of the inference trajectory, the inference instance of the inference engine corresponding to the historical session requests of the inference trajectory is determined; The input word sequence is sent to the inference instance.

7. The method according to any one of claims 1-6, further comprising: Based on the output word sequence and the log probability of each word, generate the response text; The response text is sent to the agent.

8. The method according to claim 7, further comprising: In response to detecting a tool call command sent by the agent based on the response text, determine whether there is a sandbox bound to the inference trajectory corresponding to the response text; In response to existence, the tool invocation command is executed using the sandbox bound to the inference trajectory.

9. A model training method, comprising: Based on the task type corresponding to the sample group, multiple sample groups are divided into multiple task units, wherein each sample group includes multiple training samples corresponding to multiple inference trajectories of the same inference task; the training samples are training samples obtained by the method according to any one of claims 1-8. For each task unit, the multiple sample groups included in the task unit are assigned to multiple graphics processing units (GPUs), wherein one GPU is assigned at least one sample group of the task unit. Model training is performed using multiple GPUs and allocated sample sets.

10. The method according to claim 9, wherein, For each task unit, allocating the multiple sample groups included in the task unit to multiple graphics processing units (GPUs) includes: For each task unit, the multiple sample groups included in the task unit are evenly distributed to multiple GPUs; the total word length load of each GPU is balanced.

11. The method according to claim 10, wherein, The sample groups allocated to each task unit of the GPU are divided equally according to the total number of training subsets and distributed to each training subset; each training subset is arranged according to the training execution order; within the same training subset, each sample group is arranged sequentially according to the preset task type arrangement rules, and sample groups of the same task type are arranged consecutively.

12. The method according to claim 9, wherein model training using multiple GPUs and allocated sample groups comprises: For each GPU, the sample groups assigned to that GPU are flattened and expanded to generate a one-dimensional array; The one-dimensional array is stored in shared memory, and a pointer to the shared memory is passed to the training process so that the training process can read the one-dimensional array based on the pointer and train the model.

13. The method according to claim 9, wherein training the model using multiple GPUs and allocated sample groups comprises: During training, each training sample is automatically routed to the teacher model associated with the task type of that training sample for online distillation. Different teacher models are associated with different task types.

14. A training sample construction apparatus, comprising: The first receiving module is configured to receive a session request sent by the intelligent agent that belongs to the same inference trajectory, and generate an input word sequence based on the text carried by the session request. The sequence sending module is configured to send the input word sequence to the inference engine; The second receiving module is configured to receive the output word sequence and the log probability of each word returned by the inference engine based on the large language model; The sample generation module is configured to generate training samples corresponding to the inference trajectory based on the input word sequence, the output word sequence, and the log probability of each word.

15. A model training device, comprising: The first allocation module is configured to divide multiple sample groups into multiple task units based on the task type corresponding to the sample groups, wherein each sample group includes multiple training samples corresponding to multiple inference trajectories of the same inference task; the training samples are training samples obtained by the method according to any one of claims 1-8. The second allocation module is configured to allocate multiple sample groups included in each task unit to multiple graphics processing units (GPUs) for each task unit, wherein one GPU is allocated at least one sample group of the task unit. The model training module is configured to train the model using multiple GPUs and allocated sample sets.

16. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

17. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-13.

Images