A learning training method for a code-oriented agent and related equipment
Patent Information
- Application Number
- CN202611199675.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-07
- Publication Date
- 2026-10-16
AI Technical Summary
但SRL对轨迹中所有位置的步骤均匀采样,以固定的单步(K=1)预测方式进行训练,忽略了不同步骤之间在学习难度上的内在差异,使得代码智能体的部分样本无法被有效学习,代码智能体的大语言模型训练效率低下
[0016]The embodiments of this application include at least the following beneficial effects: This application provides a learning and training method, apparatus, electronic device, storage medium, and program product for code-based intelligent agents. This scheme obtains annotation results by preprocessing the trajectory of expert code-based intelligent agents, executes the annotation results in a sandbox, and constructs training samples for each expert code-based intelligent agent trajectory. The training samples include several expert sequences, normal samples, and recovery samples, enhancing the robustness and autonomous error correction ability of the code-based intelligent agent in real engineering scenarios. It evaluates the step position difficulty and prediction span difficulty of the training samples, and obtains a joint difficulty function by weighting the step position difficulty and prediction span difficulty. This results in faster training convergence, a more stable early learning curve, improved learning efficiency in the early stages of training, and effectiveness of parameter updates, avoiding overfitting of hyperparameters to specific datasets. It obtains the initial temperature and current training step count, determines the current temperature based on the initial temperature and current training step count, and determines the training stage of the training samples based on the current temperature. The temperature gradually shifts the training focus towards more difficult samples, achieving a course from easy to difficult. The process involves: predicting the action sequences of training samples; aggregating the similarity of predicted action sequences with decreasing weights to obtain text similarity rewards; comparing the predicted action sequences with expert sequences to obtain functional equivalence rewards; and linearly fusing text similarity rewards and functional equivalence rewards to form a multi-step look-ahead hybrid reward, which expands the reward coverage of reasonable actions, reduces the error penalty for reasonable exploration paths of the model, and enables dense reward signals to more fairly and accurately measure the actual quality of the code agent's actions. This also reduces noise in the reward signal caused by single-step random bias, making the policy gradient update direction more stable. Based on group-relative policy optimization, the parameters of the large language model are updated using the training phase of the training samples and the multi-step look-ahead hybrid reward, resulting in an updated large language model. The current training step number is incremented by one to obtain a new current training step number. Based on the new current training step number and the training step threshold, the large language model is iterated to obtain a trained large language model. The trained large language model is then deployed to the code agent, improving the signal quality, learning and training efficiency, and data utilization of the code agent's reinforcement learning training.
Smart Images

Figure CN122927419A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a learning and training method and related equipment for code-based intelligent agents. Background Technology
[0002] In related technologies, supervised reinforcement learning (SRL) decomposes the trajectory of an expert code agent into a sequence of progressive actions. It uses the similarity between the model-generated actions and the expert's actions as a dense reward signal to guide the large language model of the code agent in policy optimization within the reinforcement learning framework. However, SRL uniformly samples steps at all positions in the trajectory and trains using a fixed single-step (K=1) prediction method, ignoring the inherent differences in learning difficulty between different steps. This results in some samples of the code agent not being effectively learned, leading to low training efficiency for the large language model of the code agent.
[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention
[0004] The main objective of this application is to propose a learning and training method and related equipment for code-based intelligent agents, aiming to improve the signal quality, learning and training efficiency, and data utilization rate of reinforcement learning and training for code-based intelligent agents.
[0005] To achieve the above objectives, one aspect of this application proposes a learning and training method for code-based intelligent agents, the method comprising: The expert code agent trajectory is preprocessed to obtain the annotation result. The annotation result is executed based on the sandbox. Training samples are constructed for each expert code agent trajectory. The training samples include several expert sequences and normal samples and recovery samples. The step position difficulty and prediction span difficulty of the training samples are evaluated, and the joint difficulty function is obtained by weighting the step position difficulty and the prediction span difficulty. Get the initial temperature and the current training step number, determine the current temperature based on the initial temperature and the current training step number, and determine the training stage of the training sample based on the current temperature; Predict the predicted action sequence of the training samples, aggregate the similarity of the predicted action sequence by decreasing weight to obtain a text similarity reward, compare the predicted action sequence with the expert sequence to obtain a functional equivalence reward, and linearly fuse the text similarity reward and the functional equivalence reward to form a multi-step look-ahead hybrid reward. Based on group relative strategy optimization, the parameters of the large language model are updated using the training phase of the training samples and the multi-step look-ahead hybrid reward, resulting in the updated large language model. The current training step number is incremented by one to obtain a new current training step number. Based on the new current training step number and the training step number threshold, the large language model is iterated to obtain a trained large language model. The trained large language model is then deployed to the code agent.
[0006] In some embodiments, the preprocessing of the expert code agent trajectory to obtain annotation results, and the execution of the annotation results based on a sandbox to construct training samples for each expert code agent trajectory, includes: Each action in the trajectory of the expert code agent is labeled, and the labeling results are recorded, including execution success and execution failure. Based on the labeled results, a number of normal samples are constructed for consecutive actions that have been successfully executed. If an action fails to be executed, the sample is truncated at the failed action. The length of the normal training sample is the actual number of consecutive successful steps. For actions that fail to execute, the failed actions are retained in the prefix context, and the first successfully executed action after the failed action is used as the recovery sample. The prefix context is used to train the model on how to respond correctly.
[0007] In some embodiments, the step of obtaining the initial temperature and the current training step number, determining the current temperature based on the initial temperature and the current training step number, and determining the training stage of the training sample based on the current temperature includes: The initial temperature and the current training step number are obtained, and the current temperature is determined based on the initial temperature and the current training step number. The current temperature decreases monotonically as the training progresses. The current average reward is calculated based on the validation set using the current temperature and the joint difficulty function. The training stage of the training sample is determined by the current average reward based on the stage switching reward threshold, which includes a high stage threshold and a low stage threshold. The training stage includes a warm-up stage, a transition stage, and a reinforcement stage.
[0008] In some embodiments, predicting the predicted action sequence of the training samples involves aggregating the similarity of the predicted action sequences by decreasing weights to obtain a text similarity reward, comparing the predicted action sequence with the expert sequence to obtain a functional equivalence reward, and linearly fusing the text similarity reward and the functional equivalence reward to form a multi-step look-ahead hybrid reward, including: The predicted action sequence is obtained by predicting the expert action sequence using a large language model. The text similarity reward is obtained by aggregating the similarity of the predicted action sequences of several steps by decreasing weights. Execute the expert action sequence and the predicted action sequence in the sandbox to obtain the file status and test pass rate after execution. Compare the file status with the test pass rate, and calculate the functional equivalence reward based on the file status and the test pass rate; The text similarity reward and the functional equivalence reward are linearly fused to form a multi-step look-ahead hybrid reward. If the aforementioned multi-step expert action sequence covers the end of the trajectory, then the multi-step forward hybrid reward degradation is determined as the final test pass rate reward.
[0009] In some embodiments, the method further includes: Obtain several large language models that have been trained; The task resolution rate of the large language model was evaluated using a validation set. The success rate of the large language model in recovering from erroneous actions was evaluated using a validation set. Based on the task resolution rate and the recovery success rate, the optimal large language model is selected, and the optimal large language model is deployed to the code agent.
[0010] In some embodiments, the plurality of steps are for predicting the span, and the evaluation of the step position difficulty and the predicted span difficulty of the training samples, and the weighted combination of the step position difficulty and the predicted span difficulty to obtain a joint difficulty function, includes: The difficulty of the step position in the training sample is evaluated by using the step position index of the training sample. The difficulty of predicting the span of the training samples is assessed by evaluating the predicted span of the training samples. The joint difficulty function is obtained by weighting the difficulty of the step location and the difficulty of the predicted span based on the weight hyperparameters.
[0011] In some embodiments, the step of incrementing the current training step number by one to obtain a new current training step number, and iterating the large language model based on the new current training step number and a training step threshold to obtain a trained large language model includes: The current training step number is incremented by one to obtain a new current training step number. If the new current training step number is less than the training step number threshold, the process of obtaining the initial temperature and the current training step number, determining the current temperature based on the initial temperature and the current training step number, and determining the training stage of the training sample based on the current temperature is repeated until the new current training step number is greater than or equal to the training step number threshold, thus obtaining the trained large language model.
[0012] To achieve the above objectives, another aspect of this application proposes a learning and training apparatus for code-based intelligent agents, the apparatus comprising: The annotation module is used to preprocess the expert code agent trajectory to obtain annotation results, execute the annotation results based on the sandbox, and construct training samples for each expert code agent trajectory. The training samples include several expert sequences and normal samples and recovery samples. The difficulty module is used to evaluate the step position difficulty and the predicted span difficulty of the training sample, and to obtain a joint difficulty function by weighting the step position difficulty and the predicted span difficulty. The stage module is used to obtain the initial temperature and the current training step number, determine the current temperature based on the initial temperature and the current training step number, and determine the training stage of the training sample based on the current temperature. The reward module is used to predict the predicted action sequence of the training samples, aggregate the similarity of the predicted action sequence by decreasing weight to obtain a text similarity reward, compare the predicted action sequence with the expert sequence to obtain a functional equivalence reward, and linearly fuse the text similarity reward and the functional equivalence reward to form a multi-step look-ahead hybrid reward. The training module is used for group relative policy optimization, which uses the training phase of the training samples and the multi-step look-ahead mixed reward to update the parameters of the large language model, so as to obtain the updated large language model. An iteration module is used to increment the current training step number by one to obtain a new current training step number, and iterate the large language model based on the new current training step number and the training step number threshold to obtain a trained large language model. The trained large language model is used to deploy to the code agent.
[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.
[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.
[0016] The embodiments of this application include at least the following beneficial effects: This application provides a learning and training method, apparatus, electronic device, storage medium, and program product for code-based intelligent agents. This scheme obtains annotation results by preprocessing the trajectory of expert code-based intelligent agents, executes the annotation results in a sandbox, and constructs training samples for each expert code-based intelligent agent trajectory. The training samples include several expert sequences, normal samples, and recovery samples, enhancing the robustness and autonomous error correction ability of the code-based intelligent agent in real engineering scenarios. It evaluates the step position difficulty and prediction span difficulty of the training samples, and obtains a joint difficulty function by weighting the step position difficulty and prediction span difficulty. This results in faster training convergence, a more stable early learning curve, improved learning efficiency in the early stages of training, and effectiveness of parameter updates, avoiding overfitting of hyperparameters to specific datasets. It obtains the initial temperature and current training step count, determines the current temperature based on the initial temperature and current training step count, and determines the training stage of the training samples based on the current temperature. The temperature gradually shifts the training focus towards more difficult samples, achieving a course from easy to difficult. The process involves: predicting the action sequences of training samples; aggregating the similarity of predicted action sequences with decreasing weights to obtain text similarity rewards; comparing the predicted action sequences with expert sequences to obtain functional equivalence rewards; and linearly fusing text similarity rewards and functional equivalence rewards to form a multi-step look-ahead hybrid reward, which expands the reward coverage of reasonable actions, reduces the error penalty for reasonable exploration paths of the model, and enables dense reward signals to more fairly and accurately measure the actual quality of the code agent's actions. This also reduces noise in the reward signal caused by single-step random bias, making the policy gradient update direction more stable. Based on group-relative policy optimization, the parameters of the large language model are updated using the training phase of the training samples and the multi-step look-ahead hybrid reward, resulting in an updated large language model. The current training step number is incremented by one to obtain a new current training step number. Based on the new current training step number and the training step threshold, the large language model is iterated to obtain a trained large language model. The trained large language model is then deployed to the code agent, improving the signal quality, learning and training efficiency, and data utilization of the code agent's reinforcement learning training. Attached Figure Description
[0017] Figure 1 This is a flowchart of the learning and training method for code-oriented intelligent agents provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S101 in the text; Figure 3 yes Figure 1 The flowchart of step S103 in the process; Figure 4 yes Figure 1 The flowchart of step S104 in the process; Figure 5This is a flowchart of the optimal deployment steps of the learning and training method for code-oriented intelligent agents provided in the embodiments of this application; Figure 6 yes Figure 1 The flowchart of step S102 in the document; Figure 7 This is a flowchart illustrating a specific implementation of the code-based intelligent agent learning and training method provided in this application when applied to the SWE-Bench-Lite code defect repair task; Figure 8 This is a schematic diagram of the modules provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the learning and training device for code-based intelligent agents provided in the embodiments of this application; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.
[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”
[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.
[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0022] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.
[0023] 1) Code Agent: A large-scale language model system capable of performing complex software development tasks in a real code repository environment through multiple rounds of tool calls (such as file reading and writing, code searching, terminal command execution, etc.), driven by natural language instructions. Its core feature is decomposing the problem-solving process into a series of executable actions and iteratively advancing the task through interaction with the external environment.
[0024] 2) Expert Code Agent Trajectory: This refers to the complete sequence of interactions recorded by a capable code agent (such as Claude-3.7-Sonnet) during the completion of a software engineering task, including querying questions, step-by-step actions, and final code patching. This invention uses the expert code agent trajectory as a source of supervision to extract progressively learning signals.
[0025] 3) Supervised Reinforcement Learning (SRL): A training framework that combines the advantages of supervised learning and reinforcement learning. It decomposes the trajectory of the expert code agent into a step-by-step action sequence and uses the similarity between the model-generated action and the expert action as a dense reward signal to guide the model to optimize the policy under the reinforcement learning framework. It is particularly effective in reward sparse scenarios.
[0026] 4) Reward Sparsity: In reinforcement learning training, the agent only receives a reward after completing the entire trajectory (e.g., code test pass rate). This results in a lack of supervision signals for many intermediate steps, unclear direction of model parameter updates, and difficulty in effectively advancing training. Code agent tasks are particularly prone to reward sparsity due to their long interaction chains and high testing and verification costs.
[0027] 5) Curriculum Learning: A machine learning training strategy inspired by cognitive science. By rationally designing the presentation order of training samples, the model follows a learning path from easy to difficult, thereby improving training efficiency and final performance.
[0028] 6) Functional Equivalence: A concept proposed in this invention, referring to the equivalence between two text segments that differ significantly but produce the same functional effect on the codebase after execution. Unlike sequence similarity at the text level, functional equivalence is measured by comparing the state after execution in the code sandbox (differences in file changes, unit test pass / fail status).
[0029] 7) Sandbox Execution: A mechanism that runs code agent actions in an isolated and controlled environment, records execution results (success / failure), changes in code state, and test pass rate. It is the basic component of this invention for step quality scoring and functional equivalence measurement.
[0030] 8) Error Target Shifting: The training data processing method proposed in this invention is as follows: If the k-th step in the teacher trajectory fails, the failed step is not used as the training target. Instead, the prediction target is shifted to the first successfully executed step y_{k+j} (j≥1) after it. At the same time, the failed step y_k^{error} is retained in the prefix context, so that the model learns the ability to "make correct recovery after encountering an error context", which fundamentally solves the problem of meaningless supervision caused by directly learning failed steps.
[0031] 9) Group Relative Policy Optimization (GRPO): A reinforcement learning algorithm suitable for large language models. It estimates the advantage by sampling G responses from the current policy and calculating the relative reward within the group. It does not require an additional value function network, is stable in training and computationally efficient, and is the basic algorithm for the policy optimization stage of this invention.
[0032] 10) Multi-step Lookahead Prediction: The training method proposed in this invention refers to having the model continuously predict K subsequent steps (instead of only predicting 1 step) under the given prefix context, so as to obtain the aggregated reward signal across steps, thereby alleviating the problem of excessively strict single-step similarity constraints.
[0033] In related technologies, supervised reinforcement learning (SRL) decomposes the trajectory of an expert code agent into a sequence of progressive actions. It uses the similarity between the model-generated actions and the expert's actions as a dense reward signal to guide the large language model of the code agent in policy optimization within the reinforcement learning framework. However, SRL uniformly samples steps at all positions in the trajectory and trains using a fixed single-step (K=1) prediction method, ignoring the inherent differences in learning difficulty between different steps. This results in some samples of the code agent not being effectively learned, leading to low training efficiency for the large language model of the code agent.
[0034] In summary, the technical problems existing in the relevant technologies need to be improved.
[0035] In view of this, this application provides a learning and training method and related equipment for code-based intelligent agents. This method preprocesses the expert code agent's trajectory to obtain annotation results, executes the annotation results in a sandbox, and constructs training samples for each expert code agent trajectory. The training samples include several expert sequences, normal samples, and recovery samples, enhancing the robustness and autonomous error correction capabilities of the code agent in real-world engineering scenarios. It evaluates the step position difficulty and prediction span difficulty of the training samples, and obtains a joint difficulty function by weighting the step position difficulty and prediction span difficulty. This results in faster training convergence, a more stable early learning curve, improved early training efficiency and the effectiveness of parameter updates, and avoids overfitting of hyperparameters to specific datasets. It obtains the initial temperature and current training step count, determines the current temperature based on the initial temperature and current training step count, and determines the training stage of the training samples based on the current temperature. Temperature is used to gradually shift the training focus towards more difficult samples, achieving a course progression from easy to difficult. It also predicts the pre-... The algorithm measures action sequences and aggregates predicted action sequence similarity using decreasing weights to obtain text similarity rewards. It then compares predicted action sequences with expert sequences to obtain functional equivalence rewards. Linearly fusing text similarity and functional equivalence rewards forms a multi-step look-ahead hybrid reward, expanding the reward coverage for reasonable actions, reducing penalties for errors in the model's reasonable exploration paths, and enabling dense reward signals to more fairly and accurately measure the actual quality of the code agent's actions. This also reduces noise in the reward signal caused by single-step random bias, making the policy gradient update direction more stable. Based on group-relative policy optimization, the parameters of the large language model are updated using the training phase of training samples and the multi-step look-ahead hybrid reward, resulting in an updated large language model. The current training step number is incremented by one to obtain a new current training step number. Based on this new current training step number and a training step threshold, the large language model is iterated to obtain a trained large language model. This trained large language model is then deployed to the code agent, improving the signal quality, training efficiency, and data utilization of the code agent's reinforcement learning training.
[0036] The code-based intelligent agent learning and training method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the code-based intelligent agent learning and training method, but is not limited to the above forms.
[0037] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0038] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.
[0039] Figure 1This is an optional flowchart of the learning and training method for code-oriented intelligent agents provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.
[0040] Step S101: Preprocess the expert code agent trajectory to obtain the annotation result, and construct training samples for each expert code agent trajectory based on the sandbox execution annotation result.
[0041] The training samples include several-step (K-step) expert sequences, and the training samples include normal samples and recovery samples.
[0042] In some embodiments, each action (i.e. step) in the trajectory of the expert code agent is executed and labeled, and the labeling results are recorded. The labeling results include successful execution and failed execution. Several normal samples are constructed based on the consecutive actions with successful labeling results. If an action fails to execute, it is truncated at the failed action. The length of the normal training sample is the actual number of consecutive successful steps.
[0043] Optionally, for actions that fail to execute, the failed actions are retained in the prefix context, and the first successful action after the failed action is used as the recovery sample. The prefix context is used to train the model on how to respond correctly.
[0044] Understandably, the failed steps in the trajectory of the expert code agent are rejected as training targets, and the prediction target is moved forward to the first successful step, eliminating the interference of meaningless supervision signals. At the same time, the steps are given a three-level quality score based on sandbox execution, and a quality weight is applied to the entire trajectory, so as to comprehensively improve the effective utilization of training data.
[0045] In this embodiment, the expert code agent trajectory is preprocessed to obtain the annotation result. Based on the sandbox execution annotation result, training samples are constructed for each expert code agent trajectory, which enhances the robustness and autonomous error correction capability of the code agent in real engineering scenarios.
[0046] Step S102: Evaluate the step position difficulty and prediction span difficulty of the training samples, and obtain a joint difficulty function by weighting the step position difficulty and prediction span difficulty.
[0047] In some embodiments, the step position difficulty of a training sample is evaluated by the step position index of the training sample; the prediction span difficulty of a training sample is evaluated by the prediction span of the training sample.
[0048] Optionally, a joint difficulty function is obtained by weighting the step position difficulty and the prediction span difficulty based on the weight hyperparameters.
[0049] In this embodiment, the step position difficulty and prediction span difficulty of the training samples are evaluated, and a joint difficulty function is obtained by weighting and combining the step position difficulty and prediction span difficulty. This solves the problem of interference from high-difficulty samples in the early stage of training caused by uniform sampling in SRL. The training convergence speed is faster, the early learning curve is more stable, and the learning efficiency and parameter update effectiveness in the early stage of training are improved. Furthermore, the two-dimensional joint curriculum and dense mixed rewards are organically combined, which significantly improves the completeness of curriculum learning and the adaptability to code agent tasks. At the same time, the adaptive stage switching mechanism automatically matches the curriculum progress with the actual learning progress of the model, avoiding overfitting of curriculum hyperparameters to specific datasets.
[0050] Step S103: Obtain the initial temperature and the current training step number, determine the current temperature based on the initial temperature and the current training step number, and determine the training stage of the training sample based on the current temperature.
[0051] In some embodiments, an initial temperature and the current training step number are obtained, and the current temperature is determined based on the initial temperature and the current training step number. The current temperature decreases monotonically as the training progresses, and the current temperature is used for annealing sampling.
[0052] It is understandable that this embodiment adopts a course sampling strategy based on temperature annealing. In the early stage of training, the temperature is relatively high, and the probability difference between each sample is small, which is close to uniform sampling. As the current temperature decreases monotonically, the samples with a larger joint difficulty Ω(k, K) (the more difficult) have a faster exponential growth in their exp(+Ω / τ) term, and the probability of being sampled continues to rise. The training focus gradually shifts to difficult samples, realizing the course advancement from easy to difficult.
[0053] Optionally, the current average reward can be calculated based on the validation set using the current temperature and the joint difficulty function.
[0054] Furthermore, the training phase of the training samples is determined based on the current average reward and a phase switching reward threshold. The phase switching reward threshold includes a high phase threshold and a low phase threshold, and the training phase includes a warm-up phase, a transition phase, and a reinforcement phase.
[0055] In this embodiment, the initial temperature and the current number of training steps are obtained, the current temperature is determined based on the initial temperature and the current number of training steps, and the training stage of the training sample is determined based on the current temperature. The training focus is gradually shifted to difficult samples by temperature, so as to realize the course advancement from easy to difficult.
[0056] Step S104: Predict the predicted action sequence of the training samples, aggregate the similarity of the predicted action sequence by decreasing weight to obtain the text similarity reward, compare the predicted action sequence with the expert sequence to obtain the functional equivalence reward, and linearly fuse the text similarity reward and the functional equivalence reward to form a multi-step look-ahead hybrid reward.
[0057] In some embodiments, an initial large language model or the large language model updated in the last iteration is obtained, and the large language model is used to predict several expert action sequences to obtain several predicted action sequences.
[0058] Optionally, a text similarity reward can be obtained by aggregating the similarity of several predicted action sequences by decreasing weights.
[0059] In some embodiments, several expert action sequences and several predicted action sequences are executed in a sandbox to obtain the file status and test pass rate after execution; the file status and test pass rate are compared, and a functional equivalence reward is calculated based on the file status and test pass rate.
[0060] Furthermore, text similarity rewards and functional equivalence rewards are linearly fused to form a multi-step look-ahead hybrid reward.
[0061] It should be noted that if a series of expert action sequences cover the end of the trajectory, the multi-step look-ahead mixed reward degradation will be determined as the final test pass rate reward.
[0062] In this embodiment, the predicted action sequence of the training samples is predicted. The similarity of the predicted action sequence is aggregated by decreasing weights to obtain a text similarity reward. The predicted action sequence is compared with the expert sequence to obtain a functional equivalence reward. The text similarity reward and the functional equivalence reward are linearly fused to form a multi-step look-ahead hybrid reward. This solves the problem that single-step text similarity in SRL cannot identify equivalent but text-different code action paths, expands the reward coverage of reasonable actions, reduces the erroneous penalty for reasonable exploration paths of the model, and enables dense reward signals to more fairly and accurately measure the actual quality of the code agent's actions. At the same time, the introduction of functional equivalence measurement directly links the reward signal to the actual execution effect of the code action. The accuracy and fairness of the reward signal are significantly better than the pure text similarity scheme. Multi-step aggregation reduces the noise in the reward signal caused by single-step random bias, making the policy gradient update direction more stable.
[0063] Step S105: Based on group relative policy optimization, the parameters of the large language model are updated using the training phase of the training samples and multi-step look-ahead mixed reward, resulting in the updated large language model.
[0064] In some embodiments, reinforcement learning is performed using the training samples of the constructed training set with the GRPO algorithm, and the policy parameters of the large language model are updated with within-group relative reward estimation to obtain the updated large language model.
[0065] In this embodiment, based on group relative policy optimization, the parameters of the large language model are updated by using the training phase of training samples and multi-step look-ahead mixed rewards to obtain the updated large language model. This is beneficial for realizing the training and updating of the model, thereby improving the signal quality, learning efficiency and data utilization of the code agent reinforcement learning training.
[0066] Step S106: Increment the current training step number by one to obtain a new current training step number. Iterate the large language model based on the new current training step number and the training step number threshold to obtain the trained large language model.
[0067] Specifically, the trained large language model is used to deploy to code-based intelligent agents.
[0068] In some embodiments, the updated large language model is iteratively updated based on the new current training steps and the training step threshold to obtain the trained large language model.
[0069] It is understandable that the current training step count is incremented by one to obtain the new current training step count.
[0070] Furthermore, if the new current training step number is less than the training step number threshold, the process returns to the steps of obtaining the initial temperature and the current training step number, determining the current temperature based on the initial temperature and the current training step number, and determining the training stage of the training sample based on the current temperature.
[0071] Optionally, if the new current training step is greater than or equal to the training step threshold, a large language model that has been trained is obtained.
[0072] In some embodiments, several trained large language models are obtained; the task resolution rate of the large language models is evaluated using a validation set; the recovery success rate of the large language models after erroneous actions is evaluated using a validation set; the optimal large language model is selected based on the task resolution rate and recovery success rate, and the optimal large language model is deployed to the code agent.
[0073] In this embodiment, the current training step number is incremented by one to obtain a new current training step number. Based on the new current training step number and the training step number threshold, the large language model is iterated to obtain the trained large language model. The trained large language model is then deployed to the code agent to improve the signal quality, learning efficiency, and data utilization of the code agent's reinforcement learning training.
[0074] Steps S101 to S106 as shown in this embodiment of the application preprocess the expert code agent trajectory to obtain annotation results. Based on the sandbox execution of the annotation results, training samples are constructed for each expert code agent trajectory. The training samples include several expert sequences and normal samples and recovery samples, which enhances the robustness and autonomous error correction ability of the code agent in real engineering scenarios. The step position difficulty and prediction span difficulty of the training samples are evaluated. The joint difficulty function is obtained by weighting the step position difficulty and prediction span difficulty, which results in faster training convergence speed, more stable early learning curve, improved learning efficiency and parameter update effectiveness in the early stage of training, and avoids overfitting of hyperparameters to specific datasets. The initial temperature and current training step number are obtained. The current temperature is determined based on the initial temperature and current training step number. The training stage of the training sample is determined based on the current temperature. The temperature gradually shifts the training focus to difficult samples, realizing the course progression from easy to difficult. The predicted action sequence of the training sample is predicted. Text similarity rewards are obtained by aggregating predicted action sequence similarity through decreasing weights. Functional equivalence rewards are obtained by comparing predicted action sequences with expert sequences. Text similarity rewards and functional equivalence rewards are linearly fused to form a multi-step look-ahead hybrid reward, which expands the reward coverage of reasonable actions, reduces the error penalty for reasonable exploration paths of the model, and enables dense reward signals to more fairly and accurately measure the actual quality of the code agent's actions. It also reduces noise caused by single-step random bias in the reward signal and makes the policy gradient update direction more stable. Based on group relative policy optimization, the parameters of the large language model are updated using the training phase of training samples and multi-step look-ahead hybrid rewards to obtain the updated large language model. The current training step number is incremented by one to obtain a new current training step number. The large language model is iterated based on the new current training step number and the training step number threshold to obtain the trained large language model. The trained large language model is used to deploy to the code agent, improving the signal quality, learning and training efficiency, and data utilization of the code agent's reinforcement learning training.
[0075] Please see Figure 2 In some embodiments, step S101 may include, but is not limited to, steps S201 to S203: Step S201: Perform execution annotations on each action in the trajectory of the expert code agent and record the annotation results.
[0076] Specifically, the results are labeled as either successful or unsuccessful.
[0077] In step S201 of some embodiments, for each action yk in the expert code agent trajectory, the execution is replayed in a Docker isolation sandbox, and the execution result (i.e., the annotation result) is recorded. k∈ {success, fail} (a command exit code of 0 indicates success, otherwise it indicates failure).
[0078] The annotation results are used to construct the following two types of training samples: determine the cutoff position of the K-step window, and identify the failure-success inflection point of constructable recoverable samples.
[0079] Step S202: Construct several normal samples based on the annotation results of consecutive actions that have been successfully executed. If an action fails to be executed, truncate it at the point of failure.
[0080] Specifically, the length of a normal training sample is equal to the actual number of consecutive successful steps.
[0081] In step S202 of some embodiments, based on the sandbox execution annotation results, the following two types of training samples are constructed for each expert code agent trajectory: (1) Normal samples: from each successfully executed step k(e k Starting from a success step, a prediction window is formed by taking consecutive successful steps. If a failed step is encountered in the window, it is truncated at that step. The window length is the actual number of consecutive successful steps (maximum K steps). All steps within the window are considered successful steps, and the reward is directly derived from the hybrid reward R in Module 1. mixed calculate.
[0082] (2) Recover the sample: For each failed step k(e) in the trajectory k = fail), if there exists a first successful step yk+j (j is the one that satisfies e) k+j If the value of success is the smallest positive integer, then the following specialized recovery training samples are constructed: the failed step yk is retained in the prefix context, allowing the model to learn how to respond correctly after encountering the error context; the prediction target is directly yk+j, skipping all intermediate failed steps: , Among them, y k For steps that fail to execute, y k+j Then there is a first successful step, X shift For the prefix context after the erroneous target is moved forward, j is the one that satisfies e k+j = The smallest positive integer of success.
[0083] Understandably, this constructed recovery sample enables the model to learn to directly predict the first correct recovery action in a context where an error has occurred, thereby enabling it to overcome errors and autonomously correct itself.
[0084] Step S203: For actions that fail to execute, retain the failed actions in the prefix context and take the first successfully executed action after the failed action as the recovery sample.
[0085] The prefix context is used to train the model on how to respond correctly.
[0086] Understandably, the construction of recovered samples fundamentally solves the problem of meaningless supervision of failed steps in the trajectory of expert code agents as training targets: directly learning known erroneous actions is not only unhelpful but also pollutes the model's strategy; shifting the erroneous target forward makes the supervision target of each training sample an effective action, greatly improving the quality of the training signal; and retaining the context of failed steps in the prefix enables the model to learn how to overcome errors and make correct responses when encountering them, enhancing the robustness and autonomous error correction ability of the code agent in real engineering scenarios.
[0087] Please see Figure 3 In some embodiments, step S103 may include, but is not limited to, steps S301 to S303: Step S301: Obtain the initial temperature and the current training step number, and determine the current temperature based on the initial temperature and the current training step number.
[0088] The current temperature decreases monotonically as the training progresses.
[0089] In step S301 of some embodiments, the course sampling strategy based on temperature annealing performs weighted sampling of (k, K) using a Boltzmann distribution at training step t: In the early training stage, the temperature τ is relatively high, and the probability difference between samples is small, approaching uniform sampling; as τ monotonically decreases, the samples with a larger joint difficulty Ω(k, K) (more difficult) experience a faster exponential increase in their exp(+Ω / τ) term, leading to a continuous increase in their sampling probability. The training focus gradually shifts towards more difficult samples, achieving a course progression from easy to difficult. , Where Ω(k, K) is the joint difficulty, τ(t) is the temperature parameter that monotonically decreases with the training process (annealing scheduling), and P(k,K;t) is the probability that a training sample with step position k and prediction span K is sampled at training step t, which is obtained by normalizing the joint difficulty using the Boltzmann distribution.
[0090] , Where τ(t) is a temperature parameter that monotonically decreases with the training process (annealing scheduling), τ0 is the initial temperature (default 5.0), and T maxτ represents the total number of training steps, and η is the decay exponent (default 2.0). When τ is large, the sampling probability of samples of different difficulty levels is relatively small, tending to be uniform. As τ decreases monotonically, the exp(+Ω / τ) value corresponding to samples with a larger joint difficulty Ω (difficult) increases exponentially, the sampling weight increases relatively, and the training focus gradually shifts to the high difficulty region (small k, large K), thereby achieving an orderly course progression from easy to difficult.
[0091] Step S302: Calculate the current average reward based on the validation set using the current temperature and the joint difficulty function.
[0092] It should be noted that, to avoid premature advancement caused by dividing stages into fixed steps, an adaptive switching condition based on validation set rewards is introduced: In step S302 of some embodiments, the temperature τ is further reduced only when the average reward within the current stage difficulty range exceeds the threshold ρ (default 0.6); otherwise, the current temperature is maintained, and the current stage training continues to ensure that the course progress speed and the model learning progress are adaptively matched.
[0093] Step S303: Determine the training stage of the training sample based on the current average reward and the stage switching reward threshold.
[0094] The phase switching reward thresholds include high phase thresholds and low phase thresholds, and the training phases include warm-up phase, transition phase and reinforcement phase.
[0095] Alternatively, the training phase can be determined by the current average reward or by the current temperature.
[0096] In step S303 of some embodiments, based on the annealing process of τ(t), the joint course learning is naturally divided into three training phases: Phase 1 (Warm-up Phase, t ∈ [0, T1]): τ is relatively large, and training focuses on simple samples with larger step positions k (i.e., late steps, sufficient prefixes) and a prediction span K=1. In this phase, the model quickly learns the ability to generate basic steps and establishes a good initial strategy.
[0097] The second stage (transition stage, t ∈ [T1, T2]): τ gradually decreases, the sampling range expands to steps with moderate k values and short sequence predictions with K>1. The model begins to attempt short-term multi-step planning while gradually touching upon the steps in the early part of the trajectory.
[0098] Phase 3 (Reinforcement Phase, t ∈ [T2, T]) max ]): τ decreases to a low value, sampling covers the entire trajectory range (including samples with small values in the early steps k) and a larger prediction span K (maximum K). maxThe model performs full policy optimization on all difficulty samples.
[0099] Please see Figure 4 In some embodiments, step S104 may include, but is not limited to, steps S401 to S406: Step S401: Predict several expert action sequences using a large language model to obtain several predicted action sequences.
[0100] In some embodiments, given an expert code agent trajectory y=[y1,y2,…,y] containing N steps, N (abbreviated y) k The training samples constructed by SRL in step k (representing the Teacher's expert action at step k) and the corresponding problem query x (code defect description or functional requirement) are: prefix context x. k =[x,y1,…,y k-1 The target for prediction is y. k The model is based on ~p θ (·|x k Predictions are generated using the method SeqSim(), and the reward for SRL is SeqSim( ,y k This embodiment extends this to a K-step lookahead framework (uppercase, prediction span).
[0101] In step S401 of some embodiments, K-step lookahead training sample construction is performed. For a training sample starting from the k-th step with a prediction span of K: the prefix context remains x. k =[x,y1,…,y k-1 (Same as SRL), the prediction target is expanded to a sequence of K Teacher actions from step k to step k+K-1, i.e., the target sequence is [y k ,y k+1 ,…,y k+K-1 The model generates K steps sequentially in an autoregressive manner, given a prefix, with a special delimiter inserted between each step to obtain the predicted sequence. When K=1, it degenerates into a single-step prediction of SRL; when K>1, the model needs to plan multiple subsequent steps simultaneously, and the task difficulty increases with K.
[0102] Step S402: The similarity of several predicted action sequences is aggregated by decreasing weights to obtain a text similarity reward.
[0103] In step S402 of some embodiments, the multi-step text similarity reward component calculates the sequence similarity progressively for the K-step prediction results and aggregates them with decreasing weights (steps closer to the prefix have larger weights, with the largest weight when i=0 and the smallest weight when i=K-1): , Among them, R text (k,K) represents the text similarity reward, and i represents the step offset (i=0 corresponds to the kth step itself, and i=K-1 corresponds to the k+K-1th step). The sequence similarity between the model's (k+i) step prediction and the expert's (k+i) step action is calculated using Python's difflib.SequenceMatcher; the weighting coefficient (Ki) makes the alignment accuracy of the proximal steps contribute more.
[0104] Step S403: Execute several expert action sequences and several predicted action sequences in the sandbox to obtain the file status and test pass rate after execution.
[0105] In step S403 of some embodiments, a functional equivalence reward component is applied based on the sandbox execution. The core value of the code agent's actions lies in the functional effect after execution, rather than the textual representation. This invention executes the K-step sequence generated by the model in a sandbox environment. Functional equivalence is measured by comparing the post-execution code repository state with the expert's K-step sequence y, and consists of two sub-components: 1) File State Matching (FileMatch): Compare the diff results of the target files after execution, using normalized edit distance to measure the textual differences in the modifications made to the codebase by both files—the smaller the difference, the more similar the functional effects. , Where FileMatch is the file state matching degree, Δfile(·) represents the file difference (unified diff) of the code repository after execution, and NormEditDist is the normalized string edit distance.
[0106] 2) Test Match: Run relevant test cases in the sandbox and use the difference in the pass rates of the two tests to measure functional equivalence—the smaller the difference, the closer the functional effects are. , Where TestMatch is the test pass rate matching degree, and PassRate is the test pass rate.
[0107] Step S404: Compare file status and test pass rate, and calculate functional equivalence reward based on file status and test pass rate.
[0108] In some embodiments, the comprehensive functional equivalence reward is obtained by linearly weighting the above two components: , Among them, R exec To reward the overall functional equivalence, FileMatch represents the file status matching degree, TestMatch represents the test pass rate matching degree, and α1 and α2 are hyperparameters. The default settings are α1 = 0.3 and α2 = 0.7, which give higher weight to the test pass rate to reflect the priority of functional verification.
[0109] Step S405: Linearly fuse the text similarity reward and the functional equivalence reward to form a multi-step look-ahead hybrid reward.
[0110] In some embodiments, a multi-step look-ahead hybrid reward linearly fuses text similarity rewards and functional equivalence rewards to obtain the final multi-step look-ahead reward signal: , Among them, R mixed For multi-step forward hybrid rewards, R exec For comprehensive functional equivalence rewards, R text (k,K) represents the text similarity reward, and β ∈ [0,1] is the mixing coefficient, which controls the weight ratio of the two signals: text similarity and functional equivalence.
[0111] It should be noted that in the early stages of training, the text similarity weight (with a larger β) can be appropriately increased to ensure learning stability; as the model's capabilities increase, β can be gradually reduced to shift towards functional equivalence rewards, encouraging the model to explore equivalent but not completely imitative action paths of experts.
[0112] Step S406: If several expert action sequences cover the end of the trajectory, then the multi-step look-ahead mixed reward degradation is determined as the final test pass rate reward.
[0113] In some embodiments, when the K-step prediction covers the end of the trajectory (i.e., k + K ≥ N), R mixed It degenerates into a final state verification reward, directly using the final test pass rate as the reward signal, thus achieving a natural transition from dense process rewards to sparse result rewards.
[0114] Please see Figure 5 In some embodiments, Figure 5 The learning and training method for code-based intelligent agents provided in this application embodiment also includes an optimal deployment step, which may include, but is not limited to, steps S501 to S504: Step S501: Obtain several large language models that have been trained.
[0115] In some embodiments, multiple large language models that have been trained are obtained.
[0116] Step S502: Evaluate the task resolution rate of the large language model using the validation set.
[0117] In step S502 of some embodiments, after training is completed, the model's task resolution rate is evaluated on the SWE-Bench-Lite validation set. Step S503: Evaluate the recovery success rate of the large language model after erroneous actions using a validation set.
[0118] In step S503 of some embodiments, the model’s recovery success rate after an error step is evaluated on the SWE-Bench-Lite validation set.
[0119] Step S504: Select the optimal large language model based on the task resolution rate and recovery success rate, and deploy the optimal large language model to the code agent.
[0120] In step S504 of some embodiments, the optimal checkpoint is selected for final deployment. In the TeleChat code intelligence system, this solution serves as the core training framework, integrated with the TeleChat basic model and code toolchain, supporting application scenarios such as automatic code defect repair and intelligent code review.
[0121] Please see Figure 6 In some embodiments, step S102 may also include, but is not limited to, steps S601 to S603: Step S601: Evaluate the difficulty of the step position of the training sample by using the step position index of the training sample.
[0122] Specifically, several steps are used to predict the span.
[0123] In step S601 of some embodiments, the learning difficulty of the training samples is quantified from two dimensions: step position difficulty and prediction span difficulty, and a joint difficulty function is constructed: Specifically, the difficulty level of the steps is D. pos (k) (k is the step position index): Earlier steps have shorter prefix contexts, more uncertain reasoning paths, and higher difficulty; later steps (closer to N) have stronger constraints and lower difficulty. It is defined as: , Among them, D pos (k) represents the step position difficulty, k represents the step position index, and N represents the length of the expert code agent trajectory.
[0124] Step S602: Evaluate the difficulty of predicting the span of the training samples by assessing the predicted span of the training samples.
[0125] Specifically, the difficulty of predicting the span D hor (K) (K is the number of prediction steps): The larger the number of prediction steps, the more steps the model needs to plan simultaneously, the larger the search space, and the higher the difficulty; the difficulty is lowest when K=1 (i.e., the original SRL method). It is defined as: , Among them, D hor (K) represents the difficulty of predicting the span, and K represents the number of prediction steps.
[0126] Step S603: Based on the weight hyperparameters, the difficulty of the step position and the difficulty of the predicted span are weighted and combined to obtain the joint difficulty function.
[0127] In some embodiments, the joint difficulty function Ω(k, K) is a weighted combination of the difficulty of the two dimensions (where k is the step position and K is the number of predicted steps): , Among them, D pos (k) represents the difficulty of the step location, D hor (K) represents the predicted span difficulty, k represents the step position, K represents the predicted number of steps, and γ ∈ [0,1] represents the weight hyperparameter of position difficulty and span difficulty. The default setting is γ = 0.6, which gives position difficulty a slightly higher weight.
[0128] It is understandable that the two-dimensional difficulty space forms a difficulty gradient field from the bottom right corner (large k, K=1, the easiest) to the top left corner (small k, K=Kmax, the hardest).
[0129] Figure 7 This is a flowchart illustrating a specific implementation of the code-based intelligent agent learning and training method provided in this application when applied to the SWE-Bench-Lite code defect repair task. Figure 7 The methods may include, but are not limited to, the following steps: Step 1: Expert trajectory preprocessing and step quality labeling.
[0130] Specifically, 5000 expert code agent trajector ...
[0131] Furthermore, for each step (i.e., action) in each trajectory, yk Replay the execution in a Docker isolation sandbox (starting from the state of the repository created in the previous k-1 steps), and record the execution result e. k (A command exit code of 0 indicates success, otherwise it indicates failure). After annotation, approximately 18,000 failed execution steps were identified from 5,000 trajectories. Among them, approximately 12,000 had subsequent successful recovery steps (which could be used to construct erroneous target forward samples) and were included in the erroneous target forward sample pool.
[0132] Step 2: Constructing multi-step prospective training data.
[0133] Specifically, let the upper limit of the predicted span be K. max = 5. For each trajectory, construct K ∈ {1, 2, 3, 4, 5} (uppercase K represents the predicted span) of multi-step lookahead training samples (truncated at failed steps). Each starting position k (lowercase, position index) is combined with each span K to form a training sample. Additionally, the error target forward-moving specific samples extracted in step 1 are added to form the complete training dataset, totaling approximately 8.5 million samples. 1% (approximately 85,000 samples) of these are randomly selected as the validation set for monitoring the training process and stage switching decisions.
[0134] Step 3, Configure the two-dimensional course learning schedule.
[0135] In some embodiments, the hyperparameter settings are as follows: initial temperature τ0 = 5.0, decay exponent η = 2.0, location difficulty weight γ = 0.6, and total training steps T. max = 3000 steps (GRPO iteration), stage switching reward threshold ρ = 0.6. The three stages roughly correspond to T1≈ 500 steps and T2≈ 1500 steps, but are regulated by an adaptive switching mechanism. The actual switching timing is based on the average reward of the validation set reaching the threshold. Batch sampling within each stage is dynamically calculated based on P(k, K; t) ∝ exp(+Ω(k, K) / τ(t)), and the batch size B = 64.
[0136] Step 4: GRPO strategy optimization training.
[0137] Optionally, the base model uses TeleChat3-36B and is trained using the GRPO algorithm on the training set constructed in step 2. Each input sample (prefix xk, predicted span K) undergoes G = 8 rollouts. The advantage estimate is calculated based on the mixed reward Rmixed(k, K), while dynamic filtering is performed on samples with a reward standard deviation below the threshold ε = 0.01. The mixed reward hyperparameters are: β = 0.7 (early training), linearly decreasing to β = 0.4 (late training); α1 = 0.3, α2 = 0.7. The learning rate is 3e-6, and the KL penalty coefficient is 0.01.
[0138] Step 5, Model Evaluation and Deployment.
[0139] In some embodiments, after training, the model's task resolution rate and recovery success rate after erroneous steps are evaluated on the SWE-Bench-Lite validation set, and the optimal checkpoint is selected for final deployment. In the TeleChat code agent system, this solution serves as the core training framework, integrated with the TeleChat basic model and code toolchain, supporting application scenarios such as automatic code defect repair and intelligent code review.
[0140] Specifically, the above learning and training method for code-based intelligent agents is implemented through the following modules, as shown in the module diagram below. Figure 8 As shown, the correspondence between modules and process steps is as follows: Figure 7 As shown in the diagram, the solid lines represent the main process steps, the dashed lines represent course scheduling / recovery samples, and [Module X] represents the corresponding module. Module 1: Multi-step look-ahead function equivalent trajectory similarity reward mechanism.
[0141] (1) Formal definition of the problem. Given an expert trajectory (i.e., the trajectory of the expert agent) containing N steps, y = [y1, y2, …, y N (abbreviated y) k The training samples constructed by SRL in step k (representing the Teacher's expert action at step k) and the corresponding problem query x (code defect description or functional requirement) are: prefix context x. k = [x, y1,…, y k-1 The target for prediction is y. k The model is based on ~ p θ (·|x k The prediction is generated in the manner described above, and the reward for SRL is... This invention extends this to a K-step lookahead framework (uppercase, prediction span).
[0142] (2) Construction of K-step lookahead training samples. For training samples starting from the k-th step and with a prediction span of K: the prefix context is still x k = [x, y1, …, y k-1 (Same as SRL), the prediction target is expanded to a sequence of K teacher actions from step k to step k+K-1, i.e., the target sequence is [y k , y k+1 , …, y k+K-1 The model generates K steps sequentially in an autoregressive manner, given a prefix, with a special delimiter inserted between each step to obtain the predicted sequence. When K=1, it degenerates into a single-step prediction of SRL; when K>1, the model needs to plan multiple subsequent steps simultaneously, and the task difficulty increases with K.
[0143] (3) Multi-step text similarity reward component. For the K-step prediction results, the sequence similarity is calculated step by step and aggregated with decreasing weights (the closer the step is to the prefix, the greater the weight, with the maximum weight when i=0 and the minimum weight when i=K-1): , In the formula, R text (k,K) represents the text similarity reward, and i represents the step offset (i=0 corresponds to the k-th step itself, and i=K-1 corresponds to the k+K-1-th step). The sequence similarity between the model's (k+i)-th step prediction and the expert's (k+i)-th step action is calculated using Python's difflib.SequenceMatcher; the weighting coefficient (Ki) makes the alignment accuracy of the proximal steps contribute more.
[0144] (4) Functional equivalence reward component based on sandbox execution. The core value of a code agent's actions lies in the functional effect after execution, rather than the textual representation. This invention executes the K-step sequence generated by the model in a sandbox environment. Functional equivalence is measured by comparing the K-step sequence y with the state of the code repository after execution, and consists of two sub-components: FileMatch: Compares the diff results of the target files after execution, using normalized edit distance to measure the textual differences in the modifications made to the codebase by both files—the smaller the difference, the more similar the functional effects. , Where FileMatch is the file state matching degree, Δfile(·) represents the file difference (unified diff) of the code repository after execution, and NormEditDist is the normalized string edit distance.
[0145] TestMatch pass rate matching: Run relevant test cases in a sandbox and measure functional equivalence by the difference in pass rates between the two tests—the smaller the difference, the closer the functional effects are. , Where TestMatch is the test pass rate matching degree, and PassRate is the test pass rate.
[0146] The overall functional equivalence reward is obtained by linearly weighting the above two components: , Among them, R exec To reward the overall functional equivalence, FileMatch is the file status matching degree, TestMatch is the test pass rate matching degree, and α1 and α2 are hyperparameters. The default settings are α1 = 0.3 and α2 = 0.7, which give higher weight to the test pass rate to reflect the priority of functional verification.
[0147] (5) Multi-step look-ahead hybrid reward. The text similarity reward and the functional equivalence reward are linearly fused to obtain the final multi-step look-ahead reward signal: , Among them, R mixed For multi-step forward hybrid rewards, R exec For comprehensive functional equivalence rewards, R text (k,K) represents the text similarity reward, and β ∈ [0,1] is the mixing coefficient, which controls the weight ratio of the two signals: text similarity and functional equivalence.
[0148] In the early stages of training, the text similarity weight (with a larger β) can be appropriately increased to ensure learning stability. As the model's capabilities grow, β can be gradually decreased, shifting towards functional equivalence rewards to encourage the model to explore equivalent but not completely imitative expert action paths. When the prediction covers the end of the trajectory in K steps (i.e., k + K ≥ N), R... mixed It degenerates into a final state verification reward, directly using the final test pass rate as the reward signal, thus achieving a natural transition from dense process rewards to sparse result rewards.
[0149] Module 2, Step Location and Predicted Span Joint Course Learning Strategy.
[0150] (1) Two-dimensional difficulty measurement. This invention innovatively quantifies the learning difficulty of training samples from two dimensions, constructing a joint difficulty function: Step location difficulty D pos (k) (k is the step position index): Earlier steps have shorter prefix contexts, more uncertain reasoning paths, and higher difficulty; later steps (closer to N) have stronger constraints and lower difficulty. It is defined as: , Among them, D pos (k) represents the difficulty of the step position, k represents the step position index, and N represents the length of the expert code agent trajectory.
[0151] Predicting the span difficulty D hor (K) (K is the number of prediction steps): The larger the number of prediction steps, the more steps the model needs to plan simultaneously, the larger the search space, and the higher the difficulty; the difficulty is lowest when K=1 (i.e., the original SRL method). It is defined as: , Among them, D hor (K) represents the difficulty of predicting the span, and K represents the number of prediction steps.
[0152] The joint difficulty function Ω(k, K) is a weighted combination of the difficulty from the two dimensions (where k is the step position and K is the number of prediction steps): , Among them, D pos (k) represents the difficulty of the step location, D hor (K) represents the predicted span difficulty, k is the step position, K is the predicted number of steps, and γ ∈ [0,1] is the weight hyperparameter of position difficulty and span difficulty. The default setting is γ = 0.6, which assigns a slightly higher weight to position difficulty. The two-dimensional difficulty space forms a range from the bottom right corner (large k, K=1, simplest) to the top left corner (small k, K=K). max The most difficult difficulty gradient field.
[0153] (2) Course sampling strategy based on temperature annealing. At time t, weighted sampling is performed on (k, K) using the Boltzmann distribution: In the early stage of training, the temperature τ is relatively large, and the probability difference between samples is small, which is close to uniform sampling; as τ decreases monotonically, the samples with larger joint difficulty Ω(k, K) (the more difficult) have a faster exponential growth in their exp(+Ω / τ) term, and the probability of being sampled continues to rise. The training focus gradually shifts to difficult samples, realizing the course advancement from easy to difficult: , Where Ω(k, K) is the joint difficulty, τ(t) is the temperature parameter that monotonically decreases with the training process (annealing scheduling), and P(k,K;t) is the probability that a training sample with step position k and prediction span K is sampled at training step t, which is obtained by normalizing the joint difficulty using the Boltzmann distribution.
[0154] , Where τ(t) is a temperature parameter that monotonically decreases with the training process (annealing scheduling), τ0 is the initial temperature (default 5.0), and T max τ represents the total number of training steps, and η is the decay exponent (default 2.0). When τ is large, the sampling probability of samples of different difficulty levels is relatively small, tending to be uniform. As τ decreases monotonically, the exp(+Ω / τ) value corresponding to samples with a larger joint difficulty Ω (difficult) increases exponentially, the sampling weight increases relatively, and the training focus gradually shifts to the high difficulty region (small k, large K), thereby achieving an orderly course progression from easy to difficult.
[0155] (3) Three-stage course training process. Based on the annealing process of τ(t), the joint course learning is naturally divided into three training stages: Phase 1 (Warm-up Phase, t ∈ [0, T1]): τ is relatively large, and training focuses on simple samples with larger step positions k (i.e., late steps, sufficient prefixes) and a prediction span K=1. In this phase, the model quickly learns the ability to generate basic steps and establishes a good initial strategy.
[0156] The second stage (transition stage, t ∈ [T1, T2]): τ gradually decreases, the sampling range expands to steps with moderate k values and short sequence predictions with K>1. The model begins to attempt short-term multi-step planning while gradually touching upon the steps in the early part of the trajectory.
[0157] Phase 3 (Reinforcement Phase, t ∈ [T2, T]) max ]): τ decreases to a low value, sampling covers the entire trajectory range (including samples with small values in the early steps k) and a larger prediction span K (maximum K). max The model performs full policy optimization on all difficulty samples.
[0158] (4) Adaptive mechanism for stage switching. To avoid premature advancement caused by dividing stages into fixed steps, this invention introduces an adaptive switching condition based on validation set rewards: the temperature τ is further reduced only when the average reward within the current stage difficulty range exceeds the threshold ρ (default 0.6); otherwise, the current temperature is maintained, and training in the current stage continues, ensuring that the course advancement speed and model learning progress are adaptively matched.
[0159] Module 3: Execution of Perception Step Filtering and Error Target Forwarding Mechanism.
[0160] (1) Sandbox execution annotation. For each action y in the expert (Teacher) trajectory. k Replay the execution in a Docker isolation sandbox and record the execution results. k ∈ {success, fail} (a command exit code of 0 indicates success, otherwise it indicates failure). The annotation results are used to construct the subsequent two types of training samples: to determine the cutoff position of the K-step window, and to identify the failure-success inflection point for constructing recoverable samples.
[0161] (2) Construction of two types of training samples. Based on the sandbox execution annotation results, the following two types of training samples are constructed for each expert trajectory: (1) Normal sample: from each successfully executed step k(e k Starting from a success step, a prediction window is formed by taking consecutive successful steps. If a failed step is encountered in the window, it is truncated at that step. The window length is the actual number of consecutive successful steps (maximum K steps). All steps within the window are considered successful steps, and the reward is directly derived from the hybrid reward R in Module 1. mixed calculate.
[0162] (2) Recover the sample: For each failed step k(e) in the trajectory k = fail), if there is a first successful step y afterwards. k+j (j is to satisfy e) k+j If the smallest positive integer representing success is y, then the following specialized recovery training samples are constructed: the failed step y is retained in the prefix context. k This allows the model to learn how to respond correctly after encountering an erroneous context; the prediction target is directly y. k+j Skip all failed steps in the middle: , Among them, y k For steps that fail to execute, y k+j Then there is a first successful step, X shift For the prefix context after the erroneous target is moved forward, j is the one that satisfies e k+j = the smallest positive integer of success. Recovery samples constructed in this way enable the model to learn to directly predict the first correct recovery action in the context of an error, thereby possessing the ability to overcome errors and autonomously correct deviations.
[0163] It should be noted that the technical effects of this embodiment are as follows: (1) The multi-step look-ahead functional equivalent trajectory similarity reward mechanism proposed in this embodiment solves the core problem of excessively strict SRL single-step text similarity constraints and inability to identify functionally equivalent action paths by K-step aggregated similarity and sandbox-based functional equivalence measurement, enabling dense reward signals to more accurately measure the actual quality of code actions and effectively expanding the reward coverage of reasonable actions.
[0164] (2) The joint course learning strategy of step position and prediction span proposed in this embodiment realizes orderly progressive training from long prefix single step to short prefix multi step through two-dimensional difficulty measurement and Boltzmann temperature sampling, which significantly improves training efficiency and the effectiveness of parameter update.
[0165] (3) The execution-aware step filtering and error target forwarding mechanism proposed in this embodiment completely excludes the failed steps in the trajectory from the normal training target through sandbox execution annotation, and at the same time transforms the context of the failed steps into the prefix of the recovery sample, so that the model can learn to make the correct response when encountering the error context without encountering any error supervision signals, thereby eliminating the pollution of meaningless supervision signals from the root and significantly improving data utilization.
[0166] This embodiment has been validated in the TeleChat code capability training of the Starry Sky large model. The solution rate on the SWE-Bench-Lite benchmark has been significantly improved compared with the original SRL baseline, which fully verifies the practicality and effectiveness of this embodiment.
[0167] Specifically, this invention proposes three innovative technologies for reinforcement learning training of code-based intelligent agents: ① a multi-step look-ahead functional equivalent trajectory similarity reward mechanism; ② a joint course learning strategy for step position and prediction span; and ③ an execution-aware step filtering and erroneous target forward shifting mechanism. All three innovative technologies are proposed for the first time in this patent, and their specific technical highlights and beneficial effects are as follows: ① Multi-step look-ahead function equivalent trajectory similarity reward mechanism Key technical points: Given a prefix context, the model continuously predicts a K-step action sequence, and aggregates the similarity of the K-step step sequence with decreasing weights to obtain a text similarity reward. Simultaneously, the model-generated K-step actions and expert K-step actions are executed in a sandbox, and the file status and test pass rate after execution are compared to calculate the functional equivalence reward. The two are linearly fused to form a multi-step look-ahead hybrid reward. When the K-covered trajectory ends, it degenerates into a final test pass rate reward, achieving a natural transition between dense process rewards and sparse result rewards.
[0168] Beneficial effects: (i) It solves the problem that single-step text similarity in SRL cannot identify equivalent but textually different code action paths, expands the reward coverage of reasonable actions, and reduces the erroneous penalties for reasonable exploration paths of the model; (ii) It introduces a functional equivalence metric, which directly links the reward signal with the actual execution effect of the code action, and the accuracy and fairness of the reward signal are significantly better than the pure text similarity scheme; (iii) Multi-step aggregation reduces the noise in the reward signal caused by single-step random bias, making the policy gradient update direction more stable.
[0169] ② Step-by-step location and predicted span joint course learning strategy Technical points: Defining the position of steps (Difficulty D) pos (k)=1 k / N and the difficulty of predicting the span D hor (K)=(K 1) / (K max 1) A joint difficulty function Ω(k,K) is constructed using weighted combinations; training samples are weighted using the Boltzmann distribution P(k,K;t)∝exp(+Ω(k,K) / τ(t)), with the temperature τ(t) decreasing monotonically as a power function during training (initially τ0=5.0). Initially, sampling is nearly uniform, but as the temperature decreases, the sampling focus shifts towards higher difficulty regions. Combined with an adaptive stage switching mechanism based on the validation set reward threshold, the training process is divided into three stages: warm-up, transition, and reinforcement, achieving a transition from "large k single step (K=1)" to "small k multi step (K=K)". max The two-dimensional orderly progression of ")".
[0170] Beneficial effects: (i) It solves the problem of interference from high-difficulty samples in the early stage of training caused by uniform sampling in SRL, resulting in faster training convergence and more stable early learning curves; (ii) Compared with the R³ method, which only implements the course in the position dimension and relies on sparse result rewards, the two-dimensional joint course and dense mixed rewards of this invention are organically combined, which significantly improves the completeness of course learning and adaptability to code agent tasks; (iii) The adaptive stage switching mechanism enables the course progress to automatically match the actual learning progress of the model, avoiding overfitting of course hyperparameters to specific datasets.
[0171] ③ Execution of perception step filtering and error target forward shifting mechanism Key technical points: Annotate each action in the teacher's trajectory within a sandbox environment and record the execution results. k ∈ {success, fail}; construct K normal training samples only for consecutive successful steps (truncate the window upon encountering a failure); for the failed step k, reject it as the training target and instead move the prediction target to the first subsequent successful step y.k+j (j ≥ 1), and retain the context of the failed steps in the prefix to construct the error target forward-moving special training samples.
[0172] Beneficial effects: (i) It fundamentally solves the problem of meaningless supervision of failed steps in the teacher's trajectory as training targets: directly learning known erroneous actions is not only useless, but also pollutes the model's strategy; shifting the erroneous target forward makes the supervision target of each training sample an effective action, and greatly improves the quality of the training signal; (ii) It preserves the context of failed steps in the prefix, enabling the model to learn how to overcome and respond correctly when encountering errors, and enhances the robustness and autonomous error correction ability of the code agent in real engineering scenarios.
[0173] Please see Figure 9 This application also provides a learning and training device for code-based intelligent agents, which can implement the above-described method. The device includes: The annotation module 901 is used to preprocess the expert code agent trajectory to obtain the annotation result, execute the annotation result based on the sandbox, and construct training samples for each expert code agent trajectory. The training samples include several expert sequences and normal samples and recovery samples. Difficulty module 902 is used to evaluate the step position difficulty and prediction span difficulty of the training samples. The joint difficulty function is obtained by weighting and combining the step position difficulty and prediction span difficulty. Stage module 903 is used to obtain the initial temperature and the current training step number, determine the current temperature based on the initial temperature and the current training step number, and determine the training stage of the training sample based on the current temperature; The reward module 904 is used to predict the predicted action sequence of the training samples. It obtains the text similarity reward by aggregating the similarity of the predicted action sequence through decreasing weights, and obtains the functional equivalence reward by comparing the predicted action sequence with the expert sequence. The text similarity reward and the functional equivalence reward are linearly fused to form a multi-step look-ahead hybrid reward. Training module 905 is used for group-relative policy optimization, which uses the training phase of training samples and multi-step look-ahead mixed reward to update the parameters of the large language model, resulting in the updated large language model. The iteration module 906 is used to increment the current training step number by one to obtain a new current training step number. Based on the new current training step number and the training step number threshold, the large language model is iterated to obtain the trained large language model. The trained large language model is then deployed to the code agent.
[0174] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0175] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0176] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0177] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the methods described in the embodiments of this application. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.
[0178] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0179] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0180] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0181] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0182] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0183] The learning and training method, apparatus, electronic device, storage medium, and program product for code-based intelligent agents provided in this application preprocess the expert code-based intelligent agent trajectory to obtain annotation results, execute the annotation results in a sandbox, construct training samples for each expert code-based intelligent agent trajectory, and construct training samples including several expert sequences, normal samples, and recovery samples, thereby enhancing the robustness and autonomous error correction ability of the code-based intelligent agent in real engineering scenarios. It evaluates the step position difficulty and prediction span difficulty of the training samples, and obtains a joint difficulty function by weighting the step position difficulty and prediction span difficulty, resulting in faster training convergence, more stable early learning curves, improved early training efficiency and parameter update effectiveness, and avoids overfitting of hyperparameters to specific datasets. It obtains the initial temperature and current training step number, determines the current temperature based on the initial temperature and current training step number, and determines the training stage of the training samples based on the current temperature, gradually shifting the training focus towards more difficult samples through temperature control, achieving a course progression from easy to difficult. It also predicts training samples. The predicted action sequence is aggregated with decreasing weights to obtain a text similarity reward. A functional equivalence reward is obtained by comparing the predicted action sequence with the expert sequence. The text similarity reward and the functional equivalence reward are linearly fused to form a multi-step look-ahead hybrid reward, expanding the reward coverage for reasonable actions, reducing errors in the model's reasonable exploration paths, and enabling dense reward signals to more fairly and accurately measure the actual quality of the code agent's actions. This also reduces noise in the reward signal caused by single-step random bias, making the policy gradient update direction more stable. Based on group-relative policy optimization, the parameters of the large language model are updated using the training phase of the training samples and the multi-step look-ahead hybrid reward, resulting in an updated large language model. The current training step number is incremented by one to obtain a new current training step number. Based on the new current training step number and a training step threshold, the large language model is iterated to obtain a trained large language model. The trained large language model is then deployed to the code agent, improving the signal quality, learning efficiency, and data utilization of the code agent's reinforcement learning training.
[0184] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0185] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0186] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0187] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0188] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0189] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0190] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0191] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0192] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0194] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A learning and training method for code-based intelligent agents, characterized in that, The method includes the following steps: The expert code agent trajectory is preprocessed to obtain the annotation result. The annotation result is executed based on the sandbox. Training samples are constructed for each expert code agent trajectory. The training samples include several expert sequences and normal samples and recovery samples. The step position difficulty and prediction span difficulty of the training samples are evaluated, and the joint difficulty function is obtained by weighting the step position difficulty and the prediction span difficulty. The initial temperature and the current training step number are obtained. The current temperature is determined based on the initial temperature and the current training step number. The training stage of the training sample is determined based on the current temperature. The current temperature is used for annealing sampling. Predict the predicted action sequence of the training samples, aggregate the similarity of the predicted action sequence by decreasing weight to obtain a text similarity reward, compare the predicted action sequence with the expert sequence to obtain a functional equivalence reward, and linearly fuse the text similarity reward and the functional equivalence reward to form a multi-step look-ahead hybrid reward. Based on group relative strategy optimization, the parameters of the large language model are updated using the training phase of the training samples and the multi-step look-ahead hybrid reward, resulting in the updated large language model. The current training step number is incremented by one to obtain a new current training step number. Based on the new current training step number and the training step number threshold, the large language model is iterated to obtain a trained large language model. The trained large language model is then deployed to the code agent.
2. The method according to claim 1, characterized in that, The process of preprocessing the expert code agent trajectory to obtain annotation results, executing the annotation results based on a sandbox, and constructing training samples for each expert code agent trajectory includes: Each action in the trajectory of the expert code agent is labeled, and the labeling results are recorded, including execution success and execution failure. Based on the labeled results, a number of normal samples are constructed for consecutive actions that have been successfully executed. If an action fails to be executed, the sample is truncated at the failed action. The length of the normal training sample is the actual number of consecutive successful steps. For actions that fail to execute, the failed actions are retained in the prefix context, and the first successfully executed action after the failed action is used as the recovery sample. The prefix context is used to train the model on how to respond correctly.
3. The method according to claim 1, characterized in that, The steps of obtaining the initial temperature and the current training step number, determining the current temperature based on the initial temperature and the current training step number, and determining the training stage of the training sample based on the current temperature include: The initial temperature and the current training step number are obtained, and the current temperature is determined based on the initial temperature and the current training step number. The current temperature decreases monotonically as the training progresses. The current average reward is calculated based on the validation set using the current temperature and the joint difficulty function. The training stage of the training sample is determined by the current average reward based on the stage switching reward threshold, which includes a high stage threshold and a low stage threshold. The training stage includes a warm-up stage, a transition stage, and a reinforcement stage.
4. The method according to claim 1, characterized in that, The prediction of the predicted action sequence of the training samples is performed by aggregating the similarity of the predicted action sequence with decreasing weights to obtain a text similarity reward. A functional equivalence reward is obtained by comparing the predicted action sequence with the expert sequence. The text similarity reward and the functional equivalence reward are linearly fused to form a multi-step look-ahead hybrid reward, including: The predicted action sequence is obtained by predicting the expert action sequence using a large language model. The text similarity reward is obtained by aggregating the similarity of the predicted action sequences of several steps by decreasing weights. Execute the expert action sequence and the predicted action sequence in the sandbox to obtain the file status and test pass rate after execution. Compare the file status with the test pass rate, and calculate the functional equivalence reward based on the file status and the test pass rate; The text similarity reward and the functional equivalence reward are linearly fused to form a multi-step look-ahead hybrid reward. If the aforementioned multi-step expert action sequence covers the end of the trajectory, then the multi-step forward hybrid reward degradation is determined as the final test pass rate reward.
5. The method according to claim 1, characterized in that, The method further includes: Obtain several large language models that have been trained; The task resolution rate of the large language model was evaluated using a validation set. The success rate of the large language model in recovering from erroneous actions was evaluated using a validation set. Based on the task resolution rate and the recovery success rate, the optimal large language model is selected, and the optimal large language model is deployed to the code agent.
6. The method according to any one of claims 1 to 5, characterized in that, The aforementioned steps constitute the predicted span. The evaluation of the step position difficulty and predicted span difficulty of the training samples, and the weighted combination of the step position difficulty and the predicted span difficulty to obtain a joint difficulty function, include: The difficulty of the step position in the training sample is evaluated by using the step position index of the training sample. The difficulty of predicting the span of the training samples is assessed by evaluating the predicted span of the training samples. The joint difficulty function is obtained by weighting the difficulty of the step location and the difficulty of the predicted span based on the weight hyperparameters.
7. A learning and training device for code-based intelligent agents, characterized in that, The device includes: The annotation module is used to preprocess the expert code agent trajectory to obtain annotation results, execute the annotation results based on the sandbox, and construct training samples for each expert code agent trajectory. The training samples include several expert sequences and normal samples and recovery samples. The difficulty module is used to evaluate the step position difficulty and the predicted span difficulty of the training sample, and to obtain a joint difficulty function by weighting the step position difficulty and the predicted span difficulty. The stage module is used to obtain the initial temperature and the current training step number, determine the current temperature based on the initial temperature and the current training step number, and determine the training stage of the training sample based on the current temperature. The reward module is used to predict the predicted action sequence of the training samples, aggregate the similarity of the predicted action sequence by decreasing weight to obtain a text similarity reward, compare the predicted action sequence with the expert sequence to obtain a functional equivalence reward, and linearly fuse the text similarity reward and the functional equivalence reward to form a multi-step look-ahead hybrid reward. The training module is used for group relative policy optimization, which uses the training phase of the training samples and the multi-step look-ahead mixed reward to update the parameters of the large language model, so as to obtain the updated large language model. An iteration module is used to increment the current training step number by one to obtain a new current training step number, and iterate the large language model based on the new current training step number and the training step number threshold to obtain a trained large language model. The trained large language model is used to deploy to the code agent.
8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.
