Multi-round automatic machine learning agent system based on reinforcement learning optimization

Abstract

The invention provides a multi-round automatic machine learning agent system based on reinforcement learning optimization. Comprising a task analysis module used for generating an initial prompt for an MLE agent to call; the MLE agent module is used for generating an executable code; the code executor is used for generating an execution result; the evaluator is used for outputting a normalized value of each index and a code correctness identifier; the reward construction module is used for generating a reward value; the reinforcement learning optimizer is used for calculating group average return and candidate advantages and updating strategy parameters of the MLE intelligent agent module based on the candidate advantages; and the multi-round interaction control module is used for feeding back the execution result of the previous round and the reward value to the MLE agent module in the multi-round interaction process, and controlling code generation of the next round until a preset termination condition is met. According to the invention, strategy adaptive evolution, reinforcement learning optimization of fine-grained credit distribution and multi-round closed-loop automatic process improvement can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-round automated machine learning agent system based on reinforcement learning optimization. BACKGROUND

[0002] In recent years, automated machine learning (AutoML) aims to reduce human intervention in feature engineering, model selection, hyperparameter tuning, etc., and to realize end-to-end modeling process automation. With large-scale language models (LLM) having the ability to integrate cross-domain knowledge and generate code, researchers have begun to try to use LLM as an agent to intervene in the AutoML process, drive code generation and process arrangement through natural language prompts, and thus achieve automation at a higher level. At the same time, the application of reinforcement learning (RL) in complex reasoning and multi-step decision-making tasks shows that continuous improvement of strategies using environmental feedback can improve the performance of LLM in reasoning, planning, etc.

[0003] Traditional AutoML solutions (such as Auto-sklearn, Auto-Keras) generally rely on pre-defined operator libraries and fixed pipeline structures to optimize in a static search space through grid search, random search or Bayesian optimization, and lack the ability to dynamically adjust strategies for different tasks. Recently emerging LLM-driven AutoML agents (such as AutoML-GPT, MLCopilot, and AIDE, SELA, etc. using a round-by-round trial-and-error strategy) can generate or iteratively modify machine learning process codes through prompts, achieving a certain degree of multi-round optimization, but their core decision logic usually relies on artificially designed prompt templates or heuristic rules, and the strategies remain static between different tasks. On the other hand, reinforcement learning optimization for LLM reasoning ability (such as DeepSeek-R1 and other works) often uses reinforcement learning to improve the quality of the reasoning link, but in the context of automated machine learning, which involves multi-round interaction and multiple sub-tasks, reinforcement learning has not yet formed a mature systematic integration framework. Existing multi-round decision-making RL methods often treat the entire reasoning process as a single-step decision, only relying on the final result for training, and ignoring the reward distribution of intermediate steps, resulting in rough credit distribution and sparse learning signals. SUMMARY

[0004] Therefore, the embodiments of the present application provide a multi-round automated machine learning agent system based on reinforcement learning optimization to solve the problems of static strategies and difficulty in self-optimization, sparse reward signals leading to rough credit distribution, and lack of multi-round closed-loop process optimization mechanism in the prior art.

[0005] The embodiment of the application provides a multi-round automatic machine learning intelligent agent system based on reinforcement learning optimization, which comprises: a task analysis module, which is used for receiving task description and constraint conditions input by a user and analyzing target indicators, resource limits and dataset information to generate an initial prompt for calling an MLE intelligent agent; an MLE intelligent agent module, which is used for receiving the initial prompt and multi-round feedback information based on a large-scale pre-trained language model to generate executable code; a code executor, which is used for running the executable code in a controlled execution environment, collecting running logs, model performance indicators, running time, resource occupation and error information to form an execution result; an evaluator, which is used for calculating multi-dimensional indicators of performance and running classes according to the execution result and outputting normalized values of the indicators and a code correctness identifier; a reward construction module, which is used for combining the normalized indicators and the code correctness identifier according to a preset rule to generate corresponding reward values; a reinforcement learning optimizer, which is used for calculating group average returns and candidate advantages for reward values of multiple candidate outputs generated by the MLE intelligent agent module for the same task input and updating policy parameters of the MLE intelligent agent module based on the candidate advantages; and a multi-round interaction control module, which is used for feeding back the execution result and the reward value of the last round to the MLE intelligent agent module in a multi-round interaction process to control code generation of the next round until a preset termination condition is met.

[0006] The above at least one technical solution adopted by the embodiment of the application can achieve the following beneficial effects: Through the task analysis module, the task description and constraint conditions input by the user are received and analyzed to obtain target indicators, resource limits and dataset information, and an initial prompt for calling the MLE intelligent agent is generated. The MLE intelligent agent module is used for receiving the initial prompt and multi-round feedback information based on a large-scale pre-trained language model to generate executable code. The code executor is used for running the executable code in a controlled execution environment, collecting running logs, model performance indicators, running time, resource occupation and error information to form an execution result. The evaluator is used for calculating multi-dimensional indicators of performance and running classes according to the execution result and outputting normalized values of the indicators and a code correctness identifier. The reward construction module is used for combining the normalized indicators and the code correctness identifier according to a preset rule to generate corresponding reward values. The reinforcement learning optimizer is used for calculating group average returns and candidate advantages for reward values of multiple candidate outputs generated by the MLE intelligent agent module for the same task input and updating policy parameters of the MLE intelligent agent module based on the candidate advantages. The multi-round interaction control module is used for feeding back the execution result and the reward value of the last round to the MLE intelligent agent module in a multi-round interaction process to control code generation of the next round until a preset termination condition is met. The application can realize reinforcement learning optimization of strategy adaptive evolution and fine-grained credit allocation and multi-round closed-loop automatic process improvement. BRIEF DESCRIPTION OF DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0008] Figure 1 is a schematic diagram of a framework of a multi-round automatic machine learning agent system based on reinforcement learning optimization provided by the embodiments of the present application; Figure 2 is a flowchart of a multi-round automatic machine learning method based on reinforcement learning optimization provided by the embodiments of the present application. DETAILED DESCRIPTION

[0009] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application, but it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0010] In recent years, automatic machine learning (AutoML) aims to reduce human intervention and automate the machine learning process. Traditional AutoML methods (such as Auto-sklearn, Auto-Keras, etc.) rely on pre-defined search spaces and static configurations, mainly using grid search, Bayesian optimization or fixed pipeline strategies, lacking multi-round iteration and flexible generation capabilities. Unlike this, using large-scale language models (LLM) as agents to automatically generate and optimize machine learning processes has become a new direction. For example, AutoML-GPT, MLCopilot, etc. generate complete pipelines through prompts, and AIDE, SELA, etc. optimize model design through trial and error search.

[0011] However, these LLM agents mainly rely on artificial prompt engineering and do not learn and optimize based on their own experience. Their strategies remain static across different tasks and cannot improve themselves from historical experience, severely restricting adaptability and efficiency. Reinforcement learning (RL) has been proven to significantly improve the ability of LLM in complex tasks, improving strategies through feedback-driven exploration and utilization. For example, DeepSeek-R1 and other works use reinforcement learning to train LLM to improve reasoning ability.

[0012] However, applying reinforcement learning to LLM agents to automate machine learning processes is still a blank. Existing RL methods for multi-round decision-making tasks often treat the entire reasoning process as a single-step decision, relying only on the final result for training, ignoring the intermediate rewards of each step, making it difficult for agents to distinguish the contribution of each round of operation to the final result. Existing research shows that introducing round-level rewards and fine-grained credit allocation can significantly improve the performance of multi-round reasoning tasks. Therefore, there is an urgent need for a new framework that combines LLM agents with reinforcement learning to optimize the design of strategies in a multi-round interactive environment through dynamic rewards.

[0013] In view of the problems existing in the prior art, the present application proposes a multi-round MLEAgent automatic machine learning system based on reinforcement learning optimization, which can autonomously select the best data processing scheme and feature engineering and the corresponding model for task input and data, and continuously optimize the parameters through multi-round iteration to ultimately obtain better prediction results. The technical scheme of the present application mainly includes the following contents: 1) MLE agent end-to-end process generation: a language model driven MLE agent is introduced, which can automatically generate and execute complete machine learning process code according to the input task prompt, covering feature engineering, model selection, hyperparameter tuning and model training steps. The code generated by the agent runs in the code execution environment to automatically complete the model development task.

[0014] 2) GRPO reinforcement learning optimization: the system uses the group relative policy optimization (GRPO) algorithm proposed by DeepSeek-R1 as the reinforcement learning framework. In each training, the MLE agent generates multiple candidate outputs for the same input, and calculates the advantage value according to the group return of these candidate schemes to update the policy model. Without additional value network, the policy optimization can be guided, greatly reducing the training overhead and improving the convergence efficiency.

[0015] 3) Multi-round interactive iterative optimization: the system supports multi-round interactive reasoning, and the agent generates or modifies code based on the current prompt and the feedback of the last round in each round, and obtains new feedback information after execution. Reinforcement learning is used to guide the MLE agent to improve decision-making in each iteration, making multi-round generation and optimization possible. This multi-round mode allows the agent to think and correct repeatedly like a human engineer, rather than outputting once, significantly improving the quality of the final scheme.

[0016] 4) Multi-dimensional reward signal construction: the evaluator performs multi-dimensional evaluation on the code execution results, and outputs performance indicators (such as accuracy, F1 score) and running performance indicators (such as execution time, resource utilization) of the model. The system combines these evaluation indicators with code correctness and other factors to construct a rule-based reward mechanism. For example, if the code is successfully executed and the model performance is improved, a positive reward is given; if there is a running error or performance degradation, a penalty signal is given. This rule-based reward signal ensures that the quality of the code and the effect of the model are considered in the reinforcement learning process, making the reward design comprehensive and controllable.

[0017] 5) Closed-loop training architecture: the system is composed of task analysis module, MLE agent, code executor, evaluator and reinforcement learning optimizer, etc. Through the closed loop of training and deployment, it is continuously iterated and optimized. The task analysis module parses the user input machine learning requirements into the initial prompt of the agent; the MLE agent generates code according to the prompt; the code executor executes the code and outputs the results; the evaluator evaluates the results and calculates the reward; the reinforcement learning optimizer updates the agent strategy using the reward signal, and the optimized agent can be deployed to new tasks, forming a continuous learning closed loop. The whole process can continuously evolve in the real environment, thereby enhancing the automation and intelligence of the system.

[0018] The technical solutions of the present application will be described in detail below in combination with the drawings and specific embodiments.

[0019] Figure 1 is the framework diagram of the multi-round automatic machine learning agent system based on reinforcement learning optimization provided by the embodiment of the present application. As Figure 1 shown, the multi-round automatic machine learning agent system based on reinforcement learning optimization can specifically include: The task analysis module 101 is used to receive the task description and constraint conditions input by the user and parse the target indicators, resource limitations and dataset information to generate an initial prompt for the MLE agent to call; The MLE agent module 102 is used to receive the initial prompt and multi-round feedback information based on a large-scale pre-trained language model, and generate executable code; The code executor 103 is used to run the executable code in a controlled execution environment, collect running logs, model performance indicators, running time, resource occupation and error information, and form execution results; The evaluator 104 is used to calculate multi-dimensional indicators of performance and running types according to the execution results, and output the normalized values of each indicator and the code correctness identifier; The reward construction module 105 is used to combine the normalized indicators and the code correctness identifier according to the preset rules to generate the corresponding reward value; The reinforcement learning optimizer 106 is configured to input a plurality of candidate outputs sampled by the MLE agent module for the same task, calculate the group average return and the candidate advantage of the reward value, and update the policy parameters of the MLE agent module based on the candidate advantage; The multi-round interaction control module 107 is configured to feed back the execution result of the last round and the reward value to the MLE agent module in the multi-round interaction process, control the code generation of the next round, and terminate until a preset termination condition is met.

[0020] In some embodiments, the task analysis module is specifically configured to: perform semantic analysis and field identification on the task description and the constraint condition, and extract key parameters corresponding to the target index, the resource limit, and the dataset information; organize the key parameters into a structured expression according to a preset field mapping rule and a prompt template; perform format checking and parameter normalization processing on the structured expression to obtain an initial prompt meeting the calling requirements.

[0021] Specifically, after receiving the task description and the constraint condition, the task analysis module first performs semantic analysis and field identification to locate the key parameters corresponding to the target index, the resource limit, and the dataset information; then organizes the key parameters according to the preset field mapping rule and the prompt template to generate a structured expression with uniform field names and hierarchical structure; then performs format checking and parameter normalization processing on the structured expression to make the parameter values meet the uniform value range, consistent time and resource units, and default items filled or set to default values; finally outputs an initial prompt meeting the calling requirements and passes it to the MLE agent module.

[0022] Some key technical terms involved in the present embodiment are explained as follows: Semantic analysis and field identification: refers to converting free text in the task description into recognizable semantic units based on natural language understanding technology, and mapping them to a predefined field set, such as "target index", "resource limit", "dataset information", etc.

[0023] Key parameters: refer to parameter items that directly affect the subsequent machine learning process generation and constraint control, such as accuracy upper limit, training time upper limit, memory upper limit, dataset name and storage path, etc.

[0024] Field mapping rule: refers to a rule set for establishing a mapping relationship between natural language expressions and standard field names and their value analysis methods, such as mapping "as high as possible accuracy" to "accuracy" as the main optimization indicator, and analyzing "training time not exceeding 30 seconds" to the training time upper limit field and converting it to seconds.

[0025] Prompt Template: refers to the prompt structure framework prepared for the MLE agent, usually including task target section, constraint condition section, data description section and optional output requirement section, each section uses fixed placeholders waiting for key parameter filling.

[0026] Structured Expression: refers to the internal representation of the identified key parameters organized in a hierarchical, key-value pair or other standardized form, which facilitates verification and direct calling of downstream modules.

[0027] Format Verification: refers to the process of checking whether the fields in the structured expression are complete, the types are matched, the numerical values are within the range, and the units are consistent.

[0028] Parameter Standardization Processing: refers to the conversion of parameter values to a unified standard, such as converting "30 seconds", "0.5 minutes" and "≤30s" to "30 seconds", and converting "accuracy" and "F1 score" to a pre-defined index name.

[0029] Initial Prompt: refers to the prompt text or equivalent calling data structure formed by filling the standardized key parameters into the prompt template, which can be directly input into the MLE agent module.

[0030] In a specific example, the user input task description is: "Please build a classification model for breast cancer dataset based on sklearn, the goal is to maximize accuracy, and the model training time should not exceed 30 seconds." (1) Semantic Analysis and Field Recognition Stage, the task analysis module first performs syntactic division and keyword extraction on the text, identifies the dataset information corresponding to "breast cancer dataset", the target index for "maximize accuracy" as accuracy, and the upper limit of training time in resource constraints corresponding to "training time not exceeding 30 seconds"; At the same time, it identifies the tool stack preference as "sklearn".

[0031] (2) Field Mapping Stage, the module maps "breast cancer dataset" to the standard identifier "BreastCancer Wisconsin Dataset" according to the field mapping rules; maps "maximize accuracy" to the main optimization index field accuracy; converts "training time not exceeding 30 seconds" to the training time upper limit field and records the value 30, with the unit unified as seconds; "Based on sklearn" as the model development library preference field is written into the environment constraints part of the structured expression.

[0032] (3) After generating the structured expression, the module performs format verification to confirm that all required fields have been filled, there is no out-of-range situation for numerical fields, and string fields meet the coding specifications; then performs parameter standardization processing, such as converting the accuracy identifier to "accuracy_score" and converting the time field to an integer number of seconds.

[0033] (4) In the prompt template filling stage, the module inserts the above-mentioned normalized parameters into the corresponding positions according to the template requirements to form an initial prompt containing a task target segment (improve accuracy), a resource limit segment (training time ≤ 30 seconds), a data description segment (use the Breast Cancer Wisconsin Dataset and indicate that it can be loaded through sklearn), and a tool stack constraint segment (preferentially use the sklearn library). The initial prompt is stored in a unified data structure internally and is sent to the MLE agent module in text form to drive the subsequent process of generating executable code.

[0034] In another example, if the user does not explicitly specify the data set position or give the resource limit, the task parsing module will fill in the default items in the parameter normalization process according to the preset default value strategy, for example, the default upper limit of training time is 60 seconds and the default main optimization indicator is accuracy, to ensure the integrity of the initial prompt. If the user input contains multiple target indicators (such as paying attention to both accuracy and F1 score), the module will record the corresponding weight proportion in the structured expression according to the weight rule and explicitly indicate the multi-indicator optimization requirement in the prompt template.

[0035] Through the above steps, the task parsing module converts the task requirements in free text into standardized initial prompts without relying on manual adjustment, laying a complete input foundation for the subsequent multi-round code generation and reinforcement learning optimization of the MLE agent module.

[0036] In some embodiments, the MLE agent module is specifically used for: performing semantic analysis on the initial prompt and the multi-round feedback information to determine the machine learning process elements to be generated or modified; generating code segments corresponding to the process elements according to the preset code generation rules and operator flow template set; processing the code segments to generate complete executable code; in the subsequent rounds, positioning the code position to be adjusted according to the execution result and reward value of the last round and performing incremental generation or replacement, and outputting the updated executable code to the code executor.

[0037] Specifically, the MLE agent module receives an initial prompt and multi-round feedback information based on a large-scale pre-trained language model, first performs semantic analysis on the two to extract machine learning process elements to be generated or modified, then generates corresponding code snippets for each process element according to the preset code generation rules and operator process template set, and performs combination, parameter filling, dependency completion and syntax consistency checking on the code snippets after generation to form complete executable code. In subsequent rounds, the module locates the code position that needs to be adjusted according to the last round execution result and reward value, updates the corresponding fragment using incremental generation or replacement, and outputs the updated executable code to the code executor to complete the next round of interaction.

[0038] The concepts of some key technical terms involved in the present embodiment are explained as follows: Multi-round feedback information: refers to the execution results, reward values and normalized indicator sets output by the evaluator generated by multiple rounds of interaction, which are used as input basis for the subsequent generation or modification of code by the agent.

[0039] Machine learning process elements: refer to the core links and configuration items in the machine learning process, including feature engineering steps, model category selection, hyperparameter configuration, training and validation division, evaluation index calculation, etc.

[0040] Code generation rules: refer to the generation constraints and splicing logic formulated for different process elements, such as input and output variable naming rules, operator call order, parameter value range limitation, exception handling structure, etc.

[0041] Operator process template set: refers to abstracting commonly used feature processing operators, model construction operators, evaluation operators, etc. into reusable template fragments, and marking the input and output interfaces and adjustable parameter positions for the agent to call and fill.

[0042] Code snippet processing: includes connection relationship analysis between code snippets, variable scope unification, dependency library declaration completion, redundant fragment removal, semantic consistency check, and execution order rationalization, etc.

[0043] Incremental generation or replacement: refers to generating a new local fragment or replacing the original fragment locally only for the located adjustment position on the premise of preserving effective fragments, avoiding repeated generation of the entire code, reducing rewriting cost and ensuring context consistency.

[0044] Locating the code position to be adjusted: refers to mapping the change range to the corresponding process element and then to its code snippet position by comparing the last round execution result and reward value, combining the key fields in the feedback data packet, and realizing targeted modification.

[0045] For example, in one specific example, the initial prompt indicates that a classification model needs to be built on the Breast Cancer Wisconsin Dataset, with accuracy as the primary target and a training time limit of no more than 30 seconds.

[0046] First, the MLE agent module performs semantic analysis on the initial prompt and identifies data loading, data division, model selection, feature engineering, training and evaluation, and other process elements. The module retrieves the standard data loading template, training set and test set division template, and commonly used model template from the operator process template set suitable for classification tasks, and selects a model type with appropriate computational load according to the resource limit information in the prompt. According to the code generation rules, the module fills in the key parameters from the initial prompt for the parameter positions in each template, such as mapping the upper limit of training time to the timing monitoring structure, and adding accuracy as the primary indicator to the indicator calculation list in the evaluation link. Subsequently, the module performs combination and dependency completion on the generated segments, unifies the naming of the output variables of the data loading segment and the input variables of the model training segment, ensures the continuity of variable flow, and supplements the missing library import statements and exception capture structures. After completing the combination, the module performs syntax consistency checking on the overall executable code to confirm that there are no spelling errors, undefined variables, duplicate imports, etc., and then sends the executable code to the code executor.

[0047] After the completion of this round of execution, the evaluator outputs the execution results and reward values. Suppose the reward value shows that the model performance is acceptable but lacks a feature standardization step, and the training time record is missing. After receiving the multi-round feedback information, the MLE agent module first locates the process elements to be adjusted according to the missing link indicated in the feedback data packet, i.e., feature engineering and running time record. The module then finds the corresponding code segment location through the mapping relationship, calls the template segment of feature standardization and the time record segment, and inserts them into the position before model training according to the variable naming and data flow of the existing code, maintaining the input and output interface consistency with the original segment. The incrementally generated segment replaces the original non-standardization process, and the module performs dependency completion and syntax checking again, and the updated executable code is sent to the code executor again.

[0048] When the second round of execution results returns, suppose the reward value continues to prompt to reduce the number of features without reducing performance. The MLE agent module locates the feature engineering process element, selects the feature selection operator template, and performs feature selection on the original standardized output variable, and updates the input variable reference in the model training segment. The module replaces the relevant segments in an incremental manner, keeping the other parts unchanged, and then outputs the executable code again. Thereafter, with the continuation of multi-round interaction, the module locates the modification range according to the feedback, calls the corresponding template and performs incremental generation at each round, until the multi-round interaction control module judges that the preset termination condition is reached.

[0049] In another example, if the multi-round feedback information shows that there is a dependency missing or a syntax error, the MLE agent module parses the error type and error stack information into the syntax or dependency element to be modified, calls the error repair template or dependency completion template, inserts the necessary import statement, exception capture structure, and processes the error paragraph in priority during incremental generation. In this way, the MLE agent module performs semantic-driven code generation and updating based on process elements in a multi-round interaction scenario, ensuring the integrity and consistency of executable code.

[0050] In some embodiments, the code executor is specifically used for: loading the dependent libraries and data resources required by the executable code, and initializing the execution container according to the preset resource quota; triggering the execution of the executable code and performing event listening during the execution process to record the running log; calculating the indicators according to the preset evaluation interface to obtain the model performance indicators; statistically calculating the running time and resource occupation by timestamp markers and resource monitoring probes respectively; generating error information by classifying the exception types, error codes and interrupt states captured during the execution process; summarizing and packaging the running log, model performance indicators, running time, resource occupation and error information into an execution result according to the preset data structure.

[0051] Specifically, after receiving the executable code output by the MLE agent module, the code executor first loads the dependent libraries and data resources required by the code, and initializes the execution container according to the preset resource quota; then triggers the code to run and performs event listening during the entire execution process, continuously records the standard output, standard error and exception stack information to form the running log; calculates the indicators according to the preset evaluation interface to obtain the model performance indicators; statistically calculates the running time and resource occupation by timestamp markers and resource monitoring probes respectively; generates error information by classifying the exception types, error codes and interrupt states captured during the execution process; finally, the running log, model performance indicators, running time, resource occupation and error information are summarized and packaged into an execution result according to the preset data structure and output to the evaluator.

[0052] The concepts of some key technical terms involved in this embodiment are explained as follows: Execution container: refers to an isolated environment for running executable code, which loads dependent libraries, configures data paths and applies resource quotas such as CPU, memory, time length during initialization.

[0053] Event Listening: Refers to the mechanism of capturing and recording key events (such as stage start / end, exception throwing, resource peak change) during code execution, used to form a complete running log.

[0054] Pre-set Evaluation Interface: Refers to the agreed specification for index calculation between the code executor and the evaluator, including input and output formats, calling timing, and index name set. The executor directly calculates or collects model performance indicators according to this interface.

[0055] Timestamp Marking: Refers to recording a uniform format timestamp at key nodes such as code startup, training start, training end, and prediction output, to accurately obtain running time.

[0056] Resource Monitoring Probe: A monitoring process or hook deployed in the execution container, used to periodically collect resource data such as CPU occupancy, memory usage, and disk read / write volume.

[0057] Error Information: Includes exception types (syntax error, dependency missing, interruption during execution, etc.), error codes, and interruption status captured during execution, used for subsequent feedback and penalty strategy construction.

[0058] Pre-set Data Structure: Refers to the uniform format definition used to encapsulate execution results, including field name, nesting level, and serialization rules, to ensure that the evaluator can directly parse.

[0059] For example, in a specific example, the executable code output by the MLE agent module is used to train a classification model on the BreastCancer Wisconsin Dataset and calculate the accuracy. After the code executor receives this executable code, it performs the following operations: First, the execution container initialization phase loads the required dependent libraries and data resources. The code executor sets the CPU core limit and memory quota in the container according to the resource limit information passed by the task parsing module, and mounts the data set path or calls the data loading interface to ensure that the data can be accessed by the code. If there are libraries with unspecified versions in the dependent libraries, the execution container selects compatible versions according to the pre-set version locking strategy and installs them during initialization.

[0060] Subsequently, the code execution is triggered. The event listening sub-module starts working, capturing standard output, standard error, and exception information in chronological order and writing them into the running log buffer. For example, when training starts, the "training phase starts" event and corresponding timestamp are recorded, and when model prediction ends, the "prediction phase completes" event and corresponding timestamp are recorded. If there is a dependency missing or data reading exception during execution, the event listening sub-module will capture the exception stack information and error code together and classify them into the error information cache.

[0061] After the model training and prediction phase ends, the code executor calculates the output results according to the preset evaluation interface. For classification tasks, the executor calls the performance index calculation unit to obtain accuracy, F1 score, precision, recall, and other performance indicators, and the results are stored in the model performance index field.

[0062] At the same time, the timestamp marking unit calculates the training time by reading the training start and end timestamps, and calculates the total running time according to the overall execution start and end time; the resource monitoring probe periodically collects CPU occupancy and memory usage during code execution and calculates the peak and average values, and these data are sorted into the resource occupation field.

[0063] When the execution flow ends, the error information classification unit classifies the exception types captured during the entire running period, such as classifying syntax errors as compilation errors, classifying missing dependencies as environment errors, and classifying timeout termination as resource errors, and records the corresponding error codes and interruption states.

[0064] Finally, the execution result packaging unit aggregates, renames, and serializes the running log, model performance index, running time, resource occupation, and error information according to the preset data structure, forms an execution result object, and outputs it to the evaluator. The evaluator then calculates the normalized indicators based on the execution result and generates the reward input.

[0065] In another example, if resource occupation is detected close to the upper limit of the quota during execution and is about to trigger forced termination, the resource monitoring probe sends a warning event to the event listening submodule, and the code executor can extend the monitoring period within the allowed range or trigger an early termination strategy, and record the "resource warning" state in the error information. Such information will be transmitted back to the MLE agent module by the multi-round interaction control module in subsequent rounds of feedback, so that it actively reduces the model complexity or reduces the feature dimension when generating code in the next round.

[0066] Through the above process, the code executor completely collects and packages various information during the execution process under the premise of ensuring that the execution environment is controlled, providing a reliable data basis for the evaluator and reinforcement learning optimizer.

[0067] In some embodiments, the evaluator is specifically used for: performing field analysis and data cleaning on the execution results to obtain an original index data set; comparing the model prediction output and the labeled data according to the preset performance index calculation rule to calculate performance indicators; aggregating the time records and resource monitoring data according to the preset running index statistical rule to calculate running indicators; determining the code correctness identifier according to the exception type, error code, and execution interruption state in the error information; The performance type indicators and the running type indicators are normalized according to a unified normalization strategy to generate a normalized indicator set.

[0068] Specifically, after receiving the execution result output by the code executor, the evaluator first performs field analysis and data cleaning on the running log, model prediction output, time record, resource monitoring data and error information in the execution result to form an original indicator data set; then compares the model prediction output with the labeled data according to a preset performance indicator calculation rule to obtain performance type indicators, and aggregates the time record and resource monitoring data according to a preset running indicator statistical rule to obtain running type indicators; then generates a code correctness identifier according to the abnormal type, error code and execution interruption state in the error information; finally, the performance type indicators and the running type indicators are normalized according to a unified normalization strategy to obtain a normalized indicator set and output to the reward construction module.

[0069] The concepts of some key technical terms involved in the embodiment are explained as follows: Field analysis and data cleaning: refers to identifying, disassembling and formatting the various original fields in the execution result, while eliminating missing values, duplicate records or abnormal format entries, to ensure the consistency and integrity of the subsequent indicator calculation input.

[0070] Original indicator data set: refers to the performance type and running type basic data set obtained after field analysis and data cleaning, which has not been normalized or weighted combined, including prediction result array, true label, time interval record, resource occupation sequence and error record entry, etc.

[0071] Preset performance indicator calculation rule: refers to the indicator calculation formula and process formulated for classification, regression or other task types, such as the calculation method of accuracy, F1 score, mean square error, etc. and its input and output specification.

[0072] Preset running indicator statistical rule: refers to the rule for statistical calculation of running time, memory usage, CPU occupancy, etc. running type data, including peak value calculation, mean value calculation, interval summation, unit conversion, etc.

[0073] Code correctness identifier: refers to the state identifier for judging whether the code is successfully executed according to the error information, which can include "executable", "execution exception", "partial interruption", etc. and is used for quickly screening invalid candidates in reward construction.

[0074] Unified normalization strategy: refers to using a unified normalization method (such as interval scaling, logarithmic transformation or piecewise linear mapping) for indicators of different dimensions and ranges, so that each indicator can participate in subsequent weight combination on the same scale.

[0075] For example, in one specific example, the execution result output by the code executor contains the following data: the running log records the key events and standard output information of the training phase and the prediction phase, the model prediction output is a set of binary classification prediction labels, the real label data is stored by the data loading process, the time record contains the total execution start and end time and the training phase start and end time, the resource monitoring data is the CPU occupancy rate and memory usage sequence collected every second during execution, and the error information field is empty or only records alarm information.

[0076] The evaluator first performs field parsing and data cleaning on the execution result, removes redundant text in the running log that is irrelevant to the indicators, and only retains the timestamp, phase marker, and exception summary; performs alignment checking on the prediction output and real label to ensure consistency in the number of samples and remove possible null entries; and performs missing value interpolation and unit unification on the resource monitoring sequence, and resamples data of different time granularities to a uniform time step. After completing the above processing, the evaluator forms the original indicator dataset.

[0077] In the performance indicator calculation stage, the evaluator compares the prediction output and real label according to the preset rules of the classification task, and calculates the accuracy and F1 score. The accuracy calculation uses the formula of the number of correctly predicted samples divided by the total number of samples, and the F1 score uses the harmonic mean formula of precision and recall. If the task type is regression, the evaluator will automatically switch to rules such as mean square error and mean absolute error.

[0078] In the running indicator calculation stage, the evaluator calculates the training time and total running time according to the start and end times marked in the time record; and calculates the peak and average values of CPU occupancy rate and memory usage according to the resource monitoring sequence. All running indicators are represented in their original dimensions in this stage, such as seconds, percentages, and megabytes.

[0079] In the code correctness identification determination stage, the evaluator checks the error information field. If no exception type or error code is detected and the execution is not interrupted, the identification is set to executable; if a dependency is missing, a syntax error is detected, etc., it is set to execution exception; if a forced termination occurs due to resource exhaustion in the later running period, it is set to partial interruption and the corresponding error code is recorded.

[0080] In the normalization stage, the evaluator converts the performance indicators and running indicators according to a unified normalization strategy. Taking accuracy and F1 score as an example, the numerical value itself can be used directly or scaled to the range of 0 to 1 according to the interval; for indicators such as running time and memory occupancy, the reverse normalization strategy (the shorter the time, the higher the value) or linear mapping through a previously set maximum value can be used. During the normalization process, the evaluator applies the same processing flow to all indicators to ensure comparability between different tasks. Finally, the evaluator forms a normalized indicator set and outputs it to the reward construction module together with the code correctness identification.

[0081] In another example, if there are a large number of abnormal logs in the execution result and the error information field contains multiple error types, the evaluator can prioritize the code correctness identification according to the error type priority, such as syntax errors being prioritized over resource warnings, and runtime interruptions being prioritized over poor performance. In the normalization strategy, the evaluator can append a specific placeholder value to the abnormal state or directly set the normalization result to zero, so as to impose a penalty on the candidate output in the reward construction stage.

[0082] Through the above process, the evaluator completes the calculation, normalization and code correctness identification of performance and running indicators under the premise of ensuring the consistency of input data, providing uniform and directly usable indicator input for the reward construction module.

[0083] In some embodiments, the reward construction module is specifically configured to: call a preset indicator weight mapping table and a combination rule library to assign weight coefficients to each normalized indicator and determine combination operators; divide the normalized indicators into performance and running categories to generate corresponding sub-reward components; calculate a penalty factor or an addition factor according to the code correctness identification; perform weighted combination and piecewise function operation on the sub-reward components and the penalty factor or the addition factor to obtain an initial reward value; interval clipping and quantization encoding are performed on the initial reward value to form a reward value.

[0084] Specifically, after receiving the normalized indicator set and the code correctness identification output by the evaluator, the reward construction module first calls a preset indicator weight mapping table and a combination rule library to assign weight coefficients to each normalized indicator and determine combination operators; then, according to the indicator type, the normalized indicators are divided into two groups of performance and running categories, and the corresponding sub-reward components are calculated; then, a penalty factor or an addition factor is generated according to the code correctness identification; on this basis, weighted combination and piecewise function operation are performed on the sub-reward components and the penalty factor or the addition factor to obtain an initial reward value; finally, interval clipping and quantization encoding are performed on the initial reward value to form a reward value and output to the reinforcement learning optimizer.

[0085] The concepts of some key technical terms involved in the present embodiment are explained as follows: Indicator weight mapping table: a set of weight configurations preset for each normalized indicator, which can be dynamically adjusted according to task priority, scene type or historical experience, and is used to allocate contribution proportion in weighted combination.

[0086] Combination rule library: refers to a set of rules for defining the combination method between different indicators, including linear weighting, piecewise function, multiplicative penalty, threshold truncation, etc.

[0087] Sub-reward component: refers to the performance class sub-reward component and the running class sub-reward component calculated according to the type of indicators, used to maintain the independent controllability of different categories of indicators before final combination.

[0088] Penalty factor or additive factor: refers to the coefficient determined according to the code correctness identifier, used to apply negative punishment or positive addition in reward calculation, such as reducing the overall reward to a minimum value when there is an execution error, or giving a small addition when the code is executable and structurally complete.

[0089] Piecewise function operation: refers to the function operation of setting different calculation formulas according to the value interval or threshold of the indicator, for example, using a steeper penalty segment when the performance class sub-reward component is below a certain threshold, and using a flat segment when it is above the threshold.

[0090] Interval clipping and quantization coding: refers to limiting the initial reward value within a preset upper and lower limit range, and quantizing the continuous value to a specified precision or discrete level according to the input requirements of the reinforcement learning optimizer, in order to stabilize the training.

[0091] For example, in one specific example, the normalized indicator set output by the evaluator includes performance class indicators accuracy_norm, f1_norm, running class indicators time_norm, memory_norm, and code correctness identifier code_ok. The reward construction module performs the following process: First, the module reads the corresponding weights from the indicator weight mapping table, such as accuracy_norm corresponding to weight w_acc, f1_norm corresponding to weight w_f1, time_norm corresponding to weight w_time, and memory_norm corresponding to weight w_mem. This weight mapping table can be dynamically adjusted according to the target indicator priority provided by the task analysis module, for example, when the task only emphasizes accuracy, w_acc can be set to the highest. The combination rule library indicates that the performance class indicators are linearly weighted and summed, and the running class indicators are inversely weighted (the shorter the time, the lower the resources, the higher the score) and then linearly combined.

[0092] Subsequently, the module combines accuracy_norm with f1_norm to obtain a performance class sub-reward component R_perf, and combines time_norm with memory_norm to obtain a running class sub-reward component R_run. In this process, a segmented function is used for the running class indicators: if time_norm is less than a preset threshold T_low, an additional bonus segment is given, if it is greater than a threshold T_high, it enters the penalty segment, and in the middle range, a linear segment is used to ensure that the time constraint is reasonably reflected.

[0093] Next, the module generates a penalty factor or a bonus factor according to code_ok. If code_ok is an executable state, the factor is 1 or a bonus value slightly greater than 1; if code_ok identifies an execution exception, the factor is directly set to a penalty value close to 0; if it is a partial interruption, it is set to a reduction coefficient between 0 and 1.

[0094] After that, the module combines the sub-reward components and the penalty factor. Specifically, first, perform weighted summation on R_perf and R_run according to the operators defined in the combination rule library to obtain an intermediate value R_mid; then multiply R_mid by the penalty factor, and according to the need, smooth the extreme values through a segmented function to form an initial reward value R_init.

[0095] Finally, the module performs interval clipping on R_init, limiting it to [0, 1] or other preset ranges, and quantization encoding according to the input requirements of the reinforcement learning optimizer, such as retaining the last two digits after the decimal point or mapping to 0~100 integer levels, to form the final reward value R. The reward value and the identifier of the corresponding candidate output are output to the reinforcement learning optimizer for subsequent group average return and candidate advantage calculation.

[0096] In another example, when the task contains multiple objectives and there is a need for dynamic weight adjustment, the reward construction module updates the indicator weight mapping table according to the latest task feedback or policy convergence before each iteration. For example, when the running time exceeds the limit multiple times, the module automatically increases the weight of the running class indicators and reduces the weight of the performance class indicators, so that subsequent reward calculation pays more attention to resource constraints; if the running class indicators are then stable within the threshold, the weights are gradually restored or redistributed. Such dynamic adjustment is defined in the form of policy entries in the combination rule library, and the module switches according to the entry instructions when executing.

[0097] Through the above process, the reward construction module ensures the uniformity of the input indicators and provides the reinforcement learning optimizer with controllable and discriminative reward value inputs, providing fine-grained quantitative basis for multiple rounds of policy updates.

[0098] In some embodiments, the reinforcement learning optimizer is specifically used for: receive the reward values and group the candidate outputs according to the sampling batch; statistically process the reward values in the candidate output group to obtain a group average return value; differ the reward value of each candidate output from the group average return value to obtain a corresponding candidate advantage value; process the candidate advantage values to generate a processed candidate advantage set; associate the processed candidate advantage set with the generation probability of the corresponding candidate output.

[0099] Specifically, after receiving the reward values output by the reward construction module, the reinforcement learning optimizer first groups multiple candidate outputs according to the same task input or the same sampling batch to establish a candidate output group; then performs statistical operations on the reward values in the group to obtain a group average return value; then differentiates the reward value of each candidate output from the group average return value to obtain a candidate advantage value; processes the candidate advantage values by clipping, normalization or weight adjustment to form a processed candidate advantage set; finally, one-to-one associates the processed candidate advantage set with the probability of the corresponding candidate output when generated, providing input for subsequent policy parameter update.

[0100] The concepts of some key technical terms involved in this embodiment are explained as follows: Candidate output group: refers to a set of multiple executable code schemes generated by the MLE agent module for the same task input in one policy update cycle, used for relative comparison within the group.

[0101] Group average return value: refers to the statistical mean of all reward values in the candidate output group, used to characterize the overall level of the group.

[0102] Candidate advantage value: refers to the difference between the reward value of a single candidate output and the group average return value, used to measure the degree to which the candidate output is better or worse than the group average level.

[0103] Processed candidate advantage set: refers to the advantage value set obtained after clipping (to prevent extreme values from affecting training), normalization (to unify the scale) or weight adjustment (to emphasize certain samples) of the original candidate advantage values.

[0104] Generation probability: refers to the strategy probability value given by the MLE agent module when generating the corresponding candidate output, used for probability ratio or importance sampling factor in subsequent policy gradient calculation.

[0105] One-to-one association: refers to mapping each processed candidate advantage value with its corresponding candidate output generation probability and identification at the data structure level, ensuring accurate indexing in the subsequent gradient update stage.

[0106] For example, in one specific example, for a certain classification task input, the MLE agent module generates 3 candidate executable codes in one sampling process, and the reward construction module outputs the corresponding reward values R1=0.50, R2=0.85, R3=0.92. The reinforcement learning optimizer performs the following operations: First, the candidate output grouping stage, the optimizer establishes the above 3 candidate outputs as a candidate output group G={O1, O2, O3} in the same batch, and records the probability p1, p2, p3 of each candidate output when it is generated.

[0107] Subsequently, the group average return statistics stage, the optimizer averages R1, R2, R3 to obtain the group average return value μ≈0.76.

[0108] Next, the candidate advantage calculation stage, the optimizer calculates A1=R1-μ, A2=R2-μ, A3=R3-μ, respectively, to obtain the original candidate advantage value set A={A1, A2, A3}.

[0109] In the candidate advantage processing stage, the optimizer performs clipping and normalization operations on the set A. For example, when |Ai| exceeds the preset threshold, it is truncated to within the threshold range; then all Ai are scaled to the maximum absolute value or standard deviation to obtain the processed candidate advantage setÂ={Â1,Â2,Â3}. If high reward samples need to be emphasized, positive advantage samples can be amplified and negative advantage samples can be smoothed in this stage.

[0110] Finally, the advantage probability association stage, the optimizer maps eachÂi with the corresponding generation probability pi and candidate output identifier Oi to form a set of association entries like (Â1, p1, O1), (Â2, p2, O2), (Â3, p3, O3) for subsequent direct calling by the policy parameter update unit. In this process, the optimizer caches the set in a temporary storage structure and cleans up or archives it after parameter update is completed.

[0111] In another example, if the number of candidate outputs is greater than the preset upper limit or the reward distribution is severely skewed, the reinforcement learning optimizer can first perform quantile screening on the reward values to remove abnormal extreme samples, and then calculate the group average return value to reduce the influence of abnormal values on advantage estimation. For the case where there are multiple task inputs within a batch, the optimizer will first establish candidate output groups according to task inputs, and then calculate the average return and advantage value for each group independently to avoid cross-task interference.

[0112] Through the above process, the reinforcement learning optimizer generates a candidate advantage set and probability mapping that can be directly used for policy update under the premise of ensuring that the candidate sample group is relatively reasonable, providing an accurate basis for subsequent gradient calculation and parameter synchronization.

[0113] In some embodiments, the reinforcement learning optimizer is further configured to: receive the candidate advantage value and the generation probability of the corresponding candidate output; build a policy gradient objective function with the generation probability and the candidate advantage value as independent variables; superimpose a probability ratio clipping term and / or a KL divergence-based regularization term in the objective function to obtain a final loss function; perform backpropagation on the final loss function to calculate the gradient; update the policy parameters according to a preset optimization algorithm, and perform gradient clipping and learning rate adjustment during the updating process; write the updated policy parameters to the MLE agent module and complete parameter synchronization.

[0114] Specifically, after receiving the candidate advantage value and the generation probability of the corresponding candidate output, the reinforcement learning optimizer first builds a policy gradient objective function with the generation probability and the candidate advantage value as independent variables; then superimposes a probability ratio clipping term and / or a KL divergence-based regularization term in the objective function to form a final loss function; then performs backpropagation on the final loss function to calculate the gradient, and updates the policy parameters according to a preset optimization algorithm, while performing gradient clipping and learning rate adjustment during the updating process; finally, writes the updated policy parameters to the MLE agent module and completes parameter synchronization.

[0115] The concepts of some key technical terms involved in this embodiment are explained as follows: Policy gradient objective function: refers to a target function established with the policy probability of the candidate output when generated and the corresponding candidate advantage value as input variables, used to achieve policy optimization through gradient ascent or descent.

[0116] Probability ratio clipping term: refers to a term that clips the ratio of the current policy probability to the old policy probability, used to limit the update amplitude and avoid large deviation of the policy.

[0117] KL divergence-based regularization term: refers to a constraint or penalty term of the KL divergence between the current policy and the old policy added in the loss function, used to maintain the stability of policy updating.

[0118] Final loss function: refers to an optimization target obtained by superimposing a probability ratio clipping term and / or a KL divergence regularization term on the basis of a policy gradient objective function, used to guide backpropagation.

[0119] Backpropagation: refers to the process of performing chain rule derivation on policy network parameters according to the final loss function, used to calculate parameter gradients.

[0120] Gradient Clipping: refers to the operation of limiting the gradient magnitude within a pre-set threshold range before parameter update to prevent gradient explosion.

[0121] Learning Rate Adjustment: refers to the strategy of dynamically adjusting the learning rate size according to the training stage or gradient changes during optimization, such as using decay or adaptive adjustment.

[0122] Parameter Synchronization: refers to the process of writing the updated policy parameters back to the MLE agent module and ensuring parameter consistency in multi-process or distributed environments.

[0123] For example, in a specific example, the reinforcement learning optimizer has obtained the processed candidate advantage set Â={Â1,Â2,Â3} and the corresponding candidate output generation probability set P={p1,p2,p3} calculated in the previous stage. The optimizer first constructs a policy gradient term for each pair (Âi,pi), taking pi as the probability under the old policy and the corresponding probability p'i output by the current policy network as the new policy probability, and multiplying p'i / pi with Âi to form the basic target term.

[0124] Subsequently, the optimizer reads the old policy probability distribution from the history buffer, calculates the KL divergence between the current policy distribution and the old policy distribution, and adds the KL divergence regularization term in the loss function; at the same time, according to the pre-set truncation threshold, the upper and lower limits of p'i / pi are executed to form the probability ratio truncation term, so that the final loss function contains two parts: "truncated probability ratio x candidate advantage" and "KL divergence regularization".

[0125] Next, the optimizer performs backpropagation on the final loss function to obtain the gradient vector of each layer parameter of the policy network. If it is detected that any gradient magnitude exceeds the pre-set threshold, the optimizer immediately performs gradient clipping to scale the gradient within the threshold range to avoid instability in parameter update. After completing gradient clipping, the optimizer updates the policy parameters according to the pre-set optimization algorithm (e.g. Adam or RMSProp). During the update process, if the loss convergence is slow or oscillatory, the optimizer will trigger the learning rate adjustment strategy to decay or fine-tune the learning rate to ensure smooth updating.

[0126] After completing parameter update, the optimizer writes the new policy parameters to the MLE agent module, and in the multi-process or distributed scenario, it completes parameter synchronization through parameter broadcast or synchronization barrier mechanism to ensure that the policy network weights called during subsequent code generation are the latest version.

[0127] In another example, if a batch of candidate advantage distribution is extreme (most close to zero or all positive), the optimizer will add a layer of smoothing or scaling operation to the advantage value in the loss function construction stage to ensure that the gradient signal of the objective function is sufficient and not dominated by a single direction; if it is detected that the KL divergence is too large, the optimizer will increase the KL regularization term weight or reduce the probability ratio threshold to constrain the policy shift speed.

[0128] Through the above process, the reinforcement learning optimizer constructs a loss around the candidate advantage and the generated probability, and completes parameter update and synchronization, providing the latest policy support for code generation of the MLE agent module in subsequent rounds of interaction.

[0129] In some embodiments, the multi-round interaction control module is specifically used for: receiving the execution result and reward value of the last round, and generating a feedback data packet according to a preset data structure; fusing the feedback data packet based on the initial prompt or the prompt of the last round to form an updated prompt for code generation of the next round; sending the updated prompt to the MLE agent module and triggering a code generation request; recording the interaction identifier of the current round, associating the generated executable code with the corresponding feedback data packet, and updating the interaction state; judging the interaction state according to a preset termination condition, and if the termination condition is not met, the execution result and the corresponding reward value of the current round are again encapsulated as feedback to enter the next round of interaction.

[0130] Specifically, the multi-round interaction control module generates a feedback data packet containing key fields according to a preset data structure after receiving the execution result and reward value of the last round; then the feedback data packet is semantically fused with the initial prompt or the prompt of the last round to form an updated prompt for code generation of the next round; after the updated prompt is generated, the module sends it to the MLE agent module and triggers a code generation request; at the same time, the interaction identifier of the current round is recorded, the executable code generated in the current round is associated with the corresponding feedback data packet, and the interaction state is updated; finally, the interaction state is judged according to a preset termination condition, and if the termination condition is not met, the execution result and the corresponding reward value of the current round are again encapsulated as feedback to enter the next round of interaction until the termination condition is met.

[0131] The concepts of some key technical terms involved in this embodiment are explained as follows: Feedback data packet: refers to a feedback entity formed by field extraction, archiving and serialization of the execution result and reward value of the last round according to a preset data structure, containing key fields such as execution result summary, normalized index set, reward value, error information overview, round identifier, etc.

[0132] Update prompt: refers to the prompt text or equivalent structured instructions formed by integrating the change requirements, missing links and parameter adjustment suggestions reflected in the feedback data packet on the basis of the initial prompt or the last round of prompt, used to guide the MLE agent module to generate or modify the code in the next round.

[0133] Interaction identifier: refers to the identifier used to distinguish and track the interaction process of each round, which can include round number, timestamp, task ID, etc., used to establish a one-to-one correspondence between code output and feedback information in multiple rounds of process.

[0134] Interaction state: refers to the state set recording the current stage of the multi-round cycle, the number of completed rounds, the latest reward value, whether the termination condition is met, etc., helping the module to decide whether to enter the next round.

[0135] Pre-set termination condition: refers to the termination rule set before the start of the multi-round interaction, such as reaching the maximum number of rounds, the main performance indicator converging to a threshold, the resource budget being exhausted or an unrecoverable error occurring, etc.

[0136] Fusion strategy: refers to the rule set for content merging, priority analysis and conflict resolution of the initial prompt and feedback data packet when generating the update prompt, ensuring that the update prompt not only inherits the original task target but also accurately reflects the process elements that need to be adjusted in this round.

[0137] For example, in a specific example, the initial prompt requires building a classification model on the BreastCancerWisconsinDataset, with the goal of maximizing accuracy and training time not exceeding 30 seconds. After the first round of execution, the code executor outputs the execution results, and the evaluator and reward construction module gives the normalized indicators and reward value R1=0.50, and points out that the feature standardization step is missing and the training time is not recorded.

[0138] The multi-round interaction control module first receives the execution results and R1, and generates a feedback data packet according to the pre-set data structure. The data packet contains: the model accuracy of the last round, the missing label of the training time field, the missing label of the feature engineering, the reward value R1, the error information summary (if any) and the round identifier ID1.

[0139] Then, the module executes the fusion strategy: based on the initial prompt, inserts the two change requirements "add standardization" and "record training time" into the corresponding paragraphs of the prompt, forming an updated prompt. For example, add "please record the training time and control it within 30 seconds" in the task constraint section, and insert "perform feature standardization before model training" in the process section. After fusion, the module sends the updated prompt to the MLE agent module and triggers the second round of code generation request.

[0140] At the same time of sending the request, the module records the interaction identification ID2, and associates the executable code generated in the second round with the feedback data packet in the last round, updates the "completed round number" in the interaction state to 2, the "latest reward value" to R1, and the "termination condition state" to not met.

[0141] After the completion of the second round of execution, the execution result shows that the training time is 12.3 seconds and the accuracy is improved to 0.96, and the reward value R2 = 0.85. The module receives the execution result and R2 again, generates a new feedback data packet, which marks "feature standardization has been added" and "training time record has been completed", and at the same time the evaluator indicates that the feature dimension can be further compressed by feature selection. The module fuses the feedback data packet with the last round of update prompts to form a third round of update prompts, adds the requirement of "performing feature selection to reduce input dimension" in the prompts, and triggers the third round of code generation.

[0142] After the third round, assuming that the reward value R3 = 0.92, the module updates the interaction state: the latest reward value R3 has reached the preset threshold, and the round number does not exceed the maximum allowed number of times, meeting the "performance meets the standard" termination condition. The module determines that the termination condition is met, stops further iteration, and submits the finally generated executable code and related execution results to the subsequent deployment process.

[0143] In another example, if an execution interruption or serious error occurs in a certain round, the feedback data packet will contain the error type and error code, and the update prompt will preferentially guide the MLE agent module to repair the error segment. If the resource budget is gradually consumed in multiple rounds of interaction, the interaction state will record the remaining budget, and when the budget is lower than the threshold, the termination condition is triggered, and the module will terminate the subsequent rounds and output the latest available version of the final executable code.

[0144] Through the above process, the multi-round interaction control module implements key operations such as feedback generation, prompt update, interaction record, and termination judgment in each round, ensuring that the multi-round iteration process progresses in an orderly manner under controlled conditions.

[0145] In some embodiments, the system further comprises: An experience storage module for storing task states, candidate codes, execution results, and reward values generated during the multi-round interaction process, and providing playback data to the reinforcement learning optimizer.

[0146] Specifically, the experience storage module receives and stores data elements such as task state, candidate code, execution result, and reward value during multiple rounds of interaction. First, it organizes, serializes, and labels the above elements according to the pre-set data field specification to generate retrievable experience entries. Then, it establishes multi-dimensional indexes according to task identification, round identification, and timestamp, and performs write and version management on the experience entries. When the reinforcement learning optimizer requests playback data, the module performs filtering and sampling according to its playback strategy requirements (such as reward interval, task type, round stage), combines the matching experience entries into a playback batch, and outputs it. In the long run, the module implements archiving, compression, or elimination of experience data according to the storage strategy to maintain stable storage.

[0147] The concepts of some key technical terms involved in this embodiment are explained as follows: Task state: refers to the context information set of the system at a certain round of interaction, including task description summary, current round number, last round feedback summary, strategy parameter version number, etc.

[0148] Candidate code: refers to the executable code text or its structured representation generated by the MLE agent module at a certain round, which usually corresponds one-to-one with the execution result of the code executor.

[0149] Execution result: refers to the encapsulated result object of the running log, model performance indicator, running time, resource occupation, and error information output by the code executor.

[0150] Reward value: refers to the quantitative reward calculated by the reward construction module for the candidate output at this round, used for reinforcement learning optimization.

[0151] Experience entry: refers to the storage unit aggregated in a unified format from task state, candidate code, execution result, and reward value, which is the basic access object in the experience storage module.

[0152] Multi-dimensional index: refers to the composite index key set established for the experience entry, including task ID, round ID, timestamp, reward interval label, error type label, etc., used for efficient retrieval.

[0153] Playback data: refers to the set of experience entries sampled from the experience storage module according to specified filtering conditions, used by the reinforcement learning optimizer in offline or online training phase.

[0154] Version management: refers to versioned recording of multiple writes for the same task or the same round, ensuring traceability of historical strategy parameters and corresponding experience data.

[0155] Archiving and elimination strategy: refers to the set of strategies for hierarchical storage, compression, or deletion of experience entries according to data importance, time span, or storage capacity limitations.

[0156] For example, in one specific example, the system completes three rounds of interaction for a breast cancer dataset classification task. The experience storage module performs the following operations at the end of each round: First, the module obtains the task status of the current round from the multi-round interaction control module, which records information such as task ID, round number, update prompt summary, policy parameter version number, etc.; obtains the candidate code text for this round from the MLE agent module; obtains the execution result object from the code executor; obtains the reward value R from the reward construction module; and obtains the group average return value and candidate advantage value (if needed to be archived) within the candidate output group from the reinforcement learning optimizer.

[0157] Subsequently, the module encapsulates the above data elements into experience entries according to the preset field specifications. For example, the entry contains fields: task_id, round_id, timestamp, code_snippet_id, exec_result_ref, reward_value, advantage_value, error_flag, policy_version, etc. The module performs uniform encoding and length truncation on the text fields, type checking on the numerical fields, and adds error type labels to the error information fields.

[0158] After encapsulation, the module establishes a main index according to task_id and round_id, and establishes a reward interval label index according to the interval in which reward_value is located, and an error category index according to error_flag. If there are multiple candidate outputs in this round, the module generates independent entries for each candidate output and establishes intra-group association marks between the entries, so that subsequent playback can be sampled by group.

[0159] When the reinforcement learning optimizer needs to replay high-reward samples to strengthen the strategy in the offline stage, the experience storage module receives its replay request, which specifies the filtering condition as "reward_value≥0.85 and error_flag=0". The module quickly locates the entry set that meets the conditions according to the multi-dimensional index, and if the number of sets exceeds the batch size, it performs secondary filtering or random sampling according to the timestamp or advantage value, packs the selected entries into a replay batch, and returns it. If the optimizer requests diversified playback (both high-reward and low-reward samples need to be covered), the module extracts entries in different intervals according to the reward interval label in proportion to form a more balanced replay data.

[0160] In another embodiment, to control the storage size, the experience storage module performs compression and archiving operations on entries that exceed the set retention period. Specifically, the complete running log and code text can be moved to cold storage, and only the summary and key indicators are retained for quick screening; for entries with very low reward values and errors identified as serious errors, they can be directly deleted according to the elimination strategy after the strategy is stable. The module updates the index mapping before archiving and eliminating to ensure that there are no dangling references.

[0161] In addition, when the policy network version is updated, the module will add the policy version number field to the experience entries of adjacent rounds before and after the update to ensure that the optimizer can select consistent policy stage experience according to the version when replaying, avoiding replay bias due to too large differences in policy distribution.

[0162] Through the above process, the experience storage module implements standardized storage, efficient retrieval, and policy-based replay of key data throughout the entire process of multiple rounds of interaction, providing a sustainable experience data basis for the reinforcement learning optimizer.

[0163] The foregoing embodiments have described in detail the implementation of the multi-round automatic machine learning agent system based on reinforcement learning optimization of the present application. Based on the system, the present embodiment further provides a multi-round automatic machine learning method based on reinforcement learning optimization. The implementation steps of the multi-round automatic machine learning method based on reinforcement learning optimization of the present application will be explained and described below in conjunction with specific embodiments and drawings.

[0164] Figure 2 is a flowchart of the multi-round automatic machine learning method based on reinforcement learning optimization provided by the embodiments of the present application. As shown in Figure 2 , the method can specifically include the following steps: S201, task analysis step: receiving the task description and constraint conditions input by the user, analyzing to obtain target indicators, resource limitations, and dataset information, and generating an initial prompt according to a preset field mapping rule and a prompt template; S202, code generation step: the MLE agent based on a large-scale pre-trained language model receives the initial prompt and multiple rounds of feedback information, and generates or modifies executable code according to a code generation rule and an operator flow template set; S203, code execution step: running the executable code in a controlled execution environment, collecting running logs, model performance indicators, running time, resource occupation, and error information to form an execution result; S204, indicator calculation step: calculating multi-dimensional indicators of performance and running types according to the execution result, outputting the normalized values of each indicator and generating a code correctness identifier to obtain a normalized indicator set; S205, reward construction step: combine the normalized indicator set with the code correctness identifier according to the preset rule to generate the corresponding reward value; S206, group return and advantage calculation step: for the reward values of multiple candidate outputs generated by the MLE agent sampling for the same task input, calculate the group average return value, and difference the reward value of each candidate output from the group average return value to obtain the candidate advantage value, process the candidate advantage value to obtain the processed candidate advantage set; S207, policy parameter update step: receive the processed candidate advantage set and the generation probability of the corresponding candidate output, construct a policy gradient objective function with the generation probability and the candidate advantage value as the independent variables, add a probability ratio truncation term and / or a regularization term based on KL divergence to the objective function to obtain a final loss function, perform back propagation to calculate the gradient and update the policy parameters according to the preset optimization algorithm, and perform gradient clipping and learning rate adjustment, and write the updated policy parameters to the MLE agent; S208, multi-round interaction control step: encapsulate the execution result and the reward value of the last round as a feedback data packet and fuse it with the initial prompt or the last round prompt to form an updated prompt, send the updated prompt to the MLE agent to trigger the next round of code generation, and determine whether to end according to the preset termination condition, and repeat the above steps if not.

[0165] The implementation process of the technical scheme of the present application will be described in detail below with specific examples, which specifically includes the following operations: I. Input Input task: the user wants to build a classification model for breast cancer diagnosis (Breast Cancer Wisconsin Dataset), and requires to improve the accuracy as much as possible and limit the model training time within 30 seconds.

[0166] Task prompt (Prompt): please build a classification model based on sklearn to predict the breast cancer dataset. The goal is to maximize the accuracy and limit the model training time to no more than 30 seconds.

[0167] II. Multi-round reasoning First round generation (initial output): The MLE Agent generates the following Python code: from sklearn.datasets import load_breast_cancer from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score data = load_breast_cancer() X_train, X_test, y_train, y_test = train_test_split(data.data,data.target) model = LogisticRegression() model.fit(X_train, y_train) y_pred = model.predict(X_test) print("Accuracy:", accuracy_score(y_test, y_pred)) Evaluation feedback: Runs normally, no syntax errors; accuracy is 0.92, but the model default parameters are not tuned; no feature normalization is performed, and training time is not recorded; Reward = 0.5 (runnable + average results + lack of indicator control).

[0168] Second round of correction output: from sklearn.preprocessing import StandardScaler from sklearn.ensemble import RandomForestClassifier from time import time scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) start_time = time() model=RandomForestClassifier(n_estimators=100,max_depth=6, random_state=42) model.fit(X_train_scaled, y_train) print("Train time:", time() - start_time) y_pred = model.predict(X_test_scaled) print("Accuracy:", accuracy_score(y_test, y_pred)) Evaluation feedback: Standardization added; training time recorded as 12.3 seconds; accuracy improved to 0.96; Reward=0.85 (significant improvement).

[0169] Round 3 improvement output: from sklearn.feature_selection import SelectKBest, f_classif selector = SelectKBest(score_func=f_classif, k=10) X_train_selected = selector.fit_transform(X_train_scaled, y_train) X_test_selected = selector.transform(X_test_scaled) model=RandomForestClassifier(n_estimators=100,max_depth=6, random_state=42) model.fit(X_train_selected, y_train) y_pred = model.predict(X_test_selected) print("Accuracy:", accuracy_score(y_test, y_pred)) Evaluation feedback: Feature selection added, dimension reduced from 30 to 10; training faster, performance maintained at accuracy 0.95+; code structure clear; Reward=0.92 (refinement + efficiency).

[0170] III. GRPO Training Instructions (Brief) The three rounds above are "candidate output groups", and the system constructs a reward distribution based on the result of each round: R1=0.5, R2=0.85, R3=0.92; Calculate the group average reward (μ≈0.76) using GRPO, and then calculate the advantage A_i=R_i-μ; Optimize the output probability of the policy network through the GRPO algorithm, so that the future is more inclined to generate high-reward codes like R3.

[0171] It should be understood that the order of the steps in the above embodiments does not mean the order of execution, and the order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0172] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the technical solutions of the present application are described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A multi-round automatic machine learning agent system based on reinforcement learning optimization, characterized in that: include: The task parsing module receives the task description and constraints input by the user and parses them to obtain the target indicators, resource limitations and data set information, and generates the initial prompts for the MLE agent to call; An MLE agent module is configured to receive the initial prompt and multiple rounds of feedback information based on a large-scale pre-trained language model and generate executable code; A code executor is used to run the executable code in a controlled execution environment, collect operation logs, model performance indicators, running time, resource usage and error information, and generate execution results; An evaluator, configured to calculate multi-dimensional indicators of performance and operation based on the execution results, and output a normalized value of each indicator and a code correctness indicator; A reward construction module, configured to combine the normalized index with the code correctness identifier according to a preset rule to generate a corresponding reward value; A reinforcement learning optimizer is configured to calculate the reward values ​​of multiple candidate outputs sampled and generated by the MLE agent module for the same task input, calculate the group average return and the candidate advantage, and update the policy parameters of the MLE agent module based on the candidate advantage; The multi-round interaction control module is used to feed back the execution result of the previous round and the reward value to the MLE agent module during the multi-round interaction process, and control the next round of code generation until the preset termination condition is met.

2. The system according to claim 1, wherein: The task parsing module is specifically used to: Perform semantic parsing and field recognition on the task description and constraints to extract key parameters corresponding to target indicators, resource constraints, and data set information; Organizing the key parameters into structured expressions according to preset field mapping rules and prompt templates; The structured expression is format checked and parameter normalized to obtain an initial prompt that meets the call requirements.

3. The system according to claim 1, wherein: The MLE agent module is specifically used for: Performing semantic analysis on the initial prompt and the multiple rounds of feedback information to determine machine learning process elements to be generated or modified; Generate code snippets corresponding to the process elements according to the preset code generation rules and operator process template set; Processing the code snippet to generate a complete executable code; In subsequent rounds, the code position to be adjusted is located according to the execution result and reward value of the previous round, and incremental generation or replacement is performed, and the updated executable code is output to the code executor.

4. The system according to claim 1, wherein: The code executor is specifically used for: Load the dependent libraries and data resources required by the executable code, and initialize the execution container according to the preset resource quota; Triggering the execution of the executable code and monitoring events during the execution process, and recording the operation log; Calculate the indicators of the training and prediction phase outputs according to the preset evaluation interface to obtain the model performance indicators; The running time and resource usage are counted respectively by using timestamp marking and resource monitoring probes; Classifying the exception type, error code, and interrupt status captured during the execution process and generating the error information; The operation log, model performance indicators, operation time, resource usage and error information are summarized and packaged as the execution result according to the preset data structure.

5. The system according to claim 1, wherein: The evaluator is specifically used for: Perform field parsing and data cleaning on the execution results to obtain the original indicator data set; Compare the model prediction output with the labeled data according to the preset performance indicator calculation rules to calculate the performance indicators; Aggregate time records and resource monitoring data according to preset operation indicator statistical rules to calculate operation indicators; Determine the code correctness flag based on the exception type, error code and execution interrupt status in the error information; The performance indicators and operation indicators are normalized according to a unified normalization strategy to generate a normalized indicator set.

6. The system according to claim 1, wherein: The reward building module is specifically used to: Call the preset indicator weight mapping table and combination rule library to assign weight coefficients to each normalized indicator and determine the combination operator; Divide the normalized indicators into performance and operation categories to generate corresponding sub-reward components; Calculating a penalty factor or a bonus factor according to the code correctness identifier; Performing a weighted combination and piecewise function operation on the sub-reward components and the penalty factor or the bonus factor to obtain an initial reward value; The initial reward value is interval-clipped and quantized to form the reward value.

7. The system according to claim 1, wherein: The reinforcement learning optimizer is specifically used to: receiving the reward value and establishing a candidate output group according to the sampling batch; Counting the reward values ​​within the candidate output group to obtain the group average reward value; Difference the reward value of each candidate output with the group average reward value to obtain the corresponding candidate advantage value; Processing the candidate advantage values ​​to generate a processed candidate advantage set; The processed candidate advantage set is associated with the generation probability of the corresponding candidate output.

8. The system according to claim 1, wherein: The reinforcement learning optimizer is also used to: receiving the candidate advantage value and the generation probability of the corresponding candidate output; Constructing a policy gradient objective function with the generation probability and the candidate advantage value as independent variables; Superimposing a probability ratio truncation term and / or a KL divergence-based regularization term in the objective function to obtain a final loss function; Performing backpropagation on the final loss function to calculate gradients; Perform gradient updates on the policy parameters according to a preset optimization algorithm, and perform gradient clipping and learning rate adjustment during the update process; The updated strategy parameters are written into the MLE agent module and parameter synchronization is completed.

9. The system according to claim 1, wherein: The multi-round interaction control module is specifically used to: Receive the execution results and reward values ​​of the previous round, and generate a feedback data packet according to the preset data structure; fusing the feedback data packets based on the initial prompt or the previous round of prompts to form an updated prompt for the next round of code generation; Sending the update prompt to the MLE agent module and triggering a code generation request; Record the interaction identifier of the current round, associate the generated executable code with the corresponding feedback data packet, and update the interaction status; The interaction state is judged according to a preset termination condition. If the termination condition is not met, the execution result and corresponding reward value generated in this round are used as feedback to enter the next round of interaction.

10. The system according to claim 1, wherein: The system further comprises: The experience storage module is used to store the task status, candidate codes, execution results and reward values ​​generated during multiple rounds of interaction, and provide replay data to the reinforcement learning optimizer.

Images