Multi-round automated machine learning agent system based on reinforcement learning optimization

Abstract

The application provides a multi-round automatic machine learning intelligent agent system based on reinforcement learning optimization. It includes a task analysis module for generating an initial prompt for MLE agent call; an MLE agent module for generating executable code; a code executor for generating execution results; an evaluator for outputting normalized values of each indicator and code correctness identification; a reward construction module for generating reward values; a reinforcement learning optimizer for calculating group average return and candidate advantage, and updating the strategy parameters of the MLE agent module based on the candidate advantage; a multi-round interaction control module for feeding back the execution results and reward values of the last round to the MLE agent module during the multi-round interaction process, controlling the next round of code generation, and stopping until the preset termination condition is met. The application can realize strategy adaptive evolution, reinforcement learning optimization of fine-grained credit allocation, and multi-round closed-loop automation process improvement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-round automated machine learning agent system based on reinforcement learning optimization. BACKGROUND

[0002] In recent years, automated machine learning (AutoML) aims to reduce human intervention in feature engineering, model selection, hyperparameter tuning, etc., and to realize end-to-end modeling process automation. With large-scale language models (LLM) having the ability to integrate cross-domain knowledge and generate code, researchers have begun to try to use LLM as an agent to intervene in the AutoML process, drive code generation and process arrangement through natural language prompts, and thus achieve automation at a higher level. At the same time, the application of reinforcement learning (RL) in complex reasoning and multi-step decision-making tasks shows that continuous improvement of strategies using environmental feedback can improve the performance of LLM in reasoning, planning, etc.

[0003] Traditional AutoML solutions (such as Auto-sklearn, Auto-Keras) generally rely on pre-defined operator libraries and fixed pipeline structures to optimize in a static search space through grid search, random search or Bayesian optimization, and lack the ability to dynamically adjust strategies for different tasks. Recently emerging LLM-driven AutoML agents (such as AutoML-GPT, MLCopilot, and AIDE, SELA, etc. using a round-by-round trial-and-error strategy) can generate or iteratively modify machine learning process codes through prompts, achieving a certain degree of multi-round optimization, but their core decision logic usually relies on artificially designed prompt templates or heuristic rules, and the strategies remain static between different tasks. On the other hand, reinforcement learning optimization for LLM reasoning ability (such as DeepSeek-R1 and other works) often uses reinforcement learning to improve the quality of the reasoning link, but in the context of automated machine learning, which involves multi-round interaction and multiple sub-tasks, reinforcement learning has not yet formed a mature systematic integration framework. Existing multi-round decision-making RL methods often treat the entire reasoning process as a single-step decision, only relying on the final result for training, and ignoring the reward distribution of intermediate steps, resulting in rough credit distribution and sparse learning signals. SUMMARY

[0004] Therefore, the embodiments of the present application provide a multi-round automated machine learning agent system based on reinforcement learning optimization to solve the problems of static strategies and difficulty in self-optimization, sparse reward signals leading to rough credit distribution, and lack of multi-round closed-loop process optimization mechanism in the prior art.

[0005] The embodiment of the application provides a multi-round automatic machine learning intelligent agent system based on reinforcement learning optimization, comprising: a task analysis module, configured to receive a task description and constraint conditions input by a user, and analyze target indicators, resource limits and dataset information to generate an initial prompt for calling an MLE intelligent agent; an MLE intelligent agent module, configured to receive the initial prompt and multi-round feedback information based on a large-scale pre-trained language model, and generate executable code; a code executor, configured to run the executable code in a controlled execution environment, collect running logs, model performance indicators, running time, resource occupation and error information to form an execution result; an evaluator, configured to calculate multi-dimensional indicators of performance and running classes according to the execution result, and output normalized values of the indicators and a code correctness identifier; a reward construction module, configured to combine the normalized indicators and the code correctness identifier according to a preset rule to generate a corresponding reward value; a reinforcement learning optimizer, configured to calculate group average returns and candidate advantages for reward values of multiple candidate outputs generated by the MLE intelligent agent module for the same task input, and update policy parameters of the MLE intelligent agent module based on the candidate advantages; and a multi-round interaction control module, configured to feed back the execution result and the reward value of the last round to the MLE intelligent agent module in a multi-round interaction process, control code generation of the next round until a preset termination condition is met.

[0006] The above at least one technical solution adopted by the embodiment of the application can achieve the following beneficial effects:

[0007] Through the task analysis module, the task description and constraint conditions input by the user are received and analyzed to obtain target indicators, resource limits and dataset information, and an initial prompt for calling the MLE intelligent agent is generated. The MLE intelligent agent module is used to receive the initial prompt and multi-round feedback information based on a large-scale pre-trained language model, and generate executable code. The code executor is used to run the executable code in a controlled execution environment, collect running logs, model performance indicators, running time, resource occupation and error information to form an execution result. The evaluator is used to calculate multi-dimensional indicators of performance and running classes according to the execution result, and output normalized values of the indicators and a code correctness identifier. The reward construction module is used to combine the normalized indicators and the code correctness identifier according to a preset rule to generate a corresponding reward value. The reinforcement learning optimizer is used to calculate group average returns and candidate advantages for reward values of multiple candidate outputs generated by the MLE intelligent agent module for the same task input, and update policy parameters of the MLE intelligent agent module based on the candidate advantages. The multi-round interaction control module is used to feed back the execution result and the reward value of the last round to the MLE intelligent agent module in a multi-round interaction process, control code generation of the next round until a preset termination condition is met. The application can realize policy adaptive evolution, reinforcement learning optimization of fine-grained credit allocation and multi-round closed-loop automatic process improvement. BRIEF DESCRIPTION OF DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0009] Figure 1 is a schematic diagram of a framework of a multi-round automatic machine learning agent system based on reinforcement learning optimization provided by the embodiments of the present application;

[0010] Figure 2 is a flowchart of a multi-round automatic machine learning method based on reinforcement learning optimization provided by the embodiments of the present application. DETAILED DESCRIPTION

[0011] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application, but it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0012] In recent years, automatic machine learning (AutoML) aims to reduce human intervention and automate the machine learning process. Traditional AutoML methods (such as Auto-sklearn, Auto-Keras, etc.) rely on pre-defined search spaces and static configurations, mainly using grid search, Bayesian optimization or fixed pipeline strategies, lacking multi-round iteration and flexible generation capabilities. Unlike this, using large-scale language models (LLM) as agents to automatically generate and optimize machine learning processes has become a new direction. For example, AutoML-GPT, MLCopilot, etc. generate complete pipelines through prompts, and AIDE, SELA, etc. optimize model design through trial and error search.

[0013] However, these LLM agents mainly rely on artificial prompt engineering and do not learn and optimize based on their own experience. Their strategies remain static across different tasks and cannot improve themselves from historical experience, severely restricting adaptability and efficiency. Reinforcement learning (RL) has been proven to significantly improve the ability of LLM in complex tasks, improving strategies through feedback-driven exploration and utilization. For example, DeepSeek-R1 and other works use reinforcement learning to train LLM to improve reasoning ability.

[0014] However, applying reinforcement learning to LLM agents to automate machine learning processes is still a blank. Existing RL methods for multi-round decision-making tasks often treat the entire reasoning process as a single-step decision, relying only on the final result for training, ignoring the intermediate rewards of each step, making it difficult for agents to distinguish the contribution of each round of operation to the final result. Existing research shows that introducing round-level rewards and fine-grained credit allocation can significantly improve the performance of multi-round reasoning tasks. Therefore, there is an urgent need for a new framework that combines LLM agents with reinforcement learning to design strategies through dynamic reward optimization in a multi-round interaction environment.

[0015] In view of the problems existing in the prior art, the present application proposes a multi-round MLEAgent automatic machine learning system based on reinforcement learning optimization, which can autonomously select the best data processing scheme and feature engineering and the corresponding model for task input and data, and continuously optimize parameters through multi-round iteration to ultimately obtain better prediction results. The technical scheme of the present application mainly includes the following contents:

[0016] 1) MLE agent end-to-end process generation: a language model driven MLE agent is introduced, which can automatically generate and execute complete machine learning process code according to the input task prompt, covering feature engineering, model selection, hyperparameter tuning and model training steps. The code generated by the agent runs in the code execution environment to automatically complete the model development task.

[0017] 2) GRPO reinforcement learning optimization: the system uses the group relative policy optimization (GRPO) algorithm proposed by DeepSeek-R1 as the reinforcement learning framework. In each training, the MLE agent generates multiple candidate outputs for the same input, and calculates the advantage value according to the group return of these candidate schemes to update the policy model. Without additional value network, the policy optimization can be guided, greatly reducing the training overhead and improving the convergence efficiency.

[0018] 3) Multi-round interaction iterative optimization: the system supports multi-round interaction reasoning, and the agent generates or modifies code based on the current prompt and the feedback of the last round in each round, and obtains new feedback information after execution. Reinforcement learning is used to guide the MLE agent to improve decision-making in each iteration, making multi-round generation and optimization possible. This multi-round mode allows the agent to think and correct repeatedly like a human engineer, rather than outputting once, significantly improving the quality of the final scheme.

[0019] 4) Multi-dimensional reward signal construction: the evaluator performs multi-dimensional evaluation on the code execution results, outputting performance indicators (such as accuracy, F1 score) and running performance indicators (such as execution time, resource utilization). The system combines these evaluation indicators with code correctness and other factors to construct a rule-based reward mechanism. For example, if the code is successfully executed and the model performance is improved, a positive reward is given; if there is a running error or performance degradation, a penalty signal is given. This rule-based reward signal ensures that the quality of the code and the effect of the model are considered in the reinforcement learning process, making the reward design comprehensive and controllable.

[0020] 5) Closed-loop training architecture: the system is composed of task analysis module, MLE agent, code executor, evaluator and reinforcement learning optimizer, etc. Through the training and deployment of closed loop, it is continuously iterated and optimized. The task analysis module parses the user input machine learning requirements into the initial prompt of the agent; the MLE agent generates code according to the prompt; the code executor executes the code and outputs the results; the evaluator evaluates the results and calculates the reward; the reinforcement learning optimizer updates the agent strategy using the reward signal, and the optimized agent can be deployed to new tasks, thus forming a continuous learning closed loop. The whole process can continuously evolve in the real environment, thereby enhancing the automation and intelligence of the system.

[0021] The technical solutions of the present application will be described in detail below in combination with the drawings and specific embodiments.

[0022] Figure 1 is the framework schematic diagram of the multi-round automatic machine learning agent system based on reinforcement learning optimization provided by the embodiment of the present application. As shown in Figure 1 , the multi-round automatic machine learning agent system based on reinforcement learning optimization can specifically include:

[0023] The task analysis module 101 is used for receiving the task description and constraint conditions input by the user and analyzing the target indicators, resource limitations and dataset information to generate an initial prompt for calling the MLE agent;

[0024] The MLE agent module 102 is used for receiving the initial prompt and multi-round feedback information based on a large-scale pre-trained language model to generate executable code;

[0025] The code executor 103 is used for running the executable code in a controlled execution environment, collecting running logs, model performance indicators, running time, resource occupation and error information to form execution results;

[0026] The evaluator 104 is used for calculating multi-dimensional indicators of performance and running types according to the execution results, and outputting the normalized values of the indicators and the code correctness identifier;

[0027] The reward construction module 105 is configured to combine the normalized indicators and the code correctness identification according to preset rules to generate corresponding reward values.

[0028] The reinforcement learning optimizer 106 is configured to calculate the group average return and the candidate advantage for the reward values of the multiple candidate outputs sampled by the MLE agent module for the same task input, and update the policy parameters of the MLE agent module based on the candidate advantage.

[0029] The multi-round interaction control module 107 is configured to feed back the execution result of the last round and the reward value to the MLE agent module in the multi-round interaction process, control the code generation of the next round until the preset termination condition is met.

[0030] In some embodiments, the task analysis module is specifically configured to:

[0031] perform semantic analysis and field identification on the task description and the constraint condition, and extract key parameters corresponding to the target indicators, the resource limits and the dataset information;

[0032] organize the key parameters into a structured expression according to a preset field mapping rule and a prompt template;

[0033] perform format checking and parameter normalization processing on the structured expression to obtain an initial prompt meeting the calling requirements.

[0034] Specifically, after receiving the task description and the constraint condition, the task analysis module first performs semantic analysis and field identification to locate the key parameters corresponding to the target indicators, the resource limits and the dataset information; then organizes the key parameters according to a preset field mapping rule and a prompt template to generate a structured expression with uniform field names and hierarchical structure; then performs format checking and parameter normalization processing on the structured expression to make the parameter values meet the uniform value range, the consistent time and resource units, the default items being filled or set to default values; finally outputs an initial prompt meeting the calling requirements and passes it to the MLE agent module.

[0035] The concepts of some key technical terms involved in the present embodiment are explained as follows:

[0036] Semantic analysis and field identification: refers to converting free text in the task description into recognizable semantic units based on natural language understanding technology, and mapping them to a predefined field set, such as "target indicators", "resource limits", "dataset information", etc.

[0037] Key parameters: refer to parameter items that directly affect the subsequent machine learning process generation and constraint control, such as accuracy upper limit, training time upper limit, memory upper limit, dataset name and storage path, etc.

[0038] Field mapping rule: refers to the rule set of establishing the mapping relationship of natural language table to standard field name and its value analysis method, for example, mapping "as high as possible accuracy" to "accuracy" as the main optimization indicator, and analyzing "training time not more than 30 seconds" to the training time upper limit field and converting it to seconds.

[0039] Prompt template: refers to the prompt structure framework prepared for the MLE agent, which usually includes task target section, constraint condition section, data description section and optional output requirement section, each section uses fixed placeholders waiting for key parameter filling.

[0040] Structured expression: refers to the internal representation of the identified key parameters organized in a hierarchical, key-value pair or other standardized form, which is convenient for verification and direct calling of downstream modules.

[0041] Format verification: refers to the process of checking whether the fields in the structured expression are complete, the types are matched, the values are out of bounds, and the units are consistent.

[0042] Parameter normalization processing: refers to converting parameter values into a unified standard, such as converting "30 seconds", "0.5 minutes" and "≤30s" to "30 seconds", and converting "accuracy" and "F1 score" to a pre-defined indicator name.

[0043] Initial prompt: refers to the prompt text or equivalent calling data structure formed by filling the normalized key parameters into the prompt template, which can be directly input into the MLE agent module.

[0044] In a specific example, the user input task description is: "Please build a classification model for breast cancer dataset based on sklearn, the goal is to maximize accuracy, and the model training time should not exceed 30 seconds."

[0045] (1) Semantic analysis and field identification stage, the task analysis module first performs syntactic division and key word extraction on the text, identifies the dataset information corresponding to "breast cancer dataset", the target indicator corresponding to "maximize accuracy" is accuracy, and the training time upper limit in resource constraints corresponding to "training time not more than 30 seconds"; at the same time, it identifies the tool stack preference as "sklearn".

[0046] (2) Field mapping stage, the module maps "breast cancer dataset" to the standard identifier "BreastCancer Wisconsin Dataset" according to the field mapping rule; maps "maximize accuracy" to the main optimization indicator field accuracy; converts "training time not more than 30 seconds" to the training time upper limit field and records the value 30, the unit is unified to seconds; "based on sklearn" is written as model development library preference field in the environment constraint part of structured expression.

[0047] (3) After the structured expression is generated, the module performs format verification to confirm that all mandatory fields have been filled in, that there are no out-of-bound situations for numerical fields, and that string fields conform to coding specifications. Subsequently, parameter normalization processing is performed, such as uniformly setting the accuracy identifier to "accuracy_score" and uniformly setting the time field to an integer number of seconds.

[0048] (4) In the prompt template filling stage, the module inserts the above-mentioned normalized parameters into the corresponding positions according to the template requirements to form an initial prompt containing a task objective section (to improve accuracy), a resource limit section (training time ≤ 30 seconds), a data description section (use the Breast Cancer Wisconsin Dataset and indicate that it can be loaded through sklearn), and a tool stack constraint section (preferentially use the sklearn library). The initial prompt is stored in a unified data structure internally and is sent to the MLE agent module in text form to drive the subsequent process of generating executable code.

[0049] In another example, if the user does not explicitly specify the dataset location or does not give a resource limit, the task parsing module will fill in the default items in the parameter normalization process according to the preset default value strategy, such as setting the default upper limit of training time to 60 seconds and the default main optimization indicator to accuracy, to ensure the completeness of the initial prompt. If the user input contains multiple target indicators (such as focusing on both accuracy and F1 score), the module will record the corresponding weight proportion in the structured expression according to the weight rules and explicitly indicate the multi-indicator optimization requirement in the prompt template.

[0050] Through the above steps, the task parsing module converts the task requirements in free text into a standardized initial prompt without relying on manual adjustment, laying a complete input foundation for the subsequent multi-round code generation and reinforcement learning optimization of the MLE agent module.

[0051] In some embodiments, the MLE agent module is specifically used for:

[0052] performing semantic analysis on the initial prompt and the multi-round feedback information to determine the machine learning process elements to be generated or modified;

[0053] generating code segments corresponding to the process elements according to a preset code generation rule and a set of operator flow templates;

[0054] processing the code segments to generate complete executable code;

[0055] in subsequent rounds, positioning the code position to be adjusted according to the execution result and the reward value of the last round and performing incremental generation or replacement, and outputting the updated executable code to the code executor.

[0056] Specifically, the MLE agent module receives an initial prompt and multi-round feedback information based on a large-scale pre-trained language model, first performs semantic analysis on the two to extract machine learning process elements to be generated or modified; then generates corresponding code snippets for each process element according to the preset code generation rules and operator process template set, and performs combination, parameter filling, dependency completion and syntax consistency checking on the code snippets after generation to form complete executable code; in subsequent rounds, the module locates the code position that needs to be adjusted according to the last round execution result and reward value, updates the corresponding fragment by using incremental generation or replacement, and outputs the updated executable code to the code executor to complete the next round of interaction.

[0057] The concepts of some key technical terms involved in the present embodiment are explained as follows:

[0058] Multi-round feedback information: refers to the data such as execution results, reward values and normalized indicator sets output by the evaluator generated by multiple rounds of interaction, which serves as the input basis for the subsequent generation or modification of code by the agent.

[0059] Machine learning process elements: refer to the core links and configuration items in the machine learning process, including feature engineering steps, model category selection, hyperparameter configuration, training and validation division, evaluation index calculation, etc.

[0060] Code generation rules: refer to the generation constraints and splicing logic formulated for different process elements, such as input and output variable naming rules, operator calling order, parameter value range limitation, exception handling structure, etc.

[0061] Operator process template set: refers to abstracting commonly used feature processing operators, model construction operators, evaluation operators, etc. into reusable template fragments, and labeling the input and output interfaces and adjustable parameter positions for the agent to call and fill.

[0062] Code snippet processing: includes connection relationship analysis between code snippets, variable scope unification, dependency library declaration completion, redundant fragment elimination, semantic consistency check and execution order rationalization, etc.

[0063] Incremental generation or replacement: refers to generating a new local fragment or replacing the original fragment locally only for the located position to be adjusted on the premise of preserving effective fragments, avoiding repeated generation of the entire code, reducing rewriting cost and ensuring context consistency.

[0064] Locating the code position to be adjusted: refers to mapping the change range to the corresponding process element and then to its code snippet position by comparing the last round execution result and reward value, combining the key fields in the feedback data packet, and realizing targeted modification.

[0065] For example, in one specific example, the initial prompt indicates that a classification model needs to be built on the Breast Cancer Wisconsin Dataset, with accuracy as the primary target and a training time limit of no more than 30 seconds.

[0066] First, the MLE agent module performs semantic analysis on the initial prompt and identifies data loading, data division, model selection, feature engineering, training and evaluation, and other process elements. The module retrieves the standard data loading template, training set and test set division template, and commonly used model template from the operator process template set suitable for classification tasks, and selects a model type with appropriate computational load according to the resource limit information in the prompt. According to the code generation rules, the module fills in the key parameters from the initial prompt for the parameter positions in each template, such as mapping the upper limit of training time to the timing monitoring structure, and adding accuracy as the primary indicator to the indicator calculation list in the evaluation link. Subsequently, the module performs combination and dependency completion on the generated segments, unifies the naming of the output variables of the data loading segment and the input variables of the model training segment, ensures the continuity of variable flow, and supplements the missing library import statements and exception capture structures. After completing the combination, the module performs syntax consistency checking on the overall executable code to confirm that there are no spelling errors, undefined variables, duplicate imports, etc., and then sends the executable code to the code executor.

[0067] After the completion of this round of execution, the evaluator outputs the execution results and reward values. Suppose the reward value shows that the model performance is acceptable but lacks a feature standardization step, and the training time record is missing. After receiving the multi-round feedback information, the MLE agent module first locates the process elements to be adjusted according to the missing link indicated in the feedback data packet, i.e., feature engineering and running time record. The module then finds the corresponding code segment location through the mapping relationship, calls the template segment of feature standardization and the time record segment, and inserts them into the position before model training according to the variable naming and data flow of the existing code, maintaining the input and output interface consistency with the original segment. The incrementally generated segment replaces the original non-standardization process, and the module performs dependency completion and syntax checking again, and the updated executable code is sent to the code executor again.

[0068] When the second round of execution results returns, suppose the reward value continues to prompt to reduce the number of features without reducing performance. The MLE agent module locates the feature engineering process element, selects the feature selection operator template, and performs feature selection on the original standardized output variable, and updates the input variable reference in the model training segment. The module replaces the relevant segments in an incremental manner, keeping the other parts unchanged, and then outputs the executable code again. Thereafter, with the continuation of multi-round interaction, the module locates the modification range according to the feedback, calls the corresponding template and performs incremental generation at each round, until the multi-round interaction control module judges that the preset termination condition is reached.

[0069] In another example, if the multi-round feedback information shows that there is a dependency missing or a syntax error, the MLE agent module parses the error type and error stack information into the syntax or dependency element to be modified, calls the error repair template or dependency completion template, inserts the necessary import statement, exception capture structure, and processes the error paragraph in priority during incremental generation. In this way, the MLE agent module performs semantic-driven code generation and updating in a multi-round interaction scenario at the granularity of process elements, ensuring the integrity and consistency of executable code.

[0070] In some embodiments, the code executor is specifically used for:

[0071] loading the dependent libraries and data resources required by the executable code, and initializing the execution container according to the preset resource quota;

[0072] triggering the execution of the executable code and performing event listening during the execution process to record the running log;

[0073] calculating the indicators according to the preset evaluation interface to obtain the model performance indicators;

[0074] statistically calculating the running time and resource occupation by timestamp markers and resource monitoring probes respectively;

[0075] classifying the exception types, error codes and interrupt states captured during the execution process to generate error information;

[0076] summarizing and packaging the running log, model performance indicators, running time, resource occupation and error information into an execution result according to the preset data structure.

[0077] Specifically, after receiving the executable code output by the MLE agent module, the code executor first loads the dependent libraries and data resources required by the code, and initializes the execution container according to the preset resource quota; then triggers the code to run and performs event listening during the entire execution process, continuously records the standard output, standard error and exception stack information to form the running log; calculates the indicators according to the preset evaluation interface to obtain the model performance indicators; statistically calculates the running time and resource occupation by timestamp markers and resource monitoring probes respectively; classifies the exception types, error codes and interrupt states captured during the execution process to generate error information; finally, the running log, model performance indicators, running time, resource occupation and error information are summarized and packaged into an execution result according to the preset data structure, and output to the evaluator.

[0078] The concepts of some key technical terms involved in this embodiment are explained as follows:

[0079] Execution container: refers to an isolated environment for running executable code, which loads dependent libraries, configures data paths, and applies CPU, memory, time, and other resource quotas during initialization.

[0080] Event listening: refers to the mechanism of capturing and recording key events (such as stage start / end, exception throwing, resource peak change) during code execution, forming a complete running log.

[0081] Pre-set evaluation interface: refers to the agreed specification for index calculation between the code executor and the evaluator, including input and output formats, calling timing, and index name set, the executor directly calculates or collects model performance indicators according to the interface.

[0082] Timestamp marking: refers to recording a uniform format timestamp at key nodes such as code startup, training start, training end, and prediction output, to accurately obtain running time.

[0083] Resource monitoring probe: a monitoring process or hook deployed in the execution container, used to periodically collect CPU occupancy, memory usage, disk read / write volume, and other resource data.

[0084] Error information: includes exception types (syntax error, dependency missing, interruption during running, etc.), error codes, and interruption status captured during execution, used for subsequent feedback and penalty strategy construction.

[0085] Pre-set data structure: refers to the uniform format definition for encapsulating execution results, including field name, nesting level, and serialization rules, to ensure that the evaluator can directly parse.

[0086] For example, in a specific example, the executable code output by the MLE agent module is used to train a classification model on the BreastCancer Wisconsin Dataset and calculate the accuracy. After the code executor receives the executable code, it performs the following operations:

[0087] First, the execution container initialization phase loads the required dependent libraries and data resources. The code executor sets the CPU core upper limit and memory quota in the container according to the resource limit information passed by the task parsing module, and mounts the dataset path or calls the data loading interface to ensure that the data can be accessed by the code. If there are libraries with unspecified versions in the dependent libraries, the execution container selects compatible versions and installs them according to the pre-set version locking strategy during initialization.

[0088] Subsequently, code execution is triggered. The event listening submodule starts working, capturing standard output, standard error, and exception information, and writing them in chronological order into the running log buffer. For example, when training begins, the event "training phase begins" and the corresponding timestamp are recorded, and when the model prediction ends, the event "prediction phase completes" and the corresponding timestamp are recorded. If a dependency is missing or data reading is abnormal during execution, the event listening submodule will capture the exception stack information and error code together and classify them into the error information cache.

[0089] After the model training and prediction phases end, the code executor calculates the output results according to the preset evaluation interface. For classification tasks, the executor calls the performance index calculation unit to obtain accuracy, F1 score, precision, recall, and other performance indicators, and the results are stored in the model performance index field.

[0090] At the same time, the timestamp marking unit calculates the training time by reading the training start and end timestamps, and calculates the total running time according to the overall execution start and end time; the resource monitoring probe periodically collects CPU occupancy and memory usage during code execution and calculates the peak and average values, and these data are sorted into the resource occupation field.

[0091] When the execution flow ends, the error information classification unit classifies the types of exceptions captured during the entire running period, such as classifying syntax errors as compilation errors, classifying dependency missing as environment errors, and classifying timeout termination as resource errors, and records the corresponding error code and interruption state.

[0092] Finally, the execution result packaging unit aggregates, renames, and serializes the running log, model performance index, running time, resource occupation, and error information according to the preset data structure, forms an execution result object, and outputs it to the evaluator. The evaluator then calculates the normalized index according to the execution result and generates a reward input.

[0093] In another example, if resource occupation is detected close to the upper limit of the quota during execution and is about to trigger forced termination, the resource monitoring probe sends a warning event to the event listening submodule, and the code executor can extend the monitoring period within the allowed range or trigger an early termination strategy, and record the "resource warning" state in the error information. Such information will be returned to the MLE agent module by the multi-round interaction control module in subsequent rounds of feedback, so that it actively reduces the model complexity or reduces the feature dimension when generating code in the next round.

[0094] Through the above process, the code executor completely collects and packages various information during the execution process under the premise of ensuring that the execution environment is controlled, providing a reliable data basis for the evaluator and reinforcement learning optimizer.

[0095] In some embodiments, the evaluator is specifically used for:

[0096] The execution results are parsed and cleaned to obtain the original indicator dataset;

[0097] The model's predicted output is compared with the labeled data according to the preset performance index calculation rules, and performance indicators are calculated.

[0098] Based on preset operational indicator statistical rules, time records and resource monitoring data are aggregated to calculate operational indicators;

[0099] Determine the code's correctness based on the exception type, error code, and execution interruption status in the error message;

[0100] Performance and operational metrics are normalized using a unified normalization strategy to generate a set of normalized metrics.

[0101] Specifically, after receiving the execution results output by the code executor, the evaluator first performs field parsing and data cleaning on the execution logs, model prediction outputs, time records, resource monitoring data, and error messages in the execution results to form a raw indicator dataset. Then, it compares the model prediction outputs with the labeled data according to the preset performance indicator calculation rules to obtain performance indicators, and aggregates the time records and resource monitoring data according to the preset operation indicator statistical rules to obtain operation indicators. Next, it generates code correctness identifiers based on the exception type, error code, and execution interruption status in the error messages. Finally, it normalizes the performance indicators and operation indicators according to a unified normalization strategy to obtain a normalized indicator set and outputs it to the reward construction module.

[0102] The following explains the concepts of some key technical terms involved in this embodiment:

[0103] Field parsing and data cleaning: This refers to identifying, disassembling, and standardizing the format of various raw fields in the execution results, while removing missing values, duplicate records, or entries with abnormal formats to ensure the consistency and completeness of inputs for subsequent indicator calculations.

[0104] Original indicator dataset: refers to the basic data set of performance and operation categories obtained after field parsing and data cleaning, which has not yet been normalized or weighted. It includes prediction result arrays, real labels, time interval records, resource usage sequences, and error record entries.

[0105] Preset performance index calculation rules: These refer to the calculation formulas and processes for indicators defined for classification, regression, or other task types, such as the calculation methods and input / output specifications for accuracy, F1 score, mean squared error, etc.

[0106] Pre-set running index statistics rule: refers to the rule of statistics on running time, memory usage, CPU occupancy and other running data, including peak calculation, mean calculation, interval summation, unit conversion, etc.

[0107] Code correctness identifier: refers to the state identifier of whether the code is successfully executed according to the error information, which can include states such as "executable", "execution exception", "partial interruption", etc., which is used for quickly filtering invalid candidates during reward construction.

[0108] Unified normalization strategy: refers to the use of a unified normalization method (such as interval scaling, logarithmic transformation or piecewise linear mapping) for indicators of different dimensions and ranges, so that each indicator can participate in subsequent weight combination on the same scale.

[0109] For example, in a specific example, the execution result output by the code executor contains the following data: the running log records the key events and standard output information of the training phase and the prediction phase, the model prediction output is a set of binary classification prediction labels, the real label data is stored by the data loading process, the time record contains the total execution start and end time and the training phase start and end time, the resource monitoring data is the CPU occupancy and memory usage sequence collected every second during execution, and the error information field is empty or only records alarm information.

[0110] The evaluator first performs field parsing and data cleaning on the execution result, removes redundant text in the running log that is irrelevant to the indicators, and only retains the timestamp, phase marker and exception summary; aligns the prediction output and real label, ensures the same number of samples and removes possible null entries; and performs missing value interpolation and unit unification on the resource monitoring sequence, and resamples data of different time granularities to a unified time step. After the above processing, the evaluator forms the original indicator dataset.

[0111] In the performance indicator calculation stage, the evaluator compares the prediction output and real label according to the pre-set rules of the classification task, and calculates the accuracy and F1 score. The accuracy calculation uses the formula of the number of correctly predicted samples divided by the total number of samples, and the F1 score uses the harmonic mean formula of precision and recall. If the task type is regression, the evaluator will automatically switch to rules such as mean square error and mean absolute error.

[0112] In the running indicator calculation stage, the evaluator calculates the training time and total running time according to the start and end times marked in the time record; and calculates the peak and mean of CPU occupancy and memory usage according to the resource monitoring sequence. All running indicators are represented in the original dimension in this stage, such as seconds, percentages, megabytes, etc.

[0113] The determination phase of the code correctness identification, the evaluator checks the error information field, if no abnormal type or error code is detected and the execution is not interrupted, the identification is set to executable; if dependency loss, syntax error and the like are detected, the execution is set to abnormal; if forced termination due to resource exhaustion occurs in the later running stage, it is set to partial interruption and the corresponding error code is recorded.

[0114] In the normalization phase, the evaluator converts the performance indicators and the running indicators according to the unified normalization strategy. For example, the accuracy and the F1 score can be directly used as the numerical value itself or scaled to the range of 0 to 1 according to the interval; for the running time, memory occupation and the like, the reverse normalization strategy (the shorter the time, the higher the value) or linear mapping through the preset maximum value can be used. In the normalization process, the evaluator applies the same processing flow to all indicators to ensure comparability between different tasks. Finally, the evaluator forms a normalized indicator set and outputs it to the reward construction module together with the code correctness identification.

[0115] In another example, if there are a large number of abnormal logs in the execution result and the error information field contains multiple error types, the evaluator will prioritize the code correctness identification according to the error type priority, such as syntax error before resource warning and running interruption before poor performance. In the normalization strategy, the evaluator can add a specific placeholder value to the abnormal state or directly set the normalized result to zero, so as to impose a penalty on the candidate output in the reward construction stage.

[0116] Through the above process, the evaluator completes the calculation, normalization and code correctness identification of the performance indicators and the running indicators under the premise of ensuring the consistency of the input data, and provides uniform and directly usable indicator input for the reward construction module.

[0117] In some embodiments, the reward construction module is specifically used for:

[0118] calling a preset indicator weight mapping table and a combination rule library to assign weight coefficients to each normalized indicator and determine a combination operator;

[0119] dividing the normalized indicators into performance type and running type to generate corresponding sub-reward components;

[0120] calculating a penalty factor or an addition factor according to the code correctness identification;

[0121] performing weighted combination and segmented function operation on the sub-reward components and the penalty factor or the addition factor to obtain an initial reward value;

[0122] performing interval clipping and quantization encoding on the initial reward value to form a reward value.

[0123] Specifically, after receiving the normalized indicator set and the code correctness identifier output by the evaluator, the reward construction module first calls the preset indicator weight mapping table and the combination rule library to assign weight coefficients to each normalized indicator and determine the combination operator; then, the normalized indicators are divided into performance and operation classes according to the indicator types, and the corresponding sub-reward components are calculated; then, the penalty factor or the addition factor is generated according to the code correctness identifier; on this basis, the weighted combination and the segmented function operation are performed on the sub-reward components and the penalty factor or the addition factor to obtain the initial reward value; finally, the initial reward value is interval clipped and quantization encoded to form the reward value and output to the reinforcement learning optimizer.

[0124] The concepts of some key technical terms involved in the present embodiment are explained as follows:

[0125] Indicator weight mapping table: a set of weight configurations preset for each normalized indicator, which can be dynamically adjusted according to task priority, scene type or historical experience, and is used to allocate contribution proportion when weighted combination.

[0126] Combination rule library: a set of rules for defining the combination method between different indicators, including linear weighting, segmented function, multiplicative penalty, threshold truncation, etc.

[0127] Sub-reward component: the performance and operation sub-reward components calculated according to the indicator types, used to maintain the independent controllability of different types of indicators before final combination.

[0128] Penalty factor or addition factor: a coefficient determined according to the code correctness identifier, used to apply negative punishment or positive addition in reward calculation, such as reducing the overall reward to a minimum value when an error is executed, or giving a small addition when the structure is complete.

[0129] Segmented function operation: a function operation method that uses different calculation formulas according to the indicator value interval or threshold setting, for example, using a steeper penalty segment when the performance sub-reward component is below a certain threshold, and using a flat segment when it is above the threshold.

[0130] Interval clipping and quantization encoding: limiting the initial reward value within the preset upper and lower bound range, and quantizing the continuous value to a specified precision or discrete level according to the input requirements of the reinforcement learning optimizer, so as to stabilize the training.

[0131] For example, in a specific example, the normalized indicator set output by the evaluator includes performance indicators accuracy_norm and f1_norm, operation indicators time_norm and memory_norm, and the code correctness identifier code_ok. The reward construction module performs the following process:

[0132] Firstly, the module reads corresponding weights from the index weight mapping table, such as the accuracy_norm corresponding weight w_acc, the f1_norm corresponding weight w_f1, the time_norm corresponding weight w_time, and the memory_norm corresponding weight w_mem. The weight mapping table can be dynamically adjusted according to the target index priority provided by the task analysis module, for example, when the task only emphasizes accuracy, w_acc can be set to the highest. The combination rule library indicates that the performance type index adopts linear weighted summation, and the running type index adopts reverse weighting (the shorter the time, the lower the resource, and the higher the score) and then linear synthesis.

[0133] Subsequently, the module combines accuracy_norm and f1_norm to obtain the performance type sub-reward component R_perf, and combines time_norm and memory_norm to obtain the running type sub-reward component R_run. In this process, a segmented function is used for the running type index: if time_norm is less than a preset threshold T_low, an additional bonus segment is given, if it is greater than the threshold T_high, it enters the penalty segment, and a linear segment is used in the middle range to ensure that the time constraint is reasonably reflected.

[0134] Next, the module generates a penalty factor or a bonus factor according to code_ok. If code_ok is an executable state, the factor is 1 or a bonus value slightly greater than 1; if code_ok identifies an execution exception, the factor is directly set to a penalty value close to 0; if it is a partial interruption, it is set to a reduction coefficient between 0 and 1.

[0135] After that, the module combines the sub-reward components and the penalty factor. Specifically, first, perform weighted summation on R_perf and R_run according to the operator defined in the combination rule library to obtain an intermediate value R_mid; then multiply R_mid by the penalty factor, and according to the need, smooth the extreme values through a segmented function to form an initial reward value R_init.

[0136] Finally, the module clips the interval of R_init to limit it within [0, 1] or other preset ranges, and quantizes the code according to the input requirements of the reinforcement learning optimizer, such as retaining the last two digits or mapping to 0~100 integer levels, to form the final reward value R. The reward value and the identifier of the corresponding candidate output are output to the reinforcement learning optimizer for subsequent group average return and candidate advantage calculation.

[0137] In another example, when the task contains multiple targets and there is a need for dynamic weight adjustment, the reward construction module updates the index weight mapping table according to the latest task feedback or policy convergence before each iteration. For example, when the running time exceeds the limit multiple times, the module automatically increases the weight of the running class index and reduces the weight of the performance class index, so that the subsequent reward calculation pays more attention to resource constraints; if the running class index is stable within the threshold, the weight is gradually restored or redistributed. Such dynamic adjustment is defined in the form of policy entries in the rule library, and the module switches according to the entry instructions when executing.

[0138] Through the above process, the reward construction module ensures the uniformity of the input indicators and provides the reinforcement learning optimizer with controllable and discriminative reward value inputs, providing fine-grained quantitative basis for multiple rounds of policy updates.

[0139] In some embodiments, the reinforcement learning optimizer is specifically used for:

[0140] receiving reward values and establishing candidate output groups according to sampling batches;

[0141] statistically processing the reward values in the candidate output groups to obtain a group average return value;

[0142] differencing the reward value of each candidate output from the group average return value to obtain a corresponding candidate advantage value;

[0143] processing the candidate advantage values to generate a processed candidate advantage set;

[0144] associating the processed candidate advantage set with the generation probability of the corresponding candidate output.

[0145] Specifically, after receiving the reward values output by the reward construction module, the reinforcement learning optimizer first groups multiple candidate outputs according to the same task input or the same sampling batch to establish a candidate output group; then performs statistical operations on the reward values in the group to obtain a group average return value; then differentiates the reward value of each candidate output from the group average return value to obtain a candidate advantage value; processes the candidate advantage values by clipping, normalization or weight adjustment to form a processed candidate advantage set; finally, associates the processed candidate advantage set with the generation probability of the corresponding candidate output to provide input for subsequent policy parameter updates.

[0146] The concepts of some key technical terms involved in this embodiment are explained as follows:

[0147] Candidate output group: refers to a set of multiple executable code schemes generated by the MLE agent module for the same task input within one policy update period, used for relative comparison within the group.

[0148] Group average return: refers to the statistical mean of all reward values within a candidate output group, used to characterize the overall level of the group.

[0149] Candidate advantage value: refers to the differential result of a single candidate output reward value relative to the group average return, used to measure the degree to which the candidate output is superior or inferior to the group average level.

[0150] Processed candidate advantage set: refers to the advantage value set obtained after clipping (to prevent extreme values from affecting training), normalization (to unify the scale), or weight adjustment (to emphasize certain samples) on the original candidate advantage values.

[0151] Generation probability: refers to the strategy probability value given by the MLE agent module when generating the corresponding candidate output, used for probability ratio or importance sampling factor in subsequent policy gradient calculation.

[0152] One-to-one association: refers to mapping each processed candidate advantage value to its corresponding candidate output generation probability and identification at the data structure level, ensuring accurate indexing during subsequent gradient update phase.

[0153] For example, in a specific example, for a certain classification task input, the MLE agent module generates 3 candidate executable codes in one sampling process, and the reward construction module outputs corresponding reward values R1=0.50, R2=0.85, R3=0.92. The reinforcement learning optimizer performs the following operations:

[0154] First, the candidate output grouping stage, the optimizer establishes a candidate output group G={O1, O2, O3} for the above 3 candidate outputs in the same batch, and records the probability p1, p2, p3 of each candidate output when it is generated.

[0155] Then, the group average return statistics stage, the optimizer calculates the mean of R1, R2, R3 to obtain the group average return value μ≈0.76.

[0156] Next, the candidate advantage calculation stage, the optimizer calculates A1=R1-μ, A2=R2-μ, A3=R3-μ respectively, obtaining the original candidate advantage value set A={A1, A2, A3}.

[0157] In the candidate advantage processing stage, the optimizer performs clipping and normalization operations on the set A. For example, when |Ai| exceeds the preset threshold, it is truncated to within the threshold range; then all Ai are scaled to the maximum absolute value or standard deviation to obtain the processed candidate advantage setÂ={Â1,Â2,Â3}. If high reward samples need to be emphasized, positive advantage samples can be amplified and negative advantage samples can be smoothed in this stage.

[0158] Finally, in the advantage probability association stage, the optimizer maps each Ai to the corresponding generation probability pi and candidate output identifier Oi, forming an association entry set like (Ai, pi, Oi), (A2, p2, O2), (A3, p3, O3) for subsequent direct calling by the policy parameter updating unit. In this process, the optimizer caches the set in a temporary storage structure and cleans up or archives it after parameter updating is complete.

[0159] In another example, if the number of candidate outputs is greater than the preset upper limit or the reward distribution is severely skewed, the reinforcement learning optimizer can first perform quantile screening on the reward values to remove abnormal extreme samples, and then calculate the group average return value to reduce the influence of abnormal values on advantage estimation. For the case where there are multiple task inputs in a batch, the optimizer first establishes candidate output groups separately according to task inputs, and then calculates the average return and advantage value for each group independently to avoid cross-task interference.

[0160] Through the above process, the reinforcement learning optimizer generates a candidate advantage set and a probability mapping that can be directly used for policy updating under the premise of ensuring that the candidate sample group is relatively reasonable, providing an accurate basis for subsequent gradient calculation and parameter synchronization.

[0161] In some embodiments, the reinforcement learning optimizer is also used for:

[0162] receiving candidate advantage values and generation probabilities of corresponding candidate outputs;

[0163] constructing a policy gradient objective function with generation probabilities and candidate advantage values as independent variables;

[0164] adding a probability ratio truncation term and / or a KL divergence-based regularization term in the objective function to obtain a final loss function;

[0165] performing backpropagation on the final loss function to calculate the gradient;

[0166] performing gradient clipping and learning rate adjustment during the updating process according to a preset optimization algorithm;

[0167] writing the updated policy parameters into the MLE agent module and completing parameter synchronization.

[0168] Specifically, after receiving the candidate advantage value and the generated probability of the corresponding candidate output, the reinforcement learning optimizer first constructs a policy gradient objective function with the generated probability and the candidate advantage value as independent variables; then superimposes a probability ratio truncation term and / or a KL divergence-based regularization term in the objective function to form a final loss function; then performs backpropagation on the final loss function to calculate the gradient, and updates the policy parameters according to the preset optimization algorithm, while performing gradient clipping and learning rate adjustment during the update process; finally, the updated policy parameters are written into the MLE agent module and parameter synchronization is completed.

[0169] The concepts of some key technical terms involved in the present embodiment are explained as follows:

[0170] Policy gradient objective function: refers to a target function established with the policy probability of the candidate output when generated and the corresponding candidate advantage value as input variables, used to achieve policy optimization through gradient ascent or descent.

[0171] Probability ratio truncation term: refers to a term that truncates the ratio of the current policy probability to the old policy probability, used to limit the update amplitude and avoid large deviation of the policy.

[0172] KL divergence-based regularization term: refers to a constraint or penalty term of the KL divergence between the current policy and the old policy added in the loss function, used to maintain the stability of policy update.

[0173] Final loss function: refers to the optimization target obtained by superimposing the probability ratio truncation term and / or the KL divergence regularization term on the basis of the policy gradient objective function, used to guide backpropagation.

[0174] Backpropagation: refers to the process of performing chain rule derivation on policy network parameters according to the final loss function, used to calculate parameter gradients.

[0175] Gradient clipping: refers to the operation of limiting the gradient amplitude within a preset threshold range before parameter update, to prevent gradient explosion.

[0176] Learning rate adjustment: refers to the strategy of dynamically adjusting the learning rate according to the training stage or gradient change during optimization, such as decay or adaptive adjustment.

[0177] Parameter synchronization: refers to the process of writing the updated policy parameters back to the MLE agent module and ensuring parameter consistency in each computing node or multi-process environment.

[0178] For example, in one specific example, the reinforcement learning optimizer has obtained the processed candidate advantage set {A1, A2, A3} and the corresponding candidate output generation probability set P = {p1, p2, p3} calculated in the previous stage. The optimizer first constructs a policy gradient term for each pair (Ai, pi), taking pi as the probability under the old policy, the corresponding probability p’i output by the current policy network as the new policy probability, and multiplying p’i / pi with Ai to form the basic target term.

[0179] Subsequently, the optimizer reads the old policy probability distribution from the history buffer, calculates the KL divergence between the current policy distribution and the old policy distribution, and superimposes the KL divergence regularization term in the loss function; at the same time, according to the preset truncation threshold, the upper and lower limit of p’i / pi is limited to form the probability ratio truncation term, so that the final loss function contains two parts of “truncated probability ratio x candidate advantage” and “KL divergence regularization”.

[0180] Next, the optimizer performs backpropagation on the final loss function to obtain the gradient vector of each layer parameter of the policy network. If it is detected that any gradient amplitude exceeds the preset threshold, the optimizer immediately performs gradient clipping to scale the gradient to the threshold range to avoid unstable parameter updates. After completing the gradient clipping, the optimizer updates the policy parameters according to the preset optimization algorithm (such as Adam or RMSProp). During the update process, if the loss is monitored to converge slowly or oscillate, the optimizer will trigger the learning rate adjustment strategy to decay or fine-tune the learning rate to ensure a smooth update process.

[0181] After completing the parameter update, the optimizer writes the new policy parameters to the MLE agent module and completes parameter synchronization through parameter broadcast or synchronization barrier mechanism in a multi-process or distributed scenario to ensure that the policy network weights called during subsequent code generation are the latest version.

[0182] In another example, if the candidate advantage distribution of a batch is extreme (most close to zero or all positive), the optimizer will add another layer of smoothing or scaling operation to the advantage value during the loss function construction stage to ensure that the gradient signal of the objective function is sufficient and not dominated by a single direction; if it is detected that the KL divergence is continuously too large, the optimizer will increase the KL regularization term weight or reduce the probability ratio truncation threshold to constrain the policy shift speed.

[0183] Through the above process, the reinforcement learning optimizer constructs the loss around the candidate advantage and the generation probability, completes the parameter update and synchronization, and provides the latest policy support for the code generation of the MLE agent module in subsequent multiple rounds of interaction.

[0184] In some embodiments, the multiple-round interaction control module is specifically configured to:

[0185] receive the execution result and the reward value of the previous round, and generate a feedback data packet according to a preset data structure;

[0186] fuse the feedback data packet based on the initial prompt or the prompt of the previous round to form an updated prompt for code generation of the next round;

[0187] send the updated prompt to the MLE agent module and trigger a code generation request;

[0188] record the interaction identifier of the current round, associate the generated executable code with the corresponding feedback data packet, and update the interaction state;

[0189] According to the preset termination condition, if the termination condition is not met, the execution result and the corresponding reward value of the current round are encapsulated as feedback data packets and entered into the next round of interaction.

[0190] Specifically, after receiving the execution result and the reward value of the previous round, the multi-round interaction control module generates a feedback data packet containing key fields according to a preset data structure; then performs semantic fusion on the feedback data packet and the initial prompt or the prompt of the previous round to form an updated prompt for code generation of the next round; after the updated prompt is generated, the module sends it to the MLE agent module and triggers a code generation request; at the same time, record the interaction identifier of the current round, associate the generated executable code with the corresponding feedback data packet, and update the interaction state; finally, according to the preset termination condition, if the termination condition is not met, the execution result and the corresponding reward value of the current round are encapsulated as feedback data packets and entered into the next round of interaction, until the termination condition is met.

[0191] Some key technical terms involved in this embodiment are explained as follows:

[0192] Feedback data packet: refers to a feedback entity formed by field extraction, archiving and serialization of the execution result and the reward value of the previous round according to a preset data structure, containing key fields such as execution result summary, normalized index set, reward value, error information overview, round identifier, etc.

[0193] Updated prompt: refers to a prompt text or equivalent structured instruction formed by fusing the feedback data packet reflecting the change requirements, missing links and parameter adjustment suggestions on the basis of the initial prompt or the prompt of the previous round, used to guide the MLE agent module to generate or modify the code in the next round.

[0194] Interaction identifier: refers to an identifier used to distinguish and track the interaction process of each round, which can include round number, timestamp, task ID, etc., and is used to establish a one-to-one correspondence between code output and feedback information in the multi-round process.

[0195] Interaction state: refers to the state set recording the current stage of the multi-round cycle, the number of completed rounds, the latest reward value, whether the termination condition is met, and other information, helping the module to decide whether to enter the next round.

[0196] Pre-set termination condition: refers to the termination rule set before the start of multi-round interaction, such as reaching the maximum number of rounds, the main performance index converging to the threshold, the resource budget being exhausted, or an unrecoverable error occurring.

[0197] Fusion strategy: refers to the rule set for content merging, priority analysis, and conflict resolution of the initial prompt and feedback data packet when generating an update prompt, ensuring that the update prompt not only inherits the original task goal but also accurately reflects the process elements that need to be adjusted in this round.

[0198] For example, in a specific example, the initial prompt requires building a classification model on the BreastCancerWisconsinDataset, with the goal of maximizing accuracy and training time not exceeding 30 seconds. After the first round of execution, the code executor outputs the execution results, and the evaluator and reward construction module give the normalized indicators and reward value R1=0.50, and point out that the feature standardization step is missing and the training time is not recorded.

[0199] The multi-round interaction control module first receives the execution results and R1, and generates a feedback data packet according to the pre-set data structure. The data packet contains: the model accuracy of the last round, the missing label of the training time field, the missing label of the feature engineering, the reward value R1, the error information summary (if any), and the round identifier ID1.

[0200] Then, the module executes the fusion strategy: based on the initial prompt, insert the two modification requirements of "need to add standardization" and "need to record training time" into the corresponding paragraphs of the prompt to form an updated prompt. For example, add "please record the training time and control it within 30 seconds" in the task constraint section, and insert "perform feature standardization before model training" in the process section. After fusion, the module sends the updated prompt to the MLE agent module and triggers the second round of code generation request.

[0201] At the same time of sending the request, the module records the interaction identifier ID2, and associates the executable code generated in the second round with the feedback data packet of the last round, updates the "number of completed rounds" in the interaction state to 2, the "latest reward value" to R1, and the "termination condition state" to not met.

[0202] After the second round of execution is completed, the execution result shows that the training time is 12.3 seconds and the accuracy is improved to 0.96, and the reward value R2 = 0.85. The module receives the execution result and R2 again, generates a new feedback data packet, which marks that "feature standardization has been added" and "training time record has been completed", and the evaluator indicates that the feature dimension can be further compressed by feature selection. The module fuses the feedback data packet with the last round of update prompts to form the third round of update prompts, adds the requirement of "performing feature selection to reduce input dimension" in the prompts, and triggers the third round of code generation.

[0203] After the third round ends, assuming that the reward value R3 = 0.92, the module updates the interaction state: the latest reward value R3 has reached the preset threshold, and the number of rounds does not exceed the maximum allowed number of times, which meets the "performance meets the standard" termination condition. The module determines that the termination condition is met, stops further iteration, and submits the finally generated executable code and related execution results to the subsequent deployment process.

[0204] In another example, if an execution interruption or serious error occurs in a certain round, the feedback data packet will contain the error type and error code, and the update prompt will prioritize guiding the MLE agent module to repair the error segment. If the resource budget is gradually consumed in multiple rounds of interaction, the interaction state will record the remaining budget, and when the budget is below the threshold, the termination condition is triggered, and the module will terminate subsequent rounds and output the latest available version of the final executable code.

[0205] Through the above process, the multi-round interaction control module implements key operations such as feedback generation, prompt update, interaction record, and termination judgment in each round, ensuring that the multi-round iteration process progresses in an orderly manner under controlled conditions.

[0206] In some embodiments, the system further comprises:

[0207] An experience storage module for storing task states, candidate codes, execution results, and reward values generated during the multi-round interaction process, and providing playback data to the reinforcement learning optimizer.

[0208] Specifically, the experience storage module receives and stores data elements such as task states, candidate codes, execution results, and reward values during the multi-round interaction process. First, the above elements are sorted, serialized, and labeled according to the preset data field specification to generate retrievable experience entries; then, multi-dimensional indexes are established according to task identifiers, round identifiers, and timestamps, and experience entries are written and version managed; when the reinforcement learning optimizer requests playback data, the module performs filtering and sampling according to its playback strategy requirements (such as reward interval, task type, round stage), combines the matching experience entries into a playback batch, and outputs; in the long run, the module archives, compresses, or eliminates experience data according to the storage strategy to maintain stable storage.

[0209] The concepts of some key technical terms involved in this embodiment are explained as follows:

[0210] Task state: refers to the context information set of the system at a certain round of interaction, including task description summary, current round number, last round feedback summary, strategy parameter version number, etc.

[0211] Candidate code: refers to the executable code text or its structured representation generated by the MLE agent module at a certain round, which usually corresponds one-to-one with the execution result of the code executor.

[0212] Execution result: refers to the encapsulated result object of the running log, model performance index, running time, resource occupation and error information output by the code executor.

[0213] Reward value: refers to the quantitative reward calculated by the reward construction module for the candidate output at this round, which is used for reinforcement learning optimization.

[0214] Experience entry: refers to the storage unit aggregated in a unified format by task state, candidate code, execution result and reward value, which is the basic access object in the experience storage module.

[0215] Multi-dimensional index: refers to the composite index key set established for the experience entry, including task ID, round ID, timestamp, reward interval label, error type label, etc., which is used for efficient retrieval.

[0216] Playback data: refers to the experience entry set sampled from the experience storage module according to the specified filtering conditions, which is used by the reinforcement learning optimizer in the offline or online training phase.

[0217] Version management: refers to versioning records of multiple writes for the same task or the same round, ensuring traceability of historical strategy parameters and corresponding experience data.

[0218] Archiving and elimination strategy: refers to a set of strategies for hierarchical storage, compression or deletion of experience entries according to data importance, time span or storage capacity limit.

[0219] For example, in a specific example, the system completes three rounds of interaction for the breast cancer dataset classification task. The experience storage module performs the following operations after each round:

[0220] First, the module obtains the task state of the current round from the multi-round interaction control module, which records the task ID, round number, update prompt summary, strategy parameter version number, etc.; obtains the candidate code text from the MLE agent module; obtains the execution result object from the code executor; obtains the reward value R from the reward construction module; and obtains the group average return value and candidate advantage value (if needed) for the candidate output group from the reinforcement learning optimizer.

[0221] Subsequently, the module encapsulates the above data elements into experience entries according to preset field specifications. For example, an entry contains fields such as task_id, round_id, timestamp, code_snippet_id, exec_result_ref, reward_value, advantage_value, error_flag, policy_version, etc. The module performs uniform encoding and length truncation on text fields, type checking on numerical value fields, and adds error type labels to error information fields.

[0222] After encapsulation is completed, the module establishes a main index according to task_id and round_id, and establishes reward interval label index according to the interval in which reward_value is located, and error category index according to error_flag. If there are multiple candidate outputs in this round, the module generates independent entries for each candidate output, and establishes intra-group association marks between the entries, so that subsequent playback can sample by group.

[0223] When the reinforcement learning optimizer needs to playback high-reward samples to strengthen the policy in the offline phase, the experience storage module receives its playback request, which specifies the filtering condition as "reward_value≥0.85 and error_flag=0". The module quickly locates the entry set that meets the condition according to the multi-dimensional index. If the number of sets exceeds the batch size, secondary filtering or random sampling is performed according to the timestamp or advantage value, the selected entries are packaged into a playback batch, and returned. If the optimizer requests diversified playback (both high-reward and low-reward samples need to be covered), the module extracts entries in different intervals according to the reward interval label, forming a more balanced playback data.

[0224] In another embodiment, to control the storage size, the experience storage module performs compression and archiving operations on entries that exceed the set retention period. Specifically, complete running logs and code texts can be moved to cold storage, and only summaries and key indicators are retained for quick filtering; for entries with very low reward values and error flags indicating serious errors, they can be directly deleted according to the elimination strategy after the policy is stable. The module updates the index mapping before archiving and eliminating to ensure that there are no dangling references.

[0225] In addition, when the policy network version is updated, the module will add the policy version number field to the experience entries of adjacent rounds before and after the update, to ensure that the optimizer can select experience of the same policy stage according to the version when playing back, avoiding playback bias caused by too large policy distribution difference.

[0226] Through the above process, the experience storage module realizes standardized storage, efficient retrieval and strategic playback of key data in the whole process of multiple rounds of interaction, thereby providing sustainable experience data basis for the reinforcement learning optimizer.

[0227] The foregoing embodiments introduce in detail the implementation of the multi-round automatic machine learning agent system based on reinforcement learning optimization of the present application. Based on the system, the embodiments further provide a multi-round automatic machine learning method based on reinforcement learning optimization. The implementation steps of the multi-round automatic machine learning method based on reinforcement learning optimization of the present application are explained and described below in combination with specific embodiments and drawings.

[0228] Figure 2 is a flowchart of the multi-round automatic machine learning method based on reinforcement learning optimization provided by the embodiments of the present application. As shown in Figure 2 , the method can specifically include the following steps:

[0229] S201, task analysis step: receiving the task description and constraint conditions input by the user, analyzing to obtain target indicators, resource limitations and dataset information, and generating an initial prompt according to a preset field mapping rule and a prompt template;

[0230] S202, code generation step: receiving the initial prompt and multiple rounds of feedback information by the MLE agent based on a large-scale pre-training language model, generating or modifying executable code according to a code generation rule and an operator flow template set;

[0231] S203, code execution step: running the executable code in a controlled execution environment, collecting running logs, model performance indicators, running time, resource occupation and error information, and forming an execution result;

[0232] S204, indicator calculation step: calculating multi-dimensional indicators of performance and running classes according to the execution result, outputting the normalized values of each indicator and generating a code correctness identifier, and obtaining a normalized indicator set;

[0233] S205, reward construction step: combining the normalized indicator set and the code correctness identifier according to a preset rule to generate a corresponding reward value;

[0234] S206, group return and advantage calculation step: calculating the group average return value for the reward values of multiple candidate outputs generated by the MLE agent for the same task input, and differentiating the reward value of each candidate output from the group average return value to obtain a candidate advantage value, and processing the candidate advantage value to obtain a processed candidate advantage set;

[0235] S207, policy parameter updating step: receive the processed candidate advantage set and the corresponding candidate output generation probability, construct a policy gradient objective function with the generation probability and the candidate advantage value as the independent variables, superimpose the probability ratio truncation term and / or the KL divergence-based regularization term in the objective function to obtain the final loss function, perform back propagation to calculate the gradient and update the policy parameters according to the preset optimization algorithm, and simultaneously perform gradient clipping and learning rate adjustment, and write the updated policy parameters to the MLE agent;

[0236] S208, multi-round interaction control step: encapsulate the execution result of the last round and the reward value as a feedback data packet and fuse it with the initial prompt or the last round prompt to form an updated prompt, send the updated prompt to the MLE agent to trigger the next round of code generation, and judge whether to end according to the preset termination condition, and repeat the above steps if not.

[0237] The implementation process of the technical scheme of the present application will be described in detail below in conjunction with specific examples, which specifically includes the following operations:

[0238] I. Input

[0239] Input task: The user hopes to build a classification model for breast cancer diagnosis (Breast Cancer Wisconsin Dataset), and requires to improve the accuracy as much as possible and limit the model training time within 30 seconds.

[0240] Task prompt (Prompt): Please build a classification model based on sklearn to predict the breast cancer dataset. The goal is to maximize the accuracy and limit the model training time to no more than 30 seconds.

[0241] II. Multi-round reasoning

[0242] First round generation (initial output):

[0243] The MLE Agent generates the following Python code:

[0244] from sklearn.datasets import load_breast_cancer

[0245] from sklearn.linear_model import LogisticRegression

[0246] from sklearn.model_selection import train_test_split

[0247] from sklearn.metrics import accuracy_score

[0248] data = load_breast_cancer()

[0249] X_train, X_test, y_train, y_test = train_test_split(data.data,data.target)

[0250] model = LogisticRegression()

[0251] model.fit(X_train, y_train)

[0252] y_pred = model.predict(X_test)

[0253] print("Accuracy:", accuracy_score(y_test, y_pred))

[0254] Evaluation feedback: Runs normally, no syntax errors; accuracy is 0.92, but the default parameters of the model are not tuned; no feature standardization processing and no training time recorded; Reward = 0.5 (runs normally + results are average + lack of metric control).

[0255] Second round of correction output:

[0256] from sklearn.preprocessing import StandardScaler

[0257] from sklearn.ensemble import RandomForestClassifier

[0258] import time from time

[0259] scaler = StandardScaler()

[0260] X_train_scaled = scaler.fit_transform(X_train)

[0261] X_test_scaled = scaler.transform(X_test)

[0262] start_time = time()

[0263] model=RandomForestClassifier(n_estimators=100,max_depth=6, random_state=42)

[0264] model.fit(X_train_scaled, y_train)

[0265] print("Train time:", time() - start_time)

[0266] y_pred = model.predict(X_test_scaled)

[0267] print("Accuracy:", accuracy_score(y_test, y_pred))

[0268] Evaluation feedback: Standardization was added; training time was recorded as 12.3 seconds; accuracy improved to 0.96; Reward = 0.85 (significant improvement).

[0269] Third round of improvement output:

[0270] from sklearn.feature_selection import SelectKBest, f_classif

[0271] selector = SelectKBest(score_func=f_classif, k=10)

[0272] X_train_selected = selector.fit_transform(X_train_scaled, y_train)

[0273] X_test_selected = selector.transform(X_test_scaled)

[0274] model=RandomForestClassifier(n_estimators=100,max_depth=6, random_state=42)

[0275] model.fit(X_train_selected, y_train)

[0276] y_pred = model.predict(X_test_selected)

[0277] print("Accuracy:", accuracy_score(y_test, y_pred))

[0278] Evaluation feedback: feature selection added, dimension reduced from 30 to 10; training faster, performance maintained at 0.95+ accuracy; code structure clear; Reward=0.92 (simplified + efficient).

[0279] III. GRPO training instructions (brief)

[0280] The above 3 rounds are regarded as "candidate output groups", and the system constructs a reward distribution based on the execution result of each round:

[0281] R1=0.5, R2=0.85, R3=0.92;

[0282] Calculate the group average reward (μ≈0.76) using GRPO, and then calculate the advantage A_i=R_i-μ;

[0283] Optimize the output probability of the policy network through the GRPO algorithm, so that the future is more inclined to generate high-reward codes like R3.

[0284] It should be understood that the order of the steps in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0285] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the technical solutions of the present application are described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A multi-round automated machine learning agent system based on reinforcement learning optimization, characterized in that, The method comprises the following steps: a task analysis module for receiving user input task description and constraint conditions and analyzing target indicators, resource limits and data set information to generate initial prompts for MLE agent calls; an MLE agent module for receiving the initial prompts and multiple rounds of feedback information based on a large-scale pre-trained language model to generate executable code; a code executor for running the executable code in a controlled execution environment, collecting running logs, model performance indicators, running time, resource occupation and error information to form execution results; an evaluator for calculating multi-dimensional indicators of performance and running classes according to the execution results and outputting normalized values of each indicator and code correctness identification; a reward construction module for combining normalized indicators and the code correctness identification according to preset rules to generate corresponding reward values; a reinforcement learning optimizer for calculating group average return and candidate advantage for multiple candidate outputs generated by the MLE agent module for the same task input, and updating policy parameters of the MLE agent module based on the candidate advantage; a multi-round interaction control module for feeding back the execution results of the previous round and the reward values to the MLE agent module in the multi-round interaction process to control the next round of code generation until the preset termination condition is met; wherein the code executor is specifically configured to: load dependent libraries and data resources required by the executable code and initialize an execution container according to a preset resource quota; trigger execution of the executable code and perform event listening during execution to record running logs; calculate the model performance indicators according to the preset evaluation interface for training and prediction stage outputs; statistically calculate the running time and resource occupation through timestamp markers and resource monitoring probes respectively; classify the exception types, error codes and interruption states captured during execution to generate the error information; aggregate and encapsulate the running logs, model performance indicators, running time, resource occupation and error information into the execution results according to the preset data structure; the evaluator is specifically configured to: perform field analysis and data cleaning on the execution results to obtain an original indicator data set; calculate performance indicators by comparing the model prediction output and the labeled data according to the preset performance indicator calculation rules; aggregate time records and resource monitoring data according to the preset running indicator statistical rules to calculate running indicators; determine the code correctness identification according to the exception types, error codes and interruption states in the error information; normalize the performance indicators and running indicators according to a unified normalization strategy to generate a normalized indicator set; the reward construction module is specifically configured to: call a preset indicator weight mapping table and a combination rule library to assign weight coefficients to each normalized indicator and determine combination operators; divide the normalized indicators into performance and running classes to generate corresponding sub-reward components; calculate a penalty factor or an additive factor according to the code correctness identification; performing a weighted combination and a segment function operation on the sub-reward component and the penalty factor or the bonus factor to obtain an initial reward value; performing interval clipping and quantization coding on the initial reward value to form the reward value.

2. The system of claim 1, wherein, The task analysis module is specifically configured to: perform semantic analysis and field identification on the task description and the constraint condition, and extract key parameters corresponding to target indicators, resource limits and dataset information; organize the key parameters into a structured expression according to a preset field mapping rule and a prompt template; perform format checking and parameter normalization processing on the structured expression to obtain an initial prompt meeting the calling requirements.

3. The system of claim 1, wherein, The MLE agent module is specifically configured to: perform semantic analysis on the initial prompt and the multi-round feedback information to determine machine learning process elements to be generated or modified; generate a code segment corresponding to the process elements according to a preset code generation rule and an operator process template set; perform processing on the code segment to generate a complete executable code; in subsequent rounds, locate the code position to be adjusted according to the execution result and the reward value of the last round, and perform incremental generation or replacement, and output the updated executable code to the code executor.

4. The system of claim 1, wherein, The reinforcement learning optimizer is specifically configured to: receive the reward value and establish a candidate output group according to a sampling batch; perform statistics on the reward values in the candidate output group to obtain a group average return value; differ the reward value of each candidate output from the group average return value to obtain a corresponding candidate advantage value; perform processing on the candidate advantage value to generate a processed candidate advantage set; associate the processed candidate advantage set with the generation probability of the corresponding candidate output.

5. The system of claim 4, wherein, The reinforcement learning optimizer is further configured to: receive the candidate advantage value and the generation probability of the corresponding candidate output; build a policy gradient objective function with the generation probability and the candidate advantage value as independent variables; add a probability ratio truncation term and / or a regularization term based on KL divergence to the objective function to obtain a final loss function; perform back propagation on the final loss function to calculate the gradient; update the policy parameters according to a preset optimization algorithm, and perform gradient clipping and learning rate adjustment during the update process; write the updated policy parameters to the MLE agent module and complete parameter synchronization.

6. The system of claim 1, wherein, The multi-round interaction control module is specifically configured to: receive the execution result and the reward value of the last round, and generate a feedback data packet according to a preset data structure; fuse the feedback data packet based on the initial prompt or the last round prompt to form an updated prompt for the next round of code generation; send the updated prompt to the MLE agent module and trigger a code generation request; record the interaction identifier of the current round, associate the generated executable code with the corresponding feedback data packet, and update the interaction state; determine whether the interaction state meets the preset termination condition, and if not, enter the next round of interaction with the execution result and the corresponding reward value of the current round as feedback.

7. The system of claim 1, wherein, The system further comprises: An experience storage module is configured to store task states, candidate codes, execution results and reward values generated in a multi-round interaction process, and provide playback data to the reinforcement learning optimizer.

Images