An automated machine learning method based on reinforcement learning agents

Abstract

The application discloses an automatic machine learning method based on a reinforcement learning intelligent agent. The original data of initial input is diagnosed and corrected to obtain corrected data; the corrected data is encoded to obtain encoded data; a reinforcement learning intelligent agent is constructed, the encoded data is input into the reinforcement learning intelligent agent to automatically learn machine to obtain data with better performance, and the data with better performance can be used for actual application control. The whole search learning process is carried out in parallel, no additional resource consumption and time cost are generated, the model establishment process does not need intervention, the model precision is improved while the labor cost is reduced, the degree of manual participation is reduced, and the model generalization is higher.

Description

Technical Field

[0001] This invention relates to a machine learning method in the field of artificial intelligence, specifically to an automated machine learning method based on reinforcement learning agents. Background Technology

[0002] Automated Machine Learning (AutoML) aims to reduce the need for data scientists and enable domain experts to automatically build machine learning models to solve problems in real-world industrial applications without requiring extensive statistical and machine learning knowledge.

[0003] Several existing methods have demonstrated the feasibility of automated machine learning, but most fully automated machine learning frameworks focus primarily on model and hyperparameter search, rarely incorporating feature engineering. Examples include Auto-Sklearn based on Bayesian optimization and TPOT based on genetic algorithms. There is some independent research on feature automation, exemplified by the state-of-the-art DIFER and the influential Neural Feature Search (NFS). Current challenges include: 1. Lack of high-order complex features. Due to efficiency limitations or fixed network structures, most methods cannot generate high-order complex features. For instance, NFS cannot generate complex features like (A+B) / (C+D), where A, B, C, and D are features of the original data. 2. Ignoring the coupling between features. Both NFS and DIFER focus on finding high-performance individual features and then combining them through simple feature selection. This leads to the neglect or loss of some excellent feature sets. 3. High feature sample complexity. Feature engineering explores structured data at the finest granularity (i.e., individual features). Each feature requires a separate controller to perform exploration and development, resulting in increased time and resource consumption. 4. Uncontrollable feature explosion. A rapid increase in the number of generated features leads to huge resource consumption (time and space). DIFER addresses this issue by limiting the feature space, but this results in the loss of some high-quality features. Summary of the Invention

[0004] To address the problems existing in the background technology, the purpose of this invention is to solve the shortcomings of current automatic machine learning technology for structured data by building a reinforcement learning agent composed of a deep network, thereby simultaneously solving problems such as feature engineering, model search, and hyperparameter optimization.

[0005] To address the shortcomings of the prior art, the present invention provides the following technical solution:

[0006] 1) First, perform diagnostic correction on the initial input raw data to obtain the corrected data;

[0007] 2) Encode the corrected data to obtain encoded data;

[0008] 3) Construct a reinforcement learning agent. Input the encoded data into the reinforcement learning agent for automated machine learning to obtain data with better performance. This better data can then be used for practical application control. Define the original dataset as an initial state sequence of actions without any operations.

[0009] The raw data mentioned is structured data, such as process parameter data for 3D printing.

[0010] In practice, the process parameters of 3D printing can be optimized to obtain the optimal process parameters for 3D printing, which can then be applied to the 3D printer for 3D printing.

[0011] Step 1) specifically involves: diagnosing whether the original data meets the search requirements, diagnosing and processing data such as duplicates and missing values ​​to make it qualified data for model training, which is then used as corrected data.

[0012] The qualified data comprises a matrix of data consisting of multiple features. Apart from the label column, one column in the data serves as a feature.

[0013] Step 2) specifically involves: unordering the column names of each feature column in the corrected data, and then adding the operation parameters for fusing the feature columns and the models to be selected and their hyperparameters to form the encoded data; the encoded data includes five parts: the result encoded by the column names of the feature columns, the operation parameters for fusing the feature columns, the result encoded by all the models to be selected, the result encoded by the combination method for combining different models, and the hyperparameters of all the models to be selected.

[0014] The feature columns of the original dataset are encoded, and this set is defined as the initial feature set without any further processing.

[0015] The models to be selected are several models chosen from all possible models.

[0016] The reinforcement learning agent in step 3) consists of multiple time steps performed sequentially with increasing order. Each time step takes the encoded action sequence as input and passes through each time step of the reinforcement learning agent. Each time step processes and optimizes the actions output by each module (feature generation action, feature fusion action, model selection action, model hyperparameter) and splices them together to form the action sequence of that time step, which is then input into the next time step. The action sequences generated by all time steps are merged to form the final output action sequence.

[0017] Except for the last time step, each time step mainly consists of the following actions performed sequentially: feature generation, feature fusion, model selection, and model hyperparameters. The last time step consists of the model fusion action.

[0018] The feature generation action involves generating several transformation functions and then using these transformation functions to construct more new feature columns from the input data. The transformation functions include, but are not limited to, one or more combinations of unary operations such as value transformation (e.g., logarithmic operations, exponential operations, etc.) and binary operations such as addition, subtraction, multiplication, and division.

[0019] The feature fusion action is to combine the new feature column generated by the feature generation action with the feature column in the original input data only by replacing or merging. Replace means replacing the feature column in the original input data with the new feature column, and merge means adding the new feature column to the feature column in the original input data.

[0020] The model selection process involves searching among all possible models to find the N models that best suit the data task.

[0021] The model fusion action is the selection of a combination method for the models searched by the model selection action.

[0022] The model hyperparameter action is a set of built-in hyperparameters that best perform on the input data, which is searched for the model selection action;

[0023] Furthermore, the weights are shared among the feature generation actions at different time steps, the weights are shared among the feature fusion actions at different time steps, the weights are shared among the model selection actions at different time steps, and the weights are shared among the model hyperparameter actions at different time steps.

[0024] The feature generation process involves transforming the input feature columns according to preset operations to generate new feature columns, thereby generating higher-order features.

[0025] The higher-order features refer to features generated by performing multiple transformations on the original features.

[0026] The encoded data is divided into training and test sets according to 5-fold cross-validation. The model is trained on the training set using the final merged action sequence, and the reward parameter is obtained on the test set and fed back to the reinforcement learning agent.

[0027] Furthermore, after the time step outputs new data using feature generation and feature fusion actions on the original data, it uses a verification method to filter invalid feature columns through a threshold. The ratio between the number of invalid feature columns and the total number of feature columns is used as the invalid feature ratio. The invalid feature ratio is used as a penalty factor to penalize the reward parameter. Then, the penalized reward parameter is fed back to the reinforcement learning agent, so that the reinforcement learning agent gradually learns to search for feature combinations that contribute more to the search in the continuous optimization process.

[0028] The reward parameter is penalized based on the ratio of invalid features according to the following formula:

[0029] reward′=0.001*γ*reward

[0030]

[0031] Where γ represents the invalid feature ratio, num(X) del ) represents the number of invalid feature columns, num(X) represents the number of all feature columns after the search, reward represents the reward parameter before the penalty, and reward′ represents the reward parameter after the penalty.

[0032] This invention has the following technical features:

[0033] (1) Use a single controller to perform feature engineering, model and hyperparameter search, avoiding the time and resource consumption caused by each module using the controller independently;

[0034] (2) Addressing some challenges in existing automated feature engineering:

[0035] a) Integration of a replacement-append mechanism in feature engineering. The search is performed only at the original feature locations, determining whether to replace the original feature or add a new one. Continuous replacement and exploration ensure the possibility of generating higher-order complex features, and searching only at the initial locations guarantees that the number of features increases linearly at its maximum.

[0036] b) By employing a collaborative feature search approach, the coupling effect of features is enhanced, thus addressing the problem that NAS cannot generate complex features. The beneficial effects of this invention are:

[0037] The entire search and learning process of this invention is carried out in parallel, without generating additional resource consumption and time costs. The model building process requires no intervention, reducing human resources costs while improving model accuracy. It can reduce the degree of human involvement and has higher model generalization ability.

[0038] In 3D printing, this method can be used to generate experimental data with as few experiments as possible. Through automated machine learning, some latent features can be derived from existing parameter features. Based on this, models and hyperparameters can be automatically selected for training to fit the real machine tool printing environment. By inputting different parameters, the printing effect can be simulated, and the parameters with excellent performance can be recommended. This can save a lot of raw materials and time consumption in real experiments.

[0039] Generally, the manual trial and error cycle for new materials involves at least 90 different sets of printing parameters. With the recommended parameters, only 3 sets of printing parameters are needed to find the printing parameters that meet the product requirements. Attached Figure Description

[0040] Figure 1 This is a flowchart of the optimized search process for reinforcement learning and automated machine learning that integrates feature engineering in this invention.

[0041] Figure 2 This is the core structure of the reinforcement learning agent policy network of this invention.

[0042] Figure 3 This is the search and sampling process of the strategy network controller of the present invention at a single time step;

[0043] Figure 4 This is the performance of the present invention (HAFE) on 21 datasets;

[0044] Figure 5 This invention demonstrates the prediction of crack and density performance on a dataset of process parameters for 3D printing. Detailed Implementation

[0045] To make the technical problems, technical solutions, and beneficial effects of the embodiments of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are only for explaining the present invention and do not limit the present invention.

[0046] To achieve the above objectives, specific embodiments of the present invention and their implementation processes include:

[0047] Specifically, the process parameters for 3D printing are optimized to obtain the best possible process parameters, which are then applied to the 3D printer for 3D printing.

[0048] 1) The user initially inputs the initial process parameters for 3D printing and all possible models to construct the original data. The original data is then diagnosed and corrected to obtain the corrected data.

[0049] The system diagnoses whether the original data meets the search requirements, identifies duplicates and missing values, and processes them into qualified data suitable for model training, which is then used as the corrected data.

[0050] In practice, evaluation indicators are set and optimized according to the type of task.

[0051] The task type is determined by the nature of the labels to be predicted: if the prediction result is a countable set of options, it is a classification task type; if the prediction result is an uncountable infinite set, it is a regression task type.

[0052] For classification tasks, optimized evaluation metrics can be accuracy, F1 score, AUC score, and KS score; for regression tasks, optimized evaluation metrics can be correlation coefficient, mean absolute error, mean relative error, and relative absolute error, etc.

[0053] 2) Encode the corrected data to obtain encoded data;

[0054] The column names of each feature column in the corrected data are unordered and then the operation parameters for fusing the feature columns and the model to be selected and its hyperparameters are added to form the encoded data.

[0055] The encoded data includes five parts: the result encoded by the column names of the feature columns, the operation parameters used to fuse the feature columns, the result encoded by all models to be selected, the result encoded by the combination method used to combine different models, and the hyperparameters of all models to be selected.

[0056] Note that this process does not directly encode the data; only the column names are encoded. The encoded column names participate in the search, and the search action is applied to the corresponding feature columns. The advantage of this approach is that regardless of the size of the original data, it does not participate in the search encoding process. Encoding can be completed with only a small number of parameters, and the encoding size is independent of the number of samples, making data processing very lightweight.

[0057] The aforementioned unordered encoding refers to shuffling the positions of feature columns on the same dataset, and the encoding of the column names will not affect the final model training score.

[0058] In this invention, "feature" refers to a feature column. A feature or feature column means defining the columns of the original dataset as {x1, x2, ... x...}. m}, where x i A feature column refers to a column in the data, which usually has a specific physical meaning, such as gender, age, or grades. It is also called a feature column. The original data has a total of m feature columns.

[0059] 3) Construct a reinforcement learning agent, input the encoded data into the reinforcement learning agent for automated machine learning to obtain data with better performance, and use the better data for implementation control.

[0060] The reinforcement learning agent in step 3) consists of multiple time steps that proceed sequentially with increasing order. The encoded data is used as input and passes through each time step of the reinforcement learning agent. At each time step, the data is processed and optimized, and the updated encoded data is then input into the input data of the next time step. The last time step outputs the updated encoded data as the final output data of the reinforcement learning agent.

[0061] In practice, reinforcement learning agents can be implemented using LSTM.

[0062] Each time step mainly consists of five complete actions performed sequentially: feature generation, feature fusion, model selection, model fusion, and model hyperparameter search. These correspond to feature search, feature combination search, model search, model combination search, and model hyperparameter search, respectively.

[0063] The feature generation action involves generating several transformation functions and then using these transformation functions to construct more new feature columns from the input data, supplementing the original data and enriching the data information obtained by the subsequent reinforcement learning agent. The transformation functions include, but are not limited to, one or more combinations of unary operations such as value transformation (e.g., logarithmic operations, exponential operations) and binary operations such as addition, subtraction, multiplication, and division.

[0064] The feature fusion action combines the new feature columns generated by the feature generation action with the feature columns in the original input data by simply replacing or merging them. Replace means replacing the feature columns in the original input data with the new feature columns, and merge means adding the new feature columns to the feature columns in the original input data. In other words, a new column is added to the feature columns in the original input data to supplement the new feature columns.

[0065] The model selection process involves searching among all possible models to find the N models most suitable for the data task. During use, the model space can be expanded and adjusted according to the actual application scenario.

[0066] All possible models include Logistic Regression, Random Forest, and Gradient Boosting Tree (XGBOOST / LightGBM); regression task models include Linear Regression, Random Forest, and Gradient Boosting Tree (XGBOOST / LightGBM), etc.

[0067] Model fusion action is the selection of a combination method for the models searched for by model selection action;

[0068] Optional combination methods include the Waterfall Model fusion method or the voting or weighted fusion method.

[0069] The waterfall fusion method uses a combination of multiple models in series. Specifically, each model is treated as a filter, and the filters of different types are connected one after the other.

[0070] Voting or weighted fusion methods use the prediction results of individual models to vote (classification) or weight (regression) to obtain the final prediction result, thereby reducing the impact of overfitting.

[0071] Model hyperparameter actions are a set of built-in hyperparameters that best perform on the input data and are selected by the model's action search. It is a combination of the model and its built-in parameters.

[0072] Furthermore, the weights are shared among feature generation actions at different time steps, feature fusion actions at different time steps, model selection actions at different time steps, model fusion actions at different time steps, and model hyperparameter actions at different time steps.

[0073] In the feature generation process, the input feature columns are transformed multiple times according to preset operations to generate new feature columns, thereby generating higher-order features. This can bring more linear and non-linear information between features to the model, enabling the model to better fit the data in business scenarios.

[0074] For example, the feature generation action in the first time step uses a logarithmic operation, that is, the 0th-order features of the original input feature column x1. After taking the logarithm, it becomes a first-order feature of feature column x1. The feature generation action in the second time step uses the square root (sqrt) operation, that is, in the first-order features of feature column x1... Based on this, take the square root (sqrt) to transform it into a second-order feature of feature column x1. The third-order and higher-order processing at the third time step follows the same pattern, with the highest order n being set before optimization.

[0075] In the present invention, the feature fusion action is only set to replace and merge for combination operation. In this way, as the time step increases, that is, as the highest order n increases, the number of feature columns in the final output data of the reinforcement learning agent increases only in a linear manner, rather than increasing explosively in feature dimension. This effectively avoids the feature dimension explosion phenomenon and avoids the generation and processing of a large amount of data.

[0076] The number of feature columns increases only linearly, specifically:

[0077] When the original feature set is {x1, x2, ..., x...} m The first feature column x1 is transformed into a new feature column through logarithmic transformation. but:

[0078] If new feature column If the combination method with the original features is "replace", then the newly generated feature column is considered to be... The information carried contains and exceeds the original feature column, replacing the position of the original feature column x1 to participate in the higher-order feature search below. At this time, the total feature set becomes The number of features will not increase;

[0079] If new feature column If the combination method with the original feature column x1 is merge, then the new feature column is considered to be merged. The information obtained is additional information that does not overlap with the original feature column information; at this point, the total feature set becomes... The number of features increases by one, and m feature columns participate in the feature search.

[0080] For higher-order feature searches, each search only searches for higher-order features, i.e., new feature columns, on the feature columns carrying the original feature information. It does not participate in the search for second-order features, thus generating at most m new feature columns that can be merged.

[0081] Therefore, the final total feature set is at most n*m (i.e., when all generated features are merged), where m is the number of feature columns in the original input data and n is the highest order.

[0082] In the specific implementation of step 3), the encoded data is divided into a training set and a test set according to 5-fold cross-validation. The training set is used to train the reinforcement learning agent, and the test set is used to test the trained reinforcement learning agent to obtain the reward parameter and feed it back to the reinforcement learning agent.

[0083] Furthermore, at each time step, after the data output by the model's hyperparameter actions, invalid feature columns are filtered out through thresholds using tests such as F-test, mutual information, and chi-square test. The ratio between the number of invalid feature columns and the total number of feature columns is used as the invalid feature ratio. The invalid feature ratio is used as a penalty factor to penalize the reward parameter. The penalized reward parameter is then fed back to the reinforcement learning agent, enabling the reinforcement learning agent to gradually learn to search only for feature column combinations that contribute more together during the continuous optimization search process.

[0084] The reward parameter is penalized based on the ratio of invalid features according to the following formula:

[0085] reward′=0.001*γ*reward

[0086]

[0087] Where γ represents the invalid feature ratio, num(X) del ) represents the number of invalid feature columns, num(X) represents the number of all feature columns after the search, reward represents the reward parameter before the penalty, and reward′ represents the reward parameter after the penalty.

[0088] The reinforcement learning agent of the present invention iterates based on the feedback reward parameter. During the iteration process, the reward parameter of the generated action combination on the dataset gradually improves. When the maximum preset number of iterations or the early stopping condition is reached, the iteration stops and returns to the action combination inferred in the current state of the reinforcement learning agent.

[0089] This is the set of actions that the reinforcement learning agent automatically searches out to find the best performing combination on the dataset, including feature engineering (high-order feature generation methods, feature combination methods), model fusion (N models and fusion methods), and model hyperparameters suitable for the dataset.

[0090] Therefore, by processing data through the method of this invention, we can learn and optimize the data for a specific application scenario to find better data results that improve the performance of data applications, and then feed them back into practical applications for implementation.

Claims

1. A 3D printing process parameter automation machine learning method based on reinforcement learning agent, characterized in that The method comprises: 1) first, the original data of the initial input 3D printing process parameters are diagnosed and corrected to obtain corrected data; 2) the corrected data is encoded to obtain encoded data; 3) an intelligent agent of reinforcement learning is constructed, and the encoded data is input into the intelligent agent of reinforcement learning to obtain data with better performance through automatic machine learning; The intelligent agent of reinforcement learning in step 3) comprises a plurality of time steps which are sequentially performed and the order of which is sequentially increased, each time step taking a sequence of actions as input, sequentially passing through each time step of the intelligent agent of reinforcement learning, each time step processing and optimizing the output actions to form a sequence of actions of the time step, and then inputting into the next time step, and the action sequences generated by all time steps are combined as the final output action sequence; Each time step is composed of a feature generation action, a feature fusion action, a model selection action and a model hyperparameter action, except for the last time step, which is composed of a model fusion action: The feature generation action is to generate a plurality of conversion functions, and then use the conversion functions to construct more new feature columns from the input data, the conversion functions including but not limited to one or a combination of value conversion unary operation and addition, subtraction, multiplication and division binary operation; The feature fusion action is to combine the new feature columns generated by the feature generation action with the feature columns in the original input data through replace and merge, replace means replacing the feature columns in the original input data with the new feature columns, and merge means appending the new feature columns to the feature columns in the original input data; The model selection action is to search for N models most suitable for the data task from all possible models; The model fusion action is to select the combination mode of the models searched by the model selection action; The model hyperparameter action is to search for a set of model built-in hyperparameters that perform best on the input data from the models searched by the model selection action; The feature generation actions of different time steps share weights, the feature fusion actions of different time steps share weights, the model selection actions of different time steps share weights, and the model hyperparameter actions of different time steps share weights; In the feature generation action, the input feature columns are transformed according to the preset operation to generate new feature columns, and higher order features are generated.

2. The method of claim 1, wherein the method is based on a reinforcement learning agent. Step 1) is specifically: diagnosing whether the original data meets the search requirements, diagnosing and processing whether there are duplicates and missing values to obtain qualified data for model training as corrected data.

3. The method of claim 1, wherein the method is based on a reinforcement learning agent for 3D printing process parameter automation machine learning. Step 2) is specifically: The column names of each feature column in the corrected data are unordered encoded, and then the operation parameters for fusing the feature columns and the to-be-selected models and their hyperparameters are added to form the encoded data; The encoded data includes the encoded results of column names of feature columns, operation parameters for fusing the feature columns, encoded results of all to-be-selected models, encoded results of combination modes for combining different models, and five parts of hyperparameters of all to-be-selected models.

4. The method of claim 1, wherein: The encoded data is divided into a training set and a test set according to 5-fold cross validation, the model is trained on the training set using the finally merged action sequence, the reward parameter reward is obtained on the test set, and the reward parameter reward is fed back to the reinforcement learning agent; And after the time step uses the feature to generate the action and the feature fusion action to output new data from the original data, the invalid feature columns are screened out through the threshold value using the test mode, the ratio between the number of invalid feature columns and the total number of feature columns is taken as an invalid feature ratio, the reward parameter reward is punished by taking the invalid feature ratio as a punishment factor, and then the punished reward parameter reward is fed back to the reinforcement learning agent, so that the reinforcement learning agent gradually learns to search out a feature combination with greater contribution together in the process of continuous optimization and search.

5. The method of claim 4, wherein: The reward parameter reward is punished according to the invalid feature ratio according to the following formula: reward'=0.001*γ* reward gamma = num(X del ) / num(X) where γ denotes the invalid feature ratio, num(X del ) is the number of invalid feature columns, num(X) is the number of all feature columns after searching, reward denotes the reward parameter before punishment, and reward' denotes the reward parameter after punishment.

Images