Automatic feature engineering method based on monte carlo tree search and large language model

Abstract

The application discloses an automatic feature engineering method based on Monte Carlo tree search and a large language model, which comprises inputting feature data such as a table data set into a computer system; the computer system performs feature initialization, evaluates the score of each initial feature input, and constructs an initial feature tree structure; the steps of candidate feature selection, expansion based on a large language model, feature score evaluation and score parameter updating are iteratively performed until the termination condition is met; based on the selected features, the large language model is called in combination with the prompt word engineering to expand and generate new feature nodes; the new feature nodes are substituted into a predefined end-to-end machine learning pipeline for evaluation, and the feature score is comprehensively calculated; the related parameters of all ancestor nodes in the feature tree are updated from bottom to top along the feature synthesis path; and after the iteration is terminated, the optimal feature set capable of improving the prediction performance of a basic machine learning model is output. The application can significantly improve the accuracy of a machine learning model in a table data prediction task.

Description

Technical Field

[0001] This invention relates to natural language processing technology, a key technology in artificial intelligence, and particularly to an automatic feature engineering method based on Monte Carlo tree search and large language models. Background Technology

[0002] Feature engineering (FE) is the process of transforming raw data to generate new features for machine learning (ML) models. Due to its significant effect on improving the predictive performance of machine learning models, this technique has long been highly regarded, especially when dealing with tabular data. Early feature engineering methods often relied on manual feature design, requiring substantial human and time investment. These manual methods could only utilize a limited number of predefined rules for feature transformation, resulting in an extremely limited search space and unsatisfactory efficiency and performance when faced with a large number of candidate features and operators.

[0003] To address the aforementioned issues, Automated Feature Engineering (AutoFE) has garnered significant attention over the past decade. Specifically, existing methods have applied various machine learning techniques to automated feature engineering tasks, including tree-based enumeration, greedy search, neural search, differentiable optimization, and reinforcement learning. However, while these methods effectively reduce manual workload, they remain limited to small, predefined search spaces during feature generation. They primarily rely on a few predefined arithmetic operators such as addition and subtraction for feature combination, neglecting more efficient conditional transformations and multivariate operators. Furthermore, these methods fail to fully leverage the domain knowledge relied upon by domain experts to construct high-value features. Because they are typically constrained by predefined combinations, they cannot meet the specific application requirements of real-world scenarios, thus impacting the effectiveness of generated features.

[0004] In recent years, large language models (LLMs) have demonstrated great potential in automatic feature engineering tasks. As disclosed in Reference 1 (Zhang, Y., Li, N., Gan, Q., Zhang, W., Wipf, D., Wang, M.: ELF-Gym: Evaluating large language models generated features for tabular prediction. In: CIKM, pp. 5420–5424 (2024)), features generated by large language models significantly outperform traditional automatic feature engineering methods in prediction performance. Typical large language model-driven methods (such as CAAFE, FeatLLM, OCTree, FEBP, ReaGen, and FELIX) can leverage the natural language understanding capabilities of large language models to extract semantic information from tabular data and integrate domain knowledge to synthesize valuable features for specific tasks.

[0005] While existing automated feature engineering methods based on large language models have shown significant effectiveness, they still face two major challenges. First, the feature synthesis process lacks stability. Specifically, it struggles to guide feature exploration towards higher quality across multiple iterations, primarily due to the inherent ambiguity in the synthesis process. Since these methods rely on combinations of existing features to generate new features, and the combined effect is highly uncertain, it's difficult to determine whether new features can better capture potential data patterns, thereby improving the predictive performance of machine learning models. Second, feature repetition frequently occurs in multiple generation rounds, such as the continuous generation of semantically or structurally similar variants. This is mainly because the structure and content of prompt words are quite similar across iterations, and large language models have limited knowledge of features generated in previous rounds, further exacerbating the problem. Furthermore, since large language models reference examples in prompt words, they may form self-reinforcing loops. This repetition can cause the search space to get stuck in local optima, reducing feature diversity.

[0006] It is worth noting that existing research has proposed various strategies to address the repetition problem in text and code generation for large language models. However, these strategies are designed for single-input scenarios and cannot effectively reduce the problem of repeated generation in multiple iterations. Summary of the Invention

[0007] The main objective of this invention is to provide an automatic feature engineering method based on Monte Carlo tree search and large language models. By integrating Monte Carlo tree search and large language models, this method solves the problems of unstable feature synthesis and feature duplication in multiple rounds of generation in existing automatic feature engineering methods, generating high-quality and diverse tabular data features, thereby improving the predictive performance of basic machine learning models.

[0008] Based on the first main aspect of the present invention, an automatic feature engineering method based on Monte Carlo tree search and large language models is provided, comprising the following steps performed by a computer system:

[0009] Input the tabular dataset, dataset metadata, basic machine learning model and feature generation configuration information into the computer system. The tabular dataset contains multiple records and corresponding original feature sets and label sets. The metadata describes the dataset and details of each original feature.

[0010] The computer system performs feature initialization, evaluates the score of each initial feature of the input, and constructs an initial feature tree structure;

[0011] Iteratively execute the steps of candidate feature selection, expansion based on the large language model, feature score evaluation, and score parameter update until the termination condition is met: When selecting candidate features, calculate the UCT value of each feature in the feature tree layer by layer, sort by UCT value, and greedily select the feature with the largest UCT value in each layer.

[0012] Based on the selected features, and combined with the prompt word engineering, a large language model is invoked to expand and generate new feature nodes;

[0013] The new feature nodes are substituted into a predefined end-to-end machine learning pipeline for evaluation, and the feature scores are calculated comprehensively.

[0014] Update the relevant parameters of all ancestor nodes in the feature tree from bottom to top along the feature synthesis path;

[0015] After terminating the iteration, the output is the optimal feature set that improves the prediction performance of the basic machine learning model.

[0016] As a further preferred embodiment, in the aforementioned method, the feature generation-related configuration information includes at least the number of iterations, batch size, retry limit, initial exploration factor, and final exploration factor;

[0017] Among them, the number of iterations limits the total number of execution rounds from candidate feature selection to parameter update, the batch size limits the number of new feature nodes generated by the large language model in each round, the retry limit limits the maximum number of retries when the generated features do not meet the requirements, and the initial exploration factor and the final exploration factor are used to dynamically adjust the balance weight of exploration and utilization in the calculation of UCT value.

[0018] As a further preferred option, in the aforementioned method, when calculating the UCT value during candidate feature selection, the exploration factor adopts the following dynamic adjustment strategy:

[0019] The exploration factor starts from the initial exploration factor and decreases linearly with each iteration until the final exploration factor, and the initial exploration factor is greater than the final exploration factor.

[0020] As a further preferred option, in the aforementioned method, the output of the large language model in the extension step includes the postfix expression of the new features, natural language interpretation, and Python code.

[0021] The Python code uses the DataFrame object corresponding to the tabular dataset as the operation carrier to clarify the operation relationship between new features and original features or historically generated features. The historically generated features are derived from the historical feature database.

[0022] As a further preferred option, in the aforementioned method, in the feature score evaluation step, the comprehensive score is calculated by combining the performance score and the complexity penalty.

[0023] Among them, the performance score is the performance metric of the basic machine learning model after adding new features on the test set, and the difference between the performance metric when only the original features are used;

[0024] The complexity is the sum of the number of operators in the new feature expression and the number of original / historical features involved in constructing the feature; and the complexity weight is adjusted by a preset penalty coefficient.

[0025] As a further preferred embodiment, in the aforementioned method, the score parameter update step uses the Q value as a comprehensive indicator to measure the feature value, and includes: along the path from the new feature node to the root node, first increasing the number of visits to each node by 1, and then updating the Q value corresponding to the node according to the average of the current Q value and the Q value of the optimal child node.

[0026] Finally, the evaluation results of the new features are transformed into global knowledge through bottom-up backpropagation.

[0027] As a further preferred embodiment, in the aforementioned method, the termination condition specifically includes at least one of the following:

[0028] First, the number of iterations reaches the total number of iterations limited in the feature generation configuration information;

[0029] Second, the feature tree reaches the preset maximum exploration depth;

[0030] Third, meet the early stopping condition: after the number of iterations reaches the minimum required number of rounds, if the performance of the test set of the basic machine learning model does not improve within the preset number of rounds, the iteration is terminated and the current optimal feature set is output.

[0031] As a further preferred option, in the aforementioned method, the extension step based on the large language model also includes feature diversity verification, namely: comparing the newly generated features with the features in the historical feature database to determine the similarity, with the similarity calculated based on the feature's attribute sequence, operator sequence, and expression sequence.

[0032] If highly similar features exist, the prompt words are reconstructed and the large language model is called again until the generated features meet the diversity requirements or the retry limit is reached.

[0033] According to a second key aspect of the present invention, an electronic device is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0034] The memory stores a computer program that, when executed by the processor, causes the processor to perform the aforementioned automatic feature engineering method based on Monte Carlo tree search and large language models.

[0035] Based on a third key aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon that, when executed, implements the aforementioned automatic feature engineering method based on Monte Carlo tree search and large language models.

[0036] Compared to existing technologies, this invention improves the stability of feature synthesis in automated feature engineering through the deep integration of Monte Carlo Tree Search (MCTS) and Large Language Model (LLM). Specifically, existing LLM-driven feature generation methods, lacking global exploration guidance, are prone to getting trapped in local optima, leading to fluctuations in feature quality across multiple iterations. This invention, however, leverages the hierarchical calculation of UCT values ​​and a greedy selection strategy of MCTS to dynamically balance exploring new feature spaces with utilizing high-value feature regions. This ensures that each round of feature generation progresses towards a better direction, avoiding the inherent ambiguity of the synthesis process and allowing feature quality to gradually optimize with iteration. This solves the core pain point of traditional methods' inability to stably guide feature exploration.

[0037] The feature diversity verification mechanism of this invention also successfully overcomes the challenge of feature repetition in multiple rounds, thereby improving feature diversity. This mechanism accurately identifies redundant features with similar semantics or structure through multi-dimensional similarity comparison of attribute sequences, operator sequences, and expression sequences, and guides LLM to generate differentiated features through prompt word reconstruction. Experimental data show that removing this mechanism reduces the average number of operators in features by 18.9% (classification task) and 25% (regression task), resulting in a significant drop in model performance. Enabling this mechanism allows features to cover more potential data patterns, avoiding the search space getting trapped in local optima and providing the model with richer and more effective input.

[0038] In this invention, the improvement in feature quality directly translates into a significant optimization of the predictive performance of the machine learning model, and this effect has broad applicability across different types of tasks and datasets. The features generated by this invention retain the advantages of LLM in integrating domain knowledge and semantic understanding, while also balancing effectiveness and simplicity through the global guidance and complexity penalty mechanism of MCTS. In experiments on 20 publicly available tabular datasets, the proposed solution achieved state-of-the-art performance on 17 datasets, with AUC values ​​for classification tasks and RMAE values ​​for regression tasks significantly outperforming existing state-of-the-art methods, achieving an average performance improvement of over 11.9%. This fully demonstrates the ability of its generated features to capture data patterns and effectively improves the predictive accuracy of the model in classification and regression scenarios.

[0039] This invention improves performance while also considering feature interpretability and engineering practicality, lowering the barrier to practical application. When generating features, the LLM simultaneously outputs postfix expressions, natural language explanations, and Python code, clearly defining the operational relationship between new features and original / historical features, making the feature source and logic traceable, and solving the problem of feature generation being a black box in some automatic feature engineering methods. Furthermore, the method supports multiple LLMs (including the closed-source GPT-4o-mini and the open-source Qwen2.5-72B, etc.), provides reasonable default values ​​for core hyperparameters, and offers flexible and controllable iteration termination conditions, adapting to different resource constraints and application scenarios. It is easy to implement in engineering and readily deployed in practice.

[0040] This invention also achieves an optimized balance between exploration efficiency and resource consumption, avoiding ineffective computation. MCTS's dynamic exploration factor strategy (high exploration in the early stages and high utilization in the later stages) reduces ineffective searches of low-value feature spaces, while the retry cap design of the feature diversity verification mechanism in this invention controls the additional overhead of feature regeneration. Experiments show that this method reaches peak performance in about 50 rounds, without excessive iteration, and can complete feature generation within a reasonable time, balancing performance improvement and resource efficiency. It is suitable for industrial-grade automated feature engineering scenarios with certain real-time requirements. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0042] Figure 1 The following is a flowchart illustrating the execution of an automatic feature engineering method based on Monte Carlo tree search and a large language model in one embodiment of the present invention.

[0043] Figure 2 This diagram illustrates the overall framework of an automatic feature engineering system based on Monte Carlo tree search and a large language model, according to one embodiment of the present invention.

[0044] Figure 3 A system framework diagram for candidate feature selection in one embodiment of the present invention is shown;

[0045] Figure 4 A system framework diagram based on a large language model is shown in one embodiment of the present invention;

[0046] Figure 5A system framework diagram for feature score evaluation in one embodiment of the present invention is shown;

[0047] Figure 6 A system framework diagram for updating the score parameter is shown in one embodiment of the present invention. Detailed Implementation

[0048] The preferred embodiments of the present invention will be described in detail below to provide a clearer understanding of the purpose, features, and advantages of the invention. It should be understood that the following embodiments are not intended to limit the scope of the invention, but are merely illustrative of the essential spirit of the technical solution of the invention.

[0049] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known techniques associated with the invention may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0050] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0051] In the following embodiments or drawings, the technical terms that may be involved in the specific implementation process of the present invention and their meanings are as follows:

[0052] AutoFE: Automated Feature Engineering, refers to the technology of generating new features from raw data through automated operations such as extraction, transformation, and selection to improve the predictive performance of machine learning models. It is the core application area of ​​this invention.

[0053] LLM: Large Language Model, is a pre-trained model (such as GPT-4o-mini, Qwen2.5-72B, etc.) with powerful natural language understanding and generation capabilities. In this invention, it is used to generate new features by combining prompt word engineering and integrating domain knowledge and data semantic information.

[0054] MCTS: Monte Carlo Tree Search, is a sampling-based global search algorithm used in this invention to construct a feature tree structure. It balances the exploration of new feature spaces with the utilization of high-value features through the UCT strategy, guiding the direction of feature generation.

[0055] UCT: Upper Confidence Bounds on Trees, a node selection strategy in MCTS, used in this invention to calculate the UCT value of features layer by layer, serving as the basis for greedily selecting high-value features, and dynamically balancing exploration and utilization.

[0056] DFS: Depth-First Search is a classic search algorithm and the core technology of early AutoFE methods. In this invention, it is used as a baseline method to illustrate the limitations of existing technologies (reliance on predefined operators and limited search space).

[0057] RFE: Recursive Feature Elimination is a feature selection technique used in this invention to quantify the value of new features and provide a basis for feature score evaluation.

[0058] AUC (Area Under the ROC Curve) is a performance evaluation metric for classification tasks. In this invention, it is used to measure the improvement effect of generated features on the prediction accuracy of the classification model. Its value ranges from [0,1], and the closer it is to 1, the better the performance.

[0059] RMAE: Relative Mean Absolute Error, is a performance evaluation metric for regression tasks. In this invention, it is used to evaluate the improvement effect of generated features on the regression model. After normalization, it takes a value of [0,1], and the closer it is to 1, the better the performance.

[0060] XGBoost: Extreme Gradient Boosting, is a commonly used ensemble learning model. In this invention, it is used as the basic machine learning model to verify the performance of generated features.

[0061] GPT-4o-mini is a closed-source large language model launched by OpenAI. The basic LLM used by default in this invention is used for feature generation and supports high token constraints and context understanding.

[0062] Qwen2.5-72B / Llama3.1-70B: Open source large language model, used in this invention to verify the generalization ability of the method and adapt to different types of LLM application scenarios.

[0063] LMTree: The name of an automatic feature engineering method framework proposed in one embodiment of the present invention. It integrates the global search capability of MCTS with the feature generation capability of LLM, and generates high-quality and diverse tabular data features through iterative optimization. It fundamentally solves the problems of unstable LLM feature synthesis and multiple rounds of repetition.

[0064] DiverseFeat verification mechanism: In one embodiment of the present invention, a feature diversity verification mechanism is designed to identify redundant features and reconstruct prompts to guide LLM regeneration by comparing the similarity between newly generated features and historical features in attribute sequences, operator sequences, and expression sequences, thereby avoiding feature duplication in multiple rounds.

[0065] PromptBuilder: A prompt word construction module in one embodiment of the present invention, used to integrate dataset information, metadata, historical feature database and target feature requirements to generate prompt words that guide LLM to generate new features.

[0066] ReconstructPrompt: A prompt word reconstruction module in one embodiment of the present invention, used to adjust the prompt word content and guide the LLM to a new feature generation direction when the DiverseFeat mechanism detects feature repetition.

[0067] Feature engineering (FE) is the process of generating new features by transforming raw data. In this invention, it specifically refers to feature engineering of tabular data, with the core objective of improving the predictive performance of machine learning models.

[0068] Candidate features: In this invention, it refers to the new set of features to be evaluated after the original features or historically generated features are transformed by LLM (such as K features generated in each round).

[0069] End-to-end machine learning pipeline: The predefined feature evaluation process in this invention includes feature input, model training, performance testing, and other steps, which are used to comprehensively calculate the score of new features.

[0070] Prompt Engineering: By designing reasonable prompt structures and content, LLM is guided to generate outputs that meet the requirements (a new feature in this invention), thereby improving the accuracy and stability of the generated results.

[0071] Historical Feature Database (H): This invention stores all generated features that have passed diversity verification. It is used for similarity comparison in the DiverseFeat mechanism to avoid feature duplication.

[0072] Q-value: In this invention, a comprehensive index is used to measure the value of a feature. It is calculated from the "feature performance score - complexity penalty" and is the core basis for calculating the UCT value and updating node parameters.

[0073] Feature complexity (Col(x)): In this invention, the index used to quantify the simplicity of features is calculated as the sum of the number of operators in the feature expression and the number of original / historical features involved in constructing the feature.

[0074] DataFrame: A data structure in Python data analysis libraries (such as pandas). In this invention, it is used to store tabular datasets. The Python code generated by LLM uses this structure as its operational carrier.

[0075] np.sqrt / np.log: Functions in the Python numerical computing library (numpy), corresponding to square root operation and logarithm operation respectively, which are commonly used operators in this invention when generating features for LLM.

[0076] Early stopping condition: One of the triggering conditions for iteration termination in this invention refers to the fact that after the number of iterations reaches the minimum required number of rounds, the model performance does not improve within a consecutive preset number of rounds. At this time, the iteration is terminated to avoid invalid calculations.

[0077] like Figure 1 As shown, in one embodiment of the present invention, an automatic feature engineering method based on Monte Carlo tree search and large language models includes the following steps 100-700 executed by a computer system:

[0078] Step 100: Input the tabular dataset, dataset metadata, basic machine learning model and feature generation related configuration information into the computer system. The tabular dataset contains multiple records and corresponding original feature sets and label sets. The metadata describes the dataset and details of each original feature.

[0079] Step 200: The computer system performs feature initialization, evaluates the score of each initial input feature, and constructs an initial feature tree structure;

[0080] Step 300: Iteratively execute the candidate feature selection, large language model-based expansion, feature score evaluation, and score parameter update steps until the termination condition is met: When selecting candidate features, calculate the UCT value of each feature in the feature tree layer by layer, sort by UCT value, and greedily select the feature with the largest UCT value in each layer.

[0081] Step 400: Based on the selected features, and in conjunction with the prompt word engineering, call the large language model to expand and generate new feature nodes;

[0082] Step 500: Substitute the new feature nodes into the predefined end-to-end machine learning pipeline for evaluation and calculate the feature scores comprehensively;

[0083] Step 600: Update the relevant parameters of all ancestor nodes in the feature tree from bottom to top along the feature synthesis path;

[0084] Step 700: After terminating the iteration, output the optimal feature set that improves the prediction performance of the basic machine learning model.

[0085] In one possible embodiment, the present invention proposes a new method and framework for automatic feature engineering of tabular data—referred to as LMTree in this embodiment.

[0086] Specifically, to improve feature quality and the stability of the synthesis process across multiple iterations, this embodiment utilizes the global exploration capability of Monte Carlo Tree Search (MCTS) to enhance the feature generation performance of the large language model. High-value feature configurations are evaluated and selected using the upper confidence bound of Monte Carlo Tree Search, while the pre-trained knowledge and contextual understanding capabilities of the large language model guide the generation of high-quality features, achieving a balance between exploration and utilization, thereby improving the performance of LMTree.

[0087] To alleviate the problem of repetitive multi-round validation in large language models, this embodiment designs a lightweight validation mechanism. This mechanism, based on similarity metrics of operator symbols and attribute definitions, matches newly generated features in each round with historical records. If a new feature highly similar to existing features is identified, prompt word engineering guides the large language model towards a new generation direction, thereby improving feature diversity.

[0088] Extensive experimental results demonstrate that, compared with current mainstream automatic feature engineering methods, the LMTree implementation in this embodiment exhibits superior performance, significantly improving the accuracy of machine learning models in tabular data prediction tasks.

[0089] In this embodiment, the main technical innovation lies in proposing an automatic feature engineering algorithm that integrates Monte Carlo tree search and large language models. This algorithm generates high-quality tabular features through a stable iterative optimization process. Furthermore, a verification mechanism with similarity measurement is designed to reduce redundant feature generation in large language models and promote feature diversity. Comprehensive experiments on 20 tabular datasets demonstrate that the LMTree method in this embodiment outperforms current mainstream automatic feature engineering methods and has broad applicability across various large language models. Ablation experiments further reveal the synergistic effect of Monte Carlo tree search and large language models and verify the effectiveness of the verification mechanism in reducing redundant features.

[0090] The overall architecture and implementation details of each component of the LMTree method in this embodiment will be described in detail in the following possible implementations.

[0091] The execution process of this embodiment first requires inputting a tabular dataset, dataset metadata, basic machine learning model and feature generation related configuration information into the computer system. The tabular dataset contains multiple records and corresponding original feature sets and label sets, while the metadata describes the dataset and details of each original feature.

[0092] In this embodiment, the tabular dataset is denoted as ,Include 1 record (also known as a sample), of which Represents the original feature set (including (characteristics) for The first in Each feature. In addition, the dataset also contains metadata. This describes the dataset itself, each raw feature, and other relevant details. For classification tasks, each record is associated with a label. (The category to which the encoded record belongs); for regression tasks, This represents the target value to be predicted.

[0093] In this embodiment, The dataset is divided into training, validation, and test sets, denoted as follows: , and The core task of automated feature engineering is: given a tabular dataset... Generate a set of high-quality features to improve the machine learning model. exist The predictive performance must meet resource and time constraints.

[0094] Formalistically, this embodiment aims to find the optimal feature configuration. The definition is as follows:

[0095] (1)

[0096] in Representation Model Performance score – This score is calculated through the following process: Train the model and The above validation determines the optimal feature construction strategy, and then the selected features are used. exist The above evaluates performance, that is .

[0097] in, The candidate feature set is represented by the Large Language Model (LLM) using operators. For original features and generated features The result is obtained through conversion, i.e. (Assuming each operator) Acting on (one or more features in the text). Indicates the first Our goal is to configure the feature space after each iteration. Medium and high efficiency exploration Find the model that maximizes A new feature set for predictive performance .

[0098] To clarify the feature transformation space, the operator set is classified in this embodiment:

[0099] Unary operators (such as) : Acts on a single feature;

[0100] Binary operators: combine two features (such as arithmetic addition (+), subtraction (-), multiplication (×), division (÷), exponentiation (...). ), absolute difference ( ), logarithmic ratio ( );

[0101] Multi-operational operators include conditional selection (where(condition, a, b)) and grouping operations (such as maximum value (Max), mean (Mean), summation (Sum), and variance (Variance)).

[0102] In this embodiment, the automatic feature engineering task does not require a predefined set of operators; instead, the operators... It can be dynamically generated from an open set based on the pre-trained knowledge and semantic understanding capabilities of a large language model. The core objective of this embodiment is to generate [something] in each round. High-efficiency exploration of feature search space Find the model that maximizes A new feature set for predictive performance .

[0103] In existing technologies, directly applying large language models for feature generation faces two major challenges: First, the combination of features and operators is highly uncertain, making it difficult to determine whether new features can reveal potential patterns in the data; second, large language models tend to perform local transformations based on the current feature space, resulting in generated features that are semantically similar to features from previous rounds, which can easily cause the model to get stuck in local optima and thus ignore high-potential unexplored regions.

[0104] To address these challenges, this embodiment introduces Monte Carlo Tree Search (MCTS) as a global search framework to guide the feature generation process based on a large language model.

[0105] Specifically, MCTS is a sampling-based search algorithm that searches the feature space. The model is constructed as a tree—each node represents a feature configuration state, and each edge represents a feature transformation operation. This modeling approach transforms the search problem for optimal feature configurations into a sequential decision problem on a transformation tree. The effectiveness of MCTS stems from the balance between "exploration" and "exploitation" during the search process: compared to a fixed exploration rate... Unlike classic strategies such as greedy algorithms, MCTS adopts an adaptive strategy based on dynamic adjustment of accumulated search information, which can avoid premature convergence to local optima and achieve a more thorough exploration of the feature space.

[0106] In this embodiment, the upper confidence bound (UCT) strategy of trees is used to guide the search process. That is, UCT systematically balances the "utilization" of high-potential feature regions and the "exploration" of underexplored regions through a node selection mechanism based on confidence.

[0107] The following possible implementation details the large language model-driven automatic feature engineering method—LMTree—based on Monte Carlo Tree Search (MCTS) proposed in this invention. Its overall architecture is as follows: Figure 2 As shown, the core process is as follows:

[0108] Step (1), Feature initialization: This involves performing feature initialization, evaluating the score of each initial feature in the input, and constructing the initial feature tree structure;

[0109] Step (2), Candidate Feature Selection: Iteratively calculate the UCT value of each feature in each layer, sort by UCT value, and greedily select the feature with the largest UCT value in each layer, such as... Figure 3 As shown;

[0110] Step (3), Expansion based on the large language model: Based on the selected features, and combined with the prompt word project, the large language model is called to expand and generate K new feature nodes, such as... Figure 4 As shown;

[0111] Step (4), Feature Score Evaluation: Substitute the newly expanded feature nodes into the predefined end-to-end machine learning pipeline for evaluation, and calculate their feature scores comprehensively, such as... Figure 5 As shown;

[0112] Step (5), score parameter update: Update the relevant parameters (such as Q value, etc.) of all ancestor nodes from bottom to top along the feature synthesis path, such as... Figure 6 As shown.

[0113] like Figure 2 As shown, LMTree iteratively executes steps (2)-(5) until the termination condition is met (such as reaching the maximum exploration depth or feature quality constraint).

[0114] A specific calculation process is given in one of the following possible implementations.

[0115] Input: Dataset metadata Historical Feature Database Basic machine learning models Number of iterations Batch size Retry limit Initial and final exploration factors ;

[0116] Output: Generated feature set .

[0117] 1: Initialize the feature tree structure It contains a root node and leaf nodes (corresponding to each) );

[0118] 2: Calculate each The score (x), Q-value (Q(x)), and UCT score (UCT(x));

[0119] 3: Initialize an empty database Set exploration factors ;

[0120] 4: Let the loop variable... Starting from 1, incrementing sequentially to :

[0121] 5: (Set the current node as the root node);

[0122] 6: When When it is not a leaf node:

[0123] 7: (Select the node with the largest UCT value among its child nodes);

[0124] 8: If ,but Otherwise, terminate the loop;

[0125] 9: (Use the prompt word builder to generate prompt words) );

[0126] 10: Initialize the number of retries ;

[0127] 11: When hour:

[0128] 12: Call the large language model and input prompt words. generate A set of features ;

[0129] 13: If (The new feature does not meet the diversity requirement):

[0130] 14: (Reconstruction prompts), and ;

[0131] 15: Otherwise:

[0132] 16: Terminate the retry loop;

[0133] 17: (Add qualified new features to the historical feature database);

[0134] 18: For each :

[0135] 19: Calculate the score ( ) and Q( ), and in the tree China Add leaf nodes;

[0136] 20: For from Each node on the path to the root node :

[0137] 21: Node Number of visits Add 1;

[0138] twenty two: Update node value);

[0139] 23: Update exploration factors (Linear adjustment of exploration factors);

[0140] 24: Return (Select the optimal feature set);

[0141] In one of the following possible implementations, combined with Figure 3-6 The iterative process of candidate feature selection, feature synthesis and expansion based on a large language model, feature score evaluation, and score parameter update is explained.

[0142] like Figure 3 As shown, the core of candidate feature selection is to establish a directional guidance mechanism to efficiently identify high-potential regions in the feature search space and focus on high-value feature nodes; at the same time, it balances "exploring new regions" with "utilizing already discovered high-value regions" to reduce redundant features. To this end, this embodiment adopts the tree upper confidence bound (UCT) algorithm—this algorithm dynamically balances "prioritizing the selection of feature branches with high prediction potential" with "prioritizing the exploration of branches that have not been fully explored in the search history," laying the foundation for feature synthesis of large language models.

[0143] Specifically, feature information is stored in a tree structure. During feature selection, the UCT value of each node is calculated layer by layer, and a greedy strategy is used to select the node with the largest UCT value in the current layer. If the UCT value of any of its child nodes is higher, the node is recursively explored downwards; otherwise, the node is expanded. Unlike the classic MCTS method, which can only expand high-value nodes, this strategy avoids missing potentially valuable information and improves the effectiveness of feature selection and synthesis.

[0144] The calculation of UCT values ​​incorporates node... Feature generation value ( value ) and exploration factors (For balancing exploration and utilization), the formula is as follows:

[0145]

[0146] in For nodes Total number of visits to all parent nodes For nodes Its own number of visits; This is for normalization. Specifically, in this embodiment, an adaptive strategy is used to dynamically adjust... The value of : during the iteration process, from linearly decreasing to ( In the early stages of the search, larger amounts were used. (Close to 1) to promote exploration, use smaller values ​​later. (Close to 0) to enhance utilization.

[0147] like Figure 4 As shown, feature synthesis extension based on a large language model is a crucial step in this invention. In this embodiment, feature synthesis is a core component of the LMTree framework, which utilizes the pre-trained knowledge of the large language model, combined with historical feature databases and contextual information, to generate high-quality and diverse new features. To avoid generating similar or redundant features, this embodiment further proposes a lightweight verification mechanism based on a "detection-repair" strategy—DiverseFeat (Diverse Feature Verification Mechanism).

[0148] In this embodiment, the feature generation process is as follows: designing prompt word templates to construct input. The dataset, metadata, original features, historical generated features, target task, and other contextual information are fed into the large language model to guide it in generating language expressions containing... A set of new features ( (For the preset batch size), and each feature uses a different operator.

[0149] Large language models output postfix expressions of features, natural language explanations, and Python code to improve the traceability and interpretability of features.

[0150] To improve output stability, the prompt word design avoids excessive constraints; the specific data verification and error handling after feature generation (such as infinite values ​​caused by log(0), missing values, etc.) are handled automatically by subsequent code, rather than forcing the large language model to avoid them during the generation process, thereby improving the success rate of feature synthesis.

[0151] The core logic of the DiverseFeat verification mechanism is as follows: to ensure feature diversity, a "detection-repair" strategy is used to process newly generated features. With historical feature database The system performs similarity comparisons on features; if duplicate or highly similar features are detected, the system guides the large language model to regenerate non-redundant features by reconstructing prompt words. The specific process is as follows: extract the attribute and operator information of each feature, calculate the similarity, and if duplicate or highly similar features are found... Existence and new features Historical features with similarity exceeding a preset threshold are incorporated into prompt words to prevent large language models from generating similar features.

[0152] Feature similarity is calculated based on two dimensions: "attribute" and "operator," primarily using the Jaccard coefficient and the longest common subsequence (LCS).

[0153] Attribute similarity: Let and Representing new features and historical characteristics The attribute sequence, Jaccard coefficient is calculated as follows:

[0154]

[0155] The similarity of the longest common subsequence is calculated as follows ( (This represents the length of the longest common subsequence of the two sequences).

[0156]

[0157] Operator similarity: Extracting new features and historical characteristics operator sequence and The Jaccard coefficient is calculated using the same method as in formulas (3) and (4). Similarity with longest common subsequence ;

[0158] Overall similarity: Treating the feature expressions as sequences, calculate the similarity of their longest common subsequence. Final overall similarity The weighted sum of similarity across all dimensions:

[0159]

[0160] The sum of the weights satisfies In this embodiment, the default value is: .

[0161] like Figure 5 As shown, feature score evaluation is a crucial step in feature engineering. In feature engineering, newly generated features need to improve model performance while maintaining simplicity and interpretability. Therefore, this embodiment designs an evaluation scheme that integrates "performance score" and "complexity penalty": feature value is quantified through recursive feature elimination (RFE), a penalty term is introduced to constrain the complexity of the feature expression, and finally, the Q-value Q(x) of the feature is jointly calculated.

[0162] Performance score Score(x): Defined as the improvement in model performance after adding feature x.

[0163]

[0164] in Representation Model Performance metrics on the test set (accuracy, F1 score, etc. for classification tasks, mean squared error, etc. for regression tasks).

[0165] Complexity Defined as the length of the sequence of characteristic expressions, i.e., the sum of the number of operators and the number of participating features:

[0166]

[0167] in Features The number of operators used To participate in the construction of features The number of original / generated features.

[0168] Q-value calculation: Combining performance score and complexity penalty, the formula is as follows:

[0169]

[0170] hyperparameters ≥0 is used to adjust the weight of the complexity factor: The larger the value, the more likely it is to generate concise features with fewer operators and historical features used; The smaller the value, the more priority should be given to improving model performance.

[0171] like Figure 6 As shown, in the score parameter update, to dynamically update the feature values ​​in the search space, the LMTree of this invention adopts a backpropagation mechanism to transform the evaluation results of new features into global knowledge: updating the Q value of the current node and propagating this value from bottom to top to all ancestor nodes of the root node along the feature synthesis path. For each non-leaf node, its Q value is updated to the average of the "current value" and the "optimal child node value", achieving continuous self-optimization.

[0172] Specifically, nodes The update process is as follows: First, check its access count. Add 1, and then recalculate the Q value of the corresponding feature x using the following formula:

[0173]

[0174] LMTree optimizes feature configuration and builds high-quality new features through the above four-step process until a preset termination condition is met (such as reaching the maximum number of iterations). Satisfying the early stopping condition—reaching the minimum number of iterations. or continuous (No performance improvement for wheel models).

[0175] Experiments show that the LMTree method of this invention significantly outperforms existing mainstream automatic feature engineering methods. In tests on 20 publicly available tabular datasets (including 12 classification tasks and 8 regression tasks), LMTree achieved state-of-the-art performance on 17 datasets. The generated new features significantly improve the prediction performance of the underlying machine learning model. The AUC value for the classification task and the RMAE value for the regression task are both outstanding, with an average performance improvement exceeding 11.9%, fully validating its effectiveness in automatic feature engineering tasks.

[0176] Ablation experiments further confirmed the crucial role of the core components of this invention. Removing Monte Carlo Tree Search (MCTS) made the feature synthesis process prone to getting trapped in local optima, resulting in a significant performance drop on six typical datasets. This demonstrates that MCTS is essential for efficiently exploring the feature space and stabilizing the feature generation direction. Removing the DiverseFeat validation mechanism reduced the average number of operators generating features by 18.9% (classification task) and 25% (regression task), increasing feature redundancy and significantly lowering model performance. This confirms the necessity of this mechanism in improving feature diversity and avoiding redundant generation. Furthermore, when dataset metadata was missing, the average model performance decreased by 4.1%, indicating that metadata can help large language models generate more semantically relevant and effective features.

[0177] This invention also possesses excellent generalization ability and practical adaptability. It can stably perform on different types of large language models (including closed-source GPT-4o-mini and DeepSeek-V3.2-Exp, and open-source Qwen2.5-72B and Llama3.1-70B), and the stronger the performance of the large language model, the more significant the enhancement effect of feature synthesis. Through hyperparameter sensitivity analysis, reasonable configuration of core parameters (such as UCT exploration factor, retry limit, and complexity penalty coefficient) can further optimize performance, and the default parameters can meet the needs of most scenarios. Iterative round experiments show that the model can reach peak performance in about 50 rounds, balancing exploration efficiency and resource consumption, and is suitable for practical engineering scenarios.

[0178] The technical terms, principles, or means related to the technical solutions of the present invention mentioned in the above embodiments, which are not described in detail above, are all well-known technologies or common practices that are known to those skilled in the art.

[0179] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic feature engineering based on Monte Carlo tree search and large language models, characterized in that, The method comprises the following steps executed by a computer system: inputting a table data set, data set metadata, a base machine learning model, and feature generation related configuration information into the computer system, wherein the table data set contains multiple records and corresponding original feature sets and label sets, and the metadata describes the data set and details of each original feature; the computer system performs feature initialization, evaluates the score of each initial feature inputted, and constructs an initial feature tree structure; iteratively performing candidate feature selection, large language model based expansion, feature score evaluation, and score parameter updating steps until a termination condition is met: in candidate feature selection, the UCT value of each feature in the feature tree is calculated layer by layer, and the feature with the maximum UCT value in each layer is selected greedily after sorting according to the UCT value; based on the selected features, a large language model is called by combining prompt words to expand and generate new feature nodes; the new feature nodes are substituted into a pre-defined end-to-end machine learning pipeline for evaluation, and the feature score is calculated comprehensively; after the iteration is terminated, the optimal feature set that can improve the prediction performance of the base machine learning model is outputted; in the calculation of the UCT value in the candidate feature selection, the exploration factor is dynamically adjusted as follows: the exploration factor starts from an initial exploration factor and decreases linearly to a final exploration factor with the increase of the iteration round, and the initial exploration factor is greater than the final exploration factor; in the feature score evaluation step, the comprehensive score is calculated by combining the performance score and the complexity penalty, and the formula is as follows: the score parameter updating step takes Q value as a comprehensive index for measuring the value of the feature, and comprises: along the path from the new feature node to the root node, first increasing the access count of each node by 1, and then updating the Q value corresponding to the node according to the average of the current Q value and the optimal child node Q value; finally, through bottom-up back propagation, the evaluation result of the new feature is converted into global knowledge; ； wherein, Score( x ) is a performance score, is a complexity, is a penalty coefficient, the performance score is the difference between the performance indicator of the base machine learning model after adding the new feature to the test set and the performance indicator when only using the original features; the complexity is the sum of the number of operators in the new feature expression and the number of original / historically generated features involved in constructing the feature; and the complexity weight is adjusted by the pre-set penalty coefficient; the large language model based expansion step further comprises feature diversity verification, which comprises: comparing the similarity of the newly generated feature with the features in the historical feature database, and calculating the similarity based on the attribute sequence, operator sequence, and expression sequence of the feature; if there is a historical feature with a similarity exceeding a preset threshold, the prompt words are reconstructed and the large language model is called again until the generated feature meets the diversity requirement or reaches the upper limit of retries. the feature generation related configuration information at least comprises the number of iterations, the batch size, the upper limit of retries, the initial exploration factor, and the final exploration factor; 2. The method of claim 1, wherein, wherein the number of iterations limits the total execution round of the candidate feature selection to the parameter updating step, the batch size limits the number of new feature nodes generated by the large language model in each round, the upper limit of retries limits the maximum number of retries when the generated feature does not meet the requirement, and the initial exploration factor and the final exploration factor are used to dynamically adjust the balance weight between exploration and utilization in the calculation of the UCT value. in the large language model based expansion step, the output content of the large language model includes the suffix expression, natural language explanation, and Python code of the new feature.

3. The method of claim 1, wherein, ​ The Python code takes a DataFrame object corresponding to a table data set as an operation carrier, and clearly defines the operation relationship between a new feature and an original feature or a historically generated feature, the historically generated feature being derived from a historical feature database, and the historical feature database being a database storing all generated features that have passed diversity verification.

4. The method of claim 1, wherein, The termination condition specifically includes at least one of the following: First, the number of iterations reaches the total number of iterations defined in the feature generation related configuration information. Second, the feature tree reaches a preset maximum exploration depth. Third, the early stopping condition is met: after the number of iterations reaches a minimum required round, the performance of the base machine learning model on the test set does not improve within a preset number of consecutive rounds, at which time the iteration is terminated and the current optimal feature set is output.

5. An electronic device comprising: A processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the automatic feature engineering method based on the Monte Carlo tree search and the large language model according to any one of claims 1-4.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed to implement the automatic feature engineering method based on the Monte Carlo tree search and the large language model according to any one of claims 1-4.

Images