Knowledge enhancement-based data preprocessing pipeline construction method, electronic device, readable storage medium and program product

Abstract

The application discloses a knowledge enhancement-based data preprocessing pipeline construction method, an electronic device, a readable storage medium and a program product, relates to machine learning and data preprocessing technology, and solves the problems of how to reduce the insufficient adaptation of historical pipeline experience to current tasks, reduce candidate evaluation cost, eliminate preference conflicts and structural redundancy when historical experience is reused, and improve search efficiency and performance stability under a limited budget in preprocessing pipeline search by using the existing pipeline construction method. By constructing a knowledge base for storing historical task meta-features and preprocessing logic, the application extracts current task meta-features for similar matching to obtain a candidate pipeline, and then performs module deletion and fusion reconstruction on the candidate pipeline. Finally, a search algorithm is used to assign an optimal processing algorithm to the reconstructed pipeline and score it, and the highest score is selected as the output. The application is suitable for industry scenarios with uneven data quality, mixed feature types and frequent modeling and optimization.

Description

Technical Field

[0001] This invention relates to machine learning and data preprocessing techniques. Background Technology

[0002] Data preprocessing transforms the original tabular data into data usable by downstream machine learning models. Common processing modules include missing value handling, categorical feature encoding, numerical feature normalization, feature transformation, and feature selection. Whether different modules need to be executed, their execution order, and the specific algorithms used in each module all affect the feasibility of model training and the final prediction performance. Because the applicable processing methods depend on the dataset features, downstream models, and evaluation metrics, a fixed pipeline is generally not suitable for all tasks.

[0003] Existing automated methods primarily employ random search, genetic programming, Bayesian optimization, reinforcement learning, or surrogate models to construct preprocessing pipelines. Direct search requires repeated execution of candidate pipelines and training of downstream models, resulting in high evaluation costs. While methods relying on surrogate models or policy networks can reduce some of the search, the recommendation results are affected by the distribution of training data and model estimation errors. Although the above methods improve the efficiency of pipeline construction and automated data preprocessing to some extent, their utilization of historical experience remains relatively crude, often merely reusing historical high-performance pipelines. However, since historical data and current data may differ in missing data, feature types, and statistical distributions, directly reusing historical high-performance pipelines can lead to poor preprocessing results for the original tabular data.

[0004] Existing methods for reusing historical experience have two specific problems. First, mixing experiences obtained from different downstream models or under different evaluation metrics may introduce conflicting pipeline preferences. Second, candidate pipelines obtained from multiple similar historical tasks often contain modules that are not needed in the current data, and after deleting these modules, a large number of duplicate candidates with the same logical structure will appear. Without adaptation and merging, subsequent searches will still consume evaluation resources on duplicate structures.

[0005] Taking five modules—missing value handling, categorical feature encoding, normalization, feature transformation, and feature selection—as examples, a pipeline can be constructed. Considering module selection, execution order, and specific implementation methods, the number of candidate physical pipelines can reach 207,360. Evaluating each candidate pipeline requires fitting the preprocessing steps and downstream model onto the training data and calculating the evaluation metric on the validation data. Therefore, the practical difficulty lies not in listing candidates, but in finding a small number of candidates worthy of realistic evaluation on the current task from a large combination space. After defining the relevant objects and symbols in the specific implementation, the calculation formula for the search space is given.

[0006] In summary, it is urgent to address the problems of insufficient adaptation between historical pipeline experience and current tasks, reduced candidate evaluation costs, and elimination of preference conflicts and structural redundancy when reusing historical experience in the preprocessing pipeline search of existing pipeline construction methods, so as to improve search efficiency and performance stability under limited budget. Summary of the Invention

[0007] The purpose of this invention is to address the problems of existing pipeline construction methods in reducing the inadequacy of adapting historical pipeline experience to the current task, reducing candidate evaluation costs, eliminating preference conflicts and structural redundancy when reusing historical experience, and improving search efficiency and performance stability under limited budgets. This invention provides a knowledge-enhanced data preprocessing pipeline construction method, electronic device, readable storage medium, and program product.

[0008] Knowledge-enhanced data preprocessing pipeline construction methods include: S1. Construct a historical task knowledge base, which stores multiple historical machine learning tasks, meta-feature vectors of datasets in each historical machine learning task, and multiple data preprocessing logic pipelines corresponding to each historical machine learning task; wherein, each historical machine learning task consists of a dataset and its corresponding downstream model. Each data preprocessing logic pipeline is composed of multiple processing modules with different functions cascaded in sequence, and each data preprocessing logic pipeline corresponds to a processing order, which is the logical execution order of the processing modules within the data preprocessing logic pipeline; S2. Extract the meta-features of the dataset in the learning task to be processed to form a meta-feature vector; the learning task to be processed consists of the dataset and its corresponding downstream model; S3. Based on the extracted meta-feature vectors, perform similar task matching with the meta-feature vectors stored in the historical task knowledge base to obtain a set of candidate data preprocessing logic pipelines. S4. Perform deletion and fusion operations on the candidate data preprocessing logic pipeline set to complete the logical reconstruction of each candidate data preprocessing logic pipeline and obtain the simplified candidate logic pipeline. S5. Using the Monte Carlo tree search algorithm, combined with the dataset in the learning task to be processed and the downstream model, the optimal processing algorithm is assigned to each processing module in each simplified candidate logic pipeline, and the pipeline performance score of the simplified candidate logic pipeline after assigning the optimal processing algorithm is output. The streamlined candidate logical pipeline with the highest pipeline performance score after being assigned the optimal processing algorithm is used as the optimal data preprocessing pipeline, and the optimal data preprocessing pipeline is also used as the optimal physical pipeline, thus completing the pipeline construction.

[0009] Preferably, in step S3, the method for performing similar task matching based on the extracted meta-feature vectors and the meta-feature vectors stored in the historical task knowledge base to obtain the candidate data preprocessing pipeline set includes: Similar task matching is performed based on the extracted meta-feature vectors and the meta-feature vectors stored in the historical task knowledge base. Based on the top N meta-feature vectors in the matching results, all data preprocessing pipelines under the learning task are used to form a candidate data preprocessing pipeline set.

[0010] Preferably, in step S4, deletion and fusion operations are performed on the candidate data preprocessing logic pipeline set to complete the logical reconstruction of each candidate data preprocessing logic pipeline, resulting in the following simplified implementation of the candidate logic pipeline: S41. Deletion operation: When the number of class features and / or the proportion of missing values ​​in the extracted meta-feature vector are 0, the corresponding class feature encoding module and / or missing value processing module corresponding to the number of class features and the proportion of missing values ​​are used as the processing modules to be pruned. The processing modules to be pruned in each candidate data preprocessing logic pipeline are deleted, resulting in a reconstructed logic pipeline. S42, Fusion Operation: For all logic pipelines after primary reconstruction with the same channel length and processing order, the processing modules with the same function are merged to obtain a logic pipeline after secondary reconstruction, and this logic pipeline after secondary reconstruction is used as a candidate logic pipeline after simplification.

[0011] Preferably, the learning task to be processed and the meta-feature vector corresponding to the learning task to be processed are completely identical, and the meta-feature vector is composed of the number of samples, the number of features, the number of numerical features, the number of categorical features, the proportion of missing values, the number of target categories, the average skewness of numerical features, the average kurtosis of numerical features, the average variance of features, the average of the pairwise absolute correlation coefficients of features, the average of the mutual information between features and the target, the maximum value of the mutual information between features and the target, and the target entropy.

[0012] An electronic device includes a storage device, a processor, and a computer program stored in the storage device and executable on the processor, the processor executing the computer program to implement the knowledge-enhanced data preprocessing pipeline construction method described above.

[0013] A computer-readable storage medium having a computer program stored thereon, which, when executed, implements the knowledge-enhanced data preprocessing pipeline construction method described above.

[0014] A computer program product includes a computer program that, when executed by a processor, implements the knowledge-enhanced data preprocessing pipeline construction method as described. The beneficial effects of this invention are: This invention utilizes a historical task knowledge base to fully leverage the experience of existing tasks, avoids redundant design, and performs similar task retrieval based on meta-feature vectors, thereby improving the efficiency of utilizing historical experience and avoiding the direct application of irrelevant historical experience.

[0015] Then, based on the actual situation of the current data, irrelevant modules are deleted and duplicate structures are merged to eliminate redundancy and conflicts caused by multi-source experience, which greatly reduces the number of candidate pipelines, reduces the redundancy of the candidate data preprocessing logic pipeline set, avoids the waste of search resources, and can effectively compress the search space while ensuring the accessibility of high-quality candidate pipelines.

[0016] Finally, Monte Carlo tree search and performance scoring are performed only on the simplified candidate logical pipelines. Within a limited computational budget, a thorough search is completed and the pipeline with the highest score is selected, thereby reducing evaluation costs while ensuring that the output pipeline is suitable for the current task and has stable performance.

[0017] This invention provides an interpretable knowledge reuse method, which first retrieves relevant historical pipelines based on the meta-feature vector of the current task, and then reconstructs the candidate data preprocessing logic pipeline according to the observable features of the current data.

[0018] Deletion operations: These only target missing value handling modules and category encoding modules whose necessity can be directly determined by data statistics. This avoids deleting normalization, feature transformation, or feature selection modules without validation. Deleting invalid modules shortens the candidate logic pipeline and correspondingly reduces the number of subsequent implementation algorithms that need to be selected.

[0019] Fusion operation: Based on the module composition and execution order of the logical pipeline, candidates from different sources but with the same logical structure are merged into a single logical pipeline, and their historical support information is aggregated to reduce redundant searches without losing traceable historical support information.

[0020] This invention constructs a historical pipeline reuse mechanism based on task feature matching without relying on surrogate model estimation errors. This reduces the computational resource overhead caused by repeated evaluation of candidate pipelines and structural redundancy during the preprocessing pipeline search process, and eliminates preference conflicts caused by cross-task and cross-model experience mixing. Thus, it steadily improves the efficiency of pipeline search and the final prediction performance within a limited evaluation budget.

[0021] This invention is primarily applied to model building tasks in data-driven industries, and is suitable for industry scenarios with inconsistent data quality, mixed feature types, and frequent modeling and optimization requirements. It provides an intelligent data preparation solution. By reusing knowledge and experience from past tasks, this method can automatically complete the construction and optimization of preprocessing pipelines such as data cleaning and feature engineering, which were originally time-consuming and reliant on manual labor. It can be widely applied to scenarios such as finance, healthcare, and industrial manufacturing that require processing complex data and building high-precision models. Attached Figure Description

[0022] Figure 1 This is a flowchart of the knowledge-enhanced data preprocessing pipeline construction method described in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0025] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0026] Specific Implementation Method 1: Combination Figure 1 This embodiment describes a data preprocessing pipeline construction method based on knowledge enhancement, which includes: S1. Construct a historical task knowledge base, which stores multiple historical machine learning tasks, meta-feature vectors of datasets in each historical machine learning task, and multiple data preprocessing logic pipelines corresponding to each historical machine learning task; wherein, each historical machine learning task consists of a dataset and its corresponding downstream model. Each data preprocessing logic pipeline is composed of multiple processing modules with different functions cascaded in sequence, and each data preprocessing logic pipeline corresponds to a processing order, which is the logical execution order of the processing modules within the data preprocessing logic pipeline; S2. Extract the meta-features of the dataset in the learning task to be processed to form a meta-feature vector; the learning task to be processed consists of the dataset and its corresponding downstream model; S3. Based on the extracted meta-feature vectors, perform similar task matching with the meta-feature vectors stored in the historical task knowledge base to obtain a set of candidate data preprocessing logic pipelines. S4. Perform deletion and fusion operations on the candidate data preprocessing logic pipeline set to complete the logical reconstruction of each candidate data preprocessing logic pipeline and obtain the simplified candidate logic pipeline. S5. Using the Monte Carlo tree search algorithm, combined with the dataset in the learning task to be processed and the downstream model, the optimal processing algorithm is assigned to each processing module in each simplified candidate logic pipeline, and the pipeline performance score of the simplified candidate logic pipeline after assigning the optimal processing algorithm is output. Among them, common processing modules include missing value processing module, categorical feature encoding module, numerical feature normalization module, feature transformation module and feature selection module. The missing value processing module and feature selection module are located at the beginning and end of the pipeline, respectively, while the execution positions of the categorical feature encoding module, numerical feature normalization module and feature transformation module in the pipeline can vary. The streamlined candidate logical pipeline with the highest pipeline performance score, after being assigned the optimal processing algorithm, is selected as the optimal data preprocessing pipeline. This optimal data preprocessing pipeline also serves as the optimal physical pipeline, thus completing the pipeline construction. Specifically, the streamlined candidate logical pipeline, after being assigned the optimal processing algorithm, is used as the candidate physical pipeline.

[0027] This invention first extracts the meta-feature vector of the current task and matches it with the historical knowledge base, selecting only pipelines corresponding to similar tasks as candidates to avoid directly applying irrelevant historical experience. Then, irrelevant modules are deleted and duplicate structures are merged based on the actual situation of the current data, eliminating redundancy and conflicts caused by multi-source experience and significantly reducing the number of candidate pipelines. Finally, Monte Carlo tree search and performance scoring are performed only on the simplified candidate set, completing a sufficient search within a limited computational budget and selecting the pipeline with the highest score, thereby reducing evaluation costs while ensuring that the output pipeline is adapted to the current task and has stable performance.

[0028] The Monte Carlo tree search algorithm used in this invention is an existing technology that terminates the search when a preset number of evaluations, a time limit, or other stopping conditions are reached. The same number of evaluations can be assigned to each simplified candidate logic pipeline, or the total evaluation budget can be set according to the implementation environment.

[0029] Pipeline performance scoring reflects the pipeline performance of each simplified candidate logic pipeline after the dataset has been preprocessed with the optimal processing algorithm. After the dataset has been preprocessed, the downstream model in the current task is trained using the preprocessed dataset. The performance of the downstream model after training reflects the pipeline performance of each simplified candidate logic pipeline after the optimal processing algorithm has been assigned.

[0030] Furthermore, in step S3, similar task matching is performed based on the extracted meta-feature vectors and the meta-feature vectors stored in the historical task knowledge base to obtain the candidate data preprocessing pipeline set. The implementation methods include: Similarity task matching is performed based on the extracted meta-feature vectors and the meta-feature vectors stored in the historical task knowledge base. A candidate data preprocessing pipeline set is formed based on the top N meta-feature vectors in the matching results and all data preprocessing pipelines corresponding to the learning tasks. In application, N is an integer ranging from 3 to 5.

[0031] This preferred approach effectively controls the size of the candidate set and compresses the subsequent search space by selecting all pipelines corresponding to the top N historical tasks with the highest matching degree. While ensuring diversity, it reduces the computational cost of subsequent search and reconstruction, thereby improving the overall efficiency of the method.

[0032] Furthermore, in step S4, deletion and fusion operations are performed on the candidate data preprocessing logic pipeline set to complete the logical reconstruction of each candidate data preprocessing logic pipeline, resulting in the following simplified implementation of the candidate logic pipeline: S41. Deletion operation: When the number of class features and / or the proportion of missing values ​​in the extracted meta-feature vector are 0, the corresponding class feature encoding module and / or missing value processing module corresponding to the number of class features and the proportion of missing values ​​are used as the processing modules to be pruned. The processing modules to be pruned in each candidate data preprocessing logic pipeline are deleted, resulting in a reconstructed logic pipeline. S42, Fusion Operation: For all logic pipelines after primary reconstruction with the same channel length and processing order, the processing modules with the same function are merged to obtain a logic pipeline after secondary reconstruction, and this logic pipeline after secondary reconstruction is used as a candidate logic pipeline after simplification.

[0033] This optimized approach achieves automatic simplification and unification of pipeline logic. During deletion operations, irrelevant modules are pruned based on data characteristics (such as no category features or no missing values) to reduce redundant calculations. During fusion operations, pipelines with similar structures are functionally merged to reduce repetitive processing and improve the simplicity and execution efficiency of the pipeline structure.

[0034] Furthermore, the learning task to be processed and the meta-feature vector corresponding to the learning task to be processed are completely identical, and the meta-feature vector is composed of the number of samples, the number of features, the number of numerical features, the number of categorical features, the proportion of missing values, the number of target categories, the average skewness of numerical features, the average kurtosis of numerical features, the average variance of features, the average of the pairwise absolute correlation coefficients of features, the average of the mutual information between features and the target, the maximum value of the mutual information between features and the target, and the target entropy.

[0035] In this preferred approach, the meta-feature vectors corresponding to the learning tasks to be processed have the same composition and dimension, ensuring the comprehensiveness and comparability of feature representations, providing a reliable data foundation for similar task matching and pipeline optimization, and enhancing the versatility and robustness of the method.

[0036] Specific Embodiment 2: An electronic device includes a storage device, a processor, and a computer program stored in the storage device and executable on the processor. The processor executes the computer program to implement the knowledge-enhanced data preprocessing pipeline construction method as described in Specific Embodiment 1.

[0037] Specific Implementation Method 3: A computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed, it implements the knowledge-enhanced data preprocessing pipeline construction method as described in Specific Implementation Method 1.

[0038] Specific Implementation Method Four: A computer program product, including a computer program that, when executed by a processor, implements the knowledge-enhanced data preprocessing pipeline construction method as described in Specific Implementation Method One. While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A data preprocessing pipeline construction method based on knowledge enhancement, characterized in that, include: S1. Construct a historical task knowledge base, which stores multiple historical machine learning tasks, meta-feature vectors of datasets in each historical machine learning task, and multiple data preprocessing logic pipelines corresponding to each historical machine learning task; wherein, each historical machine learning task consists of a dataset and its corresponding downstream model. Each data preprocessing logic pipeline is composed of multiple processing modules with different functions cascaded in sequence, and each data preprocessing logic pipeline corresponds to a processing order, which is the logical execution order of the processing modules within the data preprocessing logic pipeline; S2. Extract the meta-features of the dataset in the learning task to be processed to form a meta-feature vector; the learning task to be processed consists of the dataset and its corresponding downstream model; S3. Based on the extracted meta-feature vectors, perform similar task matching with the meta-feature vectors stored in the historical task knowledge base to obtain a set of candidate data preprocessing logic pipelines. S4. Perform deletion and fusion operations on the candidate data preprocessing logic pipeline set to complete the logical reconstruction of each candidate data preprocessing logic pipeline and obtain the simplified candidate logic pipeline. S5. Using the Monte Carlo tree search algorithm, combined with the dataset in the learning task to be processed and the downstream model, the optimal processing algorithm is assigned to each processing module in each simplified candidate logic pipeline, and the pipeline performance score of the simplified candidate logic pipeline after assigning the optimal processing algorithm is output. The streamlined candidate logical pipeline with the highest pipeline performance score after being assigned the optimal processing algorithm is used as the optimal data preprocessing pipeline, and the optimal data preprocessing pipeline is also used as the optimal physical pipeline, thus completing the pipeline construction.

2. The data preprocessing pipeline construction method based on knowledge enhancement according to claim 1, characterized in that, In step S3, similar task matching is performed based on the extracted meta-feature vectors and the meta-feature vectors stored in the historical task knowledge base to obtain the candidate data preprocessing pipeline set. The implementation methods include: Similar task matching is performed based on the extracted meta-feature vectors and the meta-feature vectors stored in the historical task knowledge base. Based on the top N meta-feature vectors in the matching results, all data preprocessing pipelines under the learning task are used to form a candidate data preprocessing pipeline set.

3. The data preprocessing pipeline construction method based on knowledge enhancement according to claim 1, characterized in that, In step S4, deletion and fusion operations are performed on the candidate data preprocessing logic pipeline set to complete the logical reconstruction of each candidate data preprocessing logic pipeline, resulting in the following simplified implementation of the candidate logic pipeline: S41. Deletion operation: When the number of class features and / or the proportion of missing values ​​in the extracted meta-feature vector are 0, the corresponding class feature encoding module and / or missing value processing module corresponding to the number of class features and the proportion of missing values ​​are used as the processing modules to be pruned. The processing modules to be pruned in each candidate data preprocessing logic pipeline are deleted, resulting in a reconstructed logic pipeline. S42, Fusion Operation: For all logic pipelines after primary reconstruction with the same channel length and processing order, the processing modules with the same function are merged to obtain a logic pipeline after secondary reconstruction, and this logic pipeline after secondary reconstruction is used as a candidate logic pipeline after simplification.

4. The data preprocessing pipeline construction method based on knowledge enhancement according to claim 1, characterized in that, The learning task to be processed and the meta-feature vector corresponding to the learning task to be processed are completely identical. The meta-feature vector consists of the number of samples, the number of features, the number of numerical features, the number of categorical features, the proportion of missing values, the number of target categories, the average skewness of numerical features, the average kurtosis of numerical features, the average variance of features, the average of the pairwise absolute correlation coefficients of features, the average of the mutual information between features and the target, the maximum value of the mutual information between features and the target, and the target entropy.

5. An electronic device comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that, The processor executes a computer program to implement the knowledge-enhanced data preprocessing pipeline construction method as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the knowledge-enhanced data preprocessing pipeline construction method as described in any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the knowledge-enhanced data preprocessing pipeline construction method as described in any one of claims 1 to 4.

Images