Context semantic perception-oriented data preparation pipeline recommendation method and system

Abstract

The invention relates to the technical field of data preparation pipeline recommendation, in particular to a context semantic perception-oriented data preparation pipeline recommendation method and system. The method comprises the following steps: performing data preprocessing on an obtained automatic pipeline construction data set; aiming at the preprocessed data set, carrying out double-view feature fusion based on a self-attention mechanism; for the preprocessed data set, context semantic information extraction based on a large language model is carried out; based on feature fusion and information extraction, a pipeline recommendation result is obtained through a deep reinforcement learning network; evaluating and optimizing a pipeline recommendation result; and an output pipeline. Through double-view feature fusion based on a self-attention mechanism, end-to-end dynamic integration is carried out on statistical attributes of a data set and a historical pipeline component sequence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data preparation pipeline recommendation, in particular to a context-aware semantic perception-oriented data preparation pipeline recommendation method and system. BACKGROUND

[0002] In the field of machine learning and data science, data preparation is an important link to transform raw data into a form that can be efficiently utilized by algorithms. Data preparation usually includes a series of operations such as data cleaning, feature engineering, missing value processing, data normalization, encoding, etc., and its purpose is to transform raw data into structured input that can be efficiently utilized by downstream algorithms. However, the construction of an ideal data preparation pipeline faces a series of extremely complex and highly coupled challenges in actual engineering and scientific scenarios.

[0003] First, modern data exhibits characteristics such as large scale, strong heterogeneity, and prominent time-varying. Whether it is structured table data, semi-structured logs, or unstructured text and images, it may be mixed with problems such as missing, redundancy, abnormality, nesting, or distribution drift. Second, the data preparation process itself is highly dependent on the specific needs of the task and model. Different machine learning tasks (such as classification, regression, clustering, etc.) and their corresponding model structures (such as tree models, neural networks, ensemble learning, etc.) have different requirements for feature construction, data conversion, and class balancing. The diversity of business objectives and the evolution of model architectures further exacerbate the dynamic and personalized challenges of data preparation solutions. At the same time, data preparation involves a variety of operations, complex sequences, and a huge parameter space, and there may be strong dependencies or implicit constraints between different processing steps. The theoretical combination space doubles with each additional component operation, leading to an exponential explosion of candidate pipelines. More seriously, existing data preparation automation and human-machine collaborative optimization methods still have obvious shortcomings in dealing with the above challenges. Most methods rely on shallow feature expression or rule engines, lack comprehensive modeling of complex dependencies between operations, and cannot fully exploit historical experience, contextual information, and the global influence of operation sequences. In recent years, data preparation automation research has gradually shifted from traditional pipeline construction methods that rely on human experience and static rules to more intelligent and adaptive dynamic decision-making paradigms, emphasizing the use of contextual information inherent in data and processing, especially the complex dependencies between data features and pipeline history, to achieve automatic optimization and efficient search of data preparation processes. However, there are still some key problems that have not been solved: First, current methods rely on shallow feature concatenation or static rules in the expression and utilization of contextual information, making it difficult to effectively model the complex global dependencies between pipeline history, data statistical features, and task requirements. Second, there is a lack of dynamic and intelligent space compression mechanism in the automatic component selection process, leading to pipeline structure search still limited by combinatorial space explosion and low decision-making efficiency. Third, existing methods have limited adaptability to heterogeneous and dynamic data scenarios, making it difficult to achieve end-to-end high-quality pipeline automatic construction. Most methods fail to capture high-order global dependencies and long-term historical influences between operations, lack dynamic feedback mechanisms, and result in pipeline optimization processes lacking closed-loop adaptation and long-term effectiveness guarantees.

[0004] Therefore, how to fully leverage the advantages of large models and their self-attention mechanisms in context understanding and complex dependency modeling to achieve deep integration of multi-source information and efficient decision-making becomes a key technical problem in promoting the continuous evolution of data preparation automation. In response to these unsolved problems, this paper (or this patent) proposes a context-aware data preparation pipeline automatic construction method based on large models and self-attention mechanisms, aiming to further improve the intelligence, generalization, and automatic optimization level of data preparation in complex scenarios. SUMMARY

[0005] To solve the above-mentioned problems, the application provides a context semantic perception-oriented data preparation pipeline recommendation method and system.

[0006] In a first aspect, the application provides a context semantic perception-oriented data preparation pipeline recommendation method, which adopts the following technical scheme: A context semantic perception-oriented data preparation pipeline recommendation method comprises: acquiring an automated pipeline construction dataset; performing data preprocessing on the acquired automated pipeline construction dataset; performing double-view feature fusion based on a self-attention mechanism for the preprocessed dataset; performing context semantic information extraction based on a large language model for the preprocessed dataset; obtaining a pipeline recommendation result through a deep reinforcement learning network based on feature fusion and information extraction; evaluating and optimizing the pipeline recommendation result; outputting a pipeline.

[0007] Further, the data preprocessing on the acquired automated pipeline construction dataset comprises identifying data fields closely related to a target label in a plurality of heterogeneous data sources according to task definition, and performing category distribution statistics through explicit label column identification after preliminary screening of features and labels; context information of tabular data is captured synchronously in the extraction stage, and accurate labeling of data fields is realized through a column name and semantic correspondence mechanism; finally, a consistency and standardization mechanism is introduced, different source label encodings are uniformly mapped to a consistent category set by aligning labels, the uniqueness of classification semantics is ensured, and uniform numerical operation is performed on features; the normalization or bucketing strategy is adopted for numerical fields, and the discrete fields are converted into dictionary indexes or embedding vectors to ensure the comparability of different features in the same space; for missing data, the completeness of the feature matrix is ensured through filling or rejection methods; after the process, the final classification task representation is obtained as follows: , wherein and are standardized features and labels respectively, is context meta-information bound with data.

[0008] ​Furthermore, the dual-view feature fusion based on the self-attention mechanism includes the statistical view feature extraction of the data set and the pipeline history component feature extraction. For the statistical view feature extraction of the data set, firstly, a detailed statistical feature extraction is performed on the input data set. For the case where the numerical feature distribution span is large, a combination of the fractional part and the exponential part is used to express all the numerical features. by The encoding is performed in the form of represents the fractional part of the feature, Represents its exponential part, and the two are To jointly characterize the original features; for the feature extraction of pipeline history components, a sequence modeling neural network is introduced as the core encoding module, which can process all executed data processing components in the historical order. , which is converted into a dense vector sequence through the embedding layer ,in is the number of components, is the embedding dimension, and then the dense vector sequence Input to the multi-layer stacked Transformer layer, the global dependencies between all components in the sequence are modeled in parallel, expressed as: , Then, weighted pooling is used to extract global features from the sequence to represent the historical context information of the current pipeline.

[0009] Furthermore, the dual-view feature fusion based on the self-attention mechanism also includes introducing a feature fusion method based on the cross-view attention mechanism to dynamically capture the interactive relationship between the data distribution view and the pipeline structure view in an end-to-end trainable manner, wherein the statistical characteristics of the data set are assumed to be and pipeline history global features As input sources of different modalities, the statistical features of the data set are used as query vectors. As a key, the attention weights of statistical features under different pipeline history representations are automatically calculated, expressed as: , in, , , It is a trainable reference, and the global representation of the pipeline components is obtained after aggregation by the attention mechanism. , expressed as: .

[0010] Furthermore, the contextual semantic information extraction based on the large language model includes introducing a contextual semantic information extraction mechanism based on the large language model, and deeply mining the multi-source contextual information of the data set in an automated manner. Specifically, by converting the original data table into a CSV format text with a header and data samples, the large language model is used to process ultra-long contexts and the token-level attention mechanism to capture the key features of field naming, value distribution patterns, and synonymous field classification, which are expressed as: , in represents the context vector generated by the large language model, It represents the embedding of the large language model into the inference function. Then, based on the context-based decision-making method, it simulates the process of experts narrowing down the scope of operation selection through domain knowledge, and realizes dynamic cropping and intelligent filtering of the candidate component space, which is expressed as: , Among them, when the candidate component In the given context The conditional probability under Greater than the preset threshold , the component is selected into the final component set By introducing contextual embedding features for each operation decision, we automatically focus on the component combination that is most likely to improve the data processing effect in the action space. At the same time, we propose an optimized target loss function. By considering the influence of recommendation accuracy and contextual information, we optimize the decision-making process in pipeline construction, which can be expressed as: , in, is the recommended pipeline component, is the actual component used, is the regularization coefficient, controlling the context information role in the optimization process.

[0011] Furthermore, the pipeline recommendation result is obtained through deep reinforcement learning network based on feature fusion and information extraction, including embedding all information in the pipeline construction process as state , in the decision-making process, through deep The network predicts all possible actions The Bellman equation is used to take into account the expectation of future multi-step cumulative rewards and achieve a dynamic balance between short-term and long-term benefits, which can be expressed as: , in, Indicates that in a given state Take action , and according to the current parameters The strategy can obtain the expected value of long-term accumulation, Indicates that an action is being performed Then transfer to the next state , Represents the discount factor, and then based on the strategy Select components that maximize rewards ,Right now , by inputting the current state s, outputting the priority or probability of each possible action, The calculation of the value follows the Bellman equation, which evaluates the component selected at the current step by the formula The expected value of the future cumulative reward is expressed as: , in Indicates the current step selection component The instant reward value obtained when is the discount factor of the reward, if A value close to 1 indicates that the agent will pay close attention to the benefits of each step in the future and take long-term effects into consideration. If the value is smaller than the set value, the agent will focus more on the immediate rewards obtained from the current action and pay less attention to future rewards.

[0012] Furthermore, the pipeline recommendation result is obtained through a deep reinforcement learning network based on feature fusion and information extraction, and the deep reinforcement learning network is trained. First, a dataset is randomly extracted from a large-scale training dataset pool containing different fields and diverse data distributions. , get the current data , based on the current statistical characteristics of the data set and the existing pipeline historical information to form a state , select an action Automatically generate an adapted data preparation pipeline and evaluate the pipeline effect through the performance indicators of the downstream model. As a reward signal Feedback is given to the agent to optimize the decision-making strategy, which is expressed as: , Under the continuous training of cross-domain heterogeneous data, the samples are Minimize the loss, expressed as: , And the learning rate Make an update , ensuring that the automated pipeline recommendation system spans diverse data environments, continuously outputs high-quality pipeline combinations, and The optimal pipeline is expressed as: , in Indicates that in all candidate pipeline sets The pipeline configuration that maximizes the expected cumulative reward; It represents the mathematical expectation of a random process and measures the average performance.

[0013] Furthermore, the evaluation and optimization of the pipeline recommendation results include: , the optional label for the supervision task is Perform input reasoning; generate multidimensional labels of numerical, categorical, text, time series, and geographic information or identifiers with the help of field type identification and consistency verification mechanisms, select representative sample rows through stratified sampling methods, perform range sketching on data features, and extract statistical views of the data as a whole Finally, similar tasks are found in the task library based on the embedded retrieval mechanism. If the similarity exceeds the set threshold, the corresponding pipeline prefix is ​​extracted as the initial pipeline history. , and in and Fusion to form the initial state vector for reasoning : , After the initial state vector is generated, in the component library Under the constraints of the candidate specification, based on the context semantic representation Calculate conditional prior probabilities , and then calculate the comprehensive score for each candidate component: , Calling pre-trained strategy and its corresponding action-value function , the optimal component sequence is gradually selected and the pipeline is constructed under a mild exploration mechanism.

[0014] Furthermore, the evaluation and optimization of the pipeline recommendation results also includes updating the current environment state in real time based on the state transfer mechanism of reinforcement learning after the candidate components are selected and executed. In each transfer, the newly generated intermediate result features are embedded into the semantic space, fused with the previous context representation, and a new context vector is formed through weighted aggregation , while the pipeline history track Updated to include a sequence description of the currently executed component, allowing subsequent candidate selection to be made with more complete context: , By the above recursive state transition, the optimal solution of the target task is gradually approached; in the pipeline construction process, a global evaluation mechanism is introduced to dynamically score the candidate pipeline in multiple dimensions, when the iteration is completed and the final pipeline is generated, the pipeline is taken as the output result of reasoning, and based on the feedback mechanism, the reasoning output is compared with the verification result after actual execution, if the prediction deviation is found, the deviation information is returned to the policy network and the value function network, so that the model parameters are gradually corrected in future reasoning, which is represented as: , Wherein represents the difference between the actual performance and the predicted performance, which is used to measure the gap between the prediction model and the true result, represents the model parameters, represents the learning rate for controlling the step size of each parameter update, represents the performance prediction error.

[0015] In a second aspect, a data preparation pipeline recommendation system oriented to context semantic perception includes: A data acquisition module configured to acquire an automated pipeline construction dataset; A preprocessing module configured to perform data preprocessing on the acquired automated pipeline construction dataset; A feature fusion module configured to perform double-view feature fusion based on a self-attention mechanism for the preprocessed dataset; An information extraction module configured to perform context semantic information extraction based on a large language model for the preprocessed dataset; A pipeline recommendation module configured to obtain a pipeline recommendation result through a deep reinforcement learning network based on feature fusion and information extraction; An optimization module configured to evaluate and optimize the pipeline recommendation result; An output module configured to output the pipeline.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a plurality of instructions stored therein, the instructions being adapted to be loaded and executed by a processor of a terminal device to implement the data preparation pipeline recommendation method oriented to context semantic perception.

[0017] In a fourth aspect, the present application provides a terminal device including a processor and a computer-readable storage medium, the processor being configured to implement the instructions; the computer-readable storage medium is configured to store a plurality of instructions, the instructions being adapted to be loaded and executed by the processor to implement the data preparation pipeline recommendation method oriented to context semantic perception.

[0018] In summary, the present application has the following beneficial technical effects: This paper dynamically integrates the statistical properties of a dataset with the historical pipeline component sequence end-to-end through dual-view feature fusion based on a self-attention mechanism. The self-attention mechanism automatically captures deep dependencies and nonlinear interactions between features without human intervention, effectively enhancing the discriminative power of the global state representation and enabling the recommendation system to maintain greater robustness and generalization capabilities in the face of complex and variable data distributions.

[0019] Secondly, by introducing contextual semantic information extraction based on a large language model, this invention enables high-dimensional embedding of heterogeneous multi-source information, such as data headers, field names, and sample content. This module not only addresses the shortcomings of traditional statistical features in modeling domain context, but also significantly reduces the search space for pipeline components, improving the targetedness, accuracy, and inference efficiency of the recommendation process.

[0020] Thirdly, by using a deep reinforcement learning network to automatically search and make optimal decisions on pipeline structures, the system can combine the multi-dimensional characteristics of the current state, dynamically balance immediate and long-term benefits, achieve optimal recommendations for the global structure of the pipeline, and effectively avoid local optimality and empiricism traps.

[0021] Multiple experiments and evaluations have shown that the present invention has significant improvements over existing technologies in terms of pipeline generation accuracy, decision delay, and generalization ability to adapt to new fields, and has broad engineering application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A framework diagram for constructing the pipeline of the present invention.

[0023] Figure 2 This is a flow chart of the automated pipeline construction method proposed in the present invention.

[0024] Figure 3 This is the flowchart of dual-view feature fusion based on the self-attention mechanism in step 1 of the present invention.

[0025] Figure 4 This is the flowchart of step 3 of the deep reinforcement learning network training of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to the accompanying drawings.

[0027] Example 1 Reference Figure 1 , a context-semantics-aware data preparation pipeline recommendation method of this embodiment includes: S1. Automated pipeline construction and dataset extraction In the construction process of the automated data preparation pipeline, the acquisition and characterization of the dataset are the front-end link of the whole process, and their quality and expression ability directly affect the performance of subsequent feature fusion, self-attention modeling and pipeline optimization. The present application proposes a systematic extraction and modeling method for the dataset of machine learning classification task, provides rich and stable context input for pipeline optimization through the joint mechanism of distribution feature modeling, semantic dependency mining and global consistency representation. The overall idea is to closely combine the input characteristics of the classification dataset with the task requirements, to ensure that the subsequent component recommendation and structure optimization can not only capture a single statistical pattern, but also perceive the potential class distribution law and context interaction effect.

[0028] S1.1 Data set statistics view feature extraction In the classification task scenario, the system first defines the target task as: , wherein, represents a sample feature matrix, represents a class label, represents the number of classes. According to the task definition, the extraction process needs to identify the data fields closely related to the target label in multiple heterogeneous data sources, such as structured records in relational databases, event attributes in log streams or indicator parameters in experimental result sets. After completing the preliminary screening of features and labels, the system will explicitly identify the label column and perform statistics on the class distribution, specifically by calculating: , to depict the frequency of each class. This can reveal the potential class imbalance and long-tail distribution characteristics in the data, such as the phenomenon that some classes have very low sample proportion. This distribution perception process not only constitutes the core context information of the classification dataset, but also becomes an important reference for model selection, resampling strategy and regularization method in the subsequent automated pipeline.

[0029] S1.2 Context information capture and semantic annotation In the construction process of the classification dataset, simply relying on the feature matrix and label vector is often insufficient to fully express the actual context of the data. Therefore, the present application proposes to capture the context information of tabular data simultaneously in the extraction stage, and realize accurate annotation of data fields through the correspondence mechanism of column name and semantics. Let the field set contained in the original data table be: , where each field corresponds to a column of the data table. The system generates a semantic description for each field in the extraction process, and then forms a mapping relationship: , This column-name-semantics correspondence not only characterizes the structure of data, but also preserves the external environment and business logic. For example, in the medical classification task, the column name "blood_pressure" is labeled as "clinical detection indicator", and the column name "collection_date" is labeled as "experimental timestamp"; in the log stream classification scenario, the column name "event_id" is labeled as "event identifier", and the column name "timestamp" is labeled as "time sequence anchor point". Through this mechanism, the system not only obtains the numerical level features and labels after extraction, but also understands the semantic role of each column in the task. This structured representation enables the automated pipeline to fully utilize the business meaning and task constraints of the field in the subsequent model and component recommendation process, thereby significantly improving the adaptability and discriminability of the classification task.

[0030] S1.3 Data consistency and standardization mechanism Since data often comes from heterogeneous environments, there are obvious differences in format, encoding method and label system between different sources. If these data are directly used, it will lead to uncontrollability of the pipeline optimization process. Therefore, in the extraction link, the invention introduces a consistency and standardization mechanism to realize the unified representation of cross-source data. This mechanism first aligns the labels, and maps the label codes of different sources to a consistent category set , ensuring the uniqueness of classification semantics. Secondly, uniform numerical operation is performed on the features, and the numerical fields use normalization or bucketing strategy, and the discrete fields are converted into dictionary index or embedding vector to ensure the comparability of different features in the same space. For missing data, the system ensures the integrity of the feature matrix through filling or rejection method. After this process, the final classification task representation is obtained: , wherein and are the standardized features and labels, is the context meta-information bound with the data. Through the above mechanism, the classification data has a unified format and stable semantics before entering the automated pipeline optimization link, so as to seamlessly connect the subsequent feature modeling and component search.

[0031] S2. Dual-view feature fusion based on self-attention mechanism S2.1 Statistical view feature extraction of data set In the automatic construction process of the data preparation pipeline, the invention first performs detailed statistical feature extraction on the input data set. Specifically, for each intermediate data set , its statistical features It consists of the field feature vectors calculated column-wise, denoted as:

[0032] in, Indicates the Statistical characteristics of the column, The maximum number of columns. When the actual number of fields is less than When , truncation or zero padding is used to ensure that the vector length of statistical features under all tasks is consistent, so as to facilitate the subsequent construction of the state vector in the reinforcement learning network. express.

[0033] Each column feature This includes a set of statistics that effectively reflect the data attributes of the field. These statistics include not only basic attributes such as column type, missing value ratio, and number of unique values, but also higher-level numerical statistical indicators such as mean, standard deviation, and median. These statistics help the model distinguish and determine the applicability and priority of different components. These column features provide rich data support for pipeline component selection.

[0034] In view of the fact that numerical features may have a large distribution span, extreme values ​​or uneven distribution, the present invention further proposes an innovative expression method: using a combination of fractional part and exponential part, namely scientific notation. Specifically, all numerical features by The encoding is performed in the form of represents the fractional part of the feature, Represents its exponential part, and the two are This design fully preserves the order of magnitude of the features, effectively avoiding the resolution loss that can result from simple normalization, and ensuring the model maintains excellent discriminative power and expression stability when processing large-scale span data. Furthermore, this representation helps the reinforcement learning model adaptively focus on key information at different orders of magnitude, improving the model's sensitivity to extreme values ​​and magnitudes of change.

[0035] In summary, systematic statistical feature extraction and scientific notation-style numerical feature encoding not only lay a solid data foundation for the subsequent deep feature fusion of the self-attention mechanism, but also significantly improve the generalization capability and practical application effect of pipeline automated recommendation.

[0036] S2.2 Pipeline History Component Feature Extraction In the automated data preparation pipeline modeling framework of the present application, the pipeline history component feature extraction focuses on the accurate expression of the context semantics, component interaction relationship and its order characteristics of the executed operation sequence. Specifically, for each piece of data set to be optimized, when selecting the next component, the system continuously tracks the order and type information of all selected operation components in the current pipeline. The order not only includes each basic data processing step, but also can cover feature engineering, model evaluation and other links, and the arrangement and combination directly affect the final data processing effect.

[0037] In order to fully extract the rich temporal information, dependency relationship and potential influence on subsequent decision making contained in the execution order of data preparation pipeline components, the present application innovatively introduces a sequence modeling neural network as the core coding module. Specifically, the system first converts all executed data processing components (including but not limited to missing value filling, standardization, feature selection, feature generation, dimension reduction, etc.) into a dense vector sequence according to their historical order , where is the number of components, is the embedding dimension. In this process, each type of component has an independent and trainable embedding expression in the vector space, so that the model can perceive and distinguish the semantic differences of different operation types.

[0038] Then the dense vector sequence is input into a multi-layer stacked Transformer module. Each layer of Transformer consists of multi-head attention mechanism and feedforward network, which can model the global dependency relationship between all components in the sequence in parallel, effectively capturing the interaction and long-term influence between different historical steps.

[0039] , After the Transformer coding is completed, the weighted pooling method is used to extract the global features from the sequence to represent the historical context information of the current pipeline. The weighted pooling method assigns a learnable weight to the hidden state of each position in the sequence, and sums all the encoded vectors by weight to obtain a global feature vector that comprehensively reflects the influence of all historical components. The formula is as follows: , , where is the hidden state vector of each component position in the pipeline history at time , w is a learnable parameter variable. is the weight of each position, is the global pipeline history component feature.

[0040] S2.3 Dual-view feature fusion To fully exploit and integrate the complementary potential of data distribution features and pipeline history structural information in automated pipeline optimization, the present invention innovatively introduces a feature fusion method based on cross-view attention. Unlike traditional simple concatenation or static weighted fusion methods, cross-view attention can dynamically capture the deep dependencies and complex interactions between "data distribution view" and "pipeline structure view" in an end-to-end trainable manner, thereby significantly improving the expression and discrimination ability of global state representation.

[0041] Specifically, let the statistical features of the data set and the global features of the pipeline history be the input sources of different modalities, respectively. By taking the statistical features as the query vector (Query), and the global features of the pipeline history as the key / value vector (Key / Value), the cross-view attention mechanism can automatically calculate the attention weights of the statistical features under different pipeline history representations, achieving efficient extraction and aggregation of key information. Its attention score is given by the following formula: , where, , , is a trainable reference. Compared with traditional concatenation or static weighted fusion methods, the introduction of cross-view attention mechanism greatly enhances the context adaptive ability, nonlinear expression and high-order interaction ability of global state representation, and the generalization of the model under different data distributions, task transfer and unknown scenarios.

[0042] , where, is the global representation of the pipeline component after aggregation by the attention mechanism, representing the deep interaction between data distribution features and pipeline history features.

[0043] The attention mechanism can automatically learn and dynamically adjust the attention weight between different information sources according to the actual task and the input distribution, so that the system is no longer limited to the simple information superposition of static channels when facing diversified data and complex pipeline structures, but realizes the organic unification of "global perception" and "local focus". This mechanism effectively captures the complex and dynamic coupling relationship between key components and statistical characteristics, improves the interaction depth and expression level between different modal views, and enriches the semantic dimension and discrimination ability of the state space. At the same time, the information reweighting ability of the cross-view attention mechanism enables the model to dynamically focus on the features and historical paths that contribute most to the current decision, which not only helps the model to exhibit stronger robustness in new data, unknown scenarios, distribution drift or task migration, but also proves the effectiveness and adaptability of the attention mechanism in improving the generalization ability of pipeline optimization and component recommendation system from both theoretical and empirical aspects.

[0044] Therefore, the dual-view feature fusion method based on the cross-view attention mechanism provides a solid theoretical foundation and innovative solution for the automated data preparation system in a complex, dynamic and variable data environment, and greatly enriches and expands the technical boundaries of data-driven pipeline modeling.

[0045] S3. Contextual semantic information extraction based on large language model In the field of structure search and component recommendation of automated data preparation pipeline, although a large number of related works have tried to use genetic programming, Bayesian optimization, reinforcement learning and differentiable search and other technical solutions to automatically select and sort pipeline components, there are still problems of exponential expansion of search space and huge reasoning overhead. The fundamental reason for this limitation is that most methods only rely on schema and basic statistical features to guide pipeline construction, lacking systematic utilization of domain context and deep data semantic information, making it difficult to effectively draw on the domain knowledge and practical experience relied on by data scientists when manually constructing the process. To address this deficiency, the invention proposes a contextual semantic information extraction mechanism based on a large language model, which automatically mines multi-source contextual information of the dataset, including but not limited to table header column names, typical field values, domain terms, type and value distribution, etc., and converts these key information into high-dimensional vector representation, injecting domain priori and rich semantics into the pipeline component selection and search process.

[0046] Unlike traditional shallow expressions that rely only on table structure and statistics, large language models can encode the table header, representative sample rows, and field content of a data table as a unified contextual input in the form of structured text, and achieve end-to-end embedding reasoning through deep neural networks. Large language models represented by DeepSeek not only have powerful cross-domain semantic understanding capabilities, but can also adaptively model entity relationships, implicit contexts, and domain conventions in complex data tables. In practical applications, we convert the original data table into CSV format text with a table header and data samples, and use the large language model's efficient processing of ultra-long contexts and token-level attention mechanism to effectively capture key features such as field naming, value distribution patterns, and synonymous field classification, so that the embedding vector contains both the structural information and domain semantics of the data.

[0047] , in represents the context vector generated by the large language model, Represents the embedding inference function of a large language model. In addition, large models such as DeepSeek further enhance the robustness and generalization of the model to dynamically changing table structures by truncating input sequences, prioritizing key fields, and comprehensively representing multi-granularity semantic information within the context window. Compared to traditional table-specific models that only predict cells, large language models are not only unaffected by physical structure disturbances, but can also flexibly adapt to highly dynamic data environments, effectively simulating the deep reasoning of human experts on field attributes and data semantics during pipeline construction, greatly improving the generalization, interpretability, and intelligence of automated pipeline recommendations.

[0048] More importantly, this invention leverages the powerful contextual understanding and representation capabilities of large language models to directly incorporate rich semantic information into the decision-making process for pipeline component selection and architecture search. This context-based decision-making approach not only effectively simulates the process by which human experts narrow down the scope of operational choices using domain knowledge, but also enables dynamic pruning and intelligent filtering of the candidate component space.

[0049] , Specifically, only when the candidate component In the given context The conditional probability under Greater than the preset threshold When the component is selected, it will be included in the final component set This process dynamically narrows down the range of component selection by introducing contextual information, helping the system efficiently focus on those components that are most likely to improve data processing effectiveness. By introducing contextual embedding features at each operation decision, the system can automatically focus on the most likely component combinations that can improve data processing effectiveness in a vast action space, significantly reducing the automated component search space, improving structure search efficiency, and reducing invalid attempts and resource consumption. In this way, even in the face of highly complex and diverse data sets and task scenarios, efficient and accurate pipeline structure recommendation and optimization can be achieved, laying a solid semantic foundation and intelligent support for data preparation automation in complex business scenarios. To further improve the accuracy and efficiency of the pipeline component recommendation process, we propose an optimization target loss function that considers the impact of recommendation accuracy and contextual information to optimize the decision-making process in pipeline construction. The optimization target loss function is as follows: , where, is the recommended pipeline component, is the actual component used, the first term is used to minimize the difference between the recommended component and the actual component, improving the accuracy of the recommendation. The second term is a regularization term, is the regularization coefficient, controlling the role of contextual information in the optimization process, ensuring that semantic information does not have too much impact on the recommendation results, avoiding overfitting.

[0050] This technical path can automatically simulate human experts' prior judgment and contextual reasoning in the pipeline construction process, significantly compressing the automated component search space and significantly improving the accuracy and explainability of the recommendation decision, and laying a solid semantic foundation and intelligent support for data preparation automation in complex business scenarios.

[0051] S4. Deep reinforcement learning network In this method, the automatic construction of the data preparation pipeline is modeled as a reinforcement learning problem, aiming to achieve the optimal combination of processing components through the continuous learning of the agent. First, the system embeds all key information in the current pipeline construction process, including the statistical characteristics of the current data set (such as the data type of each field, the proportion of missing values, the distribution, etc.), the history of the pipeline executed, and the current available data preparation operations, to form the "state" This state comprehensively reflects the overall situation of the pipeline system at the time of decision-making, and is the basis for the agent to make each operation decision.

[0052] In each decision-making process, the agent takes the state as input, and uses a deep The network (DQN) predicts the value of all possible actions (i.e., candidate components) . The value not only measures the immediate reward brought by the current action, but also takes into account the expectation of future multi-step cumulative rewards using the Bellman equation, achieving a dynamic balance between short-term and long-term benefits.

[0053] , Specifically, the action represents the selection of one from all data preparation operation components (such as missing value filling, normalization, encoding, outlier detection, etc.) to be executed in the current round of pipeline, and each action selection will change the state of the dataset, thereby affecting the construction effect of the subsequent pipeline.

[0054] The policy is the decision rule used by the agent to decide which action to choose at each step. It is essentially a mapping controlled by parameters θ (usually neural network weights) that selects the component that maximizes the reward, i.e. . The input is the current state s, and the output is the priority or probability of each possible action. The calculation of the value follows the Bellman equation, which evaluates the expected value of future cumulative rewards when selecting component at the current step: , where represents the immediate reward value obtained when selecting component at the current step, such as the improvement of downstream model performance. is a discount factor for balancing the immediate reward of the current decision and the cumulative reward that may be obtained in the future. The value of determines the degree of emphasis on "long-term returns" in the decision-making process of the agent. If is close to 1, the agent will pay high attention to the benefits brought by each step in the future, and will consider long-term effects as the main consideration; if is less than a certain value, the agent will focus more on the immediate reward obtained by the current action, and the degree of attention to future rewards will decrease. By introducing the discount factor , the balance between "quick effect" and "pursuit of long-term optimization" of the decision policy can be effectively adjusted, so that the automatic construction process of the pipeline can not only consider the direct effect of each step selection, but also fully consider the global optimality of the overall structure.

[0055] Through end-to-end neural network training, DQN automatically learns how to combine diverse data distributions with historical processing history, dynamically adapting to environmental changes and continuously optimizing component selection without requiring manual experience or fixed rules. This effectively avoids empiricism and local optimality, ultimately achieving globally optimal recommendations for the data preparation pipeline structure. The core goal of reinforcement learning is to enable intelligent agents to consistently make optimal decisions across diverse data and scenarios through continuous trial and error and policy optimization, balancing both immediate and long-term performance, thereby continuously improving the overall intelligence of the automated pipeline.

[0056] In the actual training phase, in order to make the pipeline automatically build models with strong generalization capabilities, this method learns a set of global decision strategies through continuous iteration. Specifically, the system first randomly extracts a dataset from a large-scale training dataset pool containing different fields and diverse data distributions. (Get current data ). Then, the reinforcement learning agent forms a state based on the current statistical characteristics of the data set and the existing pipeline history information. , and select an action Automatically generate an adapted data preparation pipeline. The generated pipeline will be actually applied to the dataset, and the pipeline effect will be evaluated through indicators such as the performance of the downstream model. Evaluation results As a reward signal Feedback is given to the intelligent agent to optimize its decision-making strategy, enabling it to continuously generate data preparation pipelines with better performance in subsequent training and inference.

[0057] , Under continuous training of heterogeneous data across domains, the learned global strategy can not only improve the pipeline performance of a single dataset, but also maintain good generalization and robustness in new tasks and unknown domains. DQN uses target network and experience replay for stable training. Minimize loss: , And the learning rate Make an update This training mechanism ensures that the automated pipeline recommendation system can cross diverse data environments and continuously output high-quality pipeline combinations, fully meeting the intelligent needs of actual large-scale and heterogeneous data scenarios. The optimal pipeline can be expressed as: , The present invention addresses the core bottlenecks in the field of automated data preparation pipeline construction, such as search space expansion, insufficient generalization, and limited intelligence, and proposes a three-layer structured solution that integrates a large language model and a self-attention mechanism. This method is driven by theory and guided by engineering feasibility. It breaks through the existing traditional pipeline optimization paradigm that relies solely on statistical features and table structures, and achieves a deep fusion of data distribution characteristics, historical pipeline information, and contextual semantic knowledge, thereby greatly improving the intelligence, adaptability, and generalization of pipeline automated recommendations. Overall, the present invention not only enriches the technical means in the field of automated data preparation, but also provides a new paradigm for the construction of efficient and robust data pipelines in multi-task and multi-scenarios.

[0058] S5. Reasoning and executable output To ensure that trained policies can be directly applied in real-world business scenarios, this invention incorporates the inference and delivery phase after "S4. Deep Reinforcement Learning Network." This phase automatically generates pipelines for new data and tasks, provides compliant interpretation, and produces executable output. This phase completes the end-to-end closed loop from contextual awareness to pipeline implementation without relying on human intervention.

[0059] S5.1. Inference Input and Initialization Let the new task data be , the optional label for the supervision task is (Can be empty in unsupervised / weakly supervised scenarios) Perform input reasoning; Generate multi-dimensional labels such as numerical values, categorical types, text, time series, geographic information or identifiers with the help of field type identification and consistency verification mechanisms. At the same time, the system identifies fields that may contain sensitive information based on regular expressions and domain dictionary rules, and performs security processing on the data through desensitization operations such as masking and hashing to ensure that privacy is not leaked during the reasoning process. Subsequently, representative sample rows are selected through a stratified sampling method, and a range sketch is performed on the data features. For example, quantiles, means, variances and missing rates are extracted in numerical fields, cardinality, main category distribution and long-tail ratio are extracted in categorical fields, key n-grams and keywords are extracted in text fields, and span, periodicity and trend information are extracted in time fields. At the same time, field units are automatically identified and scale normalization is performed to establish a highly comparable feature description.

[0060] The above processing results will be encoded into structured semantic text, and synonyms and terminology will be unified on this basis to ensure the semantic consistency of fields such as "gender", "sex", and "gender". Subsequently, the system will map the table header, field type, value range sketch and representative samples to the embedding space through the semantic encoding model, obtain field-level and table-level embedding vectors, and generate contextual semantic representation through attention aggregation. At the same time, a statistical view of the data as a whole is extracted , including at least field type encoding, missing rate, unique value ratio, zero variance flag, numerical distribution points, dominant category proportion of sub-type, and time column periodicity and trend factor. The statistical characteristics are encoded in a scientific notation double-channel to preserve the order of magnitude information and proportional information. Finally, the system searches for similar tasks in the task library based on the embedding retrieval mechanism, and if the similarity exceeds the set threshold, the corresponding pipeline prefix is extracted as the initial pipeline history , and is fused to form the initial state vector of reasoning : .

[0061] S5.2 Candidate Regulation and Strategy InferenceAfter the initial state vector is generated, the reasoning process enters the candidate component regulation and strategy inference phase. The present invention carries out candidate regulation under the constraint conditions of the component library . The component library maintains input-output contract, pre-condition, mutual exclusion relationship, dependency graph, and resource and security level metadata for each component. The system first calculates the conditional prior probability based on the context semantic representation , which comprehensively considers the field matching degree, the historical marginal contribution of similar tasks, and the potential risk deduction, and is calibrated by temperature to control the confidence interval. Then, the system calculates the comprehensive score of each candidate component: , , where the compatibility term reflects the semantic consistency with the current state, and the resource adaptation term reflects the feasibility under the computation power, storage, and latency budget. If the component has hard violations in compliance, security, or interpretability indicators, it is directly excluded and the rejection reason is recorded.

[0062] After the regulation of the candidate set is completed, the reasoning engine enters the decision cycle. This stage calls the pre-trained strategy and its corresponding action value function , and gradually selects the optimal component sequence and constructs the pipeline under the mild exploration mechanism (combination of greedy and temperature softening). After each action selection, the agent generates a local candidate pipeline and performs rapid verification through the agent evaluation mechanism, including task adaptation degree simulation, resource consumption estimation, and interpretability threshold test. If the local pipeline meets the threshold in multiple dimensions, it enters the next iteration, otherwise the system reverts and reselects candidates. When the iteration reaches the upper limit or the evaluation result converges, the reasoning process terminates, and the system outputs the final pipeline design as the reasoning result of the model.

[0063] S5.3 State Transition and Action Execution​ After the candidate component is selected and executed, the system will update the current environment state in real time according to the state transition mechanism of reinforcement learning . This state update not only reflects the evolution of the data processing flow at the semantic level, but also includes changes in resource consumption, performance indicators, and compliance constraints, thereby ensuring that the state vector can dynamically depict the running situation of the entire pipeline. In each transition, the system will embed the newly generated intermediate result features into the semantic space, fuse them with the previous context representation, and form a new context vector through weighted aggregation . At the same time, the pipeline history trajectory is updated to include the sequence description of the currently executed components, thereby enabling subsequent candidate selection to be performed under more complete context information.

[0064] , The system gradually approaches the optimal solution of the target task through the above recursive state transition.

[0065] S5.4 Global Evaluation and Pipeline Optimization During the gradual construction of the pipeline, the invention introduces a global evaluation mechanism to dynamically score the candidate pipeline in multiple dimensions. This score covers accuracy, generalization, resource utilization, inference latency, explainability, and compliance, etc. Specifically, the system simulates the performance of the pipeline on new data through a fast proxy model and corrects the bias based on heuristic correction methods; in terms of resources, it estimates the computational complexity, memory consumption, and storage pressure of each candidate pipeline to ensure that it meets the requirements of the constraint set ; in terms of compliance and explainability, the system verifies whether each step of the pipeline has traceability through an explainability rule base and a logic checker to avoid "black box" operations. If the global score is below the threshold, the system will backtrack to the previous state and reselect the candidate action until a performance-converged and stable pipeline sequence is obtained.

[0066] S 5.5 Global Evaluation and Pipeline Optimization When the system completes the iteration and generates the final pipeline, the pipeline is submitted to the user or subsequent module as the output result of inference. The output forms include executable component sequences, key hyperparameter configurations, resource consumption estimates, and performance prediction indicators, ensuring that the results can be directly deployed and also provide a basis for subsequent manual review and modification. At the same time, the system also establishes a feedback mechanism to compare the inference output with the verification results after actual execution, and if a prediction bias is found, the bias information is returned to the policy network and value function network, thereby gradually correcting the model parameters in future inference.

[0067] , The closed-loop optimization not only guarantees the reliability and repeatability of the output results, but also enables the self-iterative evolution of the application.

[0068] Experimental verification: To comprehensively verify the universality and robustness of the system proposed in the application, multiple public benchmark datasets and real business data are selected for experiments in this embodiment. The datasets used cover common classification and clustering datasets in machine learning, and can examine the performance of the system in classification, prediction, recommendation, and representation learning tasks from different dimensions.

[0069] The experimental environment is Ubuntu 22.04, equipped with NVIDIA RTX 4090 GPU and 256GB memory, and model training is implemented using PyTorch. In terms of hyperparameter settings, the batch size is 64, the learning rate is set to 0.001, the optimizer is Adam, and the weight decay coefficient is 1e-5. In the attention mechanism, the number of attention heads is set to 8, and the embedding dimension is set to 128. The step size of the diffusion model is set to 6, and the noise level is dynamically sampled in the range of [0.0001, 0.2]. The early stopping strategy (patience=10) is used during training to avoid overfitting In the classification task, the UCI Adult and Kaggle Credit datasets are selected. The former contains more than 48,000 samples, with features including education, occupation, marriage, and other multi-dimensional information, and the target is a binary classification of income, used to verify the effectiveness of the system in processing medium-sized structured data. The latter contains more than 300,000 samples and is aimed at credit card fraud identification, with extremely unbalanced classes, suitable for testing the robustness of the system under sparse and unbalanced conditions.

[0070] In the clustering task, typical datasets from the UCI ML Repository are used, including the Iris and Wine datasets. The Iris dataset is a classic multi-dimensional flower dataset containing 150 samples, with the goal of unsupervised clustering and three class labels corresponding; the Wine dataset contains 178 samples and 13 chemical attribute features, commonly used for clustering and classification hybrid tasks in high-dimensional feature space, effectively testing the adaptability of the system in small-scale high-dimensional data.

[0071] Table 1 Dataset Information

[0072] In addition, to further enhance the comprehensiveness of the verification, 18 multi-class public comprehensive benchmark datasets are introduced (see Table 1 for dataset information). The sample size of these datasets ranges from several thousand to several ten thousand, the number of features ranges between 6 and 81, the number of classes covers binary classification, multi-classification and clustering tasks, and some datasets also contain missing values (such as house and uscensus), further enhancing the complexity of the verification conditions. In addition, the difference in output dimension also provides a test basis for the generalization performance of the system in a multi-task context.

[0073] In terms of experimental parameter settings, the batch size is set to 64, the learning rate is set to 0.001, the optimizer is Adam, and the early stopping strategy is used during training to prevent overfitting. For the attention mechanism module, the number of attention heads is set to 8 and the embedding dimension is set to 128. The step size in the diffusion process is set to 6, and the noise level is dynamically sampled within the range of [0.0001, 0.2] to enhance the model's robustness to missing data and noisy data.

[0074] In classification tasks, the goal is to maximize prediction accuracy and F1-score; in clustering and recommendation tasks, the focus is on the system's adaptability and generalization ability under different class distributions and high-dimensional spaces. Due to the extremely unbalanced class distribution of some datasets (such as Kaggle Credit), the experiment additionally uses a combination of under-sampling and weighted loss function during training to avoid the loss of minority class information.

[0075] Table 2 Experimental Results

[0076] The comparison methods include DEF, RS, DP-Fix, DP-Flex, DL, HAI-AI, SAGA, and the proposed CtxPipe system. The experimental evaluation indicators include test accuracy (Accuracy), macro-averaged precision (Precision), recall (Recall), F1-score, running time (Running Time), etc., and in addition, significance tests (t-test and sign test) are performed on the extended datasets to ensure the statistical reliability of the results.

[0077] Table 2 shows the average results of different methods on the DiffPrep dataset and the extended dataset. From the table, it can be seen that the CtxPipe proposed in the present application achieves the best performance on most indicators, with a test accuracy of 0.806, which is 2.8% and 3.3% higher than the suboptimal methods DP-Flex (0.784) and DP-Fix (0.780), respectively; and the highest values of precision, recall and F1-score, fully verifying the effectiveness and robustness of the method in the multi-task and multi-data environment. At the same time, CtxPipe maintains an acceptable level of running efficiency, which is much better than the method of DP-Flex with large computational overhead. The significance test result (p<0.05) further shows that the improvement of the method in the present application has statistical significance.

[0078] A computer readable storage medium, wherein a plurality of instructions are stored, the instructions being adapted to be loaded and executed by a processor of a terminal device, and the instructions are adapted to implement the context-aware semantic-aware data preparation pipeline recommendation method.

[0079] A terminal device, comprising a processor and a computer readable storage medium, the processor being configured to implement instructions, and the computer readable storage medium being configured to store a plurality of instructions, the instructions being adapted to be loaded and executed by the processor, and the instructions being adapted to implement the context-aware semantic-aware data preparation pipeline recommendation method.

[0080] The above are preferred embodiments of the present application, which do not limit the protection scope of the present application, and therefore: any equivalent changes made on the structure, shape and principle of the present application should be covered within the protection scope of the present application.

Claims

1. A context-aware data preparation pipeline recommendation method, characterized in that: include: Get the dataset for automated pipeline construction; Perform data preprocessing on the acquired automated pipeline construction dataset; For the preprocessed dataset, dual-view feature fusion based on the self-attention mechanism is performed; For the preprocessed dataset, extract contextual semantic information based on a large language model; Based on feature fusion and information extraction, pipeline recommendation results are obtained through deep reinforcement learning network; Evaluate and optimize pipeline recommendation results; Output pipeline.

2. The context-semantic-aware data preparation pipeline recommendation method according to claim 1, characterized in that: The data preprocessing of the obtained automated pipeline construction dataset includes identifying data fields closely related to the target label in multiple heterogeneous data sources according to the task definition, and performing category distribution statistics by explicitly identifying the label column after completing the initial screening of features and labels; synchronously capturing the contextual information of the tabular data during the extraction phase, and accurately labeling the data fields through the correspondence mechanism between column names and semantics; finally, introducing a consistency and standardization mechanism to uniformly map the label codes from different sources to a consistent category set by aligning the labels. , ensure the uniqueness of classification semantics, and perform unified numerical operations on features. Numerical fields are normalized or bucketed, and discrete fields are converted into dictionary indexes or embedded vectors to ensure that different features are comparable in the same space. For missing data, the integrity of the feature matrix is ​​ensured by filling or removing methods. After this process, the final classification task representation is obtained: ， in and are the standardized features and labels, Contextual meta information bound to the data.

3. The context-semantic-aware data preparation pipeline recommendation method according to claim 2, characterized in that: The dual-view feature fusion based on the self-attention mechanism includes the statistical view feature extraction of the data set and the pipeline history component feature extraction. For the statistical view feature extraction of the data set, firstly, a detailed statistical feature extraction is performed on the input data set. For the case where the distribution span of numerical features is large, a combination of fractional part and exponential part is used to express all numerical features. by The encoding is performed in the form of represents the fractional part of the feature, Represents its exponential part, and the two are To jointly characterize the original features; for the feature extraction of pipeline history components, a sequence modeling neural network is introduced as the core encoding module, which can process all executed data processing components in the historical order. , which is converted into a dense vector sequence through the embedding layer , where n is the number of components and d is the embedding dimension. The dense vector sequence E is then input into a multi-layer stacked Transformer layer to model the global dependencies between all components in the sequence in parallel, which can be expressed as: ， Then, weighted pooling is used to extract global features from the sequence to represent the historical context information of the current pipeline.

4. The context-semantic-aware data preparation pipeline recommendation method according to claim 3, characterized in that: The dual-view feature fusion based on the self-attention mechanism also includes introducing a feature fusion method based on the cross-view attention mechanism to dynamically capture the interactive relationship between the data distribution view and the pipeline structure view in an end-to-end trainable manner, wherein the statistical characteristics of the data set are assumed to be and pipeline history global features As input sources of different modalities, the statistical features of the data set are used as query vectors. As a key, the attention weights of statistical features under different pipeline history representations are automatically calculated, expressed as: ， in, , , It is a trainable reference, and the global representation of the pipeline components is obtained after aggregation by the attention mechanism. , expressed as .

5. The context-semantic-aware data preparation pipeline recommendation method according to claim 4, characterized in that: The contextual semantic information extraction based on the large language model includes introducing a contextual semantic information extraction mechanism based on the large language model, and deeply mining the multi-source context information of the data set in an automated manner. Specifically, by converting the original data table into a CSV format text with a header and data samples, the large language model is used to process ultra-long contexts and the token-level attention mechanism to capture the key features of field naming, value distribution patterns, and synonymous field classification, which are expressed as: ， in represents the context vector generated by the large language model, It represents the embedding of the large language model into the inference function. Then, based on the context-based decision-making method, it simulates the process of experts narrowing down the scope of operation selection through domain knowledge, and realizes dynamic cropping and intelligent filtering of the candidate component space, which is expressed as: ， Among them, when the candidate component In the given context The conditional probability under Greater than the preset threshold , the component is selected into the final component set By introducing contextual embedding features for each operation decision, we automatically focus on the component combination that is most likely to improve the data processing effect in the action space. At the same time, we propose an optimized target loss function. By considering the influence of recommendation accuracy and contextual information, we optimize the decision-making process in pipeline construction, which can be expressed as: ， in, is the recommended pipeline component, is the actual component used, is the regularization coefficient, controlling the context information role in the optimization process.

6. The context-semantic-aware data preparation pipeline recommendation method according to claim 5, characterized in that: Based on feature fusion and information extraction, the pipeline recommendation result is obtained through deep reinforcement learning network, including embedding all information in the pipeline construction process as state , in the decision-making process, through deep The network predicts all possible actions The Bellman equation is used to take into account the expectation of future multi-step cumulative rewards and achieve a dynamic balance between short-term and long-term benefits, which can be expressed as: ， in, Indicates that in a given state Take action , and according to the current parameters The strategy can obtain the expected value of long-term accumulation, Indicates that an action is being performed Then transfer to the next state , Represents the discount factor, and then based on the strategy Select components that maximize rewards ,Right now , by inputting the current state s, outputting the priority or probability of each possible action, The calculation of the value follows the Bellman equation, which evaluates the component selected at the current step by the formula The expected value of the future cumulative reward is expressed as: ， in Indicates the current step selection component The instant reward value obtained when is the discount factor of the reward, if A value close to 1 indicates that the agent will pay close attention to the benefits of each step in the future and take long-term effects into consideration. If the value is smaller than the set value, the agent will focus more on the immediate rewards obtained from the current action and pay less attention to future rewards.

7. The context-semantic-aware data preparation pipeline recommendation method according to claim 6, characterized in that: The pipeline recommendation results are obtained through a deep reinforcement learning network based on feature fusion and information extraction, and the deep reinforcement learning network is trained. First, a dataset is randomly extracted from a large-scale training dataset pool containing different fields and diverse data distributions. , get the current data , based on the current statistical characteristics of the data set and the existing pipeline historical information to form a state , select an action Automatically generate an adapted data preparation pipeline and evaluate the pipeline effect through the performance indicators of the downstream model. As a reward signal Feedback is given to the agent to optimize the decision-making strategy, which is expressed as , under the continuous training of cross-domain heterogeneous data, the samples are Minimize the loss, expressed as: ， And the learning rate Make an update , ensuring that the automated pipeline recommendation system spans diverse data environments, continuously outputs high-quality pipeline combinations, and The optimal pipeline is expressed as: ， in Indicates that in all candidate pipeline sets The pipeline configuration that maximizes the expected cumulative reward; It represents the mathematical expectation of a random process and measures the average performance.

8. The context-semantic-aware data preparation pipeline recommendation method according to claim 7, characterized in that: The evaluation and optimization of pipeline recommendation results include the following steps: , the optional label for the supervision task is Perform input reasoning; generate multidimensional labels of numerical, categorical, text, time series, and geographic information or identifiers with the help of field type identification and consistency verification mechanisms, select representative sample rows through stratified sampling methods, perform range sketching on data features, and extract statistical views of the data as a whole Finally, similar tasks are found in the task library based on the embedded retrieval mechanism. If the similarity exceeds the set threshold, the corresponding pipeline prefix is ​​extracted as the initial pipeline history. , and in and Fusion to form the initial state vector for reasoning : ， After the initial state vector is generated, in the component library Under the constraints of the candidate specification, based on the context semantic representation Calculate conditional prior probabilities , and then calculate the comprehensive score for each candidate component: ， Calling pre-trained strategy and its corresponding action-value function , the optimal component sequence is gradually selected and the pipeline is constructed under a mild exploration mechanism.

9. The context-semantic-aware data preparation pipeline recommendation method according to claim 8, characterized in that: The evaluation and optimization of the pipeline recommendation results also includes updating the current environment state in real time based on the state transfer mechanism of reinforcement learning after the candidate components are selected and executed. In each transfer, the newly generated intermediate result features are embedded into the semantic space, fused with the previous context representation, and a new context vector is formed through weighted aggregation , while the pipeline history track Updated to include a sequence description of the currently executed component, allowing subsequent candidate selection to be made with more complete context: ， Through the above recursive state transition, the optimal solution for the target task is gradually approached. During the pipeline construction process, a global evaluation mechanism is introduced to dynamically score candidate pipelines in multiple dimensions. When the iteration is completed and the final pipeline is generated, the pipeline is used as the output of the inference. Based on the feedback mechanism, the inference output is compared with the verification result after actual execution. If a prediction deviation is found, the deviation information is fed back to the policy network and value function network, so that the model parameters can be gradually corrected in future inferences. This is expressed as: ， in It represents the difference between actual performance and predicted performance, and is used to measure the gap between the prediction model and the actual results. represents the model parameters, Indicates that the learning rate is used to control the step size of each parameter update. It is used to measure the performance prediction error.

10. A context-aware data preparation pipeline recommendation system, characterized in that: include: The data acquisition module is configured to acquire a data set for automated pipeline construction; A preprocessing module is configured to perform data preprocessing on the acquired automated pipeline construction dataset; The feature fusion module is configured to perform dual-view feature fusion based on the self-attention mechanism on the preprocessed dataset; The information extraction module is configured to extract contextual semantic information based on the large language model for the preprocessed data set; The pipeline recommendation module is configured to obtain pipeline recommendation results through a deep reinforcement learning network based on feature fusion and information extraction; The optimization module is configured to evaluate and optimize the pipeline recommendation results; Output modules are configured as output pipelines.

Images