Data processing flow optimization method and system based on large language model

Abstract

The invention discloses a data processing flow optimization method and system based on a large language model, and relates to the field of data processing, and the method comprises the steps: obtaining multi-source data of different data sources; the multi-source data is input into a field adaptation module, an initial multi-source data set is obtained, and the field adaptation module is used for recognizing the field type of the multi-source data and correcting and expanding the multi-source data according to the field type; inputting the initial multi-source data set into a large language model to obtain a data processing rule; performing data processing on the initial multi-source data set by adopting a data processing rule to obtain a processed multi-source data set; the processed multi-source data set is input into a field adaptation module, a final multi-source data set is obtained, and the field adaptation module is used for adjusting the processed multi-source data set according to the field type. The data processing efficiency and quality can be effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular, to a method and system for optimizing a data processing flow based on a large language model. Background Art

[0002] In today's digital age, data has become a core asset for enterprises and organizations. With the rapid development of technologies such as the Internet of Things, social media, and e-commerce, the amount of global data has exploded. Not only has the amount of data increased dramatically, but its types and complexity are also increasing. Structured data and unstructured data coexist, covering a variety of forms such as text, images, audio, and video. At the same time, the real-time requirements for data are also increasing, and many application scenarios require near-real-time data processing and analysis capabilities. In this context, enterprises and organizations are faced with the huge challenge of how to effectively manage, process, and utilize massive amounts of complex data.

[0003] Traditional data processing methods usually include the following steps: data collection, data cleaning, data conversion, data integration, and data loading. In the data collection stage, traditional methods mainly rely on manual operations or simple automated scripts to obtain data from various sources. In the data cleaning process, predefined rules are usually used to identify and process outliers, missing values, and duplicate data. The data conversion step involves converting data from one format or structure to another to meet specific analysis or storage requirements. Data integration is to merge data from different sources into a unified view. Finally, the data loading stage imports the processed data into the target system or data warehouse.

[0004] These traditional methods perform well when processing small-scale, structured data, but they expose many limitations and shortcomings when faced with today's complex data environment. First, traditional methods are inefficient when processing large-scale data. With the sharp increase in data volume, traditional data processing methods and processes are difficult to complete data processing tasks within an acceptable time. Second, traditional data processing methods usually rely on predefined rules and patterns, which are difficult to cope with rapid changes in data formats, structures, and semantics. With the diversification of data sources and the increase in data volume, it is difficult to fully and accurately identify and handle data quality issues relying solely on preset rules, making it difficult for traditional methods to effectively process data from different sources with different formats and semantics.

[0005] In summary, traditional data processing methods face huge challenges in terms of efficiency and quality. Summary of the invention

[0006] The embodiments of the present application provide a method and system for optimizing data processing flow based on a large language model, which are used to effectively improve the efficiency and quality of data processing.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a method for optimizing a data processing flow based on a large language model is provided, which is applied to an electronic device, wherein the electronic device is deployed with a large language model and a domain adaptation module, and the method comprises:

[0009] Obtain multi-source data from different data sources;

[0010] Inputting the multi-source data into the domain adaptation module to obtain an initial multi-source data set, wherein the domain adaptation module is used to identify the domain type of the multi-source data, and to modify and expand the multi-source data according to the domain type;

[0011] Inputting an initial multi-source data set into the large language model to obtain data processing rules, wherein the data processing rules include any n types of data cleaning strategies, feature selection strategies, data conversion strategies, anomaly detection strategies, and feature interaction strategies, 0<n≤4, and n is an integer, the large language model includes an input embedding layer, a multi-layer Transformer block, a parallel task head, and an output layer, and the data processing rules are obtained after the input embedding layer, the multi-layer Transformer block, the parallel task head, and the output layer sequentially process the initial multi-source data set;

[0012] Performing data processing on the initial multi-source data set using the data processing rule to obtain a processed multi-source data set;

[0013] The processed multi-source dataset is input into the domain adaptation module to obtain a final multi-source dataset, wherein the domain adaptation module is used to adjust the processed multi-source dataset according to the domain type.

[0014] In an implementation of the first aspect, the multi-source data includes structured data and unstructured data, the domain adaptation module includes multiple domain databases, a multi-source classification and recognition module, a knowledge graph module, and an expansion module, each of the domain databases stores data rules for each domain, and the multi-source data is input into the domain adaptation module to obtain an initial multi-source data set, including:

[0015] Inputting the structured data and the unstructured data into the multi-source classification and identification module to obtain field types corresponding to both the structured data and the unstructured data;

[0016] Inputting the domain type into the knowledge graph module, so that the knowledge graph module determines a target domain database from a plurality of domain databases according to the domain type, and determines a target knowledge graph according to the target domain database;

[0017] The target knowledge graph is input into the expansion module, so that the expansion module expands and modifies the multi-source data according to the target knowledge graph to obtain an initial multi-source data set.

[0018] In another implementation of the first aspect, inputting the initial multi-source data set into the large language model to obtain the data processing rules includes:

[0019] Inputting the initial multi-source data set into the input embedding layer to obtain a high-dimensional vector group of the initial multi-source data set;

[0020] Inputting the high-dimensional vector group into the multi-layer Transformer block to obtain a first output, wherein the first output includes data anomaly patterns, key features, and potential associations between different data sources in the high-dimensional vector group;

[0021] Inputting the first output into the parallel task head to obtain five types of data processing rules, wherein the parallel task head includes five task heads, each of which is used to generate one type of data processing rule, wherein the five types of data processing rules include the data cleaning strategy, the feature selection strategy, the data conversion strategy, the anomaly detection strategy, and the feature interaction strategy;

[0022] The five types of data processing rules are input into the output layer to obtain n types of data processing rules, and the output layer is used to determine the n types of data processing rules from the five types of data processing rules.

[0023] In another implementation of the first aspect, the parallel task heads include a data cleaning task head, a feature selection task head, a data conversion task head, an anomaly detection task head, and a feature interaction task head;

[0024] The structure of the data cleaning task head is a multi-layer perceptron structure. The data cleaning task head is used to generate multiple binary decisions according to the key features of the first output. Each binary decision corresponds to one key feature, and each binary decision is used to represent the cleaning probability of the corresponding key feature. The structure of the feature selection task head is based on the attention mechanism structure. The feature selection task head is used to assign corresponding importance weights to each key feature of the first output. The structure of the data conversion task head is a neural network structure. The data conversion task head is used to generate a [p×K] conversion matrix for each key feature of the first output, where p represents the dimension of the key feature and K represents the conversion matrix. methods, each element in the transformation matrix represents the applicable probability of the corresponding transformation method; the structure of the anomaly detection task head is an autoencoder structure, and the anomaly detection task head is used to learn the data normal mode according to the data anomaly mode in the high-dimensional vector group, and learn the correlation between the key features according to the potential correlation between the different data sources, and generate a threshold vector for identifying anomalies according to the data normal mode and the correlation; the structure of the feature interaction task head is a graph neural network structure, and the feature interaction task head is used to learn the interaction relationship between the key features according to the potential correlation between the different data sources and the key features, and output an adjacency matrix representing the interaction strength between feature pairs.

[0025] In another implementation of the first aspect, the output layer includes a scoring layer, a Transformer layer, a selection layer, and an optimization output layer, and the five types of data processing rules are input into the output layer to obtain n types of data processing rules, including:

[0026] Inputting the five types of data processing rules into the scoring layer to obtain a quality score for each of the data processing rules;

[0027] Inputting the five types of data processing rules into the Transformer layer to obtain the synergy values ​​between the data processing rules;

[0028] Input the quality score and the synergy value into the selection layer, calculate the attention weight of each of the data processing rules, and use the attention weight greater than a preset threshold as the target attention weight, wherein the number of the target attention weights is n;

[0029] Determine a data processing rule corresponding to each of the target attention weights to obtain n types of data processing rules;

[0030] The n types of data processing rules are input into the optimization output layer to obtain the data processing order of the n types of data processing rules, and the n types of data processing rules and the data processing order are output.

[0031] In another implementation of the first aspect, the using the data processing rule to process the initial multi-source data set to obtain a processed multi-source data set includes:

[0032] The data processing sequence and the n types of data processing rules are used to process the initial multi-source data set to obtain a processed multi-source data set.

[0033] In another implementation of the first aspect, before inputting the initial multi-source data set into the large language model to obtain the data processing rules, the method includes:

[0034] In the case where the domain type is a sensitive domain type, a desensitizing encryption strategy is used to perform desensitizing encryption processing on the initial multi-source data set to obtain the processed initial multi-source data set.

[0035] In another implementation of the first aspect, the desensitizing encryption strategy includes:

[0036] identifying a target sensitive data segment in the structured data of the initial multi-source data set;

[0037] Generate a corresponding token for the target sensitive data segment, wherein the token is unique;

[0038] A mapping relationship between the token and the target sensitive data segment is created in a preset mapping database, and the token is used to replace the target sensitive data segment.

[0039] In another implementation of the first aspect, inputting the processed multi-source dataset into the domain adaptation module to obtain a final multi-source dataset includes:

[0040] The processed multi-source dataset is input into the domain adaptation module, so that the domain adaptation module corrects the processed multi-source dataset according to the domain type and the target knowledge graph to obtain the final multi-source dataset.

[0041] In a second aspect, the present application provides a data processing flow optimization system, comprising:

[0042] a memory configured to store instructions; and

[0043] The processor is configured to call the instructions from the memory and implement the above-mentioned data processing flow optimization method based on the large language model when executing the instructions.

[0044] Through the above technical scheme, firstly, by utilizing the computing power and parallel processing capabilities of the large language model, the processing task of large-scale data can be completed in a relatively short time, overcoming the problem of low efficiency of traditional methods in processing massive data. Secondly, for traditional methods that often rely on predefined rules and patterns and are difficult to cope with the rapid changes in data format, structure and semantics, the large language model in this technical scheme can dynamically generate processing rules that adapt to different data features, including data cleaning strategies, feature selection strategies, data conversion strategies, anomaly detection strategies, and feature interaction strategies, etc., so that it can flexibly respond to various complex and changeable data situations, no longer limited to fixed preset rules. In addition, by introducing the domain adaptation module, the accuracy and pertinence of data processing are further enhanced. The domain adaptation module can identify the domain type of multi-source data, and correct and expand the data according to the domain characteristics, which not only improves the quality of the data, but also optimizes the data according to the needs of specific fields. After the data processing is completed, the domain adaptation module can also adjust the processing results again according to the domain type to ensure that the final output data set meets both the general data quality standards and the professional needs of specific fields. In addition, through the semantic understanding ability of the large language model, it is possible to effectively process data from different sources with different formats and semantics, realize efficient integration and unified processing of data, greatly expand the scope and depth of data processing, and enable enterprises and organizations to more comprehensively utilize various data resources. The large language model architecture, including the input embedding layer, multi-layer Transformer blocks, parallel task heads, and output layers, not only ensures the efficiency of the processing process, but also realizes multi-task parallel processing, further improving the efficiency and quality of data processing. This architectural design enables the model to consider multiple aspects of the data at the same time, such as semantic understanding, structural analysis, and anomaly detection, thereby generating more comprehensive and accurate data processing rules. In general, it can not only efficiently process large-scale data, but also flexibly respond to the diversity and variability of data, while achieving more accurate data processing through domain adaptation.

[0045] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a method for optimizing a data processing flow based on a large language model provided in an embodiment of the present application;

[0047] Figure 2 A structural framework diagram of a large language model provided in an embodiment of the present application;

[0048] Figure 3A flowchart of an embodiment of the present application for inputting multi-source data into a domain adaptation module to obtain an initial multi-source data set is provided. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0050] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0051] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0052] Figure 1 The following schematically shows a flow chart of a method for optimizing a data processing flow based on a large language model according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a data processing flow optimization method based on a large language model, which is applied to an electronic device, and the electronic device is deployed with a large language model and a domain adaptation module. The method may include the following steps.

[0053] S110, acquiring multi-source data, where the multi-source data includes structured data and unstructured data;

[0054] S120, inputting the multi-source data into a domain adaptation module to obtain an initial multi-source data set, wherein the domain adaptation module is used to identify the domain type of the multi-source data, and to correct and expand the multi-source data according to the domain type;

[0055] S130, inputting the initial multi-source data set into the large language model to obtain data processing rules, wherein the data processing rules include any n types of data cleaning strategies, feature engineering strategies, data conversion strategies, and integration strategies, 0<n≤4, and n is an integer, the large language model includes an input embedding layer, a multi-layer Transformer block, a parallel task head, and an output layer, and the data processing rules are obtained after the input embedding layer, the multi-layer Transformer block, the parallel task head, and the output layer process the initial multi-source data set in sequence;

[0056] S140, performing data processing on the initial multi-source data set using a data processing rule to obtain a processed multi-source data set;

[0057] S150 , inputting the processed multi-source dataset into a domain adaptation module to obtain a final multi-source dataset, wherein the domain adaptation module is used to adjust the processed multi-source dataset according to the domain type.

[0058] Multi-source data includes structured data and unstructured data. Structured data is usually stored in tabular form with predefined formats and fields, such as tabular data in a database, CSV files, or Excel tables. Structured data is easy to organize, query, and analyze. Unstructured data has no predefined data model or organization method, such as text documents, images, audio or video files. The process of obtaining multi-source data involves multiple steps: First, determine the data source, including internal databases, external APIs, files, web crawlers, etc. Second, establish connections with these data sources, using different protocols and authentication methods. Then, perform data extraction operations, involving SQL queries, API calls, file reading, etc. Finally, temporarily store the acquired data in memory or temporary storage space for subsequent processing.

[0059] The multi-source data is input into the domain adaptation module to obtain the initial multi-source data set. The domain adaptation module is used to identify the data domain type and modify and expand the data according to the domain characteristics. The module first performs domain identification on the input multi-source data. Pre-trained classification models, such as support vector machines (SVM) or deep learning models, can be used to analyze the characteristics of the data and classify them into predefined domain types. Domain types include finance, medical, education, engineering, etc. During the identification process, the module can consider data format, keywords, numerical range and other characteristics. After determining the domain type, the domain adaptation module can modify the data according to the specific knowledge and rules of the domain. This includes correcting common domain-specific errors, standardizing terms, adjusting numerical units, etc. For example, in the medical field, the names of drugs with different expressions are unified into standard names. The expansion process involves enriching the original data with domain-specific knowledge bases or external data sources, including adding relevant metadata, supplementing missing information, or inferring additional features based on domain knowledge. For example, in financial data, industry classification information can be added based on company codes. The output of the domain adaptation module is an initial multi-source dataset that has undergone domain-specific processing, which not only retains the core information of the original data but also is optimized and expanded according to the domain characteristics.

[0060] Afterwards, the initial multi-source dataset is input into the large language model to generate data processing rules. Figure 2 As shown in Figure 1, the structure of the large language model includes an input embedding layer, a multi-layer Transformer block, a parallel task head, and an output layer. First, the initial multi-source dataset is converted into a high-dimensional vector representation through the input embedding layer. This process involves techniques such as word embedding and positional encoding to convert the original data into a format that the model can process. Then, the high-dimensional vector is processed by a multi-layer Transformer block. The Transformer block uses a self-attention mechanism and a feedforward neural network to capture long-distance dependencies and complex patterns in the data. In this process, abnormal patterns, key features, and potential associations between different data sources can be identified in the data. Next, the parallel task head is responsible for generating specific data processing rules. The task head includes specialized sub-modules such as data cleaning, feature selection, and data transformation, each of which is optimized for a specific type of data processing task. For example, the data cleaning task head can generate strategies for removing outliers and handling missing data; the feature selection task head can propose methods for creating new features or selecting the most relevant features; and the data transformation task head can suggest appropriate data standardization or encoding methods. Finally, these generated rules are integrated and optimized through the output layer to produce the final n data processing rules (where n is an integer between 1 and 4). The output layer can consider the compatibility and overall effect between different rules to ensure that the generated rule set is coherent and efficient.

[0061] The generated data processing rules are used to process the initial multi-source data set, and finally the processed multi-source data set is obtained. The data processing rules generated by the large language model are presented in a structured form, such as JSON or XML format, which contains specific operation instructions and parameters. Various data processing operations can be performed in a specific order according to the data processing rules. Data cleaning is the first step, including processing missing values, removing outliers, correcting data formats, etc. For example, statistical methods (such as mean, median filling) or machine learning methods (such as K nearest neighbors) can be used to process missing data; outliers can be identified and processed using Z-score or IQR methods. Next, feature engineering strategies can be performed, which involves creating new features, selecting the most relevant features, or transforming existing features. For example, interaction terms can be created by combining existing features, dimensionality reduction can be performed using principal component analysis (PCA), or feature selection algorithms such as LASSO can be applied to select the most important features. Data transformation strategies include standardization (such as Z-score standardization), normalization (such as Min-Max scaling), or encoding categorical variables (such as One-Hot encoding or label encoding). If an integration strategy is included, information from different data sources can be merged or the prediction results of multiple models can be fused. For example, the outputs of multiple models can be combined using voting or averaging. Finally, the processed multi-source data sets can be temporarily stored, using distributed storage to handle large-scale data. Ultimately, the quality and availability of the data can be significantly improved.

[0062] Finally, the processed multi-source dataset is input into the domain adaptation module again to obtain the final multi-source dataset to further optimize and adjust the data to ensure that it fully meets the needs and standards of the specific domain. First, the domain adaptation module identifies the domain type of the data again. Although the domain identification has been performed in the initial processing stage, the characteristics of the data have changed after the previous processing steps, so it is necessary to confirm the domain type again. Because the data has been preliminarily cleaned and converted, this identification will be more accurate. Next, the domain adaptation module deeply adjusts the data according to the confirmed domain type. This process involves multiple aspects: First, domain-specific data standardization rules can be applied. For example, in the financial field, it is necessary to ensure that all monetary data are converted to the same currency unit and adjusted according to specific accounting standards. In the medical field, it is necessary to ensure that all diagnostic codes comply with the latest International Classification of Diseases. Second, the module can introduce additional domain-specific features. This includes derived variables calculated based on domain knowledge, or information supplemented from domain-specific external data sources. For example, in retail data analysis, external factors such as holiday tags or economic indicators can be added. Third, the module can perform semantic enhancement on the data. This involves enriching the semantic content of the data with domain-specific ontologies or knowledge graphs, making the data more consistent with domain requirements not only in structure but also in semantics. For example, in scientific research data processing, professional terms can be linked to standard academic concept libraries. In addition, the domain adaptation module can also perform some domain-specific data validation and consistency checks. This includes checking whether the logical relationship between certain variables conforms to domain rules, or verifying whether certain key indicators are within a reasonable range. For example, in demographic data, the consistency between age and education level can be checked. Finally, the module can make final format adjustments to the data to ensure that the output data set conforms to the commonly used data format and structure in the field.

[0063] This embodiment firstly utilizes the computing power and parallel processing capability of the large language model to complete the processing task of large-scale data in a relatively short time, thus overcoming the problem of low efficiency of traditional methods in processing massive data. Secondly, for traditional methods that often rely on predefined rules and patterns and are difficult to cope with the rapid changes in data format, structure and semantics, the large language model in this technical solution can dynamically generate processing rules that adapt to different data features, including data cleaning strategies, feature selection strategies, data conversion strategies, anomaly detection strategies, and feature interaction strategies, etc., so that it can flexibly cope with various complex and changeable data situations, and is no longer limited to fixed preset rules. In addition, by introducing a domain adaptation module, the accuracy and pertinence of data processing are further enhanced. The domain adaptation module can identify the domain type of multi-source data, and correct and expand the data according to the domain characteristics, which not only improves the quality of the data, but also optimizes the data according to the needs of a specific field. After the data processing is completed, the domain adaptation module can also adjust the processing results again according to the domain type to ensure that the final output data set meets both the general data quality standards and the professional needs of a specific field. In addition, through the semantic understanding ability of the large language model, it is possible to effectively process data from different sources with different formats and semantics, realize efficient integration and unified processing of data, greatly expand the scope and depth of data processing, and enable enterprises and organizations to more comprehensively utilize various data resources. The large language model architecture, including the input embedding layer, multi-layer Transformer blocks, parallel task heads, and output layers, not only ensures the efficiency of the processing process, but also realizes multi-task parallel processing, further improving the efficiency and quality of data processing. This architectural design enables the model to consider multiple aspects of the data at the same time, such as semantic understanding, structural analysis, and anomaly detection, thereby generating more comprehensive and accurate data processing rules. In general, it can not only efficiently process large-scale data, but also flexibly respond to the diversity and variability of data, while achieving more accurate data processing through domain adaptation.

[0064] In one implementation of this embodiment, the multi-source data includes structured data and unstructured data, the domain adaptation module includes multiple domain databases, a multi-source classification and recognition module, a knowledge graph module and an expansion module, each domain database stores data rules for each domain, and the multi-source data is input into the domain adaptation module to obtain an initial multi-source data set, including the following steps:

[0065] S210, inputting the structured data and the unstructured data into a multi-source classification and recognition module to obtain field types corresponding to both the structured data and the unstructured data;

[0066] S220, inputting the domain type into the knowledge graph module, so that the knowledge graph module determines a target domain database from multiple domain databases according to the domain type, and determines a target knowledge graph according to the target domain database;

[0067] S230. Input the target knowledge graph into the expansion module so that the expansion module expands and corrects the multi-source data according to the target knowledge graph to obtain an initial multi-source data set.

[0068] Figure 3 A schematic diagram of a process of inputting multi-source data into a domain adaptation module to obtain an initial multi-source data set is shown in an embodiment of the present application. In this embodiment, structured data and unstructured data are first input into a multi-source classification and recognition module to obtain a domain type corresponding to both types of data. The multi-source classification and recognition module adopts an integrated learning method, combining multiple classifiers to improve the accuracy and robustness of recognition. For structured data, a feature-based classification method such as random forest, support vector machine (SVM) or gradient boosting decision tree (GBDT) is used. First, key features are extracted from structured data, such as statistical features (mean, variance, quantile, etc.) of numerical fields, distribution features of category fields, trend features of time series data, etc. Then, these features are input into a pre-trained classification model for domain recognition.

[0069] For unstructured data, especially text data, deep learning methods can be used to automatically learn the semantic features of text, such as convolutional neural networks (CNN). For example, for a document, the text is first converted into a vector sequence using a pre-trained word embedding model, and then feature extraction and classification are performed through a deep neural network. For other types of unstructured data such as images or audio, specialized deep learning models are used, such as ResNet for images or LSTM networks for audio.

[0070] The multi-source classification recognition module can process and integrate information from different data sources. For example, for a data set containing customer information, structured data includes the customer's age, income, purchase history, etc., while unstructured data includes customer comments or customer chat records. The multi-source classification recognition module can process these two types of data separately, and then use an integration method (such as voting or weighted average) to comprehensively obtain the final domain classification result.

[0071] In practice, this step is implemented as follows: Assume that there is a dataset containing customer data, and the goal is to determine which business field (such as retail, finance, healthcare, etc.) this data belongs to. First, a random forest classifier is used on structured data (such as customer demographic information, transaction records). For example, if the data contains a large number of financial transaction records and credit scores, the random forest will tend to classify it as a financial field. At the same time, a BERT-based text classification model is used for unstructured data (such as customer reviews). If words such as "product", "delivery", and "return" appear frequently in the comments, the BERT model will identify it as a retail field. Finally, the module combines the results of the two classifiers and uses a soft voting method, that is, comprehensively considering the confidence of the two classifiers to obtain the final field classification result.

[0072] The domain type is input into the knowledge graph module to determine the target domain database and the corresponding target knowledge graph. The knowledge graph technology can be used to select and construct knowledge structures related to specific domains. Knowledge graph is a structured knowledge representation method that stores the relationship between entities in the form of a graph and can effectively capture domain-specific concepts, attributes, and relationships.

[0073] First, the knowledge graph module can match and select from multiple domain databases based on the input domain type. Each domain database contains data rules, concept hierarchies, entity relationships, and other information specific to that domain. For example, if the domain type is identified as "finance", the knowledge graph module will select a database in the financial domain, which contains relevant concepts and rules such as banking, stocks, investment, and risk assessment.

[0074] Once the target domain database is determined, the knowledge graph module begins to select the corresponding target knowledge graph. This process involves the following steps: 1. Entity extraction: Extract key entities from the target domain database. For example, in the financial field, entities such as "bank", "stock", and "investor" are extracted. 2. Relationship extraction: Identify the relationship between entities. For example, there are relationships such as "buy" and "hold" between "investor" and "stock". 3. Attribute extraction: Define relevant attributes for each entity. For example, the "stock" entity has attributes such as "price", "volume", and "market value". 4. Ontology construction: Establish a hierarchy of domain concepts. For example, "stock" is a subclass of "financial assets". 5. Rule integration: Integrate domain-specific rules and constraints into the knowledge graph. For example, some financial products have specific trading rules or risk assessment criteria.

[0075] In practical applications, this step is implemented as follows: Assume that the field type determined in the previous step is "medical". The knowledge graph module will first select the database in the medical field. Then, the knowledge graph module starts to select the knowledge graph in the medical field. In this process, the knowledge graph module extracts entities such as "disease", "symptoms", "treatment methods", "drugs", and establishes relationships between them, such as "disease-cause-symptoms", "drug-treatment-disease", etc. At the same time, the knowledge graph module can also define the attributes of these entities, such as "drugs" have attributes such as "dosage", "side effects", and "contraindications". In addition, a conceptual hierarchy can be established, such as "antibiotics" is a subclass of "drugs". Finally, integrate rules specific to the medical field, such as restrictions on the use of certain drugs or diagnostic criteria for specific diseases.

[0076] The target knowledge graph is input into the expansion module to expand and correct the multi-source data according to the target knowledge graph, and finally the initial multi-source data set is obtained. The core of this step is to use the domain-specific knowledge graph to enhance and optimize the original data to make it more complete, accurate and semantic.

[0077] The workflow of the augmentation module usually includes the following main steps: 1. Entity linking: First, the augmentation module tries to match elements in the original data with entities in the knowledge graph. The matching process uses a variety of techniques, such as string matching, semantic similarity calculation, or entity disambiguation. For example, when processing medical data, the disease names in the patient records are linked to the disease entities in the knowledge graph. 2. Relationship inference: Based on the linked entities, the augmentation module tries to infer the relationships between the data elements. These relationships are not explicitly represented in the original data, but can be inferred from the relationship structure in the knowledge graph. For example, if a symptom is mentioned in the patient data, the system will infer the potential disease associated with the symptom through the knowledge graph. 3. Attribute supplementation: The augmentation module uses information from the knowledge graph to supplement the missing or incomplete attributes in the original data. This can be achieved by adding standardized attribute values, supplementing missing fields, or adding additional relevant information. For example, when processing drug data, information such as the mechanism of action and common side effects of the drug will be supplemented from the knowledge graph. 4. Data standardization: Using the standard terminology and classification system in the knowledge graph, the expansion module can standardize the expressions in the original data. This is particularly useful for processing data from different sources, and can eliminate differences in terminology and expressions. For example, unify disease names with different expressions into standard medical terms. 5. Consistency check: The expansion module uses the rules and constraints in the knowledge graph to check the consistency of the data and correct errors. This includes correcting obvious data entry errors or marking outliers that require further verification. 6. Semantic enhancement: By adding contextual information from the knowledge graph, the expansion module can enhance the semantic content of the data. This includes adding classification labels, associated concepts, or explanatory information.

[0078] In practice, this step is implemented as follows: Assume that we are dealing with a medical dataset containing patient visit records. The original data contains basic information such as patient ID, symptom description, and diagnosis results. The expansion module first links symptoms and diagnoses to entities in the medical knowledge graph. For example, the symptom of "headache" is linked to the corresponding node in the knowledge graph. Then, based on the relationships in the knowledge graph, other related symptoms or diseases are inferred and added to the dataset. Next, detailed descriptions of each diagnosis result, common treatment methods, and other information are supplemented from the knowledge graph. In the standardization process, the same symptoms expressed differently (such as "headache" and "headache") are unified into standard terms. In the consistency check stage, some abnormal combinations can be found and marked, such as inconsistencies between certain symptoms and diagnosis results. Finally, relevant medical classification codes are added to each diagnosis to enhance the semantic content of the data.

[0079] Finally, a significantly enhanced initial multi-source dataset was generated. The initial multi-source dataset not only retains the core information of the original data, but also incorporates rich domain knowledge and semantic content. For example, the original missing, inconsistent or incorrect data has been corrected, the potential relationship between the data has been revealed, and additional relevant information has been supplemented.

[0080] This implementation method can not only accurately identify the field to which the data belongs, but also use the field-specific knowledge structure to enhance and optimize the data through the three key steps of multi-source classification and identification, knowledge graph construction, and data expansion and correction, which significantly improves the quality, integrity, and semantic richness of the data and provides a solid foundation for subsequent data analysis and decision support. By integrating machine learning, knowledge graphs, and data processing technologies, it can intelligently process complex data, extract implicit relationships, and ultimately generate a highly structured, information-rich data set, laying the foundation for various advanced analysis tasks and intelligent applications.

[0081] In one implementation of this embodiment, the initial multi-source data set includes text data, numerical data, categorical data and / or time series data. The initial multi-source data set is input into a large language model to obtain a data processing rule, including the following steps:

[0082] S310, inputting the initial multi-source data set into the input embedding layer to obtain a high-dimensional vector group of the initial multi-source data set;

[0083] S320, inputting the high-dimensional vector group into a multi-layer Transformer block to obtain a first output, where the first output includes data anomaly patterns, key features, and potential correlations between different data sources in the high-dimensional vector group;

[0084] S330, inputting the first output into the parallel task head to obtain five types of data processing rules, wherein the parallel task head includes five task heads, each of which is used to generate one type of data processing rule, wherein the five types of data processing rules include a data cleaning strategy, a feature selection strategy, a data conversion strategy, an anomaly detection strategy, and a feature interaction strategy;

[0085] S340, inputting the five types of data processing rules into the output layer to obtain data processing rules, the output layer being used to determine n types of data processing rules from the five types of data processing rules.

[0086] The processing of the initial multi-source dataset begins by inputting different types of data into the input embedding layer. The core goal of this step is to convert various heterogeneous data into a unified high-dimensional vector representation, laying the foundation for subsequent deep learning processing. Specifically, for text data, word embedding technology is used. For example, using a 300-dimensional word vector, the sentence "The catsatonthemat" will be converted into a [300×6] matrix, where each row represents a 300-dimensional vector of a word. For numerical data, it is first standardized or normalized, and then mapped to a high-dimensional space using a fully connected layer. For example, after standardizing features such as age and income, the original low-dimensional features are mapped to a 100-dimensional vector space through a linear transformation of the form y=Wx+b. Categorical data usually uses one-hot encoding or entity embedding technology. For example, the color features "red, green, blue" are encoded as [1,0,0], [0,1,0], [0,0,1], and then converted to a higher-dimensional vector through an embedding layer. Time series data uses positional encoding or special time embedding techniques to capture the temporal nature of the data. For example, for a stock price series, a combination of sine and cosine functions can be used to encode the temporal information, and then these encodings can be combined with the original data.

[0087] After the high-dimensional vector group is processed by the input embedding layer, it is then input into the multi-layer Transformer block. The core of the Transformer block is its self-attention mechanism, which can effectively capture the long-distance dependencies between data. At this stage, each Transformer layer first performs multi-head self-attention calculations on the input vectors. Specifically, for the input sequence X, the query (Q), key (K), and value (V) matrices are obtained through linear transformation, and then the attention weights are calculated:

[0088]

[0089] Where, d k is the dimension of the key, which is used to scale the dot product to prevent it from being too large. The multi-head attention mechanism allows the model to learn information from different representation subspaces simultaneously, further enhancing the expressiveness of the model.

[0090] For example, consider a stock price sequence containing 10 time steps, each of which has a feature vector dimension of 64. The multi-head attention mechanism (assuming 8 heads) can transform the [10×64] input matrix into 8 [10×8] attention matrices, each representing the aggregation of information from a different angle. These attention outputs are then concatenated and passed through a linear layer to obtain a [10×64] output, maintaining the original dimension.

[0091] After the attention calculation, each Transformer layer also contains a feed-forward neural network, which usually consists of two linear transformations with a ReLU activation function in between:

[0092] FFN(x)=max(0,xW1+b1)W2+b2;

[0093] In this formula, x is the output of the attention mechanism, the first linear transformation is xW1+b1, W1 is the first weight matrix, b1 is the first bias vector, which is used to map the input x to a higher-dimensional space. max(0,...) is the ReLU activation function, which sets all negative values ​​to 0 and keeps positive values ​​unchanged to introduce nonlinearity, allowing the network to learn more complex functions. The second linear transformation is (...)W2+b2, W2 is the second weight matrix, b2 is the second bias vector, which is used to map the data back to the original dimension. The whole process is: first, the input is mapped to a higher-dimensional space, then, nonlinearity is introduced through the ReLU activation function, and finally, the result is mapped back to the original dimension. The nonlinear expression ability of the model is further enhanced. The stacking of multiple layers of Transformer enables the model to abstract layer by layer, from low-level features to high-level semantic representations.

[0094] Through this process, the Transformer block can identify abnormal patterns in the data, such as sudden jumps in time series or uncommon word combinations in text data. At the same time, it can also extract key features of the data, such as identifying certain specific price patterns or changes in trading volume as important indicators in financial data analysis. In addition, since different types of data are mapped to the same vector space, the Transformer block can also discover potential correlations between different data sources, such as the correlation between text sentiment and stock price changes.

[0095] The output of this stage, namely the first output, contains rich information, which not only reflects the core characteristics of the original data, but also includes the complex relationships between the data and potential abnormal patterns.

[0096] After the first output is processed by the multi-layer Transformer block, it is then input into the parallel task head for further specialized processing. The design concept of the parallel task head is to generate multiple types of data processing rules at the same time through multi-task learning. This stage contains m independent task heads, each of which is responsible for generating a specific type of data processing rule, where m is greater than the number of rules n required in the end.

[0097] Each task head is equipped with a dedicated neural network structure, including a fully connected layer, a convolutional layer, or a recurrent neural network layer, to meet the needs of generating different types of rules. For example, assuming m = 5, these five task heads correspond to the following data processing tasks:

[0098] 1. Data cleaning rule generation: This task head uses a multi-layer perceptron (MLP) structure. The input is the output of the Transformer, and the output is a series of binary decisions, indicating whether certain features need to be cleaned. For example, for a data set containing 100 features, this task head outputs a 100-dimensional vector, where each element has a value between 0 and 1, indicating the probability that the corresponding feature needs to be cleaned.

[0099] 2. Feature selection rule generation: This task head uses an attention-based structure to assign importance weights to each feature. For example, for the same 100 features, the output is a 100-dimensional weight vector, where the larger the value, the more important the feature.

[0100] 3. Data transformation rule generation: This task head uses a complex neural network structure to generate a transformation plan for each feature. The output is a [100xK] matrix, where K represents the number of transformation methods (such as logarithmic transformation, square root transformation, etc.), and each element in the matrix represents the applicable probability of the corresponding transformation method.

[0101] 4. Anomaly detection rule generation: This task head uses an autoencoder structure to learn the normal patterns of the data and output a threshold vector for identifying anomalies.

[0102] 5. Feature interaction rule generation: This task head uses a graph neural network-based structure to learn the interaction relationship between features and output an adjacency matrix representing the interaction strength between feature pairs.

[0103] This parallel processing approach not only improves computational efficiency, but also allows the model to share underlying representations between different data processing tasks, resulting in a more comprehensive and coordinated data processing strategy. For example, data cleaning rules and feature selection rules can affect each other because some features that need to be cleaned happen to be less important features.

[0104] The output of this stage is m different types of data processing rules, each of which targets a specific aspect of the data processing process. These rules provide a wealth of candidate solutions for subsequent selection and integration, increasing the flexibility and adaptability of the final data processing strategy.

[0105] Finally, the m types of data processing rules are input to the output layer for integration and optimization, and finally the n most suitable data processing rules are selected. The design of the output layer aims to integrate multiple specialized data processing rules into a coherent and comprehensive data processing strategy. This process usually includes the following key links: 1. Rule evaluation: First, the output layer evaluates each data processing rule. This involves the use of a scoring network that receives the description of each rule (a vector representation) as input and outputs a score representing the quality or applicability of the rule. For example, for m = 5 rules, the scoring network outputs a 5-dimensional vector, each element ranging from 0 to 1, representing the score of the corresponding rule. 2. Modeling the relationship between rules: Next, a Transformer structure is used to simulate the mutual influence between different rules. To consider the synergy or conflict between rules. For example, if the data cleaning rule decides to delete certain samples, it will affect the effect of the feature selection rule. The self-attention mechanism of the Transformer allows the model to capture these complex interactions. 3. Rule selection: Based on the results of the first two steps, a selector based on the attention mechanism is used to determine the final n rules. This selector uses a multi-head attention mechanism, where the query (Q) is a learnable parameter vector, and the key (K) and value (V) are representations of rules. By calculating the attention weights, the model can select the most relevant n rules.

[0106] For example, assuming m=5, n=3, the selection process is as follows: Rule 1 (data cleaning): attention weight 0.3; Rule 2 (feature selection): attention weight 0.25; Rule 3 (data transformation): attention weight 0.2; Rule 4 (anomaly detection): attention weight 0.15; Rule 5 (feature interaction): attention weight 0.1.

[0107] 4. Rule optimization: For the selected n rules, the output layer also performs further optimization. This includes adjusting the parameters of the rules, or generating a combination strategy between the rules. For example, if data cleaning and feature selection rules are selected at the same time, the model will generate a serialization strategy to guide how to perform data cleaning first and then feature selection.

[0108] 5. Output generation: Finally, the output layer generates a structured output that describes in detail the n selected data processing rules and how they are applied. This includes the specific parameters of each rule and the order in which they are applied.

[0109] This implementation can automatically identify abnormal patterns, key features, and potential associations in the data, and generate customized processing strategies based on these insights. Through the design of parallel task heads, this method can simultaneously consider multiple data processing requirements, and the final rule selection process ensures the overall coordination and effectiveness of the selected strategy. This automated and intelligent data processing method can not only greatly improve the efficiency and quality of data preprocessing, but also provide a more reliable foundation for subsequent data analysis and model training.

[0110] In one implementation of this embodiment, the parallel task header includes a data cleaning task header, a feature selection task header, a data conversion task header, an anomaly detection task header, and a feature interaction task header;

[0111] The structure of the data cleaning task head is a multi-layer perceptron structure. The data cleaning task head is used to generate multiple binary decisions based on the key features of the first output. Each binary decision corresponds to a key feature, and each binary decision is used to represent the cleaning probability of the corresponding key feature. The structure of the feature selection task head is based on the attention mechanism structure. The feature selection task head is used to assign corresponding importance weights to each key feature of the first output. The structure of the data conversion task head is a neural network structure. The data conversion task head is used to generate a [p×K] transformation matrix for each key feature of the first output. p represents the dimension of the key feature, K represents the number of transformation methods, and each element in the transformation matrix represents the applicable probability of the corresponding transformation method; the structure of the anomaly detection task head is an autoencoder structure. The anomaly detection task head is used to learn the data normal mode according to the data abnormal mode in the high-dimensional vector group, and learn the correlation between key features according to the potential correlation between different data sources, and generate a threshold vector for identifying anomalies according to the data normal mode and correlation. The structure of the feature interaction task head is a graph neural network structure. The feature interaction task head is used to learn the interaction relationship between key features according to the potential correlation and key features of different data sources, and output an adjacency matrix representing the interaction strength between feature pairs.

[0112] The data cleaning task head adopts a multi-layer perceptron structure, and its core goal is to generate multiple binary decisions based on the key features of the first output. Each binary decision corresponds to a key feature, which indicates the probability of cleaning the feature. In specific implementation, the multi-layer perceptron can contain multiple hidden layers, and each layer uses a nonlinear activation function (such as ReLU) to increase the expressive power of the model. The input layer receives the key features, the middle layer performs feature conversion and combination, and the output layer generates a probability value between 0 and 1 for each feature. This probability value indicates the possibility that the feature needs to be cleaned. For example, for a data set containing age, income, and occupation, the output may be [0.1, 0.8, 0.2], indicating that the income data is most likely to need cleaning. During the training process, labeled sample data can be used to optimize the network parameters through the back propagation algorithm. The loss function can select binary cross entropy to measure the difference between the predicted cleaning probability and the actual cleaning requirement. This can effectively identify outliers, missing values, or inconsistent data, improve data quality, and lay the foundation for subsequent analysis. In addition, since probability values ​​are generated, different thresholds can be set to balance the strictness of cleaning, thereby adapting to different application scenario requirements.

[0113] The feature selection task head is based on the attention mechanism structure, and its main function is to assign corresponding importance weights to each key feature of the first output. The core idea of ​​the attention mechanism is to enable the model to dynamically pay attention to the importance of different features. In the implementation process, each key feature is first converted into three vectors: query, key, and value. Then, the attention score is obtained by calculating the similarity between the query vector and all key vectors. These scores are normalized by the softmax function to form the attention weights. Finally, these weights are multiplied and summed with the corresponding value vector to obtain the final representation of each feature. In the specific implementation, a multi-head attention mechanism can be used, that is, multiple sets of independent attention are calculated in parallel, and then the results are spliced ​​together to capture information in different subspaces. The output attention weight directly reflects the importance of each feature. For example, for a dataset containing user browsing history, search keywords, and purchase records, the attention mechanism may assign a higher weight to the purchase record because it may be more relevant to the prediction of the user's future behavior. In this way, feature importance can be learned adaptively without artificially setting rules. It can capture complex interactions between features and can handle input sequences of different lengths. In practical applications, this feature selection method can significantly improve the interpretability and performance of the model because it allows the model to focus on the most relevant features, thereby reducing the impact of noise and improving the accuracy of predictions.

[0114] The data conversion task head adopts a neural network structure, and its main purpose is to generate a [p×K] transformation matrix for each key feature of the first output, where p represents the dimension of the key feature and K represents the number of transformation methods. Each element in this transformation matrix represents the applicable probability of the corresponding transformation method. In the implementation process, a multi-layer feedforward neural network can be used, the input layer receives the key features, the hidden layer performs nonlinear transformation, and the output layer generates a [p×K]-dimensional vector, which is then reshaped into the required matrix shape. The activation function can be selected as ReLU, etc., and the last layer uses the softmax function to ensure that the sum of the probabilities of each row is 1. For example, for a numerical feature, K may include methods such as standardization, minimum-maximum scaling, and logarithmic transformation; for categorical features, it may include methods such as one-hot encoding, label encoding, and frequency encoding. During the training process, the cross entropy loss function can be used to optimize the network parameters through back propagation. In specific implementation, different transformation methods can be applied to a small part of the data first, and their impact on downstream tasks can be evaluated, and these results can be used as training labels. In this way, the most suitable transformation method can be automatically recommended for different features without manually trying each combination. It can adapt to different data distributions and model requirements, and improve the efficiency and effectiveness of feature engineering.

[0115] The anomaly detection task head adopts an autoencoder structure, which is mainly used to learn the normal pattern of data, the correlation between key features, and generate a threshold vector for identifying anomalies. The autoencoder consists of two parts: an encoder and a decoder. The encoder compresses the input data into a low-dimensional latent space, and the decoder tries to reconstruct the original data from this latent representation. In the implementation process, the encoder network is first used to compress the high-dimensional vector group into a low-dimensional latent space. This process can be achieved through a multi-layer neural network, with each layer gradually reducing the number of neurons. Then, the decoder network tries to reconstruct the original data from the latent representation. The training goal is to minimize the reconstruction error, usually using the mean square error as the loss function. In this way, the autoencoder is able to capture the main features and normal patterns of the data. In order to learn the potential correlation between different data sources, a correlation layer can be added to the latent space, and the attention mechanism or graph neural network can be used to model the interaction between features. In this way, the autoencoder not only learns the distribution of a single feature, but also captures the complex relationship between features. After the training is completed, the distribution of the reconstruction error is calculated using the validation set, and the threshold vector for anomaly detection is determined based on this distribution. Specifically, the 95% quantile of the reconstruction error of each feature can be used as the corresponding threshold. In practical applications, if the reconstruction error of a sample exceeds this threshold, it is marked as a potential anomaly. The advantage of this method is that it is unsupervised and does not require pre-labeled abnormal samples. It can adapt to complex data distributions and can detect abnormal patterns that are difficult to describe with simple rules. In practice, this anomaly detection method can effectively identify outliers, abnormal patterns, or data quality issues in the data, thereby improving the reliability of data analysis and the effect of model training. In addition, since the correlation between features is taken into account, it can detect complex abnormal patterns that may be overlooked in univariate analysis.

[0116] The feature interaction task head adopts a graph neural network structure, and its main purpose is to learn the interaction relationship between key features and output an adjacency matrix representing the interaction strength between feature pairs. In the implementation process, each key feature is first regarded as a node in the graph, and the initial feature vector is used as the attribute of the node. Then, the structure of the graph is initialized using the potential association relationship between different data sources. For example, the initial edge weights can be set based on the correlation coefficient or prior knowledge between the features. Next, a multi-layer graph convolutional network (GCN) is applied to learn the interaction between nodes. Each layer of GCN can be expressed as: in is the normalized adjacency matrix, H (l) is the node feature of the lth layer, W (l)is a learnable weight matrix, and σ is a nonlinear activation function. Through multi-layer propagation, each node can aggregate information from its neighbors, thereby capturing a wider range of feature interactions. During training, the performance of downstream tasks can be used as a guide. For example, if it is a prediction task, the output of GCN can be connected to a prediction layer and the prediction error can be used to backpropagate and update the graph structure. The output of the last layer of GCN can be used to construct the final adjacency matrix, where each element represents the interaction strength between the corresponding feature pairs. In the specific implementation, GCN can be enhanced with an attention mechanism, allowing the model to dynamically adjust the importance of different neighbors. Thus, complex nonlinear relationships between features can be automatically discovered and quantified without being restricted by artificial assumptions. It can capture high-order feature interactions that are difficult to discover with traditional methods, thereby providing richer feature combination information. In practical applications, this task head can significantly improve the effect of feature engineering and provide a more informative feature set for downstream tasks. The output adjacency matrix can not only be used to generate interactive features, but also provide interpretable insights to help understand the dependencies between features.

[0117] In one implementation of this embodiment, the output layer includes a scoring layer, a Transformer layer, a selection layer, and an optimization output layer. Five types of data processing rules are input into the output layer to obtain n types of data processing rules, including the following steps:

[0118] S510, inputting five types of data processing rules into the scoring layer to obtain a quality score for each data processing rule;

[0119] S520, inputting five types of data processing rules into the Transformer layer to obtain the synergy value between the data processing rules;

[0120] S530, input the quality score and the synergy value into the selection layer, calculate the attention weight of each data processing rule, and use the attention weight greater than the preset threshold as the target attention weight, where the number of target attention weights is n;

[0121] S540, determining a data processing rule corresponding to each target attention weight, and obtaining n types of data processing rules;

[0122] S550, inputting n types of data processing rules into the optimization output layer, obtaining the data processing sequence of the n types of data processing rules, and outputting the n types of data processing rules and the data processing sequence.

[0123] The five types of data processing rules are input into the scoring layer to obtain the quality score of each data processing rule. The scoring layer uses a multi-layer perceptron structure to evaluate the quality of each data processing rule. In specific implementation, a three-layer neural network can be designed, with the number of input layer nodes being the same as the number of features of the data processing rule, and the hidden layer using the ReLU activation function.

[0124] The output layer uses the sigmoid activation function to generate a quality score between 0 and 1. For example, assuming that each data processing rule has 10 features to describe its properties (such as complexity, scope of application, execution time, etc.), the input layer has 10 nodes, the hidden layer can be set to 20 nodes, and the output layer has 1 node. For each input data processing rule, a quality score is calculated through forward propagation. In the training phase, historical data or expert-annotated quality scores can be used as supervisory signals to optimize network parameters through back propagation. In order to improve the generalization ability of the model, dropout technology and L2 regularization can be used. In addition, an attention mechanism can be introduced to allow the model to learn to pay attention to the importance of different features. In practical applications, quality scores can be used as an important basis for screening and sorting data processing rules. For example, if the quality score of a data cleaning rule is 0.85, while the score of another feature selection rule is 0.62, this indicates that the former is more reliable or more effective. The introduction of the scoring layer enables the automatic evaluation and comparison of the quality of different types of data processing rules, providing a quantitative basis for subsequent rule selection and combination, thereby improving the efficiency and reliability of the entire data processing process.

[0125] Five types of data processing rules are input into the Transformer layer to obtain the synergy values ​​between the data processing rules. The Transformer layer uses the self-attention mechanism to learn the mutual relations between different data processing rules. When implemented, each data processing rule is first encoded into a vector representation, and then the synergy relationship between the rules is calculated through the multi-head self-attention mechanism. The specific steps are as follows: 1) Position encoding is performed on each rule to preserve the order information of the rule sequence. 2) The encoded rules are input into the multi-head self-attention layer. Assume that there are 8 attention heads, and the dimension of each head is 64. 3) For each attention head, the query matrix Q, key matrix K, and value matrix V are calculated, and then the attention score is calculated. 4) The outputs of all attention heads are concatenated and passed through a feedforward neural network. 5) Residual connections and layer normalization are added to stabilize training. This process can be stacked with multiple layers (such as 6 layers) to capture more complex relationships. Finally, the output of the Transformer layer can be interpreted as a synergy value matrix between rules, where each element represents the synergy strength between two rules. For example, if the synergy value of data cleaning rule A and feature selection rule B is 0.75, this indicates that the two rules are highly compatible or complementary. The introduction of the Transformer layer enables the model to automatically discover implicit relationships between rules, including complementarity, conflict, or sequential dependency. By learning these synergistic relationships, data processing rules can be combined and sorted more intelligently, redundant operations can be avoided, and the synergy between rules can be maximized.

[0126] The quality score and synergy value are input into the selection layer to calculate the attention weight of each data processing rule, and the attention weight greater than the preset threshold is used as the target attention weight, where the number of target attention weights is n. The core of the selection layer is to calculate the attention weight by combining the quality and synergy relationship of the rules, so as to screen out the most important and relevant rules. The implementation process is as follows: 1) For each rule i, multiply its quality score qi by the average synergy value ci of all rules to obtain the initial importance score si = qi × ci. 2) Use the softmax function to normalize these scores into attention weights: w i =exp(s i ) / ∑ j exp(s j). 3) Set a preset threshold t (such as 0.1) and select rules with attention weights greater than t. The weights of these rules are called target attention weights. 4) If the number of rules greater than the threshold is not equal to n, the threshold can be adjusted dynamically until exactly n rules are selected. For example, suppose there are 5 rules, their quality scores are [0.8, 0.7, 0.9, 0.6, 0.75], and their average synergy values ​​are [0.7, 0.65, 0.8, 0.55, 0.7]. The calculated attention weights are [0.22, 0.18, 0.28, 0.13, 0.19]. If n=3 and the threshold t=0.2, the selected target attention weights are [0.22, 0.28, 0.19], and the corresponding rules are the 1st, 3rd, and 5th. The introduction of the selection layer enables the adaptive selection of the most important and coordinated subset of rules, taking into account both the quality of individual rules and the synergy between rules. This mechanism can effectively balance the individual quality and overall synergy of rules, ensuring that the selected rule sets are both high-quality and complementary, thereby improving the efficiency and effectiveness of data processing.

[0127] Determine the data processing rules corresponding to each target attention weight, and obtain n types of data processing rules. This step is to map the target attention weight selected in the previous step back to the original data processing rules, so as to obtain the final selected n rules. The specific implementation process is as follows: 1) Maintain an index mapping to record the correspondence between each attention weight and the original rule. 2) For each target attention weight, find the corresponding original data processing rule through the index mapping. 3) Combine these rules into a new rule set, which is the final selected n types of data processing rules. For example, suppose that the original 5 rules are: data cleaning rule A, feature selection rule B, data conversion rule C, anomaly detection rule D and feature interaction rule E. If the selected target attention weight corresponds to the 1st, 3rd, and 5th rules, then the final selected n (here n = 3) types of data processing rules are: data cleaning rule A, data conversion rule C and feature interaction rule E. In practical applications, each selected rule can be attached with its attention weight value, which can be used for subsequent rule sorting or weighted processing. In addition, the detailed properties of each rule, such as parameter settings, applicable conditions, etc., can be recorded to facilitate subsequent fine-tuning and interpretation. The importance of this step is that it concretizes the abstract attention weights into actual executable data processing rules, providing a clear operating guide for subsequent data processing processes. In this way, the subset of rules that best suits the current data and tasks can be automatically screened from a large number of candidate rules, greatly improving the pertinence and efficiency of data processing.

[0128] Input n types of data processing rules into the optimization output layer, obtain the data processing order of n types of data processing rules, and output n types of data processing rules and data processing order. The main task of the optimization output layer is to determine the optimal execution order of the selected n data processing rules to maximize the overall processing effect. The implementation process is as follows: 1) Build a sequence-to-sequence (Seq2Seq) model, and both the encoder and decoder use LSTM (Long Short-Term Memory Network) structure. 2) The encoder input is the feature representation of n rules, including rule type, parameters, attention weight and other information. 3) The decoder uses the rule sequence generated by the beamsearch strategy to predict the next rule to be executed at each time step. 4) In the training phase, the model is optimized using reinforcement learning methods (such as policy gradients), and the reward function can be based on the data quality improvement after simulated execution or the performance of downstream tasks. 5) In the inference phase, the rule execution sequence with the highest score is selected as the final output. For example, suppose the three selected rules are: data cleaning rule A, data conversion rule C, and feature interaction rule E. The execution order output by the optimization output layer is: A→C→E, which means that data cleaning should be performed first, then data conversion, and finally feature interaction. In practical applications, specific execution parameters and conditions can be attached to each rule, such as which columns the data cleaning rule A is applied to, what threshold is used, etc. In addition, the expected effect and uncertainty estimate of each step can be output to facilitate manual intervention and adjustment. The introduction of the optimization output layer enables the entire data processing process to not only select the most appropriate rules, but also optimize their execution order, thereby maximizing the overall effect of data processing. This automated rule sorting mechanism can capture the complex dependencies and interaction effects between rules, far exceeding the fixed processes designed by humans, and provides higher flexibility and efficiency for data processing.

[0129] This implementation method realizes the automatic evaluation, selection and optimization of data processing rules through multi-level intelligent processing. The scoring layer performs quality assessment on individual rules to provide a basis for rule screening. The Transformer layer learns the synergistic relationship between rules and captures complex interaction effects. The selection layer combines quality and synergy information to intelligently screen out the most important subset of rules. The optimized output layer further determines the optimal execution order of the rules. This end-to-end automated process not only improves the efficiency and quality of data processing, but also can adapt to different data characteristics and task requirements. Through learning and optimization, it can continuously improve its decision-making ability, provide intelligent data processing support, and transform the traditional fixed data processing process into dynamic and adaptive intelligence, greatly improving the effect and flexibility of data preprocessing.

[0130] In one implementation of this embodiment, the initial multi-source data set is processed using a data processing rule to obtain a processed multi-source data set, including the following steps:

[0131] S610 , using a data processing sequence and n types of data processing rules to process the initial multi-source data set to obtain a processed multi-source data set.

[0132] The initial multi-source data set is processed using the data processing sequence and n types of data processing rules to obtain the processed multi-source data set. This step is the core of the entire data processing process, which actually applies the rules selected and optimized in the previous steps to the initial data set. The specific implementation process is as follows:

[0133] First, according to the data processing order given by the optimized output layer, each data processing rule is executed in turn. Assume that the selected rules include data cleaning, feature selection and data conversion, and the optimized order is: data cleaning → feature selection → data conversion.

[0134] 1. Data cleaning rules: including processing missing values, removing outliers, and eliminating duplicate data. For example, for missing values, you can choose an appropriate filling method based on the data type and distribution characteristics, such as using the mean or median to fill in numerical features, using the mode to fill in categorical features, or using more complex interpolation methods such as K-nearest neighbor interpolation. For outliers, you can use the Z-score method to detect and process them. For example, if the Z-score of a numerical feature is greater than 3 or less than -3, it can be regarded as an outlier and replaced with the upper and lower limits.

[0135] 2. Feature selection rules: This step aims to select the most relevant and important subset of features from a large number of features. Various methods can be used, such as correlation-based methods, feature importance-based methods (such as feature importance of random forests), or wrapper methods (such as recursive feature elimination). For example, using the Pearson correlation coefficient method, calculate the correlation between each feature and the target variable, and select features with an absolute value of the correlation coefficient greater than 0.3. Or use a random forest model to calculate the importance score of each feature and select the features with the top 20% scores.

[0136] 3. Data transformation rules: including standardization, normalization, logarithmic transformation and other operations. For example, the numerical features are standardized so that their mean is 0 and their standard deviation is 1: (x-μ) / σ, where x is the original value, μ is the mean, and σ is the standard deviation. For features with skewed distribution, logarithmic transformation can be applied: log(1+x) to make the distribution closer to normal distribution.

[0137] When executing each rule, it is necessary to consider the specific parameter settings and applicable conditions of the rule. For example, the outlier threshold in the data cleaning rule, the feature quantity or importance threshold in the feature selection rule, the conversion method selection in the data conversion rule, etc. These parameters may have been optimized and determined in the previous steps, or can be dynamically adjusted according to the actual situation of the data.

[0138] In addition, during the processing, the consistency and integrity of the data need to be maintained. For example, ensure the coding consistency of categorical variables, handle newly emerged categories, maintain the logical relationship between features, etc. At the same time, it is necessary to record detailed logs for each step of the processing, including the amount of data affected, the statistical characteristics of the changes, etc., for subsequent analysis and adjustment.

[0139] In practical applications, parallel computing techniques can be used to speed up the processing process, especially when dealing with large-scale data sets. For example, a distributed computing framework such as Apache Spark can be used to split the data into multiple parts for parallel processing and then merge the results.

[0140] Finally, the processed data set is evaluated for quality, including data integrity check, feature distribution analysis, feature correlation analysis, etc., to ensure that the processed data set meets the expected quality standards. If problems are found, it may be necessary to go back to the previous steps, adjust the rules or parameters, and re-execute the data processing process.

[0141] This implementation method realizes the automatic conversion from the initial multi-source data set to the high-quality processed data set through the intelligent data processing process. It not only applies the optimized selected data processing rules, but also considers the optimal execution order between the rules, thereby maximizing the effect of data processing. It can adapt to different types of data and task requirements and provide flexible and efficient data preprocessing solutions. Through the systematic cleaning, feature selection and conversion process, the data quality is significantly improved, the noise and redundant information are reduced, and the expressiveness of the data is enhanced. It provides better input for subsequent machine learning model training and data analysis.

[0142] In one implementation of this embodiment, before the initial multi-source data set is input into the large language model to obtain the data processing rules, the following steps are included:

[0143] S710. When the domain type is a sensitive domain type, a desensitizing encryption strategy is used to perform desensitizing encryption processing on the initial multi-source data set to obtain a processed initial multi-source data set.

[0144] When the domain type is a sensitive domain type, the desensitizing encryption strategy is used to desensitize and encrypt the initial multi-source data set to obtain the processed initial multi-source data set. The core goal of desensitizing and encryption processing is to preserve the availability and analytical value of the data as much as possible while protecting data privacy and security.

[0145] First, the initial multi-source data set needs to be fully scanned and analyzed to identify all data fields that may contain sensitive information. This sensitive information may include personal identity information (such as name, ID number, phone number), financial information (such as bank account number, credit card information), medical health information, etc. The identification process can be achieved through regular expression matching, keyword search, pattern recognition and other technologies. For example, use regular expressions to match China's 18-digit ID number.

[0146] Next, based on the identified sensitive information types, select appropriate desensitization strategies for each type. Desensitization strategies include: 1. Data masking: Replace some or all characters of sensitive information with specific characters, such as asterisks or pound signs. 2. Data replacement: Replace the original data with fictitious but similarly formatted data. For example, replace the real name "Zhang San" with the randomly generated "Li Si". 3. Data generalization: Replace specific values ​​with more general categories. For example, replace the specific age of 35 with the age range "30-40 years old". 4. Data shuffling: Randomly shuffle the data in the same column to destroy the association between sensitive information and other information. This method is suitable for scenarios where the overall statistical characteristics need to be retained but the specific corresponding relationship does not need to be retained. 5. Data sampling: Randomly extract a portion of the data from the original data set to reduce the total amount of sensitive information.

[0147] For data that needs stronger protection, encryption technology can also be used. Common encryption algorithms include symmetric encryption (such as AES, DES) and asymmetric encryption (such as RSA). For example, the AES-256 algorithm can be used to encrypt sensitive fields. The encrypted data will become a string of seemingly random characters that can only be decrypted by people with the correct key.

[0148] In practical applications, a combination of desensitization and encryption technologies can be used according to the sensitivity of the data and the usage scenario. For example, for personal names, data can be replaced first, and then the replaced pseudonyms can be encrypted. For bank card numbers, they can be partially shielded (only the last four digits are retained), and then the remaining visible part can be encrypted.

[0149] In addition, the consistency and availability of data also need to be considered. For example, if the same sensitive information appears in multiple data sources (such as a person's name appears in different tables), it should be ensured that the information is consistently desensitized or encrypted to maintain the relevance of the data. At the same time, for numerical sensitive data that requires statistical analysis, desensitization methods that maintain statistical characteristics can be used, such as adding noise or perturbation processing, to preserve the analytical value of the data while protecting privacy.

[0150] During the desensitization and encryption process, a complete logging system needs to be established to record the processing method, processing time, processing personnel and other information of each field for subsequent audits and problem tracking. At the same time, a strict key management strategy needs to be formulated to ensure the secure storage and appropriate use of encryption keys.

[0151] This implementation method uses a desensitizing encryption strategy in sensitive areas to fully and securely process the initial multi-source data set. This step effectively protects personal privacy and sensitive information. The processed data set greatly reduces the risk and potential harm of data leakage while maintaining basic analytical value. This method of balancing security and availability provides a safe and reliable foundation for subsequent data analysis and modeling work. At the same time, it also enhances the trust of data subjects and related parties in the data processing process, which is conducive to the widespread application and value mining of data.

[0152] In one implementation of this embodiment, the desensitization encryption strategy includes the following steps:

[0153] S810, identifying a target sensitive data segment in the structured data of the initial multi-source data set;

[0154] S820, generating a corresponding token for the target sensitive data segment, wherein the token is unique;

[0155] S830. Create a mapping relationship between the token and the target sensitive data segment in a preset mapping database, and use the token to replace the target sensitive data segment.

[0156] S840. Identify the target sensitive data segments in the structured data of the initial multi-source data set. This step is the starting point of the desensitization encryption strategy, which aims to accurately locate the sensitive information that needs to be protected. Implementing this step requires the comprehensive use of multiple technical means to ensure that all potential sensitive data can be efficiently and accurately identified.

[0157] First, a comprehensive library of sensitive data types needs to be established, including but not limited to personal identity information (such as name, ID number, passport number), contact information (such as phone number, email address), financial information (such as bank account number, credit card number), medical and health information, etc. This type library should be customized according to different industry standards and laws and regulations to ensure that all relevant sensitive data types are covered.

[0158] Next, a multi-level recognition approach is used to scan the initial multi-source dataset. This typically includes the following aspects:

[0159] 1. Regular expression matching: Use predefined regular expressions to match sensitive data in a specific format. 2. Keyword matching: Use a predefined list of sensitive words to identify potentially sensitive fields. For example, search for field names or data content containing keywords such as "password", "account", "SSN", etc. 3. Semantic analysis: Use natural language processing technology to understand the context and semantics of the field to identify sensitive information, which helps to identify sensitive data that does not follow a fixed format. 4. Machine learning models: Train specialized machine learning models to identify complex or hidden sensitive data patterns. These models can learn to recognize various forms of sensitive information, including data that has been artificially disguised or deformed. 5. Metadata analysis: Analyze database schemas, table structures, and field properties to identify fields containing sensitive information. For example, fields marked as "encrypted" or "confidential" contain sensitive data.

[0160] In actual applications, the above methods are usually used in combination. For example, for a data table containing customer information, metadata analysis can be used to first identify fields containing sensitive information, and then regular expression matching and keyword searches can be applied to these fields. For fields with complex content, semantic analysis and machine learning models can be used for further in-depth inspection.

[0161] In addition, the identification process also needs to consider the context and relevance of the data. Some data is not sensitive when viewed alone, but it can reveal sensitive information when combined with other data. For example, a person's date of birth is not sensitive on its own, but if it appears with a name and address, it constitutes sensitive personal identification information.

[0162] During the identification process, detailed log records need to be established to record the location, type and identification method of each identified sensitive data segment.

[0163] Generate a corresponding token for the target sensitive data segment, where the token is unique and is intended to create a unique alternative identifier, i.e., a token, for each identified sensitive data segment. Not only must the uniqueness of the token be ensured, but the token itself must also not contain any information that can be used to infer the original data.

[0164] The token generation process usually involves the following key elements: 1. Uniqueness guarantee: Use a random number generator or cryptographic hash function to create a token. Common methods include UUID (universally unique identifier), a cryptographically secure pseudo-random number generator (CSPRNG), or a cryptographic hash function (such as SHA-256) combined with a salt value. 2. Length and complexity: The token should have sufficient length and complexity to prevent guessing attacks. Typically, at least 128 bits (16 bytes) of random data are used to generate tokens. 3. Readability and usability: Depending on actual needs, the generated random data can be encoded into a form that is easier to use and store, such as Base64 encoding or hexadecimal representation. 4. Context relevance: In some cases, it is necessary to embed some metadata (such as a data type identifier) ​​in the token, but ensure that this information does not disclose the content of the original data. 5. Time sensitivity: For sensitive data that needs to be updated regularly, consider including timestamps or version information in the token to support regular rotation of tokens.

[0165] One implementation is: first, generate a 16-byte (128-bit) random number for each sensitive data segment, and a cryptographically secure random number generator can be used. Next, combine this random number with some contextual information (such as a data type identifier and a timestamp). For example, if the sensitive data is a card number, the prefix "CC_" and the current timestamp can be added. Then, the combined data is processed through the SHA-256 hash function to obtain a 256-bit hash value. The uniqueness and unpredictability of the token are further enhanced. Finally, the obtained hash value is encoded in Base64 format for easy storage and transmission. Base64 encoding converts binary data into a string consisting of 64 different ASCII characters.

[0166] Create a mapping relationship between the token and the target sensitive data segment in the preset mapping database, and use the token to replace the target sensitive data segment. This step is the last link of the desensitization encryption strategy. It establishes the association between the token and the original sensitive data and completes the actual data replacement operation. This process must not only ensure the accuracy and security of the mapping relationship, but also ensure the integrity and traceability of the entire replacement process.

[0167] The mapping database should have the following features: 1. High security: Use strong encryption algorithms (such as AES-256) to encrypt stored sensitive data. 2. Access control: Implement strict access rights management to allow only authorized personnel or access. 3. Audit log: Record all access and modification operations on mapping relationships. 4. High availability: Use redundant storage and backup strategies to ensure data persistence and reliability.

[0168] The mapping database can use a high-security database (such as encrypted PostgreSQL). The structure of the database includes the following fields: token, encrypted original data, data type, creation time, last access time, expiration time, etc.

[0169] The process of creating a mapping relationship is as follows: 1. Encrypt each target sensitive data segment. For example, use the AES-256 algorithm and a randomly generated key to encrypt the data. 2. Store the encrypted data together with the corresponding token in the mapping database. 3. Securely store the encryption key in a separate key management.

[0170] Next, the actual data replacement operation is performed. This process requires traversing all target sensitive data segments in the initial multi-source data set and replacing them with corresponding tokens. The specific steps are as follows: 1. For each identified sensitive data segment, find its corresponding token. 2. Replace the original data with the token. 3. Update the data structure to ensure that the replaced data format is correct and does not affect subsequent data processing. 4. Record each replacement operation, including the location, time, and data type involved.

[0171] This implementation method first accurately identifies all sensitive data segments that need to be protected, ensuring the integrity of the protection scope. Then, a unique and secure token is generated for each sensitive data segment, effectively hiding the characteristics of the original data. Finally, by establishing a secure mapping relationship and replacing the original data, data desensitization is achieved. Not only does it greatly reduce the risk of data leakage, but it also maintains the availability and integrity of the data. At the same time, the traceability and auditability of the entire process are also guaranteed, providing strong support for the compliant use and management of data. Effectively protect privacy and sensitive information.

[0172] In one implementation of this embodiment, the processed multi-source data set is input into the domain adaptation module to obtain the final multi-source data set, including the following steps:

[0173] S910. Input the processed multi-source dataset into the domain adaptation module, so that the domain adaptation module corrects the processed multi-source dataset according to the domain type and the target knowledge graph to obtain the final multi-source dataset.

[0174] The processed multi-source dataset is input into the domain adaptation module so that the module corrects the processed multi-source dataset according to the domain type and the target knowledge graph, thereby obtaining the final multi-source dataset.

[0175] The core function of the domain adaptation module is to adjust and optimize data sets from different sources that may contain inconsistent or inaccurate information according to the knowledge and rules of a specific domain. This process involves multiple complex subtasks, including data cleaning, entity alignment, relationship reasoning, and knowledge completion. First, the domain adaptation module loads the corresponding rule set and knowledge base according to the pre-defined domain type. For example, if the domain type is healthcare, the module loads professional knowledge such as medical terminology, disease classification system, and drug interaction rules. This knowledge is usually stored in a structured form, such as ontology or knowledge graph. Next, the module performs a preliminary analysis of the input multi-source data set to identify entities, attributes, and relationships in the data. This process may use natural language processing (NLP) techniques such as named entity recognition (NER) and relationship extraction. For example, in a medical record, entities such as "patient", "symptom", "diagnosis", and "treatment" and their relationships may be identified. Then, the domain adaptation module will perform entity alignment or ontology matching on the identified entities and relationships with the concepts in the target knowledge graph. Use string matching, semantic similarity calculation, structural similarity analysis, etc. For example, "heart disease" in the data may be aligned to the concept of "coronary heart disease" in the knowledge graph. Based on entity alignment, the module checks whether the relationships in the data are consistent with the domain knowledge. This includes verifying the validity of the relationship, inferring implicit relationships, resolving conflicting relationships, etc. For example, if the data shows that a patient took two drugs at the same time that are known to have serious interactions, this potential problem will be marked. Next, the domain adaptation module will perform knowledge completion. This process attempts to fill in the gaps in the data and add valuable information. It may be based on rule reasoning or machine learning models. For example, if a patient's symptoms and test results highly match a certain disease, but the diagnosis information is missing, the system may infer a possible diagnosis and add it to the data. Throughout the process, the domain adaptation module will constantly refer to the target knowledge graph to ensure that the corrected data is consistent with the domain knowledge.

[0176] In this implementation, the multi-source data set is comprehensively optimized and enriched through the processing of the domain adaptation module. First, the entities and relationships in the data are accurately identified and aligned with standard terms, eliminating possible ambiguities and inconsistencies. Secondly, through the interaction with the target knowledge graph, the data is given deep domain semantics, making it more in line with professional standards. Thirdly, the knowledge completion process fills the gaps in the data and adds potentially valuable information. Finally, the entire process ensures the consistency, integrity, and domain relevance of the data. The final multi-source data set processed in this way not only retains the core information of the original data, but also incorporates rich domain knowledge, greatly improving the quality and practicality of the data. This lays a solid foundation for subsequent data analysis, decision support, and knowledge discovery, and can support more accurate and reliable applications.

[0177] The present application also provides a data processing flow optimization system, including:

[0178] a memory configured to store instructions; and

[0179] The processor is configured to call instructions from the memory and implement the above-mentioned data processing flow optimization method based on the large language model when executing the instructions.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of the processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0184] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0185] Memory includes non-permanent memory in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0186] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0187] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0188] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing flow optimization method based on a large language model, characterized in that: Applied to an electronic device, the electronic device is deployed with a large language model and a domain adaptation module, the method includes: Obtain multi-source data from different data sources; Inputting the multi-source data into the domain adaptation module to obtain an initial multi-source data set, wherein the domain adaptation module is used to identify the domain type of the multi-source data, and to modify and expand the multi-source data according to the domain type; Inputting an initial multi-source data set into the large language model to obtain data processing rules, wherein the data processing rules include any n types of data cleaning strategies, feature selection strategies, data conversion strategies, anomaly detection strategies, and feature interaction strategies, 0<n≤4, and n is an integer, the large language model includes an input embedding layer, a multi-layer Transformer block, a parallel task head, and an output layer, and the data processing rules are obtained after the input embedding layer, the multi-layer Transformer block, the parallel task head, and the output layer sequentially process the initial multi-source data set; Performing data processing on the initial multi-source data set using the data processing rule to obtain a processed multi-source data set; The processed multi-source dataset is input into the domain adaptation module to obtain a final multi-source dataset, wherein the domain adaptation module is used to adjust the processed multi-source dataset according to the domain type.

2. The method according to claim 1, characterized in that The multi-source data includes structured data and unstructured data, the domain adaptation module includes multiple domain databases, a multi-source classification and recognition module, a knowledge graph module and an expansion module, each of the domain databases stores data rules for each domain, and the multi-source data is input into the domain adaptation module to obtain an initial multi-source data set, including: Inputting the structured data and the unstructured data into the multi-source classification and identification module to obtain field types corresponding to both the structured data and the unstructured data; Inputting the domain type into the knowledge graph module, so that the knowledge graph module determines a target domain database from a plurality of domain databases according to the domain type, and determines a target knowledge graph according to the target domain database; The target knowledge graph is input into the expansion module, so that the expansion module expands and modifies the multi-source data according to the target knowledge graph to obtain an initial multi-source data set.

3. The method according to claim 1, characterized in that The step of inputting the initial multi-source data set into the large language model to obtain data processing rules includes: Inputting the initial multi-source data set into the input embedding layer to obtain a high-dimensional vector group of the initial multi-source data set; Inputting the high-dimensional vector group into the multi-layer Transformer block to obtain a first output, wherein the first output includes data anomaly patterns, key features, and potential associations between different data sources in the high-dimensional vector group; Inputting the first output into the parallel task head to obtain five types of data processing rules, wherein the parallel task head includes five task heads, each of which is used to generate one type of data processing rule, wherein the five types of data processing rules include the data cleaning strategy, the feature selection strategy, the data conversion strategy, the anomaly detection strategy, and the feature interaction strategy; The five types of data processing rules are input into the output layer to obtain n types of data processing rules, and the output layer is used to determine the n types of data processing rules from the five types of data processing rules.

4. The method according to claim 3, characterized in that The parallel task heads include a data cleaning task head, a feature selection task head, a data conversion task head, an anomaly detection task head and a feature interaction task head; The structure of the data cleaning task head is a multi-layer perceptron structure. The data cleaning task head is used to generate multiple binary decisions according to the key features of the first output. Each binary decision corresponds to one key feature, and each binary decision is used to represent the cleaning probability of the corresponding key feature. The structure of the feature selection task head is based on the attention mechanism structure. The feature selection task head is used to assign corresponding importance weights to each key feature of the first output. The structure of the data conversion task head is a neural network structure. The data conversion task head is used to generate a [p×K] conversion matrix for each key feature of the first output, where p represents the dimension of the key feature and K represents the conversion matrix. methods, each element in the transformation matrix represents the applicable probability of the corresponding transformation method; the structure of the anomaly detection task head is an autoencoder structure, and the anomaly detection task head is used to learn the data normal mode according to the data anomaly mode in the high-dimensional vector group, and learn the correlation between the key features according to the potential correlation between the different data sources, and generate a threshold vector for identifying anomalies according to the data normal mode and the correlation; the structure of the feature interaction task head is a graph neural network structure, and the feature interaction task head is used to learn the interaction relationship between the key features according to the potential correlation between the different data sources and the key features, and output an adjacency matrix representing the interaction strength between feature pairs.

5. The method according to claim 3, characterized in that: The output layer includes a scoring layer, a Transformer layer, a selection layer and an optimized output layer. The five types of data processing rules are input into the output layer to obtain n types of data processing rules, including: Inputting the five types of data processing rules into the scoring layer to obtain a quality score for each of the data processing rules; Inputting the five types of data processing rules into the Transformer layer to obtain the synergy values ​​between the data processing rules; Input the quality score and the synergy value into the selection layer, calculate the attention weight of each of the data processing rules, and use the attention weight greater than a preset threshold as the target attention weight, wherein the number of the target attention weights is n; Determine a data processing rule corresponding to each of the target attention weights to obtain n types of data processing rules; The n types of data processing rules are input into the optimization output layer to obtain the data processing order of the n types of data processing rules, and the n types of data processing rules and the data processing order are output.

6. The method according to claim 5, characterized in that The adopting the data processing rule to process the initial multi-source data set to obtain a processed multi-source data set includes: The data processing sequence and the n types of data processing rules are used to process the initial multi-source data set to obtain a processed multi-source data set.

7. The method according to claim 2, characterized in that Before inputting the initial multi-source data set into the large language model to obtain the data processing rules, the method includes: In the case where the domain type is a sensitive domain type, a desensitizing encryption strategy is used to perform desensitizing encryption processing on the initial multi-source data set to obtain the processed initial multi-source data set.

8. The method according to claim 7, characterized in that The desensitization encryption strategy includes: identifying a target sensitive data segment in the structured data of the initial multi-source data set; Generate a corresponding token for the target sensitive data segment, wherein the token is unique; A mapping relationship between the token and the target sensitive data segment is created in a preset mapping database, and the token is used to replace the target sensitive data segment.

9. The method according to claim 2, characterized in that: The step of inputting the processed multi-source dataset into the domain adaptation module to obtain a final multi-source dataset comprises: The processed multi-source dataset is input into the domain adaptation module, so that the domain adaptation module corrects the processed multi-source dataset according to the domain type and the target knowledge graph to obtain the final multi-source dataset.

10. A data processing flow optimization system, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and to implement the data processing flow optimization method based on a large language model according to any one of claims 1 to 9 when executing the instructions.

Images