A data processing flow optimization method and system based on large language model

Abstract

The present application discloses a data processing process optimization method and system based on a large language model, which relates to the field of data processing. The method includes: obtaining multi-source data from different data sources; inputting the multi-source data into a domain adaptation module to obtain an initial multi-source data set, wherein the domain adaptation module is used to identify the domain type of the multi-source data and to correct and expand the multi-source data according to the domain type; inputting the initial multi-source data set into the large language model to obtain data processing rules; using the data processing rules to process the initial multi-source data set to obtain a processed multi-source data set; inputting the processed multi-source data set into the domain adaptation module to obtain a final multi-source data set, wherein the domain adaptation module is used to adjust the processed multi-source data set according to the domain type. The present application can effectively improve the efficiency and quality of data processing.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular to a method and system for optimizing a data processing flow based on a large language model. Background Art

[0002] In today's digital age, data has become a core asset for businesses and organizations. With the rapid development of technologies such as the Internet of Things, social media, and e-commerce, the amount of global data is exploding. Not only is the volume of data surging, but its types and complexity are also increasing. Structured and unstructured data coexist, encompassing a variety of formats, including text, images, audio, and video. At the same time, data real-time requirements are increasing, with many application scenarios requiring near-real-time data processing and analysis capabilities. Against this backdrop, businesses and organizations face the immense challenge of effectively managing, processing, and leveraging massive amounts of complex data.

[0003] Traditional data processing methods typically include the following steps: data collection, data cleaning, data transformation, data integration, and data loading. During the data collection phase, traditional methods primarily rely on manual operations or simple automated scripts to acquire data from various sources. During the data cleaning process, predefined rules are often used to identify and address outliers, missing values, and duplicate data. The data transformation step involves converting data from one format or structure to another to meet specific analytical or storage requirements. Data integration combines data from different sources into a unified view. Finally, the data loading phase imports the processed data into the target system or data warehouse.

[0004] These traditional methods perform well when processing small-scale, structured data, but they expose many limitations and shortcomings when faced with today's complex data environment. First, traditional methods are inefficient when processing large-scale data. With the rapid increase in data volume, traditional data processing methods and processes have difficulty completing data processing tasks within an acceptable time. Second, traditional data processing methods often rely on predefined rules and patterns, which are difficult to cope with the rapid changes in data format, structure, and semantics. With the diversification of data sources and the increase in data volume, it is difficult to fully and accurately identify and address data quality issues relying solely on preset rules, making it difficult for traditional methods to effectively process data from different sources with different formats and semantics.

[0005] In summary, traditional data processing methods face huge challenges in terms of efficiency and quality. Summary of the Invention

[0006] The embodiments of the present application provide a data processing flow optimization method and system based on a large language model, which are used to effectively improve the efficiency and quality of data processing.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a method for optimizing a data processing flow based on a large language model is provided, which is applied to an electronic device, wherein the electronic device is deployed with a large language model and a domain adaptation module. The method includes:

[0009] Obtain multi-source data from different data sources;

[0010] Inputting the multi-source data into the domain adaptation module to obtain an initial multi-source data set, wherein the domain adaptation module is used to identify the domain type of the multi-source data and to modify and expand the multi-source data according to the domain type;

[0011] Inputting an initial multi-source dataset into the large language model to obtain data processing rules, wherein the data processing rules include any n types of a data cleaning strategy, a feature selection strategy, a data conversion strategy, an anomaly detection strategy, and a feature interaction strategy, where 0<n≤4 and n is an integer; the large language model includes an input embedding layer, a multi-layer Transformer block, a parallel task head, and an output layer; and the data processing rules are obtained by sequentially processing the initial multi-source dataset through the input embedding layer, the multi-layer Transformer block, the parallel task head, and the output layer;

[0012] Performing data processing on the initial multi-source data set using the data processing rule to obtain a processed multi-source data set;

[0013] The processed multi-source dataset is input into the domain adaptation module to obtain a final multi-source dataset, wherein the domain adaptation module is used to adjust the processed multi-source dataset according to the domain type.

[0014] In an implementation of the first aspect, the multi-source data includes structured data and unstructured data, the domain adaptation module includes multiple domain databases, a multi-source classification and recognition module, a knowledge graph module, and an expansion module, each of the domain databases stores data rules for each domain, and inputting the multi-source data into the domain adaptation module to obtain an initial multi-source dataset includes:

[0015] Inputting the structured data and the unstructured data into the multi-source classification and recognition module to obtain domain types corresponding to both the structured data and the unstructured data;

[0016] Inputting the domain type into the knowledge graph module, so that the knowledge graph module determines a target domain database from a plurality of domain databases according to the domain type, and determines a target knowledge graph according to the target domain database;

[0017] The target knowledge graph is input into the expansion module, so that the expansion module expands and modifies the multi-source data according to the target knowledge graph to obtain an initial multi-source data set.

[0018] In another implementation of the first aspect, inputting the initial multi-source dataset into the large language model to obtain data processing rules includes:

[0019] Inputting the initial multi-source dataset into the input embedding layer to obtain a high-dimensional vector group of the initial multi-source dataset;

[0020] Inputting the high-dimensional vector group into the multi-layer Transformer block to obtain a first output, wherein the first output includes data anomaly patterns, key features, and potential associations between different data sources in the high-dimensional vector group;

[0021] Inputting the first output into the parallel task head to obtain five types of data processing rules, wherein the parallel task head includes five task heads, each of which is used to generate one type of data processing rule, wherein the five types of data processing rules include the data cleaning strategy, the feature selection strategy, the data conversion strategy, the anomaly detection strategy, and the feature interaction strategy;

[0022] The five types of data processing rules are input into the output layer to obtain n types of data processing rules. The output layer is used to determine n types of data processing rules from the five types of data processing rules.

[0023] In another implementation of the first aspect, the parallel task heads include a data cleaning task head, a feature selection task head, a data conversion task head, an anomaly detection task head, and a feature interaction task head;

[0024] The structure of the data cleaning task head is a multi-layer perceptron structure. The data cleaning task head is used to generate multiple binary decisions based on the key features of the first output. Each binary decision corresponds to a key feature, and each binary decision is used to represent the cleaning probability of the corresponding key feature. The structure of the feature selection task head is based on the attention mechanism structure. The feature selection task head is used to assign corresponding importance weights to each key feature of the first output. The structure of the data conversion task head is a neural network structure. The data conversion task head is used to generate a [p×K] conversion matrix for each key feature of the first output, where p represents the dimension of the key feature and K represents the conversion direction. methods, each element in the transformation matrix represents the applicability probability of the corresponding transformation method; the structure of the anomaly detection task head is an autoencoder structure, and the anomaly detection task head is used to learn the data normal pattern according to the data anomaly pattern in the high-dimensional vector group, and learn the correlation between the key features according to the potential correlation relationship between the different data sources, and generate a threshold vector for identifying anomalies according to the data normal pattern and the correlation; the structure of the feature interaction task head is a graph neural network structure, and the feature interaction task head is used to learn the interaction relationship between the key features according to the potential correlation relationship between the different data sources and the key features, and output an adjacency matrix representing the interaction strength between feature pairs.

[0025] In another implementation of the first aspect, the output layer includes a scoring layer, a Transformer layer, a selection layer, and an optimization output layer, and the five types of data processing rules are input into the output layer to obtain n types of data processing rules, including:

[0026] Inputting the five types of data processing rules into the scoring layer to obtain a quality score for each of the data processing rules;

[0027] Inputting the five types of data processing rules into the Transformer layer to obtain synergy values ​​between the data processing rules;

[0028] Inputting the quality score and the synergy value into the selection layer, calculating the attention weight of each of the data processing rules, and taking the attention weight greater than a preset threshold as the target attention weight, wherein the number of the target attention weights is n;

[0029] Determining a data processing rule corresponding to each of the target attention weights to obtain n types of data processing rules;

[0030] The n types of data processing rules are input into the optimization output layer to obtain the data processing sequence of the n types of data processing rules, and the n types of data processing rules and the data processing sequence are output.

[0031] In another implementation of the first aspect, processing the initial multi-source dataset using the data processing rule to obtain a processed multi-source dataset includes:

[0032] The data processing sequence and the n types of data processing rules are used to process the initial multi-source data set to obtain a processed multi-source data set.

[0033] In another implementation of the first aspect, before inputting the initial multi-source dataset into the large language model to obtain the data processing rules, the method includes:

[0034] In the case that the domain type is a sensitive domain type, a desensitizing encryption strategy is adopted to perform desensitizing encryption processing on the initial multi-source dataset to obtain the processed initial multi-source dataset.

[0035] In another implementation of the first aspect, the desensitizing encryption strategy includes:

[0036] identifying a target sensitive data segment in the structured data of the initial multi-source data set;

[0037] Generating a corresponding token for the target sensitive data segment, wherein the token is unique;

[0038] A mapping relationship between the token and the target sensitive data segment is created in a preset mapping database, and the token is used to replace the target sensitive data segment.

[0039] In another implementation of the first aspect, inputting the processed multi-source dataset into the domain adaptation module to obtain a final multi-source dataset includes:

[0040] The processed multi-source dataset is input into the domain adaptation module, so that the domain adaptation module corrects the processed multi-source dataset according to the domain type and the target knowledge graph to obtain the final multi-source dataset.

[0041] In a second aspect, the present application provides a data processing flow optimization system, comprising:

[0042] a memory configured to store instructions; and

[0043] The processor is configured to call the instructions from the memory and implement the above-mentioned data processing flow optimization method based on the large language model when executing the instructions.

[0044] Through the above technical solution, first, by utilizing the computing power and parallel processing capabilities of the large language model, large-scale data processing tasks can be completed in a relatively short period of time, overcoming the problem of low efficiency of traditional methods when processing massive amounts of data. Secondly, while traditional methods often rely on predefined rules and patterns and are difficult to cope with the rapid changes in data format, structure, and semantics, the large language model in this technical solution can dynamically generate processing rules that adapt to different data characteristics, including data cleaning strategies, feature selection strategies, data conversion strategies, anomaly detection strategies, and feature interaction strategies, enabling flexible response to various complex and changing data situations, no longer restricted by fixed preset rules. In addition, by introducing the domain adaptation module, the accuracy and pertinence of data processing are further enhanced. The domain adaptation module can identify the domain type of multi-source data and modify and expand the data based on the domain characteristics, which not only improves the data quality, but also optimizes the data processing according to the needs of specific fields. After the data processing is completed, the domain adaptation module can also adjust the processing results again according to the domain type, ensuring that the final output data set meets both general data quality standards and the professional needs of specific fields. Furthermore, the semantic understanding capabilities of the large language model enable effective processing of data from diverse sources, with varying formats and semantics, enabling efficient data integration and unified processing. This significantly expands the scope and depth of data processing, enabling businesses and organizations to more comprehensively leverage various data resources. The large language model architecture, comprising an input embedding layer, multi-layer Transformer blocks, a parallel task head, and an output layer, not only ensures efficient processing but also enables multi-task parallel processing, further improving the efficiency and quality of data processing. This architectural design enables the model to simultaneously consider multiple aspects of the data, such as semantic understanding, structural analysis, and anomaly detection, resulting in more comprehensive and accurate data processing rules. Overall, it not only efficiently processes large-scale data but also flexibly addresses data diversity and variability, while achieving more precise data processing through domain adaptation.

[0045] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a method for optimizing a data processing flow based on a large language model provided in an embodiment of the present application;

[0047] Figure 2 A structural framework diagram of a large language model provided in an embodiment of the present application;

[0048] Figure 3A flowchart of an embodiment of the present application for inputting multi-source data into a domain adaptation module to obtain an initial multi-source data set is provided. DETAILED DESCRIPTION

[0049] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0050] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0051] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0052] Figure 1 The following schematically shows a flow chart of a method for optimizing a data processing flow based on a large language model according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a data processing flow optimization method based on a large language model, which is applied to an electronic device, where the electronic device is deployed with a large language model and a domain adaptation module. The method may include the following steps.

[0053] S110, acquiring multi-source data, where the multi-source data includes structured data and unstructured data;

[0054] S120, inputting the multi-source data into a domain adaptation module to obtain an initial multi-source data set, wherein the domain adaptation module is used to identify the domain type of the multi-source data and to modify and expand the multi-source data according to the domain type;

[0055] S130: Input the initial multi-source dataset into the large language model to obtain data processing rules, where the data processing rules include any n types of data cleaning strategies, feature engineering strategies, data conversion strategies, and integration strategies, where 0 < n ≤ 4 and n is an integer; the large language model includes an input embedding layer, a multi-layer Transformer block, a parallel task head, and an output layer; the data processing rules are obtained by sequentially processing the initial multi-source dataset using the input embedding layer, the multi-layer Transformer block, the parallel task head, and the output layer;

[0056] S140, processing the initial multi-source data set using a data processing rule to obtain a processed multi-source data set;

[0057] S150 , inputting the processed multi-source dataset into a domain adaptation module to obtain a final multi-source dataset, wherein the domain adaptation module is used to adjust the processed multi-source dataset according to the domain type.

[0058] Multi-source data includes structured data and unstructured data. Structured data is typically stored in tabular form with predefined formats and fields, such as tabular data in a database, CSV files, or Excel spreadsheets. Structured data is easy to organize, query, and analyze. Unstructured data, such as text documents, images, audio, or video files, has no predefined data model or organization. The process of acquiring multi-source data involves multiple steps: First, identify the data source, which can include internal databases, external APIs, files, web crawlers, etc. Second, establish connections to these data sources using different protocols and authentication methods. Then, perform data extraction operations, which involve SQL queries, API calls, file reading, etc. Finally, temporarily store the acquired data in memory or temporary storage space for subsequent processing.

[0059] Multi-source data is input into the domain adaptation module to generate an initial multi-source dataset. The domain adaptation module is used to identify the data domain type and modify and augment the data based on domain characteristics. This module first performs domain identification on the input multi-source data. Pre-trained classification models, such as support vector machines (SVMs) or deep learning models, can be used to analyze the data's characteristics and classify it into predefined domain types. Domain types include finance, healthcare, education, and engineering. During the identification process, the module considers characteristics such as data format, keywords, and numerical ranges. After determining the domain type, the domain adaptation module modifies the data based on domain-specific knowledge and rules. This includes correcting common domain-specific errors, standardizing terminology, and adjusting numerical units. For example, in the healthcare field, drug names with different expressions can be standardized. The augmentation process involves enriching the original data using domain-specific knowledge bases or external data sources. This includes adding relevant metadata, supplementing missing information, or inferring additional features based on domain knowledge. For example, in financial data, industry classification information can be added based on company codes. The output of the domain adaptation module is an initial multi-source dataset that has undergone domain-specific processing. This dataset not only retains the core information of the original data, but is also optimized and expanded according to domain characteristics.

[0060] Afterwards, the initial multi-source dataset is fed into the large language model to generate data processing rules. Figure 2 As shown in the figure, the structure of the large language model consists of an input embedding layer, a multi-layer Transformer block, a parallel task head, and an output layer. First, the initial multi-source dataset is converted into a high-dimensional vector representation through the input embedding layer. This process, which involves techniques such as word embedding and positional encoding, converts the raw data into a format that the model can process. The high-dimensional vector is then processed by a multi-layer Transformer block. Transformer blocks use self-attention mechanisms and feedforward neural networks to capture long-range dependencies and complex patterns in the data. This process identifies unusual patterns, key features, and potential correlations between different data sources. Next, the parallel task head is responsible for generating specific data processing rules. The task head includes specialized submodules such as data cleaning, feature selection, and data transformation, each optimized for a specific type of data processing task. For example, the data cleaning task head can generate strategies for removing outliers and handling missing data; the feature selection task head can propose methods for creating new features or selecting the most relevant features; and the data transformation task head can recommend appropriate data normalization or encoding methods. Finally, these generated rules are integrated and optimized through the output layer to produce the final n data processing rules (where n is an integer between 1 and 4). The output layer can consider the compatibility and overall effect between different rules to ensure that the generated rule set is coherent and efficient.

[0061] The generated data processing rules are used to process the initial multi-source dataset, ultimately obtaining a processed multi-source dataset. The data processing rules generated by the large language model are presented in a structured format, such as JSON or XML, and contain specific operation instructions and parameters. Various data processing operations can be performed in a specific order according to the data processing rules. Data cleaning is the first step, which includes handling missing values, removing outliers, and correcting data formats. For example, statistical methods (such as mean and median imputation) or machine learning methods (such as K-nearest neighbors) can be used to handle missing data; outliers can be identified and handled using methods such as Z-score or IQR. Next, feature engineering strategies can be implemented, which involve creating new features, selecting the most relevant features, or transforming existing features. For example, existing features can be combined to create interaction terms, principal component analysis (PCA) can be used for dimensionality reduction, or feature selection algorithms such as LASSO can be applied to select the most important features. Data transformation strategies include standardization (such as Z-score standardization), normalization (such as Min-Max scaling), or encoding categorical variables (such as one-hot encoding or label encoding). If ensemble strategies are included, information from different data sources can be combined or predictions from multiple models can be fused. For example, voting or averaging can be used to combine the outputs of multiple models. Finally, processed multi-source datasets can be temporarily stored, using distributed storage to handle large-scale data. This can significantly improve data quality and availability.

[0062] Finally, the processed multi-source dataset is fed back into the domain adaptation module to generate the final multi-source dataset. This allows for further optimization and adjustment of the data to ensure it fully complies with the requirements and standards of a specific domain. First, the domain adaptation module re-identifies the data's domain type. Although domain identification was performed during the initial processing phase, the data's characteristics have changed after the previous processing steps, making reconfirmation of the domain type necessary. Because the data has already undergone preliminary cleansing and transformation, this identification is more accurate. Next, the domain adaptation module performs in-depth adjustments to the data based on the confirmed domain type. This process involves multiple aspects: First, domain-specific data standardization rules can be applied. For example, in the financial domain, this requires ensuring that all monetary data is converted to a single currency unit and aligned with specific accounting standards. In the medical domain, this requires ensuring that all diagnosis codes conform to the latest International Classification of Diseases standards. Second, the module can introduce additional domain-specific features. These can include derived variables computed based on domain knowledge or supplementary information from domain-specific external data sources. For example, in retail data analysis, external factors such as holiday markers or economic indicators can be added. Finally, the module can perform semantic enhancement on the data. This involves using domain-specific ontologies or knowledge graphs to enrich the semantic content of the data, making the data more consistent with domain requirements not only structurally but also semantically. For example, in scientific research data processing, professional terms can be linked to standard academic concept libraries. In addition, the domain adaptation module can also perform some domain-specific data validation and consistency checks. This includes checking whether the logical relationships between certain variables conform to domain rules, or verifying whether certain key indicators are within a reasonable range. For example, in demographic data, the consistency between age and education level can be checked. Finally, the module can perform final format adjustments on the data to ensure that the output dataset conforms to the data format and structure commonly used in the domain.

[0063] First, this embodiment can complete large-scale data processing tasks in a relatively short period of time by utilizing the computing power and parallel processing capabilities of the large language model, overcoming the problem of low efficiency of traditional methods when processing massive amounts of data. Secondly, for traditional methods that often rely on predefined rules and patterns and are difficult to cope with the rapid changes in data format, structure and semantics, the large language model in this technical solution can dynamically generate processing rules that adapt to different data features, including data cleaning strategies, feature selection strategies, data conversion strategies, anomaly detection strategies and feature interaction strategies, etc., so that it can flexibly respond to various complex and changeable data situations and is no longer restricted to fixed preset rules. In addition, by introducing the domain adaptation module, the accuracy and pertinence of data processing are further enhanced. The domain adaptation module can identify the domain type of multi-source data and correct and expand the data according to the domain characteristics, which not only improves the quality of the data, but also optimizes the data according to the needs of specific fields. After the data processing is completed, the domain adaptation module can also adjust the processing results again according to the domain type to ensure that the final output data set meets both general data quality standards and the professional needs of specific fields. Furthermore, the semantic understanding capabilities of the large language model enable effective processing of data from diverse sources, with varying formats and semantics, enabling efficient data integration and unified processing. This significantly expands the scope and depth of data processing, enabling businesses and organizations to more comprehensively leverage various data resources. The large language model architecture, comprising an input embedding layer, multi-layer Transformer blocks, a parallel task head, and an output layer, not only ensures efficient processing but also enables multi-task parallel processing, further improving the efficiency and quality of data processing. This architectural design enables the model to simultaneously consider multiple aspects of the data, such as semantic understanding, structural analysis, and anomaly detection, resulting in more comprehensive and accurate data processing rules. Overall, it not only efficiently processes large-scale data but also flexibly addresses data diversity and variability, while achieving more precise data processing through domain adaptation.

[0064] In one implementation of this embodiment, the multi-source data includes structured data and unstructured data. The domain adaptation module includes multiple domain databases, a multi-source classification and recognition module, a knowledge graph module, and an expansion module. Each domain database stores data rules for each domain. The multi-source data is input into the domain adaptation module to obtain an initial multi-source data set, including the following steps:

[0065] S210, inputting the structured data and the unstructured data into a multi-source classification and recognition module to obtain domain types corresponding to both the structured data and the unstructured data;

[0066] S220: Input the domain type into the knowledge graph module, so that the knowledge graph module determines a target domain database from multiple domain databases according to the domain type, and determines a target knowledge graph according to the target domain database;

[0067] S230. Input the target knowledge graph into the expansion module, so that the expansion module expands and corrects the multi-source data according to the target knowledge graph to obtain an initial multi-source data set.

[0068] Figure 3 A schematic diagram of a process of inputting multi-source data into a domain adaptation module to obtain an initial multi-source data set is shown in an embodiment of the present application. In this embodiment, structured data and unstructured data are first input into a multi-source classification and recognition module to obtain domain types corresponding to both types of data. The multi-source classification and recognition module adopts an ensemble learning method, combining multiple classifiers to improve the accuracy and robustness of recognition. For structured data, a feature-based classification method such as random forest, support vector machine (SVM) or gradient boosting decision tree (GBDT) is used. First, key features are extracted from the structured data, such as statistical features (mean, variance, quantile, etc.) of numerical fields, distribution characteristics of category fields, trend characteristics of time series data, etc. Then, these features are input into the pre-trained classification model for domain recognition.

[0069] For unstructured data, particularly text data, deep learning methods, such as convolutional neural networks (CNNs), can be used to automatically learn the semantic features of text. For example, for a document, a pre-trained word embedding model is first used to convert the text into a vector sequence, and then a deep neural network is used for feature extraction and classification. For other types of unstructured data, such as images or audio, specialized deep learning models are used, such as ResNet for images or LSTM networks for audio.

[0070] The multi-source classification and recognition module processes and integrates information from different data sources. For example, for a dataset containing customer information, structured data might include age, income, and purchase history, while unstructured data might include customer reviews or chat logs. The multi-source classification and recognition module can process these two types of data separately and then use an ensemble method (such as voting or weighted averaging) to synthesize the data to arrive at the final domain classification result.

[0071] In practical applications, this step is implemented as follows: Suppose there is a dataset containing customer data, and the goal is to determine which business field (such as retail, finance, healthcare, etc.) this data belongs to. First, a random forest classifier is used on structured data (such as customer demographic information, transaction records). For example, if the data contains a large number of financial transaction records and credit scores, the random forest will tend to classify it as belonging to the financial field. At the same time, a BERT-based text classification model is used for unstructured data (such as customer reviews). If words such as "product", "delivery", and "return" appear frequently in the reviews, the BERT model will identify it as belonging to the retail field. Finally, the module combines the results of the two classifiers and uses a soft voting method, that is, comprehensively considering the confidence of the two classifiers to obtain the final field classification result.

[0072] The domain type is input into the knowledge graph module to determine the target domain database and the corresponding target knowledge graph. Knowledge graph technology can be used to select and construct knowledge structures related to specific domains. A knowledge graph is a structured knowledge representation that stores relationships between entities in the form of a graph, effectively capturing domain-specific concepts, attributes, and relationships.

[0073] First, the knowledge graph module can match and select from multiple domain databases based on the input domain type. Each domain database contains domain-specific data rules, concept hierarchies, entity relationships, and other information. For example, if the domain type is identified as "finance," the knowledge graph module will select a database in the finance field, which contains relevant concepts and rules such as banking, stocks, investments, and risk assessment.

[0074] Once the target domain database is determined, the knowledge graph module begins to select the corresponding target knowledge graph. This process involves the following steps: 1. Entity extraction: Extract key entities from the target domain database. For example, in the financial field, entities such as "bank", "stock", and "investor" will be extracted. 2. Relationship extraction: Identify the relationship between entities. For example, there are relationships such as "buy" and "hold" between "investor" and "stock". 3. Attribute extraction: Define relevant attributes for each entity. For example, the "stock" entity has attributes such as "price", "trading volume", and "market value". 4. Ontology construction: Establish a hierarchy of domain concepts. For example, "stock" is a subclass of "financial assets". 5. Rule integration: Integrate domain-specific rules and constraints into the knowledge graph. For example, some financial products have specific trading rules or risk assessment criteria.

[0075] In actual applications, this step is implemented as follows: Assume that the domain type determined in the previous step is "medical". The knowledge graph module will first select the database in the medical field. Then, the knowledge graph module starts to select the knowledge graph in the medical field. In this process, the knowledge graph module extracts entities such as "disease", "symptoms", "treatment methods", "drugs", and establishes relationships between them, such as "disease-cause-symptoms", "drug-treatment-disease", etc. At the same time, the knowledge graph module can also define the attributes of these entities, such as "drugs" have attributes such as "dosage", "side effects", and "contraindications". In addition, a conceptual hierarchy can be established, such as "antibiotics" is a subclass of "drugs". Finally, integrate rules unique to the medical field, such as restrictions on the use of certain drugs or diagnostic criteria for specific diseases.

[0076] The target knowledge graph is fed into the augmentation module, which augments and modifies the multi-source data based on the target knowledge graph, ultimately generating the initial multi-source dataset. The core of this step is to leverage the domain-specific knowledge graph to enhance and optimize the original data, making it more complete, accurate, and semantically rich.

[0077] The augmentation module's workflow typically includes the following key steps: 1. Entity Linking: First, the augmentation module attempts to match elements in the raw data with entities in the knowledge graph. This matching process uses a variety of techniques, such as string matching, semantic similarity calculation, or entity disambiguation. For example, when processing medical data, the disease names in patient records are linked to disease entities in the knowledge graph. 2. Relationship Inference: Based on the linked entities, the augmentation module attempts to infer relationships between data elements. These relationships are not explicitly represented in the raw data but can be inferred from the relational structure in the knowledge graph. For example, if a symptom is mentioned in the patient data, the knowledge graph can be used to infer potential diseases associated with that symptom. 3. Attribute Supplementation: The augmentation module uses information from the knowledge graph to supplement missing or incomplete attributes in the raw data. This can be achieved by adding standardized attribute values, filling in missing fields, or adding additional relevant information. For example, when processing drug data, the knowledge graph can be used to supplement information such as the drug's mechanism of action and common side effects. 4. Data standardization: Using the standard terminology and classification system in the knowledge graph, the expansion module can standardize the expressions in the original data. This is particularly useful for processing data from different sources, and can eliminate differences in terminology and expression. For example, the names of diseases with different expressions can be unified into standard medical terms. 5. Consistency check: The expansion module will use the rules and constraints in the knowledge graph to check the consistency of the data and correct errors. This includes correcting obvious data entry errors or marking outliers that require further verification. 6. Semantic enhancement: By adding contextual information from the knowledge graph, the expansion module can enhance the semantic content of the data. This includes adding classification labels, associated concepts, or explanatory information.

[0078] In practice, this step is implemented as follows: Suppose we are processing a medical dataset containing patient medical records. The original data contains basic information such as patient ID, symptom description, and diagnosis results. The expansion module first links symptoms and diagnoses with entities in the medical knowledge graph. For example, the symptom of "headache" is linked to the corresponding node in the knowledge graph. Then, based on the relationships in the knowledge graph, other related symptoms or diseases are inferred and added to the dataset. Next, detailed descriptions of each diagnosis result, common treatment methods, and other information are supplemented from the knowledge graph. During the standardization process, the same symptoms with different expressions (such as "headache" and "headache") are unified into standard terms. During the consistency check stage, some abnormal combinations can be discovered and marked, such as inconsistencies between certain symptoms and diagnosis results. Finally, relevant medical classification codes are added to each diagnosis to enhance the semantic content of the data.

[0079] The result is a significantly enhanced initial multi-source dataset. This dataset not only retains the core information of the original data but also incorporates rich domain knowledge and semantic content. For example, existing data gaps, inconsistencies, and errors are corrected, potential relationships between data are revealed, and additional relevant information is added.

[0080] Through three key steps: multi-source classification and identification, knowledge graph construction, and data augmentation and correction, this implementation not only accurately identifies the domain to which the data belongs, but also leverages domain-specific knowledge structures to enhance and optimize the data, significantly improving its quality, integrity, and semantic richness, providing a solid foundation for subsequent data analysis and decision support. By integrating machine learning, knowledge graphs, and data processing technologies, it intelligently processes complex data, extracts implicit relationships, and ultimately generates a highly structured, information-rich dataset, laying the foundation for a variety of advanced analytical tasks and intelligent applications.

[0081] In one implementation of this embodiment, the initial multi-source dataset includes text data, numerical data, categorical data, and / or time series data. The initial multi-source dataset is input into a large language model to obtain data processing rules, including the following steps:

[0082] S310, inputting the initial multi-source data set into the input embedding layer to obtain a high-dimensional vector group of the initial multi-source data set;

[0083] S320: Input the high-dimensional vector group into a multi-layer Transformer block to obtain a first output, where the first output includes data anomaly patterns, key features, and potential correlations between different data sources in the high-dimensional vector group.

[0084] S330: Input the first output into a parallel task head to obtain five types of data processing rules, wherein the parallel task head includes five task heads, each task head is used to generate a type of data processing rule, wherein the five types of data processing rules include a data cleaning strategy, a feature selection strategy, a data conversion strategy, an anomaly detection strategy, and a feature interaction strategy;

[0085] S340 , inputting the five types of data processing rules into the output layer to obtain data processing rules. The output layer is used to determine n types of data processing rules from the five types of data processing rules.

[0086] Processing the initial multi-source dataset begins by feeding different types of data into the input embedding layer. The core goal of this step is to convert the heterogeneous data into a unified high-dimensional vector representation, laying the foundation for subsequent deep learning processing. Specifically, for text data, word embedding techniques are used. For example, using 300-dimensional word vectors, the sentence "The catsatonthemat" is converted into a [300×6] matrix, where each row represents a 300-dimensional vector for a word. For numerical data, standardization or normalization is first performed before mapping it to a higher-dimensional space using a fully connected layer. For example, after standardizing features such as age and income, a linear transformation of the form y = Wx + b is used to map the original low-dimensional features to a 100-dimensional vector space. Categorical data is typically processed using one-hot encoding or entity embedding techniques. For example, the color features "red, green, blue" are encoded as [1, 0, 0], [0, 1, 0], and [0, 0, 1], and then converted to a higher-dimensional vector using an embedding layer. Time series data uses positional encoding or special time embedding techniques to capture the temporal nature of the data. For example, for a stock price series, a combination of sine and cosine functions can be used to encode the temporal information, and then these encodings can be combined with the original data.

[0087] After the high-dimensional vector array is processed by the input embedding layer, it is then input into a multi-layer Transformer block. The core of the Transformer block is its self-attention mechanism, which can effectively capture long-range dependencies between data. At this stage, each Transformer layer first performs a multi-head self-attention calculation on the input vector. Specifically, for the input sequence X, a linear transformation is performed to obtain the query (Q), key (K), and value (V) matrices, and then the attention weights are calculated:

[0088]

[0089] Where, d k is the dimension of the key, which is used to scale the dot product to prevent it from being too large. The multi-head attention mechanism allows the model to learn information from different representation subspaces simultaneously, further enhancing the expressive power of the model.

[0090] For example, consider a stock price series with 10 time steps, where the feature vector dimension of each time step is 64. The multi-head attention mechanism (assuming there are 8 heads) can transform the [10×64] input matrix into 8 [10×8] attention matrices, each representing the aggregation of information from a different perspective. These attention outputs are then concatenated and passed through a linear layer to obtain a [10×64] output, preserving the original dimension.

[0091] After the attention calculation, each Transformer layer also contains a feed-forward neural network, which usually consists of two linear transformations with a ReLU activation function in the middle:

[0092] FFN(x)=max(0,xW1+b1)W2+b2;

[0093] In this formula, x is the output of the attention mechanism, the first linear transformation is xW1+b1, W1 is the first weight matrix, b1 is the first bias vector, used to map the input x to a higher-dimensional space. max(0,…) is the ReLU activation function, which sets all negative values ​​to 0 and keeps positive values ​​unchanged to introduce nonlinearity, allowing the network to learn more complex functions. The second linear transformation is (…)W2+b2, W2 is the second weight matrix, b2 is the second bias vector, used to map the data back to the original dimension. The entire process is: first, the input is mapped to a higher-dimensional space, then, nonlinearity is introduced through the ReLU activation function, and finally, the result is mapped back to the original dimension. This further enhances the nonlinear expression ability of the model. The stacking of multiple layers of Transformer enables the model to abstract layer by layer, from low-level features to high-level semantic representations.

[0094] Through this process, the Transformer block can identify unusual patterns in the data, such as sudden jumps in time series or unusual word combinations in text data. It can also extract key features of the data. For example, in financial data analysis, specific price patterns or changes in trading volume are identified as important indicators. Furthermore, because different types of data are mapped into the same vector space, the Transformer block can also discover potential connections between different data sources, such as the correlation between text sentiment and stock price fluctuations.

[0095] The output of this stage, namely the first output, contains rich information, which not only reflects the core characteristics of the original data, but also includes the complex relationships between the data and potential abnormal patterns.

[0096] After the first output is processed by multiple layers of Transformer blocks, it is fed into a parallel task head for further specialized processing. The design philosophy of the parallel task head is to simultaneously generate multiple types of data processing rules through multi-task learning. This stage consists of m independent task heads, each responsible for generating a specific type of data processing rule, where m is greater than the final number of rules required, n.

[0097] Each task head is equipped with a specialized neural network structure, including fully connected layers, convolutional layers, or recurrent neural network layers, to adapt to the generation requirements of different types of rules. For example, assuming m = 5, these five task heads correspond to the following data processing tasks:

[0098] 1. Data cleaning rule generation: This task head uses a multi-layer perceptron (MLP) architecture. Its input is the output of a Transformer, and its output is a series of binary decisions indicating whether certain features require cleaning. For example, for a dataset containing 100 features, this task head outputs a 100-dimensional vector, where each element has a value between 0 and 1, indicating the probability that the corresponding feature requires cleaning.

[0099] 2. Feature Selection Rule Generation: This task head uses an attention-based architecture to assign importance weights to each feature. For example, for the same 100 features, the output is a 100-dimensional weight vector, where larger values ​​indicate more important features.

[0100] 3. Data Transformation Rule Generation: This task head uses a complex neural network structure to generate transformation solutions for each feature. The output is a [100xK] matrix, where K represents the number of transformation methods (such as logarithmic transformation, square root transformation, etc.), and each element in the matrix represents the probability of applying the corresponding transformation method.

[0101] 4. Anomaly detection rule generation: This task head uses an autoencoder structure to learn the normal pattern of the data and output a threshold vector for identifying anomalies.

[0102] 5. Feature interaction rule generation: This task head uses a graph neural network-based structure to learn the interaction relationship between features and output an adjacency matrix representing the interaction strength between feature pairs.

[0103] This parallel processing approach not only improves computational efficiency but also allows the model to share underlying representations between different data processing tasks, resulting in a more comprehensive and coordinated data processing strategy. For example, data cleaning rules and feature selection rules can affect each other because some features that need to be cleaned happen to be less important features.

[0104] The output of this phase is m different types of data processing rules, each targeting a specific aspect of the data processing process. These rules provide a rich set of candidate solutions for subsequent selection and integration, increasing the flexibility and adaptability of the final data processing strategy.

[0105] Finally, the m types of data processing rules are input into the output layer for integration and optimization, ultimately selecting the n most suitable data processing rules. The output layer is designed to synthesize multiple specialized data processing rules into a coherent, comprehensive data processing strategy. This process typically includes the following key steps: 1. Rule Evaluation: First, the output layer evaluates each data processing rule. This involves using a scoring network that receives a description of each rule (a vector representation) as input and outputs a score representing the rule's quality or applicability. For example, for m = 5 rules, the scoring network outputs a 5-dimensional vector, with each element ranging from 0 to 1, representing the score of the corresponding rule. 2. Modeling Inter-Rule Relationships: Next, a Transformer architecture is used to model the interactions between different rules. This allows for consideration of any synergies or conflicts between rules. For example, if a data cleaning rule decides to remove certain samples, this can affect the effectiveness of a feature selection rule. The Transformer's self-attention mechanism allows the model to capture these complex interactions. 3. Rule Selection: Based on the results of the first two steps, an attention-based selector is used to determine the final n rules. This selector uses a multi-head attention mechanism, where the query (Q) is a learnable parameter vector, and the key (K) and value (V) are the representations of the rules. By calculating the attention weights, the model can select the n most relevant rules.

[0106] For example, assuming m=5, n=3, the selection process is as follows: Rule 1 (data cleaning): attention weight 0.3; Rule 2 (feature selection): attention weight 0.25; Rule 3 (data transformation): attention weight 0.2; Rule 4 (anomaly detection): attention weight 0.15; Rule 5 (feature interaction): attention weight 0.1.

[0107] 4. Rule Optimization: The output layer further optimizes the n selected rules. This includes adjusting rule parameters or generating strategies for combining rules. For example, if both data cleaning and feature selection rules are selected, the model generates a sequential strategy that guides how to perform data cleaning first and then feature selection.

[0108] 5. Output Generation: Finally, the output layer generates a structured output that details the n selected data processing rules and how they were applied. This includes the specific parameters for each rule and the order in which they should be applied.

[0109] This implementation automatically identifies unusual patterns, key features, and potential correlations in data and generates customized processing strategies based on these insights. Through the design of a parallel task head, this method can simultaneously consider multiple data processing requirements, while the final rule selection process ensures the overall coordination and effectiveness of the selected strategies. This automated and intelligent data processing approach not only significantly improves the efficiency and quality of data preprocessing but also provides a more reliable foundation for subsequent data analysis and model training.

[0110] In one implementation of this embodiment, the parallel task headers include a data cleaning task header, a feature selection task header, a data conversion task header, an anomaly detection task header, and a feature interaction task header;

[0111] The structure of the data cleaning task head is a multi-layer perceptron structure. The data cleaning task head is used to generate multiple binary decisions based on the key features of the first output. Each binary decision corresponds to a key feature, and each binary decision is used to represent the cleaning probability of the corresponding key feature. The structure of the feature selection task head is based on the attention mechanism structure. The feature selection task head is used to assign corresponding importance weights to each key feature of the first output. The structure of the data conversion task head is a neural network structure. The data conversion task head is used to generate a [p×K] transformation matrix for each key feature of the first output. p represents the dimension of the key feature, K represents the number of transformation methods, and each element in the transformation matrix represents the applicable probability of the corresponding transformation method; the structure of the anomaly detection task head is an autoencoder structure. The anomaly detection task head is used to learn the data normal pattern based on the data anomaly pattern in the high-dimensional vector group, and learn the correlation between key features based on the potential correlation relationship between different data sources, and generate a threshold vector for identifying anomalies based on the data normal pattern and correlation. The structure of the feature interaction task head is a graph neural network structure. The feature interaction task head is used to learn the interaction relationship between key features based on the potential correlation relationship and key features of different data sources, and output an adjacency matrix representing the interaction strength between feature pairs.

[0112] The data cleaning task head uses a multi-layer perceptron (MLP) architecture. Its core goal is to generate multiple binary decisions based on the key features of the first output. Each binary decision corresponds to a key feature and represents the probability of cleaning that feature. In practice, the MLP can include multiple hidden layers, each using a nonlinear activation function (such as ReLU) to increase the model's expressiveness. The input layer receives the key features, the intermediate layers perform feature transformation and combination, and the output layer generates a probability value between 0 and 1 for each feature. This probability value represents the likelihood that the feature requires cleaning. For example, for a dataset containing age, income, and occupation, the output might be [0.1, 0.8, 0.2], indicating that income data is most likely to require cleaning. During training, labeled sample data can be used to optimize network parameters using the backpropagation algorithm. The loss function can be binary cross-entropy, which measures the difference between the predicted cleaning probability and the actual cleaning requirement. This effectively identifies outliers, missing values, or inconsistent data, improves data quality, and lays the foundation for subsequent analysis. Furthermore, since the generated probability value is a probability value, different thresholds can be set to balance the stringency of the cleaning process, adapting to different application scenarios.

[0113] The feature selection task head is based on an attention mechanism. Its primary function is to assign importance weights to each key feature in the first output. The core idea of ​​the attention mechanism is to enable the model to dynamically focus on the importance of different features. In implementation, each key feature is first converted into three vectors: query, key, and value. Then, an attention score is obtained by calculating the similarity between the query vector and all key vectors. These scores are normalized using a softmax function to form attention weights. Finally, these weights are multiplied by the corresponding value vector and summed to obtain the final representation of each feature. In practice, a multi-head attention mechanism can be used, computing multiple independent attention sets in parallel and then concatenating the results to capture information from different subspaces. The output attention weights directly reflect the importance of each feature. For example, for a dataset containing a user's browsing history, search keywords, and purchase history, the attention mechanism might assign higher weights to purchase history because it is more relevant to predicting the user's future behavior. This allows for adaptive learning of feature importance without the need for manual rules. It can capture complex interactions between features and handle input sequences of varying lengths. In practical applications, this feature selection method can significantly improve the interpretability and performance of the model because it allows the model to focus on the most relevant features, thereby reducing the impact of noise and improving the accuracy of predictions.

[0114] The data transformation task head employs a neural network architecture. Its primary purpose is to generate a [p × K] transformation matrix for each key feature of the first output, where p represents the dimension of the key feature and K represents the number of transformation methods. Each element in this transformation matrix represents the probability of applying the corresponding transformation method. This can be implemented using a multi-layer feedforward neural network. The input layer receives the key features, the hidden layer performs nonlinear transformations, and the output layer generates a [p × K]-dimensional vector, which is then reshaped into the desired matrix shape. Activation functions such as ReLU can be used, and the final layer uses a softmax function to ensure that the sum of probabilities for each row is 1. For example, for a numerical feature, K might include normalization, min-max scaling, or logarithmic transformation; for categorical features, it might include one-hot encoding, label encoding, or frequency encoding. During training, the cross-entropy loss function can be used to optimize network parameters through backpropagation. In practice, different transformation methods can be applied to a small portion of the data to evaluate their impact on downstream tasks, using these results as training labels. This allows for automatic recommendation of the most suitable transformation method for different features, eliminating the need for manual trial of every combination. It can adapt to different data distributions and model requirements, and improve the efficiency and effectiveness of feature engineering.

[0115] The anomaly detection task head uses an autoencoder architecture, primarily designed to learn normal patterns in the data, correlations between key features, and generate threshold vectors for identifying anomalies. The autoencoder consists of two parts: an encoder and a decoder. The encoder compresses the input data into a low-dimensional latent space, while the decoder attempts to reconstruct the original data from this latent representation. In implementation, the encoder network first compresses a high-dimensional vector array into a low-dimensional latent space. This process can be achieved using a multi-layer neural network, with each layer progressively reducing the number of neurons. The decoder network then attempts to reconstruct the original data from the latent representation. The training objective is to minimize the reconstruction error, typically using mean squared error as a loss function. This approach allows the autoencoder to capture the key features and normal patterns in the data. To learn the underlying relationships between different data sources, a correlation layer can be added to the latent space, using an attention mechanism or graph neural networks to model the interactions between features. This allows the autoencoder to not only learn the distribution of individual features but also capture the complex relationships between them. After training, the distribution of the reconstruction error is calculated using a validation set, and the threshold vector for anomaly detection is determined based on this distribution. Specifically, the 95th percentile of the reconstruction error for each feature can be used as the corresponding threshold. In practical applications, if the reconstruction error of a sample exceeds this threshold, it is marked as a potential anomaly. The advantage of this method is that it is unsupervised and does not require pre-labeled anomaly samples. It can adapt to complex data distributions and detect anomalous patterns that are difficult to describe with simple rules. In practice, this anomaly detection method can effectively identify outliers, abnormal patterns, or data quality issues in the data, thereby improving the reliability of data analysis and the effectiveness of model training. In addition, because it considers the correlation between features, it can detect complex anomalous patterns that may be overlooked in univariate analysis.

[0116] The feature interaction task head adopts a graph neural network structure. Its main purpose is to learn the interaction relationship between key features and output an adjacency matrix representing the interaction strength between feature pairs. In the implementation process, each key feature is first regarded as a node in the graph, and the initial feature vector is used as the attribute of the node. Then, the structure of the graph is initialized using the potential correlation relationship between different data sources. For example, the initial edge weights can be set based on the correlation coefficient between features or prior knowledge. Next, a multi-layer graph convolutional network (GCN) is applied to learn the interaction between nodes. Each layer of GCN can be expressed as: in is the normalized adjacency matrix, H (l) is the node feature of the lth layer, W (l)is a learnable weight matrix, and σ is a nonlinear activation function. Through multiple layers of propagation, each node aggregates information from its neighbors, thereby capturing a wider range of feature interactions. During training, the performance of downstream tasks can be used as guidance. For example, for prediction tasks, the output of the GCN can be connected to a prediction layer, and the prediction error can be used to backpropagate updates to the graph structure. The output of the last GCN layer is used to construct the final adjacency matrix, where each element represents the strength of the interaction between corresponding feature pairs. In specific implementations, the GCN can be enhanced with an attention mechanism, allowing the model to dynamically adjust the importance of different neighbors. This allows for the automatic discovery and quantification of complex nonlinear relationships between features, free from the constraints of artificial assumptions. It can capture high-order feature interactions that are difficult to detect with traditional methods, providing richer feature combination information. In practical applications, this task head can significantly improve feature engineering results, providing more informative feature sets for downstream tasks. The output adjacency matrix can not only be used to generate interaction features but also provide interpretable insights into the dependencies between features.

[0117] In one implementation of this embodiment, the output layer includes a scoring layer, a Transformer layer, a selection layer, and an optimization output layer. Five types of data processing rules are input into the output layer to obtain n types of data processing rules, including the following steps:

[0118] S510, inputting the five types of data processing rules into the scoring layer to obtain a quality score for each data processing rule;

[0119] S520, inputting five types of data processing rules into the Transformer layer to obtain the synergy value between the data processing rules;

[0120] S530: Input the quality score and the synergy value into the selection layer, calculate the attention weight of each data processing rule, and use the attention weight greater than the preset threshold as the target attention weight, where the number of target attention weights is n;

[0121] S540, determining a data processing rule corresponding to each target attention weight, and obtaining n types of data processing rules;

[0122] S550: Input n types of data processing rules into the optimization output layer, obtain the data processing sequence of the n types of data processing rules, and output the n types of data processing rules and the data processing sequence.

[0123] Five types of data processing rules are input into the scoring layer to obtain a quality score for each data processing rule. The scoring layer uses a multi-layer perceptron structure to evaluate the quality of each data processing rule. For implementation, a three-layer neural network can be designed, with the number of nodes in the input layer equal to the number of features in the data processing rules, and the hidden layer using the ReLU activation function.

[0124] The output layer uses a sigmoid activation function to generate a quality score between 0 and 1. For example, assuming each data processing rule has 10 features describing its properties (such as complexity, scope of application, and execution time), the input layer can have 10 nodes, the hidden layer can have 20 nodes, and the output layer can have 1 node. For each input data processing rule, a quality score is calculated through forward propagation. During the training phase, historical data or expert-annotated quality scores can be used as supervisory signals, and network parameters can be optimized through backpropagation. To improve the model's generalization, dropout techniques and L2 regularization can be employed. Furthermore, an attention mechanism can be introduced to teach the model to focus on the importance of different features. In practical applications, quality scores can serve as an important basis for screening and ranking data processing rules. For example, if a data cleaning rule has a quality score of 0.85, while another feature selection rule has a score of 0.62, this indicates that the former is more reliable or effective. The introduction of a scoring layer enables automatic evaluation and comparison of the quality of different types of data processing rules, providing a quantitative basis for subsequent rule selection and combination, thereby improving the efficiency and reliability of the entire data processing pipeline.

[0125] Five types of data processing rules are input into the Transformer layer to obtain the synergy values ​​between the data processing rules. The Transformer layer uses the self-attention mechanism to learn the relationships between different data processing rules. In implementation, each data processing rule is first encoded into a vector representation, and then the synergy between the rules is calculated using a multi-head self-attention mechanism. The specific steps are as follows: 1) Positional encoding is performed on each rule to preserve the sequential information of the rule sequence. 2) The encoded rules are input into the multi-head self-attention layer. Assume there are 8 attention heads, each with a dimension of 64. 3) For each attention head, the query matrix Q, key matrix K, and value matrix V are calculated, and then the attention score is calculated. 4) The outputs of all attention heads are concatenated and passed through a feedforward neural network. 5) Residual connections and layer normalization are added to stabilize training. This process can be stacked with multiple layers (e.g., 6 layers) to capture more complex relationships. Finally, the output of the Transformer layer can be interpreted as a synergy value matrix between rules, where each element represents the synergy strength between two rules. For example, if the synergy value between data cleaning rule A and feature selection rule B is 0.75, this indicates that the two rules are highly compatible or complementary. The introduction of the Transformer layer enables the model to automatically discover implicit relationships between rules, including complementarity, conflict, or sequential dependencies. By learning these synergistic relationships, it can more intelligently combine and sort data processing rules, avoid redundant operations, and maximize the synergy between rules.

[0126] The quality score and synergy value are input into the selection layer to calculate the attention weight of each data processing rule, and the attention weight greater than the preset threshold is used as the target attention weight, where the number of target attention weights is n. The core of the selection layer is to calculate the attention weight by combining the quality and synergy relationship of the rules, so as to screen out the most important and relevant rules. The implementation process is as follows: 1) For each rule i, its quality score qi is multiplied by the average synergy value ci of all rules to obtain the initial importance score si = qi × ci. 2) Use the softmax function to normalize these scores into attention weights: w i =exp(s i ) / ∑ j exp(s j). 3) Set a preset threshold t (such as 0.1) and select rules with attention weights greater than t. The weights of these rules are called target attention weights. 4) If the number of rules greater than the threshold is not equal to n, the threshold can be dynamically adjusted until exactly n rules are selected. For example, suppose there are 5 rules, their quality scores are [0.8, 0.7, 0.9, 0.6, 0.75], and their average synergy values ​​are [0.7, 0.65, 0.8, 0.55, 0.7]. The calculated attention weights are [0.22, 0.18, 0.28, 0.13, 0.19]. If n = 3 and the threshold t = 0.2, the target attention weights selected are [0.22, 0.28, 0.19], and the corresponding rules are the 1st, 3rd, and 5th. The introduction of the selection layer enables the adaptive selection of the most important and coordinated subset of rules, taking into account both the quality of individual rules and the synergy between rules. This mechanism can effectively balance the individual quality and overall synergy of rules, ensuring that the selected rule sets are both high-quality and complementary, thereby improving the efficiency and effectiveness of data processing.

[0127] Determine the data processing rules corresponding to each target attention weight, obtaining n types of data processing rules. This step maps the target attention weights selected in the previous step back to the original data processing rules, resulting in the final n selected rules. The specific implementation process is as follows: 1) Maintain an index mapping to record the correspondence between each attention weight and the original rule. 2) For each target attention weight, find the corresponding original data processing rule through the index mapping. 3) Combine these rules into a new rule set, which is the final n types of data processing rules selected. For example, suppose the original five rules are: data cleaning rule A, feature selection rule B, data transformation rule C, anomaly detection rule D, and feature interaction rule E. If the target attention weights selected correspond to rules 1, 3, and 5, then the final n (here n = 3) types of data processing rules selected are: data cleaning rule A, data transformation rule C, and feature interaction rule E. In practical applications, each selected rule can be assigned its attention weight value, which can be used for subsequent rule sorting or weighted processing. Furthermore, detailed attributes of each rule, such as parameter settings and applicable conditions, can be recorded to facilitate subsequent fine-tuning and interpretation. This step is crucial because it crystallizes abstract attention weights into practical, executable data processing rules, providing clear operational guidance for subsequent data processing. This approach automatically selects the subset of rules most appropriate for the current data and task from a large pool of candidate rules, significantly improving the relevance and efficiency of data processing.

[0128] N types of data processing rules are input into the optimization output layer, which generates the data processing order for each of the n types of data processing rules and outputs the n types of data processing rules and their order. The optimization output layer's primary task is to determine the optimal execution order for the n selected data processing rules to maximize overall processing performance. The implementation process is as follows: 1) A sequence-to-sequence (Seq2Seq) model is constructed, with both the encoder and decoder employing LSTM (Long Short-Term Memory) architectures. 2) The encoder input is the feature representation of the n rules, including rule type, parameters, attention weights, and other information. 3) The decoder uses a beam search strategy to generate a rule sequence and predict the next rule to execute at each time step. 4) During the training phase, the model is optimized using reinforcement learning methods (such as policy gradients). The reward function can be based on improved data quality after simulated execution or downstream task performance. 5) During the inference phase, the rule execution sequence with the highest score is selected as the final output. For example, suppose the three selected rules are: data cleaning rule A, data transformation rule C, and feature interaction rule E. The optimized output layer outputs the execution order: A→C→E, meaning data cleaning should occur first, followed by data transformation, and finally feature interaction. In practical applications, specific execution parameters and conditions can be attached to each rule, such as which columns data cleaning rule A applies to and what thresholds are used. Furthermore, the expected effect and uncertainty estimates for each step can be output to facilitate manual intervention and adjustment. The introduction of the optimized output layer not only selects the most appropriate rules for the entire data processing pipeline but also optimizes their execution order, thereby maximizing the overall effectiveness of data processing. This automated rule sorting mechanism can capture the complex dependencies and interactions between rules, far surpassing fixed, manually designed processes and providing greater flexibility and efficiency for data processing.

[0129] This implementation method realizes the automatic evaluation, selection and optimization of data processing rules through multi-level intelligent processing. The scoring layer performs quality assessment on individual rules and provides a basis for rule screening. The Transformer layer learns the collaborative relationship between rules and captures complex interaction effects. The selection layer combines quality and collaborative information to intelligently screen out the most important subset of rules. The optimized output layer further determines the optimal execution order of the rules. This end-to-end automated process not only improves the efficiency and quality of data processing, but also can adapt to different data characteristics and task requirements. Through learning and optimization, its decision-making ability can be continuously improved, providing intelligent data processing support, and transforming the traditional fixed data processing process into dynamic and adaptive intelligence, greatly improving the effect and flexibility of data preprocessing.

[0130] In one implementation of this embodiment, data processing rules are used to process the initial multi-source dataset to obtain a processed multi-source dataset, including the following steps:

[0131] S610 , performing data processing on the initial multi-source data set using a data processing sequence and n types of data processing rules to obtain a processed multi-source data set.

[0132] The initial multi-source dataset is processed using a data processing sequence and n types of data processing rules to obtain a processed multi-source dataset. This step is the core of the entire data processing process, applying the rules selected and optimized in the previous steps to the initial dataset. The specific implementation process is as follows:

[0133] First, we execute each data processing rule in sequence according to the data processing order given by the optimized output layer. Assume that the selected rules include data cleaning, feature selection, and data transformation, and the optimized order is: data cleaning → feature selection → data transformation.

[0134] 1. Data cleaning rules: These include handling missing values, removing outliers, and eliminating duplicate data. For example, for missing values, you can choose an appropriate filling method based on the data type and distribution characteristics, such as using the mean or median for numerical features, the mode for categorical features, or using more complex interpolation methods such as K-nearest neighbor interpolation. For outliers, you can use the Z-score method to detect and handle them. For example, if the Z-score of a numerical feature is greater than 3 or less than -3, it can be considered an outlier and replaced with the upper and lower limits.

[0135] 2. Feature selection rules: This step aims to select the most relevant and important subset of features from a large number of features. Various methods can be used, such as correlation-based methods, feature importance-based methods (such as random forest feature importance), or wrapper methods (such as recursive feature elimination). For example, using the Pearson correlation coefficient method, calculate the correlation between each feature and the target variable and select features with an absolute value of the correlation coefficient greater than 0.3. Or using a random forest model, calculate the importance score of each feature and select features with the top 20% scores.

[0136] 3. Data transformation rules: These include operations such as standardization, normalization, and logarithmic transformation. For example, a numerical feature can be standardized to have a mean of 0 and a standard deviation of 1: (x - μ) / σ, where x is the original value, μ is the mean, and σ is the standard deviation. For features with skewed distributions, a logarithmic transformation can be applied: log(1 + x) to make the distribution closer to normal.

[0137] When executing each rule, you need to consider its specific parameter settings and applicable conditions. For example, the outlier threshold in a data cleaning rule, the feature count or importance threshold in a feature selection rule, and the transformation method selection in a data transformation rule. These parameters may have been optimized in previous steps or can be adjusted dynamically based on the actual data.

[0138] Furthermore, data consistency and integrity must be maintained during the processing process. For example, this includes ensuring consistent coding of categorical variables, handling newly emerged categories, and maintaining logical relationships between features. Furthermore, detailed logs must be kept for each processing step, including the amount of data affected and the statistical characteristics that changed, to facilitate subsequent analysis and adjustments.

[0139] In practical applications, parallel computing techniques can be used to speed up the processing process, especially when dealing with large data sets. For example, a distributed computing framework such as Apache Spark can be used to split the data into multiple parts, process them in parallel, and then merge the results.

[0140] Finally, the processed dataset is evaluated for quality, including data integrity checks, feature distribution analysis, and feature correlation analysis, to ensure that the processed dataset meets the expected quality standards. If problems are found, it may be necessary to go back to the previous steps, adjust the rules or parameters, and re-execute the data processing process.

[0141] This implementation achieves automated conversion from initial multi-source datasets to high-quality processed datasets through an intelligent data processing process. It not only applies optimally selected data processing rules but also considers the optimal execution order between rules, thereby maximizing data processing effectiveness. It can adapt to different types of data and task requirements, providing a flexible and efficient data preprocessing solution. Through a systematic cleaning, feature selection, and conversion process, data quality is significantly improved, noise and redundant information are reduced, and the data's expressiveness is enhanced. This provides better input for subsequent machine learning model training and data analysis.

[0142] In one implementation of this embodiment, before inputting the initial multi-source dataset into the large language model to obtain the data processing rules, the following steps are included:

[0143] S710: When the domain type is a sensitive domain type, a desensitizing encryption strategy is used to perform desensitizing encryption processing on the initial multi-source dataset to obtain a processed initial multi-source dataset.

[0144] If the domain type is sensitive, a desensitizing encryption strategy is used to desensitize and encrypt the initial multi-source dataset, resulting in a processed initial multi-source dataset. The core goal of desensitizing encryption is to protect data privacy and security while preserving the data's usability and analytical value as much as possible.

[0145] First, the initial multi-source dataset needs to be thoroughly scanned and analyzed to identify all data fields that may contain sensitive information. This sensitive information may include personal identification information (such as name, ID number, and phone number), financial information (such as bank account number and credit card information), and medical and health information. This identification process can be achieved through techniques such as regular expression matching, keyword search, and pattern recognition. For example, a regular expression can be used to match the 18-digit Chinese ID number.

[0146] Next, based on the identified sensitive information types, select an appropriate desensitization strategy for each type. Desensitization strategies include: 1. Data masking: Replace some or all characters of sensitive information with specific characters, such as asterisks or pound signs. 2. Data replacement: Replace the original data with fictitious but similarly formatted data. For example, replace the real name "Zhang San" with the randomly generated "Li Si". 3. Data generalization: Replace specific values ​​with more general categories. For example, replace the specific age of 35 with the age range of "30-40 years old". 4. Data shuffling: Randomly shuffle the data in the same column to destroy the association between sensitive information and other information. This method is suitable for scenarios where the overall statistical characteristics need to be retained but the specific corresponding relationships do not need to be retained. 5. Data sampling: Randomly extract a portion of data from the original data set to reduce the total amount of sensitive information.

[0147] For data requiring stronger protection, encryption technology can be used. Common encryption algorithms include symmetric encryption (such as AES and DES) and asymmetric encryption (such as RSA). For example, sensitive fields can be encrypted using the AES-256 algorithm. The encrypted data becomes a string of seemingly random characters that can only be decrypted by those with the correct key.

[0148] In practice, a combination of desensitization and encryption techniques can be used, depending on the sensitivity of the data and the usage scenario. For example, for personal names, the data can be replaced first, and then the replaced pseudonym can be encrypted. For bank card numbers, partial masking (retaining only the last four digits) can be performed first, and then the remaining visible portion can be encrypted.

[0149] Furthermore, data consistency and availability must be considered. For example, if the same sensitive information appears in multiple data sources (such as a person's name appearing in different tables), this information should be consistently desensitized or encrypted to maintain data relevance. Furthermore, for numerically sensitive data requiring statistical analysis, desensitization methods that preserve statistical characteristics, such as adding noise or perturbation, can be used to preserve the analytical value of the data while protecting privacy.

[0150] During the desensitizing and encryption process, a comprehensive logging system must be established to record information such as the processing method, processing time, and processing personnel for each field, to facilitate subsequent audits and problem tracking. Furthermore, a strict key management policy must be established to ensure the secure storage and appropriate use of encryption keys.

[0151] This implementation method utilizes a desensitizing and encryption strategy in sensitive areas, ensuring comprehensive and secure processing of the initial multi-source dataset. This step effectively protects personal privacy and sensitive information. While maintaining essential analytical value, the processed dataset significantly reduces the risk and potential harm of data leakage. This approach, balancing security and usability, provides a secure and reliable foundation for subsequent data analysis and modeling. It also enhances the trust of data subjects and stakeholders in the data processing process, facilitating widespread application and value mining of data.

[0152] In one implementation of this embodiment, the desensitization encryption strategy includes the following steps:

[0153] S810, identifying a target sensitive data segment in the structured data of the initial multi-source data set;

[0154] S820. Generate a corresponding token for the target sensitive data segment, where the token is unique.

[0155] S830. Create a mapping relationship between the token and the target sensitive data segment in a preset mapping database, and use the token to replace the target sensitive data segment.

[0156] S840: Identify target sensitive data segments within the structured data in the initial multi-source dataset. This step is the starting point for the desensitization and encryption strategy, aiming to accurately locate sensitive information that requires protection. Implementing this step requires the combined use of multiple technologies to ensure efficient and accurate identification of all potentially sensitive data.

[0157] First, a comprehensive library of sensitive data types needs to be established, including but not limited to personal identification information (such as name, ID number, passport number), contact information (such as phone number, email address), financial information (such as bank account number, credit card number), medical and health information, etc. This library should be customized according to different industry standards and laws and regulations to ensure that all relevant sensitive data types are covered.

[0158] Next, a multi-layered recognition approach is employed to scan the initial multi-source dataset. This typically involves the following aspects:

[0159] 1. Regular expression matching: Use predefined regular expressions to match sensitive data in a specific format. 2. Keyword matching: Use a predefined list of sensitive words to identify potentially sensitive fields. For example, search for field names or data content containing keywords such as "password", "account", and "SSN". 3. Semantic analysis: Use natural language processing technology to understand the context and semantics of the field to identify sensitive information, which helps to identify sensitive data that does not follow a fixed format. 4. Machine learning model: Train specialized machine learning models to identify complex or hidden sensitive data patterns. These models can learn to identify various forms of sensitive information, including data that has been artificially concealed or deformed. 5. Metadata analysis: Analyze database schemas, table structures, and field properties to identify fields containing sensitive information. For example, fields marked as "encrypted" or "confidential" contain sensitive data.

[0160] In practice, these methods are often combined. For example, for a data table containing customer information, metadata analysis can be used to identify fields containing sensitive information. Regular expression matching and keyword searches can then be applied to these fields. For fields with complex content, semantic analysis and machine learning models can be used for further inspection.

[0161] Furthermore, the identification process must consider the context and relevance of the data. Some data may not be sensitive on its own, but may reveal sensitive information when combined with other data. For example, a person's date of birth is not sensitive on its own, but when combined with their name and address, it becomes sensitive personal identification information.

[0162] During the identification process, detailed log records need to be established to record the location, type and identification method of each identified sensitive data segment.

[0163] Generate a unique token for each sensitive data segment. This token is designed to create a unique alternative identifier (or token) for each identified sensitive data segment. This uniqueness must be ensured, and the token itself must not contain any information that could lead to inference about the original data.

[0164] The token generation process usually involves the following key elements: 1. Uniqueness guarantee: Use a random number generator or cryptographic hash function to create the token. Common methods include UUID (universally unique identifier), a cryptographically secure pseudo-random number generator (CSPRNG), or a cryptographic hash function (such as SHA-256) combined with a salt value. 2. Length and complexity: The token should have sufficient length and complexity to prevent guessing attacks. Typically, at least 128 bits (16 bytes) of random data are used to generate the token. 3. Readability and usability: Depending on actual needs, the generated random data can be encoded into a form that is easier to use and store, such as Base64 encoding or hexadecimal representation. 4. Context relevance: In some cases, it is necessary to embed some metadata (such as a data type identifier) ​​in the token, but ensure that this information does not disclose the content of the original data. 5. Time sensitivity: For sensitive data that needs to be updated regularly, consider including a timestamp or version information in the token to support regular rotation of the token.

[0165] One implementation method is: first, generate a 16-byte (128-bit) random number for each sensitive data segment, and a cryptographically secure random number generator can be used. Next, combine this random number with some contextual information (such as a data type identifier and a timestamp). For example, if the sensitive data is a card number, the prefix "CC_" and the current timestamp can be added. The combined data is then processed through the SHA-256 hash function to obtain a 256-bit hash value. This further enhances the uniqueness and unpredictability of the token. Finally, the obtained hash value is encoded in Base64 format for easy storage and transmission. Base64 encoding converts binary data into a string consisting of 64 different ASCII characters.

[0166] A mapping relationship between the token and the target sensitive data segment is created in a pre-set mapping database, and the token is used to replace the target sensitive data segment. This step is the final step in the desensitization encryption strategy, establishing the association between the token and the original sensitive data and completing the actual data replacement operation. This process must not only ensure the accuracy and security of the mapping relationship, but also the integrity and traceability of the entire replacement process.

[0167] The mapping database should have the following characteristics: 1. High security: Use strong encryption algorithms (such as AES-256) to encrypt stored sensitive data. 2. Access control: Implement strict access control to limit access to authorized personnel only. 3. Audit log: Record all access and modification operations on mapping relationships. 4. High availability: Use redundant storage and backup strategies to ensure data durability and reliability.

[0168] The mapping database can use a high-security database (such as encrypted PostgreSQL). The database structure includes the following fields: token, encrypted original data, data type, creation time, last access time, expiration time, etc.

[0169] The process for creating a mapping relationship is as follows: 1. Encrypt each target sensitive data segment. For example, use the AES-256 algorithm and a randomly generated key. 2. Store the encrypted data along with the corresponding token in the mapping database. 3. Securely store the encryption key in a separate key management system.

[0170] Next, the actual data replacement operation is performed. This process involves traversing all target sensitive data segments in the initial multi-source dataset and replacing them with corresponding tokens. The specific steps are as follows: 1. For each identified sensitive data segment, find its corresponding token. 2. Replace the original data with the token. 3. Update the data structure to ensure that the replaced data is formatted correctly and does not affect subsequent data processing. 4. Record each replacement operation, including the location, time, and data type involved.

[0171] This implementation first accurately identifies all sensitive data segments requiring protection, ensuring the integrity of the protection scope. Next, a unique, secure token is generated for each sensitive data segment, effectively concealing the characteristics of the original data. Finally, data desensitization is achieved by establishing a secure mapping relationship and replacing the original data. This not only significantly reduces the risk of data leakage but also maintains data availability and integrity. Furthermore, the traceability and auditability of the entire process are ensured, providing strong support for the compliant use and management of data, effectively protecting privacy and sensitive information.

[0172] In one implementation of this embodiment, the processed multi-source dataset is input into the domain adaptation module to obtain the final multi-source dataset, including the following steps:

[0173] S910. Input the processed multi-source dataset into the domain adaptation module, so that the domain adaptation module corrects the processed multi-source dataset according to the domain type and the target knowledge graph to obtain the final multi-source dataset.

[0174] The processed multi-source dataset is input into the domain adaptation module so that the module corrects the processed multi-source dataset according to the domain type and the target knowledge graph, thereby obtaining the final multi-source dataset.

[0175] The core function of the domain adaptation module is to adapt and optimize datasets from various sources, which may contain inconsistent or inaccurate information, based on domain-specific knowledge and rules. This process involves multiple complex subtasks, including data cleaning, entity alignment, relationship reasoning, and knowledge completion. First, the domain adaptation module loads the corresponding rule sets and knowledge base based on the predefined domain type. For example, if the domain type is healthcare, the module loads specialized knowledge such as medical terminology, disease classification systems, and drug interaction rules. This knowledge is typically stored in a structured format, such as an ontology or knowledge graph. Next, the module performs a preliminary analysis of the input multi-source dataset to identify entities, attributes, and relationships within the data. This process may utilize natural language processing (NLP) techniques such as named entity recognition (NER) and relationship extraction. For example, in a medical record, entities such as "patient," "symptom," "diagnosis," and "treatment" and the relationships between them may be identified. The domain adaptation module then performs entity alignment or ontology matching of the identified entities and relationships with concepts in the target knowledge graph. This is accomplished using methods such as string matching, semantic similarity calculation, and structural similarity analysis. For example, "heart disease" in the data may be aligned to the concept of "coronary heart disease" in the knowledge graph. Based on the entity alignment, the module will check whether the relationships in the data are consistent with the domain knowledge. This includes verifying the validity of the relationship, inferring implicit relationships, resolving conflicting relationships, etc. For example, if the data shows that a patient is taking two drugs at the same time that are known to have serious interactions, this potential problem will be marked. Next, the domain adaptation module will perform knowledge completion. This process attempts to fill in the gaps in the data and add valuable information. It may be based on rule reasoning or machine learning models. For example, if a patient's symptoms and test results highly match a certain disease, but the diagnosis information is missing, the system may infer a possible diagnosis and add it to the data. Throughout the process, the domain adaptation module will constantly refer to the target knowledge graph to ensure that the corrected data is consistent with the domain knowledge.

[0176] In this embodiment, the multi-source dataset is comprehensively optimized and enriched through the processing of the domain adaptation module. First, the entities and relationships in the data are accurately identified and aligned with standard terminology, eliminating possible ambiguity and inconsistency. Second, through interaction with the target knowledge graph, the data is given deep domain semantics, making it more in line with professional standards. Third, the knowledge completion process fills the gaps in the data and adds potentially valuable information. Finally, the entire process ensures the consistency, integrity and domain relevance of the data. The final multi-source dataset processed in this way not only retains the core information of the original data, but also incorporates rich domain knowledge, greatly improving the quality and practicality of the data. This lays a solid foundation for subsequent data analysis, decision support and knowledge discovery, and can support more accurate and reliable applications.

[0177] The present application also provides a data processing flow optimization system, including:

[0178] a memory configured to store instructions; and

[0179] The processor is configured to call instructions from the memory and implement the above-mentioned data processing flow optimization method based on the large language model when executing the instructions.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of the processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0184] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0185] Memory includes non-permanent memory in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0186] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0187] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0188] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A data processing flow optimization method based on a large language model, characterized in that: Applied to electronic devices, the electronic devices are deployed with a large language model and a domain adaptation module. The large language model includes an input embedding layer, a multi-layer Transformer block, a parallel task head, and an output layer. The output layer includes a scoring layer, a Transformer layer, a selection layer, and an optimized output layer. The method includes: Obtain multi-source data from different data sources; Input the multi-source data into the domain adaptation module to obtain an initial multi-source data set, wherein the domain adaptation module is used to identify the domain type of the multi-source data and modify and expand the multi-source data according to the domain type; Input the initial multi-source dataset into the input embedding layer to obtain a high-dimensional vector group of the initial multi-source dataset; Input the high-dimensional vector group into a multi-layer Transformer block to obtain a first output, which includes data anomaly patterns, key features, and potential correlations between different data sources in the high-dimensional vector group; Input the first output into the parallel task head to obtain five types of data processing rules. The parallel task head includes five task heads, each of which is used to generate a type of data processing rule. The five types of data processing rules include data cleaning strategy, feature selection strategy, data conversion strategy, anomaly detection strategy, and feature interaction strategy. Input the five types of data processing rules into the scoring layer to obtain the quality score of each data processing rule; Input five types of data processing rules into the Transformer layer to obtain the synergy value between the data processing rules; Input the quality score and synergy value into the selection layer, calculate the attention weight of each data processing rule, and take the attention weight greater than the preset threshold as the target attention weight, where the number of target attention weights is n; Determine the data processing rules corresponding to each target attention weight, and obtain n types of data processing rules; Input n types of data processing rules into the optimized output layer, obtain the data processing order of n types of data processing rules, and output n types of data processing rules and data processing order, 0<n≤4, and n is an integer; Using data processing rules to process the initial multi-source data set to obtain a processed multi-source data set; The processed multi-source dataset is input into the domain adaptation module to obtain the final multi-source dataset, wherein the domain adaptation module is used to adjust the processed multi-source dataset according to the domain type.

2. The method according to claim 1, characterized in that Multi-source data includes structured data and unstructured data. The domain adaptation module includes multiple domain databases, a multi-source classification and recognition module, a knowledge graph module, and an expansion module. Each domain database stores the data rules of each domain. The multi-source data is input into the domain adaptation module to obtain the initial multi-source data set, including: Input structured data and unstructured data into the multi-source classification and recognition module to obtain the field types corresponding to both structured data and unstructured data; Inputting the domain type into the knowledge graph module, so that the knowledge graph module determines a target domain database from multiple domain databases according to the domain type, and determines a target knowledge graph according to the target domain database; The target knowledge graph is input into the expansion module so that the expansion module expands and corrects the multi-source data according to the target knowledge graph to obtain an initial multi-source data set.

3. The method according to claim 1, characterized in that Parallel task heads include data cleaning task head, feature selection task head, data conversion task head, anomaly detection task head and feature interaction task head; The structure of the data cleaning task head is a multi-layer perceptron structure. The data cleaning task head is used to generate multiple binary decisions based on the key features of the first output. Each binary decision corresponds to a key feature, and each binary decision is used to represent the cleaning probability of the corresponding key feature. The structure of the feature selection task head is based on the attention mechanism structure. The feature selection task head is used to assign corresponding importance weights to each key feature of the first output. The structure of the data conversion task head is a neural network structure. The data conversion task head is used to generate a [p×K] transformation matrix for each key feature of the first output. p represents the dimension of the key feature, K represents the number of transformation methods, and each element in the transformation matrix represents the applicable probability of the corresponding transformation method; the structure of the anomaly detection task head is an autoencoder structure. The anomaly detection task head is used to learn the data normal pattern based on the data anomaly pattern in the high-dimensional vector group, and learn the correlation between key features based on the potential correlation relationship between different data sources, and generate a threshold vector for identifying anomalies based on the data normal pattern and correlation. The structure of the feature interaction task head is a graph neural network structure. The feature interaction task head is used to learn the interaction relationship between key features based on the potential correlation relationship and key features of different data sources, and output an adjacency matrix representing the interaction strength between feature pairs.

4. The method according to claim 1, wherein The initial multi-source dataset is processed using data processing rules to obtain a processed multi-source dataset, including: The data processing sequence and n types of data processing rules are used to process the initial multi-source data set to obtain a processed multi-source data set.

5. The method according to claim 2, characterized in that Before inputting the initial multi-source dataset into the large language model and obtaining the data processing rules, the following steps are involved: When the domain type is a sensitive domain type, a desensitizing encryption strategy is used to perform desensitizing encryption processing on the initial multi-source dataset to obtain a processed initial multi-source dataset.

6. The method according to claim 5, characterized in that Desensitization encryption strategies include: Identifying target sensitive data segments in structured data of an initial multi-source dataset; Generate a corresponding token for the target sensitive data segment, where the token is unique; Create a mapping relationship between the token and the target sensitive data segment in a preset mapping database, and use the token to replace the target sensitive data segment.

7. The method according to claim 2, characterized in that The processed multi-source dataset is input into the domain adaptation module to obtain the final multi-source dataset, including: The processed multi-source dataset is input into the domain adaptation module so that the domain adaptation module corrects the processed multi-source dataset according to the domain type and the target knowledge graph to obtain the final multi-source dataset.

8. A data processing process optimization system, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the data processing flow optimization method based on a large language model according to any one of claims 1 to 7 when executing the instructions.

Images