A data integration method based on diffusion model and self-supervised outlier detection

Abstract

The present invention discloses a data integration method based on a diffusion model and self-supervised outlier detection, which relates to the technical field of big data processing and includes the following steps: establishing a unified metadata management system; ensuring that the meaning, format and scope of each data field are accurately described and understood by collecting and maintaining metadata of each data source; achieving a unified data format through data standardization operations; the standardization method includes defining a unified data format and standard; establishing data conversion and mapping rules to map fields in heterogeneous data sources to a unified data model; constructing a data completion scheme based on a diffusion model to fill in missing values; using a self-supervised anomaly detection algorithm to detect and delete outliers, effectively solving the problem in the prior art that a lot of effort is needed to unify and convert data from each system, and the cleaning process is complicated and time-consuming due to data quality issues, thereby providing high-quality support for multi-source data fusion empowerment.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to a data integration method based on diffusion model and self-supervised outlier detection. Background Art

[0002] With the advent of the big data era, government and corporate departments have accumulated vast amounts of heterogeneous, multi-source data. To better understand and utilize this data, breaking down departmental information silos and implementing intelligent and reliable data integration methods are urgently needed. Integrating data resources from diverse sources, breaking down information silos, and enabling data sharing and interoperability not only improves decision-making transparency and responsiveness, but also makes services more precise and efficient, better meeting people's diverse needs. Data fusion also helps unlock the potential value of data, promotes optimal resource allocation, and enhances the relevance and effectiveness of services. Furthermore, establishing a unified data fusion platform can strengthen data security management, effectively prevent the risk of data leakage and misuse, and safeguard personal information.

[0003] The process of integrating multi-source data presents numerous difficult data cleaning challenges. First, different departments and systems employ varying data formats and naming conventions, making it difficult to ensure data consistency and standardization, necessitating a significant effort in achieving unified conversion. Second, data quality issues such as duplicate data, missing values, and erroneous data are common, particularly in large-scale datasets, making the cleaning process complex and time-consuming. Furthermore, data timeliness demands high standards, requiring timely updates and synchronization of the latest information. However, the varying update frequencies and mechanisms across different data sources complicate data cleaning. Therefore, it is necessary to apply efficient cleaning algorithms and establish effective data synchronization mechanisms to address the complexity of multi-source data. Summary of the Invention

[0004] The purpose of the present invention is to provide a data integration method based on diffusion model and self-supervised outlier detection to solve the problems raised by the above background technology.

[0005] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0006] The present invention is a data integration method based on diffusion model and self-supervised outlier detection, comprising the following steps:

[0007] S1. Establish a unified metadata management system; by collecting and maintaining metadata from various data sources, ensure that the meaning, format, and scope of each data field are accurately described and understood;

[0008] S2. Achieve a unified data format through data standardization operations; standardization methods include defining unified data formats and standards;

[0009] S3. Establish data conversion and mapping rules to map fields in heterogeneous data sources into a unified data model;

[0010] S4. Build a data completion solution based on the diffusion model to fill in missing values;

[0011] S5. Use the self-supervised anomaly detection algorithm to detect and delete outliers.

[0012] Furthermore, in step S4, a data completion scheme is constructed based on the diffusion model to fill in the missing values, which includes the following sub-steps:

[0013] S41. Fill in missing values. The basic principle of the diffusion model is to gradually add noise to the data until it approaches a simple distribution. In this process, given the initial data x0, the noise data x in step t is t , which is expressed as follows,

[0014] ;

[0015] in, is the time step that controls the rate of increase of noise, I is the identity matrix, which is used to define the covariance structure of Gaussian distribution, N is Gaussian distribution; q represents the state x in the previous step. t-1 Under the condition that the current step x t The conditional probability distribution of ;

[0016] S42, reverse processing the data, from the noise data x t Restore to the original data x0. Each step in this process is described as a Markov chain, which is expressed as follows:

[0017] ;

[0018] in, is the mean parameter predicted by the model, which is adjusted through continuous learning of the model. is the covariance matrix, which is adjusted through continuous learning of the model. is the conditional probability distribution of the inverse process; N is a Gaussian distribution;

[0019] S43. In the training process of the diffusion model, the variational lower bound is used as the loss function, which is expressed as,

[0020] ;

[0021] Where L is the overall loss function of the diffusion model, and the goal is to minimize the loss to optimize the model parameters; T is the total number of time steps of the diffusion process; E q is the expectation under distribution q, D klis the Kullback-Leibler divergence, which measures the difference between two distributions, q is the conditional distribution in the forward process, is the conditional distribution in the reverse process.

[0022] Furthermore, in step S5, the self-supervised anomaly detection algorithm is used to detect and delete outliers, which includes the following sub-steps:

[0023] S51. Standardize and normalize the input data to eliminate the dimensional differences between different features, making it easier for the model to learn the essential characteristics of the data during training, thereby improving the robustness of the model and ensuring data consistency and quality.

[0024] S52, divide the data into positive sample pairs and negative sample pairs, train the model to extract meaningful feature representations; during the training process, use the NT-Xent loss function to optimize the feature extractor,

[0025] ;

[0026] Among them, z i and z j is the feature vector of the positive sample pair, representing the features extracted by the encoder, τ is the temperature parameter used to scale the vector inner product, N is the number of samples in the batch, K is the index of the negative sample, all samples in the batch except sample i; is an exponential function used to convert similarity scores into positive numbers; Represents the sum of similarities of all negative sample pairs;

[0027] S53, after feature extraction, performing cluster analysis on the features, dividing the feature space into multiple cluster centers;

[0028] S54. For each test sample, calculate its Euclidean distance to each cluster center and evaluate its abnormality based on the distance. The calculation formula is:

[0029] ;

[0030] Among them, Z is the feature vector of the test sample, is the cluster center;

[0031] S55. Calculate the score by distance, set the threshold according to the actual situation, and then determine the outliers and delete them.

[0032] The present invention has the following beneficial effects:

[0033] This application defines unified data formats and standards by establishing a unified metadata management system; establishes data conversion and mapping rules to map fields in heterogeneous data sources to a unified data model; and extracts key, high-quality data from the data after integration and puts it into the final data pool. This effectively solves the problem that the existing technology requires a lot of effort to unify and convert data from various systems, and the cleaning process is complicated and time-consuming due to data quality issues. It provides high-quality support for multi-source data fusion, helps build smart cities, and promotes the digital transformation of the entire city.

[0034] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 This is a flow chart of the data fusion method based on diffusion model and self-supervised outlier detection of the present invention.

[0037] Figure 2 Schematic diagram of the model learning reversal generation process of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0039] See also Figure 1-2 As shown, the present invention is a data integration method based on diffusion model and self-supervised outlier detection, comprising the following steps:

[0040] S1. Establish a unified metadata management system. By collecting and maintaining metadata from various data sources, ensure that the meaning, format, and scope of each data field are accurately described and understood. A metadata management system is a key tool for integrating heterogeneous data. It defines and manages the structure, attributes, and relationships of various data sources. A metadata management system not only provides a data dictionary and catalog but also helps track data sources and change history, thereby maintaining data consistency and integrity during the data integration process.

[0041] S2. Achieve a unified data format through data standardization. Standardization methods include defining unified data formats and standards, such as unified date formats, address formats, and encoding standards. In practice, regular expressions and string processing functions can be used to format data. For example, dates in different formats can be converted to a unified "YYYY-MM-DD" format, or address information can be broken down into standardized fields such as street, city, and postal code. The standardization process must fully consider the characteristics of each data source to ensure that the meaning of the original data is not lost during the conversion process.

[0042] S3. Establish data conversion and mapping rules to map fields from heterogeneous data sources into a unified data model. Because data structures across different data sources can differ significantly, directly merging data can lead to data distortion or inconsistency. Therefore, data conversion and mapping rules are necessary to map fields from heterogeneous data sources into a unified data model. The project will use data conversion scripts for data conversion. Data conversion scripts automate the data extraction, conversion, and loading processes, ensuring that data is converted and merged according to predefined rules.

[0043] S4. Build a data completion solution based on the diffusion model to fill in missing values. The basic principle of the diffusion model is to gradually add noise to the data until it approaches a simple distribution (such as a Gaussian distribution). Then, the model learns how to reverse this process and gradually restore the original data from the noisy data. This process can be regarded as a generative process, in which each step attempts to reduce the uncertainty of the data. Figure 2 shown.

[0044] In step S4, a data completion scheme is constructed based on the diffusion model to fill in missing values, which includes the following sub-steps:

[0045] S41. Fill in missing values. The basic principle of the diffusion model is to gradually add noise to the data until it approaches a simple distribution. In this process, given the initial data x0, the noise data x in step t is t , which is expressed as follows,

[0046] ;

[0047] in, is the time step that controls the rate of increase of noise, I is the identity matrix, which is used to define the covariance structure of Gaussian distribution, N is Gaussian distribution; q represents the state x in the previous step. t-1 Under the condition that the current step x t The conditional probability distribution of step S41 provides the necessary conditions for the subsequent inverse process, so that the model can gradually recover the original data from the noise data.

[0048] S42, reverse processing the data, from the noise data x t Restore to the original data x0. Each step in this process is described as a Markov chain, which is expressed as follows:

[0049] ;

[0050] in, is the mean parameter predicted by the model, which is adjusted through continuous learning of the model. is the covariance matrix, which is adjusted through continuous learning of the model. is the conditional probability distribution of the reverse process; N is a Gaussian distribution; by continuously learning and optimizing parameters and , the model can gradually improve the accuracy of recovering original data from noisy data, thereby achieving effective filling of missing values.

[0051] S43. In the training process of the diffusion model, the variational lower bound is used as the loss function, which is expressed as,

[0052] ;

[0053] Where L is the overall loss function of the diffusion model, and the goal is to minimize the loss to optimize the model parameters; T is the total number of time steps of the diffusion process; E q is the expectation under distribution q, D kl is the Kullback-Leibler divergence, which measures the difference between two distributions, q is the conditional distribution in the forward process, is the conditional distribution in the reverse process. Step S43 can effectively measure the difference between the model predicted distribution and the true distribution, and guide the optimization of the model parameters to avoid introducing too much noise or erroneous information.

[0054] S5. Use a self-supervised outlier detection algorithm to detect and remove outliers. After imputing missing values, the outlier detection and removal step is performed. Outliers are detected using the Self-Supervised Outlier Detection (SSD) algorithm, an outlier detection algorithm based on self-supervised learning. The main goal of the SSD algorithm is to use self-supervised learning techniques to identify outliers in the data by learning the inherent structure and characteristics of the data without the need for labeled data.

[0055] Step S5 uses a self-supervised anomaly detection algorithm to detect and delete outliers, which includes the following sub-steps:

[0056] S51. Standardize and normalize the input data to eliminate the dimensional differences between different features, making it easier for the model to learn the essential characteristics of the data during training, thereby improving the robustness of the model and ensuring data consistency and quality.

[0057] S52, divide the data into positive sample pairs and negative sample pairs, train the model to extract meaningful feature representations; during the training process, use the NT-Xent loss function to optimize the feature extractor,

[0058] ;

[0059] Among them, z i and z j is the feature vector of the positive sample pair, representing the features extracted by the encoder, τ is the temperature parameter used to scale the vector inner product, N is the number of samples in the batch, K is the index of the negative sample, all samples in the batch except sample i; is an exponential function used to convert similarity scores into positive numbers; Represents the sum of similarities of all negative sample pairs;

[0060] Step S52 helps the model to more accurately determine the abnormality of the test sample in subsequent steps.

[0061] S53. After feature extraction, cluster analysis is performed on the features to divide the feature space into multiple cluster centers. By dividing the feature space into multiple cluster centers, the processing flow of subsequent anomaly detection can be simplified and the detection efficiency can be improved.

[0062] S54. For each test sample, calculate its Euclidean distance to each cluster center and evaluate its abnormality based on the distance. The calculation formula is:

[0063] ;

[0064] Among them, Z is the feature vector of the test sample, is the cluster center; by calculating the distance between the test sample and the cluster center, the abnormality of the sample can be accurately assessed. Samples with a farther distance are more likely to be considered outliers.

[0065] S55. Calculate the score by distance, set the threshold according to the actual situation, and then determine the outliers and delete them.

[0066] The present invention can solve the following problems: 1. The problem of difficulty in ensuring data consistency and standardization during the multi-source data fusion process; 2. The problem of a complex and time-consuming cleaning process due to data quality issues such as duplicate data, missing values, and erroneous data; 3. The problem of different data sources updating at different frequencies and mechanisms, which increases the complexity of data cleaning; especially in complex data integration scenarios with high missing rates (>40%) and multimodal mixing, the overall performance indicators exceed the existing optimal methods by more than 20%, showing significant technological breakthroughs and industrial application value.

[0067] This application establishes a unified metadata management system; defines unified data formats and standards; establishes data conversion and mapping rules to map fields in heterogeneous data sources to a unified data model; and after data integration, extracts key, high-quality data and puts it into the final data pool. This effectively solves the problem that existing methods require a lot of effort to unify and convert data from various systems, and the cleaning process is complicated and time-consuming due to data quality issues. It provides high-quality support for multi-source data fusion, helps build smart cities, and promotes the digital transformation of cities across the board.

[0068] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0069] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A data integration method based on diffusion model and self-supervised outlier detection, characterized in that: The following steps are involved: S1. Establish a unified metadata management system; by collecting and maintaining metadata from various data sources, ensure that the meaning, format, and scope of each data field are accurately described and understood; S2. Achieve a unified data format through data standardization operations; standardization methods include defining unified data formats and standards; S3. Establish data conversion and mapping rules to map fields in heterogeneous data sources into a unified data model; S4. Build a data completion solution based on the diffusion model to fill in missing values; S41. Fill in missing values. The basic principle of the diffusion model is to gradually add noise to the data until it approaches a simple distribution. In this process, given the initial data X0, the noise data X0 in the tth step is t , which is expressed as follows, ; in, is the time step that controls the rate of increase of noise, I is the identity matrix, which is used to define the covariance structure of Gaussian distribution, N is Gaussian distribution; q represents the state x in the previous step. t-1 Under the condition that the current step x t The conditional probability distribution of ; S42, reverse processing the data, from the noise data x t Restore to the original data X0. Each step in this process is described as a Markov chain, which is expressed as follows: ; in, is the mean parameter predicted by the model, which is adjusted through continuous learning of the model. is the covariance matrix, which is adjusted through continuous learning of the model. is the conditional probability distribution of the inverse process; N is a Gaussian distribution; S43. In the training process of the diffusion model, the variational lower bound is used as the loss function, which is expressed as, ; Where L is the overall loss function of the diffusion model, and the goal is to minimize the loss to optimize the model parameters; T is the total number of time steps of the diffusion process; E q is the expectation under distribution q, D EL is the Kullback-Leibler divergence, which measures the difference between two distributions, q is the conditional distribution in the forward process, is the conditional distribution in the reverse process; S5. Use the self-supervised anomaly detection algorithm to detect and delete outliers; S51. Standardize and normalize the input data to eliminate the dimensional differences between different features, making it easier for the model to learn the essential characteristics of the data during training, thereby improving the robustness of the model and ensuring data consistency and quality. S52, divide the data into positive sample pairs and negative sample pairs, train the model to extract meaningful feature representations; during the training process, use the NT-Xent loss function to optimize the feature extractor, ; Among them, z i and z j is the feature vector of the positive sample pair, representing the features extracted by the encoder, is the temperature parameter used to scale the vector inner product, N is the number of samples in the batch, K is the index of the negative sample, all samples in the batch except sample i; is an exponential function used to convert similarity scores into positive numbers; Represents the sum of similarities of all negative sample pairs; S53, after feature extraction, performing cluster analysis on the features, dividing the feature space into multiple cluster centers; S54. For each test sample, calculate its Euclidean distance to each cluster center and evaluate its abnormality based on the distance. The calculation formula is: ; Among them, z is the feature vector of the test sample, is the cluster center; S55. Calculate the score by distance, set the threshold according to the actual situation, and then determine the outliers and delete them.

Images