Data integration method based on diffusion model and self-supervised abnormal value detection

Abstract

The invention discloses a data integration method based on a diffusion model and self-supervised abnormal value detection, and relates to the technical field of big data processing, and the method comprises the following steps: building a unified metadata management system; by collecting and maintaining metadata of each data source, the meaning, format and range of each data field are ensured to be accurately described and understood; the unified format of data is realized through data standardization operation; the standardization method comprises the following steps: defining a unified data format and standard; establishing a data conversion and mapping rule, and mapping fields in the heterogeneous data source into a unified data model; constructing a data completion scheme based on the diffusion model, and filling missing values; and a self-supervision anomaly detection algorithm is used to detect and delete abnormal values, so that the problems that in the prior art, a large amount of energy needs to be consumed to unify and convert data of all systems, and the cleaning process of data quality problems is complex and time-consuming are effectively solved, and high-quality support is provided for multi-source data fusion enabling.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to a data integration method based on diffusion model and self-supervised outlier detection. Background Art

[0002] With the advent of the big data era, government and enterprise-related functional departments have accumulated a massive amount of multi-source heterogeneous data. In order to better understand and utilize these data, break the departmental information silos, and integrate data in an intelligent and reliable way, it is urgent. Integrating data resources from different sources, breaking information silos, and realizing data sharing and interoperability can not only improve the transparency and response speed of decision-making, but also make services more accurate and efficient, and better meet people's diverse needs. At the same time, data fusion can also help to explore the potential value of data, promote the optimal allocation of resources, and enhance the pertinence and effectiveness of services. In addition, establishing a unified data fusion empowerment platform can strengthen data security management, effectively prevent data leakage and abuse risks, and protect personal information security.

[0003] In the process of multi-source data fusion, we face many difficult data cleaning problems. First, different departments and systems use different data formats and naming conventions, which makes it difficult to ensure data consistency and standardization, and requires a lot of effort for unified conversion. Secondly, data quality issues such as duplicate data, missing values, and erroneous data frequently exist, especially in large-scale data sets, and the cleaning process is complex and time-consuming. In addition, the timeliness of data is highly required, and the latest information needs to be updated and synchronized in a timely manner. However, the update frequency and mechanism of different data sources are different, which increases the complexity of data cleaning. Therefore, it is necessary to apply efficient cleaning algorithms and establish effective data synchronization mechanisms to address the complexity of multi-source data. Summary of the invention

[0004] The purpose of the present invention is to provide a data integration method based on diffusion model and self-supervised outlier detection to solve the problems raised by the above background technology.

[0005] In order to solve the above technical problems, the present invention is achieved through the following technical solutions: The present invention is a data integration method based on diffusion model and self-supervised outlier detection, comprising the following steps: S1. Establish a unified metadata management system; by collecting and maintaining metadata of each data source, ensure that the meaning, format and scope of each data field are accurately described and understood; S2. Achieve a unified data format through data standardization operations; the standardization method includes defining a unified data format and standard; S3. Establish data conversion and mapping rules to map fields in heterogeneous data sources to a unified data model; S4, build a data completion scheme based on the diffusion model to fill in missing values; S5. Use the self-supervised anomaly detection algorithm to detect and delete outliers.

[0006] Furthermore, in step S4, a data completion scheme is constructed based on the diffusion model to fill in the missing values, which includes the following sub-steps: S41. Fill in missing values. The basic principle of the diffusion model is to gradually add noise to the data until it approaches a simple distribution. In this process, given the initial data x0, the noise data x in the tth step t , which is expressed as follows, ; in, is the time step that controls the rate of noise increase, I is the identity matrix, which is used to define the covariance structure of the Gaussian distribution, N is the Gaussian distribution; q represents the state x given the previous step. t-1 Under the condition that the current step x t The conditional probability distribution of ; S42, reverse processing of the data, from the noise data x t Restore to the original data x0. Each step in this process is described as a Markov chain, as follows: ; in, is the mean parameter predicted by the model, which is adjusted through continuous learning of the model. is the covariance matrix, which is adjusted through continuous learning of the model. is the conditional probability distribution of the inverse process; N is a Gaussian distribution; S43. In the training process of the diffusion model, the variational lower bound is used as the loss function, which is expressed as, ; Where L is the overall loss function of the diffusion model, and the goal is to minimize the loss to optimize the model parameters; T is the total time steps of the diffusion process; E q is the expectation under distribution q, D kl is the Kullback-Leibler divergence, which measures the difference between two distributions, q is the conditional distribution in the forward process, is the conditional distribution in the reverse process.

[0007] Furthermore, in step S5, the self-supervised anomaly detection algorithm is used to detect and delete outliers, which includes the following sub-steps: S51. Standardize and normalize the input data to eliminate the dimensional differences between different features, so that the model can more easily learn the essential characteristics of the data during training, improve the robustness of the model, and ensure data consistency and quality; S52, divide the data into positive sample pairs and negative sample pairs, train the model to extract meaningful feature representations; during the training process, use the NT-Xent loss function to optimize the feature extractor, ; Among them, z i and z j is the feature vector of the positive sample pair, representing the features extracted by the encoder, τ is the temperature parameter used to scale the vector inner product, N is the number of samples in the batch, K is the index of the negative samples, all samples in the batch except sample i; is an exponential function used to convert similarity scores into positive numbers; Represents the sum of similarities of all negative sample pairs; S53, after feature extraction, performing cluster analysis on the features, dividing the feature space into multiple cluster centers; S54. For each test sample, calculate its Euclidean distance from each cluster center, and evaluate its abnormality based on the distance. The calculation formula is: ; Among them, Z is the feature vector of the test sample, is the cluster center; S55. Calculate the score by distance, set the threshold according to the actual situation, and then determine the outliers and delete them.

[0008] The present invention has the following beneficial effects: This application defines unified data formats and standards by establishing a unified metadata management system; establishes data conversion and mapping rules to map fields in heterogeneous data sources to a unified data model; after data integration, extracts key, high-quality data and puts it into the final data pool. This effectively solves the problem that the existing technology requires a lot of effort to unify and convert data from various systems, and the cleaning process is complicated and time-consuming due to data quality issues. It provides high-quality support for multi-source data fusion empowerment, helps build smart cities, and promotes the digital transformation of cities across the board.

[0009] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0011] Figure 1 The flowchart of the data fusion method based on diffusion model and self-supervised outlier detection of the present invention is shown in FIG.

[0012] Figure 2 Schematic diagram of the model learning reversal generation process of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] See also Figure 1-2 As shown, the present invention is a data integration method based on diffusion model and self-supervised outlier detection, comprising the following steps: S1. Establish a unified metadata management system; by collecting and maintaining metadata of each data source, ensure that the meaning, format and scope of each data field are accurately described and understood; the metadata management system is a key tool for integrating heterogeneous data. It can define and manage the structure, attributes and relationships of each data source. The metadata management system not only provides a dictionary and directory of data, but also helps track the source and change history of data, thereby maintaining data consistency and integrity during the data integration process.

[0015] S2. Achieve a unified data format through data standardization operations; standardization methods include defining unified data formats and standards; such as unified date formats, address formats, and coding standards. In actual operations, regular expressions and string processing functions can be used to format data. For example, dates in different formats can be uniformly converted to the "YYYY-MM-DD" format, or address information can be decomposed into standardized fields such as streets, cities, and postal codes. The standardization process needs to fully consider the characteristics of each data source to ensure that the meaning of the original data is not lost during the conversion process.

[0016] S3. Establish data conversion and mapping rules to map the fields in heterogeneous data sources to a unified data model. Since the data structures of different data sources may differ significantly, directly merging data will result in data distortion or inconsistency. Therefore, it is necessary to establish data conversion and mapping rules to map the fields in heterogeneous data sources to a unified data model. The project will use data conversion scripts for data conversion. Data conversion scripts can automatically handle the data extraction, conversion, and loading process to ensure that data is converted and merged according to predetermined rules.

[0017] S4. Build a data completion solution based on the diffusion model to fill in missing values. The basic principle of the diffusion model is to gradually add noise to the data until it approaches a simple distribution (such as a Gaussian distribution). Then, the model learns how to reverse this process and gradually restore the original data from the noisy data. This process can be regarded as a generative process, in which each step attempts to reduce the uncertainty of the data. Figure 2 shown.

[0018] In step S4, a data completion scheme is constructed based on the diffusion model to fill in the missing values, which includes the following sub-steps: S41. Fill in missing values. The basic principle of the diffusion model is to gradually add noise to the data until it approaches a simple distribution. In this process, given the initial data x0, the noise data x in the tth step t , which is expressed as follows, ; in, is the time step that controls the rate of noise increase, I is the identity matrix, which is used to define the covariance structure of the Gaussian distribution, N is the Gaussian distribution; q represents the state x given the previous step. t-1 Under the condition that the current step x t conditional probability distribution; step S41 provides the necessary conditions for the subsequent inverse process, so that the model can gradually recover the original data from the noise data.

[0019] S42, reverse processing of the data, from the noise data x t Restore to the original data x0. Each step in this process is described as a Markov chain, as follows: ; in, is the mean parameter predicted by the model, which is adjusted through continuous learning of the model. is the covariance matrix, which is adjusted through continuous learning of the model. is the conditional probability distribution of the reverse process; N is a Gaussian distribution; by continuously learning and optimizing parameters and , the model can gradually improve the accuracy of recovering the original data from the noisy data, thereby achieving effective filling of missing values.

[0020] S43. In the training process of the diffusion model, the variational lower bound is used as the loss function, which is expressed as, ; Where L is the overall loss function of the diffusion model, and the goal is to minimize the loss to optimize the model parameters; T is the total time steps of the diffusion process; E q is the expectation under distribution q, D kl is the Kullback-Leibler divergence, which measures the difference between two distributions, q is the conditional distribution in the forward process, is the conditional distribution in the reverse process. Step S43 can effectively measure the difference between the model prediction distribution and the true distribution, and guide the optimization of the model parameters to avoid introducing too much noise or erroneous information.

[0021] S5. Use the self-supervised outlier detection algorithm to detect and delete outliers. After missing values ​​are filled, the outlier detection and deletion step will be performed. The self-supervised outlier detection (SSD) algorithm is used to detect outliers. SSD is an outlier detection algorithm based on self-supervised learning. The main goal of the SSD algorithm is to use self-supervised learning technology to identify outliers in the data by learning the intrinsic structure and characteristics of the data without the need for labeled data.

[0022] The step S5 uses a self-supervised anomaly detection algorithm to detect and delete outliers, and includes the following sub-steps: S51. Standardize and normalize the input data to eliminate the dimensional differences between different features, so that the model can more easily learn the essential characteristics of the data during the training process, which is used to improve the robustness of the model and ensure data consistency and quality; S52, divide the data into positive sample pairs and negative sample pairs, train the model to extract meaningful feature representations; during the training process, use the NT-Xent loss function to optimize the feature extractor, ; Among them, z i and z j is the feature vector of the positive sample pair, representing the features extracted by the encoder, τ is the temperature parameter used to scale the vector inner product, N is the number of samples in the batch, K is the index of the negative samples, all samples in the batch except sample i; is an exponential function used to convert similarity scores into positive numbers; Represents the sum of similarities of all negative sample pairs; Step S52 helps the model to more accurately determine the abnormality of the test sample in subsequent steps.

[0023] S53. After feature extraction, cluster analysis is performed on the features to divide the feature space into multiple cluster centers. By dividing the feature space into multiple cluster centers, the processing flow of subsequent anomaly detection can be simplified and the detection efficiency can be improved.

[0024] S54. For each test sample, calculate its Euclidean distance from each cluster center, and evaluate its abnormality based on the distance. The calculation formula is: ; Among them, Z is the feature vector of the test sample, is the cluster center; by calculating the distance between the test sample and the cluster center, the abnormality of the sample can be accurately assessed. Samples with a longer distance are more likely to be considered outliers.

[0025] S55. Calculate the score by distance, set the threshold according to the actual situation, and then determine the outliers and delete them.

[0026] The present invention can solve the following problems: 1. It is difficult to ensure the consistency and standardization of data during the fusion of multi-source data; 2. The cleaning process is complicated and time-consuming due to data quality reasons such as duplicate data, missing values ​​and erroneous data; 3. The update frequencies and mechanisms of different data sources are different, which increases the complexity of data cleaning; especially in complex data integration scenarios with high missing rates (>40%) and multi-modal mixing, the overall performance indicators exceed the existing optimal methods by more than 20%, which has significant technical breakthroughs and industrial application value.

[0027] This application establishes a unified metadata management system; defines unified data formats and standards; establishes data conversion and mapping rules to map fields in heterogeneous data sources to a unified data model; after data integration, extracts key, high-quality data and puts it into the final data pool. This effectively solves the problem that existing methods require a lot of effort to unify and convert data from various systems, and the cleaning process is complicated and time-consuming due to data quality issues. It provides high-quality support for multi-source data fusion empowerment, helps build smart cities, and promotes the digital transformation of cities across the board.

[0028] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0029] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A data integration method based on diffusion model and self-supervised outlier detection, characterized in that: The following steps are involved: S1. Establish a unified metadata management system; by collecting and maintaining metadata of each data source, ensure that the meaning, format and scope of each data field are accurately described and understood; S2. Achieve a unified data format through data standardization operations; the standardization method includes defining a unified data format and standard; S3. Establish data conversion and mapping rules to map fields in heterogeneous data sources to a unified data model; S4, build a data completion scheme based on the diffusion model to fill in missing values; S5. Use the self-supervised anomaly detection algorithm to detect and delete outliers.

2. A data integration method based on diffusion model and self-supervised outlier detection according to claim 1, characterized in that: In step S4, a data completion scheme is constructed based on the diffusion model to fill in the missing values, which includes the following sub-steps: S41. Fill in missing values. The basic principle of the diffusion model is to gradually add noise to the data until it approaches a simple distribution. In this process, given the initial data x0, the noise data x in the tth step t , which is expressed as follows, ; in, is the time step that controls the rate of noise increase, I is the identity matrix, which is used to define the covariance structure of the Gaussian distribution, N is the Gaussian distribution; q represents the state x given the previous step. t-1 Under the condition that the current step x t The conditional probability distribution of ; S42, reverse processing of the data, from the noise data x t Restore to the original data x0. Each step in this process is described as a Markov chain, as follows: ； in, is the mean parameter predicted by the model, which is adjusted through continuous learning of the model. is the covariance matrix, which is adjusted through continuous learning of the model. is the conditional probability distribution of the inverse process; N is a Gaussian distribution; S43. In the training process of the diffusion model, the variational lower bound is used as the loss function, which is expressed as, ； Where L is the overall loss function of the diffusion model, and the goal is to minimize the loss to optimize the model parameters; T is the total time steps of the diffusion process; E q is the expectation under distribution q, D kl is the Kullback-Leibler divergence, which measures the difference between two distributions, q is the conditional distribution in the forward process, is the conditional distribution in the reverse process.

3. The data integration method based on diffusion model and self-supervised outlier detection according to claim 1, characterized in that: The step S5 uses a self-supervised anomaly detection algorithm to detect and delete outliers, and includes the following sub-steps: S51. Standardize and normalize the input data to eliminate the dimensional differences between different features, so that the model can more easily learn the essential characteristics of the data during training, improve the robustness of the model, and ensure data consistency and quality; S52, divide the data into positive sample pairs and negative sample pairs, train the model to extract meaningful feature representations; during the training process, use the NT-Xent loss function to optimize the feature extractor, ; Among them, z i and z j is the feature vector of the positive sample pair, representing the features extracted by the encoder, τ is the temperature parameter used to scale the vector inner product, N is the number of samples in the batch, K is the index of the negative samples, all samples in the batch except sample i; is an exponential function used to convert similarity scores into positive numbers; Represents the sum of similarities of all negative sample pairs; S53, after feature extraction, performing cluster analysis on the features, dividing the feature space into multiple cluster centers; S54. For each test sample, calculate its Euclidean distance from each cluster center, and evaluate its abnormality based on the distance. The calculation formula is: ; Among them, Z is the feature vector of the test sample, is the cluster center; S55. Calculate the score by distance, set the threshold according to the actual situation, and then determine the outliers and delete them.

Images