Multi-source risk control data cleaning processing method

Abstract

The invention relates to the field of data cleaning, in particular to a multi-source risk control data cleaning processing method. The method comprises the following steps: S1, collecting risk control data of different sources; s2, performing splitting and label extraction on the multi-source risk control data, and performing classification of the multi-source risk control data based on a data classification algorithm; s3, preprocessing the data by adopting a data processing algorithm based on the classified multi-source risk control data; s4, abnormal value detection, processing and conversion are carried out on the preprocessed data to complete data cleaning; and S5, randomly extracting a data sample, and performing data verification on the data sample. Data are accurately divided through a random seed splitting algorithm, redundant records are eliminated in combination with a data deduplication algorithm, missing information is filled through a data completion algorithm, and extreme data points are identified and adjusted through an abnormal value detection and processing algorithm, so that the purity and reliability of a data set when the data set is used for risk model construction are ensured.

Description

Technical Field

[0001] This invention relates to the field of data cleaning, and more specifically, to a method for cleaning and processing multi-source risk control data. Background Technology

[0002] In today's information-saturated era, multi-source risk control data cleaning and processing technology has emerged to address the widespread data heterogeneity and quality issues in the financial and internet industries. This technology employs advanced data processing methods to deeply clean and integrate data from different sources. Its aim is to eliminate redundant information, correct erroneous data, and ensure data accuracy and consistency, thereby providing high-quality data support for risk management and decision support systems to improve overall decision-making efficiency and risk control capabilities.

[0003] Because risk control data needs to be collected from multiple channels, this data often suffers from problems such as inconsistent formats, duplicate content, missing values, and outliers. While existing data cleaning methods can address some of these issues, they often lack a comprehensive consideration of the characteristics of multi-source data, resulting in poor cleaning effectiveness and impacting the accuracy of risk analysis. This paper proposes a comprehensive method for cleaning and processing multi-source risk control data. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-source risk control data cleaning and processing method to solve the problem that although the existing data cleaning methods mentioned in the background art can handle some data quality issues, they often lack comprehensive consideration of the characteristics of multi-source data, resulting in poor cleaning effect and affecting the accuracy of risk analysis.

[0005] To achieve the above objectives, the present invention aims to provide a method for cleaning and processing multi-source risk control data, comprising the following steps:

[0006] S1. Collect risk control data from different sources;

[0007] S2. By splitting and extracting tags from multi-source risk control data, the multi-source risk control data is classified based on a data classification algorithm;

[0008] S3. Based on the categorized multi-source risk control data, data processing algorithms are used to preprocess the data;

[0009] S4. Perform outlier detection, processing, and transformation on the preprocessed data to complete data cleaning;

[0010] S5. Randomly select data samples and perform data verification on the data samples.

[0011] As a further improvement to this technical solution, in step S2, the steps involved in splitting, extracting tags, and classifying multi-source risk control data are as follows:

[0012] S1.1 After splitting the multi-source risk control data using a random seed splitting algorithm, extract frequently occurring words in the text as data labels based on a word frequency statistics algorithm;

[0013] S1.2. Based on the extracted data tags, use a data classification algorithm to classify data tags with the same tags.

[0014] As a further improvement to this technical solution, the specific steps involved in splitting and extracting tags from multi-source risk control data in S1.1 are as follows:

[0015] The original dataset D is randomly split based on a random seed splitting algorithm, specifically as follows:

[0016] S′=Split(D, ratio, seed);

[0017] In the formula, S′ represents the split subset; Split represents the split function; D represents the original dataset; ratio represents the split ratio; and seed represents the seed for random number generation. It is used to ensure the repeatability of the split. If the same seed value is set, the same split result will be obtained every time the Split function is called; if no seed is set, the split result may be different each time.

[0018] Tag extraction uses a word frequency statistics algorithm to extract words that appear multiple times in the text, specifically:

[0019] f w =|{t|t=w and t∈tokens}|;

[0020] F(S′, threshold) = {w|f w ≥threshold};

[0021] In the formula, F represents the function for extracting the frequency of occurrence in the text; S′ represents the split subset of the text data to be processed; threshold represents the minimum threshold for word frequency occurrence; w represents the word that appears in the text; represents the frequency of word w in the text S′; and tokens represents the set containing the text tokens.

[0022] As a further improvement to this technical solution, in S1.2, by extracting frequently occurring words from the text as data labels, and based on a data classification algorithm to minimize the squared error within each cluster, similar data points are grouped into clusters. The data classification algorithm specifically includes:

[0023]

[0024] In the formula, J(S) represents the sum of squared errors within a cluster; k represents the number of clusters; S i x represents the i-th cluster; x represents cluster S. i One data point; μ i Indicates cluster S i The center point.

[0025] The sum of squared errors within a cluster is the sum of the squared distances between each data point and the centroid of its cluster. The smaller this value, the closer the data points are to the centroid of their clusters, and the better the clustering effect.

[0026] As a further improvement to this technical solution, in step S3, the data processing algorithm uses a data deduplication algorithm to find and delete duplicate data, retaining unique records; a data completion algorithm is used to insert the average value of missing linear data to repair the linear data; and a data classification algorithm is used to cluster similar data points, subdividing the dataset into clusters with similar characteristics. The specific steps involved in the data processing algorithm are as follows:

[0027] A data deduplication algorithm is used to find and delete duplicate data with unique values ​​in the dataset. Specifically:

[0028] DELETE(DISTINCT(D′)={x∈D′|xis unique});

[0029] In the formula, D′ represents the categorized multi-source risk control dataset; DISTINCT(D′) represents selecting a unique record from dataset D′; DELETE represents the deletion function;

[0030] The missing values ​​in the dataset are replaced by the mean value of the non-missing data points calculated using a data completion algorithm. Specifically:

[0031]

[0032] In the formula, μ represents the mean of the missing values ​​to be filled; x i Let represent the i-th non-missing data point; n represents the total number of non-missing data points; and i represents the index coefficient.

[0033] As a further improvement to this technical solution, in step S4, outlier detection, processing, and transformation of the preprocessed data are performed using an outlier detection algorithm, a Box-Cox statistical transformation algorithm, and a data transformation algorithm, respectively. The steps involved are as follows:

[0034] S2.1. Outlier values ​​are detected using an outlier detection algorithm;

[0035] S2.2. The detected outliers are processed using the Box-Cox statistical transformation algorithm;

[0036] S2.3. Based on the data transformation algorithm, scale the data to the [0, 1] interval and unify the measurement standard.

[0037] As a further improvement to this technical solution, in S2.1, potential outliers are identified by calculating the degree of deviation between the data points and the average value. The Z-score reflects the degree of deviation of each data point X from the standard deviation σ of the average value μ of the entire dataset. The outlier detection algorithm is as follows:

[0038]

[0039] In the formula, Z represents the degree of deviation of the data points from the mean; X represents the observed value; μ represents the mean; and σ represents the standard deviation.

[0040] As a further improvement to this technical solution, in S2.2, the outliers detected by the outlier detection algorithm are based on the Box-Cox statistical transformation algorithm, which is used to convert non-normally distributed data into a form that is closer to a normal distribution. Specifically:

[0041]

[0042] In the formula, Y represents the converted data value; λ λ represents the original data value Y raised to the power of λ; λ represents the transformation parameter; ln represents the natural logarithm function.

[0043] The parameter λ of the Box-Cox transformation is determined through maximum likelihood estimation or other optimization methods, specifically:

[0044] When λ = 0, the transformation is equivalent to the natural logarithm transformation, which is suitable for cases where the original data are all positive numbers;

[0045] When λ = 1, the transformation is equivalent to no transformation, that is, Y remains unchanged;

[0046] When λ≠0, the transformation is a power transformation, which can flexibly adjust the distribution shape of the data;

[0047] The Box-Cox transformation requires that the original data be greater than 0, because power transformation and natural logarithm transformation are not applicable to non-positive numbers. If the dataset contains zero or negative numbers, data transformation is usually required to meet the requirements of the Box-Cox transformation. The Box-Cox transformation can effectively improve the normality of the data by adjusting the value of λ.

[0048] As a further improvement to this technical solution, in S2.3, the processed outliers are transformed using a data transformation algorithm that scales the data to the [0, 1] interval for unified measurement standards, specifically as follows:

[0049]

[0050] In the formula, X norm X represents the normalized data value; X represents the original data value; X min X represents the minimum value in the dataset; max This represents the maximum value in the dataset.

[0051] By establishing a unified standard for data measurement, it is possible to directly compare different features or datasets. This normalization method simplifies the comparison process and makes it easier for analysts to identify key differences and similarities in the data.

[0052] As a further improvement to this technical solution, in step S5, a smaller subset is selected from a larger dataset as sample data through random sampling. The specific steps involved in the data verification are as follows:

[0053] From the normalized data value X norm A sample S is randomly selected from the given data, specifically as follows:

[0054] S = Sample(X) norm (n);

[0055] In the formula, S represents the sample dataset; Sample represents the sampling function; n represents the number of data samples drawn; X norm This represents the normalized data value;

[0056] The integrity verification algorithm checks whether each record in sample S is complete, specifically as follows:

[0057]

[0058] In the formula, C(S) represents the integrity verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i This represents the i-th record in sample S; n represents the size of sample S.

[0059] An accuracy verification algorithm is used to check whether each record in sample S is accurate, specifically:

[0060]

[0061] In the formula, A(S) represents the accuracy verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i This represents the i-th record in sample S; n represents the size of sample S; Schema represents a predefined data schema;

[0062] The uniqueness verification algorithm is used to check whether there are duplicate records in sample S. Specifically:

[0063] U(S)=|S|=|{Hash(S i )|S i ∈S}|;

[0064] In the formula, represents the uniqueness verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i Let represent the i-th record in sample S.

[0065] Data validation involves three specific steps: sampling, integrity verification, and accuracy and uniqueness verification. First, a sample S of size n is randomly selected from the original dataset D. Next, an integrity verification algorithm checks if each record in sample S is complete; if all records are complete, the integrity verification result is true. Then, an accuracy verification algorithm checks if each record in sample S conforms to a predefined data schema; if all records are accurate, the accuracy verification result is true. Finally, a uniqueness verification algorithm checks if there are duplicate records in sample S; if the sample size equals the size of the hashed record set, the uniqueness verification result is true.

[0066] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0067] 1. This multi-source risk control data cleaning method, through refined data splitting, label extraction, and classification steps, can more effectively identify and integrate key information from multi-source data, thereby improving the accuracy and efficiency of data cleaning. In particular, the use of random seed splitting and data classification algorithms ensures the repeatability of data processing and the consistency of data within clusters, which is crucial for the stability and predictive ability of risk control models.

[0068] 2. In this multi-source risk control data cleaning and processing method, the data is accurately divided by a random seed splitting algorithm and key data labels are extracted by a word frequency statistics algorithm to achieve accurate classification. At the same time, the data deduplication algorithm is combined to eliminate redundant records, the data completion algorithm fills in missing information, and the outlier detection and processing algorithm identifies and adjusts extreme data points. These comprehensive measures greatly improve the precision and effectiveness of data cleaning, ensuring the purity and credibility of the dataset when used for risk model construction. Attached Figure Description

[0069] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Example:

[0072] Please see Figure 1 As shown in the figure, this embodiment provides a method for cleaning and processing multi-source risk control data, including the following steps:

[0073] S1. Collect risk control data from different sources;

[0074] S2. By splitting and extracting tags from multi-source risk control data, the multi-source risk control data is categorized based on a data classification algorithm. The steps involved in splitting, extracting tags, and categorizing multi-source risk control data are as follows:

[0075] S1.1 After splitting the multi-source risk control data using a random seed splitting algorithm, words that appear frequently in the text are extracted as data labels based on a word frequency statistics algorithm. The specific steps involved in splitting and extracting labels from the multi-source risk control data are as follows:

[0076] The original dataset D is randomly split based on a random seed splitting algorithm, specifically as follows:

[0077] S′=Split(D, ratio, seed);

[0078] In the formula, S′ represents the split subset; Split represents the split function; D represents the original dataset; ratio represents the split ratio; and seed represents the seed for random number generation. It is used to ensure the repeatability of the split. If the same seed value is set, the same split result will be obtained every time the Split function is called; if no seed is set, the split result may be different each time.

[0079] Tag extraction uses a word frequency statistics algorithm to extract words that appear multiple times in the text, specifically:

[0080] f w =|{t|t=w and t∈tokens}|;

[0081] F(S′, threshold) = {w|f w ≥threshold};

[0082] In the formula, F represents the function for extracting the frequency of occurrence in the text; S′ represents the split subset of the text data to be processed; threshold represents the minimum threshold for word frequency occurrence; w represents the word that appears in the text; represents the frequency of word w in the text S′; and tokens represents the set containing the text tokens.

[0083] S1.2. Based on the extracted data labels, a data classification algorithm is used to classify data with the same labels. By extracting words that appear multiple times in the text as data labels, and minimizing the squared error within the cluster based on the data classification algorithm, similar data points are grouped into clusters. The data classification algorithm is specifically as follows:

[0084]

[0085] In the formula, J(S) represents the sum of squared errors within a cluster; k represents the number of clusters; S i x represents the i-th cluster; x represents cluster S. i One data point; μ i Indicates cluster S i The center point.

[0086] The sum of squared errors within a cluster is the sum of the squared distances between each data point and the centroid of its cluster. The smaller this value, the closer the data points are to the centroid of their clusters, and the better the clustering effect.

[0087] S3. Based on the categorized multi-source risk control data, a data processing algorithm is used to preprocess the data; the data processing algorithm uses a data deduplication algorithm to find and delete duplicate data, retaining unique records; a data completion algorithm is used to insert the average value of missing linear data to repair the linear data; based on the data classification algorithm, similar data points are clustered and divided into clusters with similar characteristics. The specific steps involved in the data processing algorithm are as follows:

[0088] A data deduplication algorithm is used to find and delete duplicate data with unique values ​​in the dataset. Specifically:

[0089] DELETE(DISTINCT(D′)={x∈D′|xis unique});

[0090] In the formula, D′ represents the categorized multi-source risk control dataset; DISTINCT(D′) represents selecting a unique record from dataset D′; DELETE represents the deletion function;

[0091] The missing values ​​in the dataset are replaced by the mean value of the non-missing data points calculated using a data completion algorithm. Specifically:

[0092]

[0093] In the formula, μ represents the mean of the missing values ​​filled; x i Let represent the i-th non-missing data point; n represents the total number of non-missing data points; and i represents the index coefficient.

[0094] S4. Data cleaning is completed by detecting, processing, and transforming outliers in the preprocessed data. The outlier detection, processing, and transformation of the preprocessed data employ outlier detection algorithms, Box-Cox statistical transformation algorithms, and data transformation algorithms, involving the following steps:

[0095] S2.1. Outlier detection algorithms are used to detect outliers. Potential outliers are identified by calculating the degree of deviation between data points and the mean. The Z-score reflects the degree of deviation of each data point X from the standard deviation σ of the mean μ of the entire dataset. The outlier detection algorithm is as follows:

[0096]

[0097] In the formula, Z represents the degree of deviation of the data points from the mean; X represents the observed value; μ represents the mean; and σ represents the standard deviation.

[0098] S2.2, The detected outliers are processed using the Box-Cox statistical transformation algorithm; the outliers detected by the outlier detection algorithm are based on the Box-Cox statistical transformation algorithm, which is used to transform non-normally distributed data into a form that is closer to a normal distribution, specifically:

[0099]

[0100] In the formula, Y represents the converted data value; λ λ represents the original data value Y raised to the power of λ; λ represents the transformation parameter; ln represents the natural logarithm function.

[0101] The parameter λ of the Box-Cox transformation is determined through maximum likelihood estimation or other optimization methods, specifically:

[0102] When λ = 0, the transformation is equivalent to the natural logarithm transformation, which is suitable for cases where the original data are all positive numbers;

[0103] When λ = 1, the transformation is equivalent to no transformation, that is, Y remains unchanged;

[0104] When λ≠0, the transformation is a power transformation, which can flexibly adjust the distribution shape of the data;

[0105] The Box-Cox transformation requires that the original data be greater than 0, because power transformation and natural logarithm transformation are not applicable to non-positive numbers. If the dataset contains zero or negative numbers, data transformation is usually required to meet the requirements of the Box-Cox transformation. The Box-Cox transformation can effectively improve the normality of the data by adjusting the value of λ.

[0106] S2.3. Based on the data transformation algorithm, scale the data to the [0, 1] interval to unify the measurement standard; the processed outliers are then scaled to the [0, 1] interval using the data transformation algorithm to unify the measurement standard, specifically:

[0107]

[0108] In the formula, X norm X represents the normalized data value; X represents the original data value; X min X represents the minimum value in the dataset; max This represents the maximum value in the dataset.

[0109] By establishing a unified standard for data measurement, it is possible to directly compare different features or datasets. This normalization method simplifies the comparison process and makes it easier for analysts to identify key differences and similarities in the data.

[0110] S5. Randomly select data samples and perform data validation on the data samples; select a smaller subset from a larger dataset as sample data through random sampling. The specific steps involved in the data validation are as follows:

[0111] From the normalized data value X norm A sample S is randomly selected from the given data, specifically as follows:

[0112] S = Sample(X) norm (n);

[0113] In the formula, S represents the sample dataset; Sample represents the sampling function; n represents the number of data samples drawn; X norm This represents the normalized data value;

[0114] The integrity verification algorithm checks whether each record in sample S is complete, specifically as follows:

[0115]

[0116] In the formula, C(S) represents the integrity verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i This represents the i-th record in sample S; n represents the size of sample S.

[0117] An accuracy verification algorithm is used to check whether each record in sample S is accurate, specifically:

[0118]

[0119] In the formula, A(S) represents the accuracy verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i This represents the i-th record in sample S; n represents the size of sample S; Schema represents a predefined data schema;

[0120] The uniqueness verification algorithm is used to check whether there are duplicate records in sample S. Specifically:

[0121] U(S)=|S|=|{Hash(S i )|S i ∈S}|;

[0122] In the formula, represents the uniqueness verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i Let represent the i-th record in sample S.

[0123] Data validation involves three specific steps: sampling, integrity verification, and accuracy and uniqueness verification. First, a sample S of size n is randomly selected from the original dataset D. Next, an integrity verification algorithm checks if each record in sample S is complete; if all records are complete, the integrity verification result is true. Then, an accuracy verification algorithm checks if each record in sample S conforms to a predefined data schema; if all records are accurate, the accuracy verification result is true. Finally, a uniqueness verification algorithm checks if there are duplicate records in sample S; if the sample size equals the size of the hashed record set, the uniqueness verification result is true.

[0124] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for cleaning and processing multi-source risk control data, characterized in that: Includes the following steps: S1. Collect risk control data from different sources; S2. By splitting and extracting tags from multi-source risk control data, the multi-source risk control data is classified based on a data classification algorithm; S3. Based on the categorized multi-source risk control data, data processing algorithms are used to preprocess the data; S4. Perform outlier detection, processing, and transformation on the preprocessed data to complete data cleaning; S5. Randomly select data samples and perform data verification on the data samples.

2. The method for cleaning and processing multi-source risk control data according to claim 1, characterized in that: In step S2, the steps involved in splitting, tagging, and classifying multi-source risk control data are as follows: S1.1 After splitting the multi-source risk control data using a random seed splitting algorithm, extract frequently occurring words in the text as data labels based on a word frequency statistics algorithm; S1.

2. Based on the extracted data tags, use a data classification algorithm to classify data tags with the same tags.

3. The method for cleaning and processing multi-source risk control data according to claim 2, characterized in that: In S1.1, the specific steps involved in splitting and extracting tags from multi-source risk control data are as follows: The original dataset D is randomly split based on a random seed splitting algorithm, specifically as follows: S′=Split(D, ratio, seed); In the formula, S′ represents the split subset; Split represents the splitting function; D represents the original dataset; ratio represents the splitting ratio; and seed represents the seed for random number generation. Tag extraction uses a word frequency statistics algorithm to extract words that appear multiple times in the text, specifically: f w =|{t|t=w and t∈tokens}|; F(S′,threshold)={w |f w ≥threshold}; In the formula, F represents the function for extracting the frequency of occurrence in the text; S′ represents the split subset; threshold represents the minimum threshold for word frequency occurrence; w represents the word that appears in the text; represents the frequency of word w in the text S′; and tokens represents the set containing the text segments.

4. The method for cleaning and processing multi-source risk control data according to claim 2, characterized in that: In step S1.2, by extracting frequently occurring words from the text as data labels, and based on a data classification algorithm to minimize the squared error within each cluster, similar data points are grouped into clusters. The data classification algorithm specifically includes: In the formula, J(S) represents the sum of squared errors within a cluster; k represents the number of clusters; S i x represents the i-th cluster; x represents cluster S. i One data point; μ i Indicates cluster S i The center point.

5. The method for cleaning and processing multi-source risk control data according to claim 1, characterized in that: In step S3, the data processing algorithm uses a data deduplication algorithm to find and delete duplicate data, retaining only unique records; it employs a data completion algorithm to insert the average value of missing linear data to repair the linearity; and it uses a data classification algorithm to cluster similar data points, subdividing the dataset into clusters with similar characteristics. The specific steps involved in the data processing algorithm are as follows: A data deduplication algorithm is used to find and delete duplicate data with unique values ​​in the dataset. Specifically: DELETE(DISTINCT(D′)={x∈D′|xis unique}); In the formula, D′ represents the categorized multi-source risk control dataset; DISTINCT(D′) represents selecting a unique record from dataset D′; DELETE represents the deletion function; The missing values ​​in the dataset are replaced by the mean value of the non-missing data points calculated using a data completion algorithm. Specifically: In the formula, μ represents the mean of the missing values ​​to be filled; x i Let represent the i-th non-missing data point; n represents the total number of non-missing data points; and i represents the index coefficient.

6. The method for cleaning and processing multi-source risk control data according to claim 1, characterized in that: In step S4, outlier detection, processing, and transformation of the preprocessed data are performed using an outlier detection algorithm, a Box-Cox statistical transformation algorithm, and a data transformation algorithm, respectively. The steps involved are as follows: S2.1 Detect outliers using an outlier detection algorithm; S2.

2. The detected outliers are processed using the Box-Cox statistical transformation algorithm; S2.

3. Based on the data transformation algorithm, scale the data to the [0, 1] interval and unify the measurement standard.

7. The method for cleaning and processing multi-source risk control data according to claim 6, characterized in that: In step S2.1, potential outliers are identified by calculating the degree of deviation between the data points and the average value. The outlier detection algorithm is as follows: In the formula, Z represents the degree of deviation of the data points from the mean; X represents the observed value; μ represents the mean; and σ represents the standard deviation.

8. The method for cleaning and processing multi-source risk control data according to claim 6, characterized in that: In step S2.2, the outliers detected by the outlier detection algorithm are based on the Box-Cox statistical transformation algorithm, which is used to transform non-normally distributed data into a form that is closer to a normal distribution. Specifically: In the formula, Y represents the converted data value; λ The original data value Y is raised to the power of λ; λ represents the transformation parameter; l n This represents the natural logarithm function.

9. The method for cleaning and processing multi-source risk control data according to claim 6, characterized in that: In step S2.3, the processed outliers are transformed using a data transformation algorithm that scales the data to the [0, 1] interval for unified measurement standards. Specifically: In the formula, X norm X represents the normalized data value; X represents the original data value; X min X represents the minimum value in the dataset; max This represents the maximum value in the dataset.

10. The method for cleaning and processing multi-source risk control data according to claim 1, characterized in that: In step S5, a subset is selected as sample data from a randomly sampled dataset. The specific steps involved in the data verification are as follows: From the normalized data value X norm A sample S is randomly selected from the given data, specifically as follows: S=Sample(X norm ,n); In the formula, S represents the sample dataset; Sample represents the sampling function; n represents the number of data samples drawn; X norm This represents the normalized data value; The integrity verification algorithm checks whether each record in sample S is complete, specifically as follows: In the formula, C(S) represents the integrity verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i This represents the i-th record in sample S; n represents the size of sample S. An accuracy verification algorithm is used to check whether each record in sample S is accurate, specifically: In the formula, A(S) represents the accuracy verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i This represents the i-th record in sample S; n represents the size of sample S; Schema represents a predefined data schema; The uniqueness verification algorithm is used to check whether there are duplicate records in sample S. Specifically: U(S)=|S|=I{Hash(S i )|S i ∈S}|; In the formula, represents the uniqueness verification result of sample S; S represents a sample randomly drawn from the original dataset D; S i Let represent the i-th record in sample S.

Images