A Blockchain-Based Distributed Data Cleaning and Preprocessing Method

Abstract

This application discloses a distributed data cleaning and preprocessing method based on blockchain, belonging to the field of blockchain technology. The key technical points include: S1, data collection; S2, analysis of the distributed blockchain data; S3, synchronous processing of each data segment; S4, retrieval of pre-stored privacy-based data keywords from the database; S5, summing the privacy values ​​of each complete data segment; and S6, comparing and analyzing the privacy values ​​of each complete data set with a preset threshold. This application, through segmented synchronous processing, reduces data memory usage and accelerates data processing speed, while also utilizing a database containing a large number of privacy-based data keywords.

Description

Technical Field

[0001] This application relates to the field of blockchain technology, and more specifically to a blockchain-based distributed data cleaning and preprocessing method. Background Technology

[0002] Blockchain is essentially a decentralized distributed ledger database. It appears in the form of a new type of protocol and can securely store various types of transaction data. This information cannot be forged or tampered with. In blockchain, data is stored in a distributed manner, that is, data is distributed and stored on multiple nodes, and each node holds a complete copy of the database. This distributed storage method makes blockchain have extremely high reliability and security.

[0003] Chinese patent CN115687318A discloses an integrated data cleaning method, apparatus, server, and storage medium, comprising: acquiring dirty data to be cleaned; performing preliminary data cleaning if a preprocessing rule corresponding to the dirty data exists in a preset rule base; performing data cleaning for the reasoning process if a structural rule corresponding to the dirty data exists in the preset rule base; generating new preprocessing rules and / or structural rules corresponding to the dirty data if no preprocessing rules and / or structural rules exist in the preset rule base; expanding and reusing the rules; and cleaning the dirty data using the new preprocessing rules and / or structural rules. This invention aims to solve the problem in existing technologies where each data cleaning operation requires manually defining rules based on data characteristics, which is time-consuming and labor-intensive, resulting in a lack of efficiency and quality improvement in data cleaning.

[0004] Chinese patent CN112883096A discloses a data preprocessing method, comprising the following steps: upon receiving a data preprocessing instruction, reading data to be processed from multiple data sources; converting unstructured data in the data to be processed into structured data; clustering the structured data according to a preprocessing matching library to obtain clustered data; and performing data standardization, data union, and data cleaning on the clustered data to obtain preprocessed data.

[0005] While the aforementioned technologies have improved the efficiency and quality of data cleaning, they lack analysis of the degree of privacy during data processing. If the data is highly private, leakage could cause losses, so the data processing methods still need improvement. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention aims to provide a blockchain-based distributed data cleaning and preprocessing method. Its advantages lie in its segmented synchronous processing, which reduces data memory usage and accelerates data processing speed, while also leveraging the database's storage of numerous private data keywords. A data privacy calculation model, formed by pre-stored private data keywords in the database, is used. Each complete data segment is input into this model synchronously, allowing for rapid calculation of the privacy value of each segment. The privacy values ​​of each segment are then summed and compared to a threshold. Data groups with higher privacy levels are encrypted, thus improving the security of critical data after blockchain parsing.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a blockchain-based distributed data cleaning and preprocessing method, comprising:

[0008] S1. Acquire a large amount of distributed blockchain data through data collection;

[0009] S2. Analyze the blockchain distributed data, divide it into several groups and several segments, and mark each segment with a symbol;

[0010] S3. Process each data segment synchronously, including data deduplication, missing value imputation, and outlier handling, to obtain each complete data segment;

[0011] S4. Retrieve a pre-stored data privacy calculation model based on privacy data keywords from the database, synchronously input each complete data segment into the data privacy calculation model, and obtain the privacy value of each complete data segment. The data privacy calculation model is as follows:

[0012]

[0013] In the formula, For data privacy, Let j be the value of the i-th privacy feature of the data. The j-th feature index value of the i-th feature of the private data pre-stored in the database. Let i be the total number of private data items.

[0014] S5. Sum the privacy values ​​of each complete data segment to obtain the privacy value of each set of complete data;

[0015] S6. Compare and analyze the privacy value of each complete data set with a preset threshold, and encrypt the high privacy data set whose privacy value is greater than or equal to the preset threshold.

[0016] A further preferred embodiment of the present invention: Step 2, which involves dividing the data into several groups and several segments, and marking each segment with a symbol, specifically includes:

[0017] S21. Analyze and group the acquired blockchain distributed data;

[0018] S22. Symbolically label each set of blockchain distributed data;

[0019] S23. Use delimiters to segment each group of blockchain distributed data, dividing each group of data into several segments.

[0020] S24. Mark each data segment with a symbol.

[0021] A further preferred embodiment of the present invention: the data deduplication in step 3 specifically includes:

[0022] S31. Arrange all data in a column according to coordinates (x, y);

[0023] S32. Sequentially check the repetition between the i-th coordinate and all subsequent coordinates. If a repetition exists, delete the subsequent duplicate coordinates. Here, i starts from 1 and increments sequentially until all coordinates have been searched.

[0024] A further preferred embodiment of the present invention: the missing value filling in step 3 specifically includes:

[0025] S311. Arrange all data in a column according to coordinates (x, y);

[0026] S312. Retrieve all data coordinates. If there are missing x or y coordinates, fill the missing coordinates by using the average of adjacent data at the same position.

[0027] S313. The formula for calculating missing value imputation is:

[0028]

[0029] in, The data is used to fill in the data; i is the number of data items to fill in. is data of different types; n is the number of data of different types.

[0030] A further preferred embodiment of the present invention: the outlier processing in step 3 specifically includes:

[0031] S3111. Arrange all data in a column according to coordinates (x, y);

[0032] S3112. Calculate the deviation value k between the target data coordinates (x, y) and the two adjacent data coordinates (a, b) and (c, d). The deviation value formula is:

[0033] ;

[0034] S3113. If the deviation value k is greater than 100%, the target data coordinates (x, y) are determined to be outliers, and the mean of the two adjacent data is used to replace the outlier.

[0035] A further preferred embodiment of the present invention: obtaining the privacy value of each set of complete data in step S5 specifically includes the following steps:

[0036] S41. Obtain the privacy value of each complete data segment after it has been calculated using the data privacy calculation model;

[0037] S42. Sum the privacy values ​​of all segments of complete data;

[0038] S43. Obtain the privacy value for each complete set of data;

[0039] S44. The formula for calculating the privacy value of each complete set of data is:

[0040] ;

[0041] in, Let n be the privacy value for each complete set of data, and n be the privacy value for each complete data segment.

[0042] A further preferred embodiment of the present invention: Step S6, analyzing the privacy value of each set of complete data, specifically includes the following steps:

[0043] S61. Obtain the privacy value of each set of complete data after calculation;

[0044] S62. Compare the privacy value of each set of complete data with the preset threshold in the database;

[0045] S63. If the privacy value is greater than or equal to the preset threshold, then the privacy value of the complete set of data is high.

[0046] S64. If the privacy value is less than the preset threshold, then the privacy value of the complete set of data is low.

[0047] S65. Mark complete data sets with high privacy values.

[0048] A further preferred embodiment of the present invention: the encryption process for data groups with a high degree of privacy in step 6 specifically includes:

[0049] S6a. Determine the data bit length of the high privacy data group and obtain the total length of the data sequence;

[0050] S6b. Based on the total length of the data sequence, the data is divided into several subsequence data of a first preset length and several subsequence data of a second preset length, wherein the first preset length is less than the second preset length;

[0051] S6c. Perform multi-directional scrambling operations on the first preset length subsequence data and the second preset length subsequence data respectively to generate corresponding multi-directional scrambling sub-data;

[0052] S6d. Randomly arrange and combine all the scrambled sub-data to obtain encrypted data.

[0053] In summary, the present invention has the following advantages:

[0054] This invention reduces data memory usage while accelerating data processing. It can quickly calculate the privacy value of each complete data segment, then sum the privacy values ​​of each segment and compare them with a threshold. Data groups with high privacy levels are encrypted, improving the security of critical data after blockchain parsing and meeting the needs of staff. Attached Figure Description

[0055] Figure 1 This is the main flowchart of the present invention;

[0056] Figure 2 This is a flowchart of the blockchain distributed data analysis of the present invention;

[0057] Figure 3 This is a flowchart of the data deduplication process of this invention;

[0058] Figure 4 This is a flowchart of the missing value filling method of the present invention;

[0059] Figure 5 This is a flowchart of the outlier handling process of this invention;

[0060] Figure 6 This is a flowchart of the privacy value of the present invention;

[0061] Figure 7 This is a flowchart of the privacy value analysis process of this invention;

[0062] Figure 8 This is a flowchart of the encryption process for the data group in this invention. Detailed Implementation

[0063] The present invention will be further described in detail below with reference to the accompanying drawings.

[0064] like Figure 1 As shown, a blockchain-based distributed data cleaning and preprocessing method includes:

[0065] S1. Acquire a large amount of distributed blockchain data through data collection;

[0066] S2. Analyze the blockchain distributed data, divide it into several groups and several segments, and mark each segment with a symbol;

[0067] S3. Process each data segment synchronously, including data deduplication, missing value imputation, and outlier handling, to obtain each complete data segment;

[0068] S4. Retrieve a pre-stored data privacy calculation model based on privacy data keywords from the database, synchronously input each complete data segment into the data privacy calculation model, and obtain the privacy value of each complete data segment. The data privacy calculation model is as follows:

[0069]

[0070] In the formula, For data privacy, Let j be the value of the i-th privacy feature of the data. The j-th feature index value of the i-th feature of the private data pre-stored in the database. Let i be the total number of private data items.

[0071] S5. Sum the privacy values ​​of each complete data segment to obtain the privacy value of each set of complete data;

[0072] S6. Compare and analyze the privacy value of each complete data set with a preset threshold, and encrypt the high privacy data set whose privacy value is greater than or equal to the preset threshold.

[0073] Preferred, such as Figure 2 As shown, step 2, which involves dividing the data into several groups and segments, and marking each segment with a symbol, specifically includes:

[0074] S21. Analyze and group the acquired blockchain distributed data;

[0075] S22. Symbolically label each set of blockchain distributed data;

[0076] S23. Use delimiters to segment each group of blockchain distributed data, dividing each group of data into several segments.

[0077] S24. Mark each data segment with a symbol.

[0078] Preferred, such as Figure 3 As shown, the data deduplication in step 3 specifically includes:

[0079] S31. Arrange all data in a column according to coordinates (x, y);

[0080] S32. Sequentially check the repetition between the i-th coordinate and all subsequent coordinates. If a repetition exists, delete the subsequent duplicate coordinates. Here, i starts from 1 and increments sequentially until all coordinates have been searched.

[0081] Preferred, such as Figure 4 As shown, the missing value filling in step 3 specifically includes:

[0082] S311. Arrange all data in a column according to coordinates (x, y);

[0083] S312. Retrieve all data coordinates. If there are missing x or y coordinates, fill the missing coordinates by using the average of adjacent data at the same position.

[0084] S313. The formula for calculating missing value imputation is:

[0085]

[0086] in, The data is used to fill in the data; i is the number of data items to fill in. is data of different types; n is the number of data of different types.

[0087] Preferred, such as Figure 5 As shown, the outlier handling in step 3 specifically includes:

[0088] S3111. Arrange all data in a column according to coordinates (x, y);

[0089] S3112. Calculate the deviation value k between the target data coordinates (x, y) and the two adjacent data coordinates (a, b) and (c, d). The deviation value formula is:

[0090] ;

[0091] S3113. If the deviation value k is greater than 100%, the target data coordinates (x, y) are determined to be outliers, and the mean of the two adjacent data is used to replace the outlier.

[0092] Preferred, such as Figure 6 As shown, obtaining the privacy value for each set of complete data in step S5 specifically includes the following steps:

[0093] S41. Obtain the privacy value of each complete data segment after it has been calculated using the data privacy calculation model;

[0094] S42. Sum the privacy values ​​of all segments of complete data;

[0095] S43. Obtain the privacy value for each complete set of data;

[0096] S44. The formula for calculating the privacy value of each complete set of data is:

[0097] ;

[0098] in, Let n be the privacy value for each complete set of data, and n be the privacy value for each complete data segment.

[0099] Preferred, such as Figure 7 As shown, step S6, which analyzes the privacy value of each set of complete data, specifically includes the following steps:

[0100] S61. Obtain the privacy value of each set of complete data after calculation;

[0101] S62. Compare the privacy value of each set of complete data with the preset threshold in the database;

[0102] S63. If the privacy value is greater than or equal to the preset threshold, then the privacy value of the complete set of data is high.

[0103] S64. If the privacy value is less than the preset threshold, then the privacy value of the complete set of data is low.

[0104] S65. Mark complete data sets with high privacy values.

[0105] Preferred, such as Figure 8 As shown, step 6, which involves encrypting data groups with a high degree of privacy, specifically includes:

[0106] S6a. Determine the data bit length of the high privacy data group and obtain the total length of the data sequence;

[0107] S6b. Based on the total length of the data sequence, the data is divided into several subsequence data of a first preset length and several subsequence data of a second preset length, wherein the first preset length is less than the second preset length;

[0108] S6c. Perform multi-directional scrambling operations on the first preset length subsequence data and the second preset length subsequence data respectively to generate corresponding multi-directional scrambling sub-data;

[0109] S6d. Randomly arrange and combine all the scrambled sub-data to obtain encrypted data.

[0110] The working process and beneficial effects of this invention are as follows:

[0111] Those skilled in the art will understand that blockchain is essentially a decentralized distributed ledger database. It appears in the form of a novel protocol, capable of securely storing various types of transaction data. This data is unforgeable and tamper-proof. Furthermore, in blockchain, data is stored in a distributed manner, meaning it is distributed across multiple nodes, each holding a complete copy of the database. This distributed storage method gives blockchain extremely high reliability and security. This invention obtains complete data segments by grouping and segmenting large amounts of data, and simultaneously performing deduplication, missing value imputation, and outlier handling on each segment. The segment-synchronous processing method reduces data memory usage and accelerates data processing speed. Furthermore, the database stores a large number of private data keywords, and a data privacy calculation model formed by these keywords is pre-stored in the database. Each complete data segment is input into this model synchronously, allowing for rapid calculation of the privacy value of each segment. The privacy values ​​of each segment are then summed and compared with a threshold. Data groups with high privacy levels are encrypted, improving the security of critical data after blockchain parsing and meeting the needs of staff.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the design concept of the present invention should be included within the protection scope of the present invention.

Claims

1. A blockchain-based distributed data cleaning and preprocessing method, characterized in that, include: S1. Acquire a large amount of distributed blockchain data through data collection; S2. Analyze the blockchain distributed data, divide it into several groups and several segments, and mark each segment with a symbol; S3. Process each data segment synchronously, including data deduplication, missing value imputation, and outlier handling, to obtain each complete data segment; S4. Retrieve a pre-stored data privacy calculation model based on privacy data keywords from the database, synchronously input each complete data segment into the data privacy calculation model, and obtain the privacy value of each complete data segment. The data privacy calculation model is as follows: In the formula, For data privacy, Let j be the value of the i-th privacy feature of the data. The j-th feature index value of the i-th feature of the private data pre-stored in the database. Let i be the total number of private data items. S5. Sum the privacy values ​​of each complete data segment to obtain the privacy value of each set of complete data; S6. Compare and analyze the privacy value of each complete data set with a preset threshold, and encrypt the high privacy data set whose privacy value is greater than or equal to the preset threshold.

2. The distributed data cleaning and preprocessing method based on blockchain according to claim 1, characterized in that, Step 2, which involves dividing the data into several groups and segments, and marking each segment with a symbol, specifically includes: S21. Analyze and group the acquired blockchain distributed data; S22. Symbolically label each set of blockchain distributed data; S23. Use delimiters to segment each group of blockchain distributed data, dividing each group of data into several segments. S24. Mark each data segment with a symbol.

3. The distributed data cleaning and preprocessing system based on blockchain according to claim 1, characterized in that, Step 3, the data deduplication process, specifically includes: S31. Arrange all data in a column according to coordinates (x, y); S32. Sequentially check the repetition between the i-th coordinate and all subsequent coordinates. If a repetition exists, delete the subsequent duplicate coordinates. Here, i starts from 1 and increments sequentially until all coordinates have been searched.

4. The distributed data cleaning and preprocessing system based on blockchain according to claim 1, characterized in that, The missing value filling described in step 3 specifically includes: S311. Arrange all data in a column according to coordinates (x, y); S312. Retrieve all data coordinates. If there are missing x or y coordinates, fill the missing coordinates by using the average of adjacent data at the same position. S313. The formula for calculating missing value imputation is: in, The data is used to fill in the data; i is the number of data items to fill in. is data of different types; n is the number of data of different types.

5. A blockchain-based distributed data cleaning and preprocessing system according to claim 1, characterized in that, The outlier handling described in step 3 specifically includes: S3111. Arrange all data in a column according to coordinates (x, y); S3112. Calculate the deviation value k between the target data coordinates (x, y) and the two adjacent data coordinates (a, b) and (c, d). The deviation value formula is: ； S3113. If the deviation value k is greater than 100%, the target data coordinates (x, y) are determined to be outliers, and the mean of the two adjacent data is used to replace the outlier.

6. The blockchain-based distributed data cleaning and preprocessing method according to claim 1, characterized in that, Step S5, obtaining the privacy value for each set of complete data, specifically includes the following steps: S41. Obtain the privacy value of each complete data segment after it has been calculated using the data privacy calculation model; S42. Sum the privacy values ​​of all segments of complete data; S43. Obtain the privacy value for each complete set of data; S44. The formula for calculating the privacy value of each complete set of data is: ； in, Let n be the privacy value for each complete set of data, and n be the privacy value for each complete data segment.

7. The blockchain-based distributed data cleaning and preprocessing method according to claim 1, characterized in that, Step S6 analyzes the privacy value of each set of complete data, specifically including the following steps: S61. Obtain the privacy value of each set of complete data after calculation; S62. Compare the privacy value of each set of complete data with the preset threshold in the database; S63. If the privacy value is greater than or equal to the preset threshold, then the privacy value of the complete set of data is high. S64. If the privacy value is less than the preset threshold, then the privacy value of the complete set of data is low. S65. Mark complete data sets with high privacy values.

8. The blockchain-based distributed data cleaning and preprocessing method according to claim 1, characterized in that, Step 6, which involves encrypting data groups with high privacy requirements, specifically includes: S6a. Determine the data bit length of the high privacy data group and obtain the total length of the data sequence; S6b. Based on the total length of the data sequence, the data is divided into several subsequence data of a first preset length and several subsequence data of a second preset length, wherein the first preset length is less than the second preset length; S6c. Perform multi-directional scrambling operations on the first preset length subsequence data and the second preset length subsequence data respectively to generate corresponding multi-directional scrambling sub-data; S6d. Randomly arrange and combine all the scrambled sub-data to obtain encrypted data.

Images