Big data management method and big data management system

Abstract

The invention discloses a big data management method and a big data management system, relates to the technical field of data processing, and solves the technical problems of lack of system analysis and processing and lack of effective protection of data in the data processing and storage process. According to the method, privacy data leakage is effectively prevented, the strict requirement of data security is met, the storage node performance is quantitatively evaluated from the three dimensions of reading speed, available storage capacity and reading error rate, the storage node with the optimal performance can be accurately selected, the data storage efficiency and stability are improved, and the data security is improved. An efficient analysis process aiming at repeated data and single data is designed, label replacement is performed on the repeated data, binary conversion and matching replacement are performed on residual data, and rapid recombination of the data is realized. And meanwhile, the data are segmented and recombined by taking the average value of the network speed of the storage nodes as a standard, so that the data storage and processing efficiency is further improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data management method and a big data management system. Background Art

[0002] With the continuous development of big data technology, the integrity and accuracy of basic communication data directly affect the accuracy and reliability of various business data analysis results.

[0003] According to a patent application with publication number CN116881340A, a big data management method and a big data management platform are disclosed, which include obtaining multiple communication basic data; determining in turn whether each of the communication basic data meets the preset data standard; determining the communication basic data that does not meet the data standard as the target basic data; judging whether the target basic data belongs to revised data, and if the target basic data belongs to revised data, revising the target basic data so that the revised target basic data meets the data standard, so that it is determined that after modification, it can meet the standard.

[0004] However, some existing big data management systems lack systematic analysis and processing when managing data, and have deficiencies in data quality improvement, privacy protection, storage performance optimization, etc. At the same time, during the data processing and storage process, there is a lack of effective protection of privacy data, and privacy data is easily leaked. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a big data management method and a big data management system, which solve the problems of lack of systematic analysis and processing, and lack of effective protection of data during data processing and storage.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a big data management method, which specifically includes the following steps:

[0007] After removing noise from the acquired big data, processing missing values, standardizing and normalizing them, data integration is performed to obtain integrated data;

[0008] The integrated data is divided into private data and regular data. The storage nodes are evaluated based on three aspects: reading speed, available storage capacity, and reading error rate. The corresponding evaluation values ​​are assigned and the performance value is calculated by summing the three.

[0009] Select the two groups of storage nodes with the highest performance values, encrypt and analyze the private data and regular data, divide the data by time, and generate duplicate data or single data analysis signals;

[0010] Analyze the repeated data analysis signal, label the repeated data and replace them to generate replacement information, convert the remaining data into binary, use four binary numbers to generate 16 groups of replacement templates, match and replace them with the converted data, and combine them to obtain recombined data. Do the same process for the single data analysis signal;

[0011] Calculate the average network speed of the storage nodes, divide and store the reorganized data accordingly, and generate storage information.

[0012] As a further solution of the present invention, the specific method of evaluating the storage node reading speed is:

[0013] Record the read speed data of the storage node in time period t, and set the assignment interval based on historical data analysis: if the average read speed is ≥ 100MB / s, assign a value of 3; if the average read speed is ≤ 50MB / s, assign a value of 3.

[0014] If the average speed is less than 100MB / s, assign a value of 2; if the average read speed is less than 50MB / s, assign a value of 1. Match the calculated average with the assigned value interval to determine the read speed assignment.

[0015] As a further solution of the present invention, a specific method for evaluating the available storage capacity of a storage node is as follows:

[0016] The available storage capacity is divided into value ranges based on past data: if the available capacity ratio is ≥80%, the value is 3; if the available capacity ratio is 50% or less and the ratio is less than 80%, the value is 2; if the ratio is less than 50%, the value is 1.

[0017] As a further solution of the present invention, the specific method of evaluating the storage node read error rate is:

[0018] The error ratio of read operations per unit time is statistically analyzed to determine the assigned value: if the error rate is ≤ 1%, the value is 3; if the error rate is 1% < error rate ≤ 5%, the value is 2; if the error rate is > 5%, the value is 1. For example, if 20 errors occur in 1000 read operations, the error rate is 2%, and the corresponding value is 2.

[0019] As a further solution of the present invention, the specific method of generating repeated data or single data analysis signals is:

[0020] Sort the private data by data time, using time T as the partitioning standard to obtain interval data sets, and label them as a, where a = 1, 2, ..., b, and b is the number of interval data sets. Check whether there is duplicate data in each interval data set.

[0021] If it exists, a repeated data analysis signal is generated; if it does not exist, a single data analysis signal is generated;

[0022] The same analysis is performed on conventional data and the same analysis signals are generated.

[0023] As a further solution of the present invention, the specific method of labeling and replacing the duplicate data to generate replacement information is as follows:

[0024] Extract duplicate data from interval datasets, label them in order of appearance, mark different types of duplicate data as n, where n = 1, 2, ..., m, where m is the number of duplicate data types, obtain all duplicate data of that type, replace them with 1 c in order of appearance, where c = 1, 2, ..., j, where j is the number of duplicate data of the same type, process all types of duplicate data according to this rule, and generate replacement information.

[0025] As a further solution of the present invention, the specific method of obtaining the recombined data is:

[0026] Convert the remaining data into conversion data using binary format, generate 16 groups of replacement templates using four binary numbers 0 and 1, sort them according to the rule that 0 takes precedence over 1, generate a replacement table, match a group of English characters for each group of templates, and match the conversion data with the replacement table according to priority;

[0027] If the same template is encountered, it is also replaced according to priority, and the replaced English characters are combined with the remaining conversion data to obtain reorganized data, completing the matching and replacement of all conversion data.

[0028] As a further solution of the present invention, the specific method of generating the storage information is:

[0029] Obtain the storage nodes corresponding to the private data and regular data, collect the network speed of these nodes within time t, and calculate the average speed. Use the average as the standard to split and reorganize the data, and then store it to generate storage information.

[0030] Big data management system, including:

[0031] A data acquisition unit, which is used to acquire big data and transmit it to a data processing and classification unit;

[0032] A data processing and classification unit, which is used to perform denoising and missing value processing on the acquired big data to obtain preprocessed data, perform standardization and normalization processing, and perform data integration to obtain integrated data. The integrated data is then classified into private data and regular data, and different access rights are set. Storage nodes are evaluated based on three aspects: reading speed, available storage capacity, and reading error rate, and corresponding values ​​are obtained. The sum of the three values ​​is calculated to obtain the storage node performance value. The storage nodes corresponding to the two groups of performance values ​​with the largest values ​​are selected to encrypt and perform storage analysis on the private data and regular data, and the selected storage nodes are transmitted to the data encryption and analysis unit;

[0033] The data encryption analysis unit is used to encrypt private data and regular data, divide the data into interval data sets according to the data time, and generate duplicate data analysis signals and single data analysis signals based on the existence of duplicate data, and analyze the two separately;

[0034] Analyze the repeated data analysis signal, obtain the repeated data and label it, replace the repeated data according to the repeated data label, generate replacement information, then perform binary conversion on the remaining data to obtain converted data, and generate 16 groups of replacement templates based on the four binary numbers 0 and 1. Match and replace the replacement templates with the converted data, and combine them with the remaining converted data to obtain reconstructed data. Similarly, analyze the single data analysis signal to generate reconstructed data, and then transmit the reconstructed data to the data storage and analysis unit;

[0035] The data storage and analysis unit is used to perform storage analysis on the reorganized data, calculate the average network speed of the storage nodes within a time period, and use it as a standard to evenly divide the reorganized data, while storing and generating storage information.

[0036] Beneficial effects

[0037] The present invention provides a big data management method and a big data management system. Compared with the existing technology, it has the following advantages:

[0038] This invention uses a systematic preprocessing process to remove noise and address missing values. It also standardizes, normalizes, and integrates data, unifying data formats and semantics. This improves data usability and accuracy, laying a solid foundation for subsequent analysis. It also performs secondary data classification, distinguishing between private and regular data and adopting different access rights settings. It also performs encryption analysis on private data, effectively preventing its leakage and meeting stringent data security requirements.

[0039] This invention quantitatively evaluates storage node performance based on three dimensions: read speed, available storage capacity, and read error rate. This allows accurate selection of the optimal storage node, improving data storage efficiency and stability. It also designs an efficient analysis process for duplicate and single data, replacing duplicate data with labels and performing binary conversion and matching replacement on the remaining data, enabling rapid data reorganization. Furthermore, data is segmented and reorganized based on the average storage node network speed, further improving data storage and processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a diagram of the steps and methods of the present invention;

[0041] Figure 2 Replacement of the present invention;

[0042] Figure 3 This is a block diagram of the system principle of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] Example 1

[0045] See also Figure 1 and Figure 2 , this application provides a big data management method, which specifically includes the following steps:

[0046] Step 1: Obtain big data, where big data refers to data from different time periods, and preprocess the resulting big data. This preprocessing includes removing noise and handling missing values ​​to obtain preprocessed data. The preprocessed data is then standardized and normalized to convert the data into a unified format and scale for comparison and analysis. For example, data of varying magnitudes can be normalized to the interval [0, 1] and integrated to obtain integrated data. Data integration specifically refers to matching and fusing data patterns from different data sources to unify the data's structure and semantics.

[0047] Preprocessing operations mainly include removing noise and processing missing values. Noise data may be caused by data acquisition equipment failure, transmission interference, etc. For example, in the data collected by a set of temperature sensors, if there are individual abnormal values ​​that obviously deviate from the normal range, such as a value of -100°C suddenly appearing when the normal temperature range is 20°C-30°C, this is very likely to be noise data, which can be removed through algorithms such as mean filtering and median filtering.

[0048] In terms of handling missing values, if the customer age field of some orders in a sales data report is empty, the multiple filling method can be used to use known customer attributes (such as purchase frequency, consumption amount, etc.) to predict the missing age value through a regression model, or directly delete records with too many missing values ​​to obtain preprocessed data.

[0049] Step 2: Perform secondary classification on the integrated data to obtain private data and regular data. The classification is specifically based on the established classification standards. Private data usually contains information that can directly or indirectly identify an individual, such as ID numbers, biometric data, and health records. Regular data refers to data that does not involve sensitive privacy and is publicly accessible and usable, such as statistical summary data and public market research data. Different access permissions are set for each of them. At the same time, all storage nodes are obtained and labeled as i, where i = 1, 2, ..., j, where j represents the number of storage nodes. Then, the storage performance of storage node i is analyzed, and the storage performance is evaluated based on three aspects: read speed, available storage capacity, and read error rate.

[0050] Obtain the average reading speed of the storage node within time t, and match the average reading speed with the corresponding assignment interval. The assignment interval is obtained based on a large amount of data analysis to obtain the reading speed assignment;

[0051] Similarly, the analysis method for read speed is used to assign values ​​to the available storage capacity and read error rate respectively, and the assigned values ​​are summed to obtain the storage node performance value;

[0052] First, record a series of read speed data of the storage node within a specific time period t. For example, within 1 hour (t = 1 h), record the read speed of the storage node every 1 minute to obtain 60 speed samples, calculate the average of the collected read speed data, and pre-set the reading speed assignment interval based on a large amount of historical data analysis. For example, when the average read speed is ≥ 100MB / s, the assignment value is 3; when 50MB / s ≤ the average read speed is < 100MB / s, the assignment value is 2; when the average read speed is < 50MB / s, the assignment value is 1. Match the calculated average read speed with these assignment intervals to obtain the read speed assignment;

[0053] Available storage capacity: Taking a cloud storage service as an example, based on past data, the available storage capacity is divided into different assignment intervals. When the proportion of available storage capacity to the total capacity is ≥80%, the assignment value is 3; when the proportion is 50% ≤ <80%, the assignment value is 2; when the proportion is <50%, the assignment value is 1. Assuming that the total capacity of a storage node is 1 TB, the current available capacity is 600GB, and its available proportion is 60%, the corresponding assignment value is 2.

[0054] Read error rate: The value is determined by counting the proportion of errors in read operations per unit time. For example, when the read error rate is ≤1%, the value is 3; when 1% < read error rate ≤5%, the value is 2; when the read error rate is >5%, the value is 1. If 20 errors occur in 1000 read operations and the error rate is 2%, the value is 2.

[0055] The performance value of a storage node is calculated by summing the values ​​assigned to read speed, available storage capacity, and read error rate. For example, if a storage node has a read speed of 2, an available storage capacity of 2, and a read error rate of 3, then the performance value of the storage node is 2 + 2 + 3 = 7.

[0056] Obtain the evaluation levels corresponding to all storage nodes i, and select the two groups of storage nodes with the largest storage node performance values. At the same time, store the private data and regular data with the selected storage nodes respectively. Specifically, store the private data in the storage node with the largest performance value in the two groups of storage nodes, and store the regular data in the remaining storage nodes.

[0057] Step 3: Perform encryption analysis on the private data, obtain all private data and label them, and obtain the data time corresponding to the private data. Then, sort the data from the front to the back according to the data time. Here, sorting from the front to the back specifically means sorting in chronological order. Then, divide the private data according to the data time using time T as the division standard to obtain interval data sets. The specific value of time T is set by the operator and labeled as a, where a=1, 2, ..., b, where b represents the number of interval data sets. Then, determine whether there is duplicate data in interval data set a. If there is duplicate data in interval data set a, a duplicate data analysis signal is generated. Otherwise, if there is no duplicate data in interval data set a, a single data analysis signal is generated.

[0058] The same analysis is performed on conventional data and the same analysis signals are generated.

[0059] With the help of professional data collection tools, private data is collected from multiple data sources. During the collection process, each piece of private data is assigned a unique identifier, and the time when the data was generated, i.e., the data time, is accurately recorded. Taking the online medical service platform as an example, every time a patient performs an operation such as a medical record query or a diagnosis record update, the system automatically collects this private data, assigns it a unique number, and records the time when the operation occurred.

[0060] Use efficient sorting algorithms (such as the quick sort algorithm) to sort the collected private data in ascending order based on the time sequence of the data. The sorted data presents a clear order in the time dimension;

[0061] Operators flexibly set the time interval T based on business needs and analysis objectives. Using time T as the criterion, the sorted private data is divided into multiple interval datasets. Each interval dataset is labeled a, with a ranging from 1 to b, where b represents the total number of interval datasets. For example, when processing user behavior data, the operator sets time T to 1 week, and the system divides the collected user behavior data into multiple interval datasets by week.

[0062] For each interval dataset a, the system compares the data within the dataset by constructing a hash table or using a similarity algorithm to determine whether there are duplicates. If duplicate data is detected, the system generates a duplicate data analysis signal; if all data in the dataset is unique, a unique data analysis signal is generated. For example, if multiple identical user updates are found within a given interval dataset, a duplicate data analysis signal is generated.

[0063] Similar analysis processes are performed on regular data, similar to the analysis process for private data. For example, during the data collection phase, each sales record is assigned a unique number and the time of sale is recorded. After sorting the sales data by time, the sales records are divided into different interval data sets based on a time period T set by the operator, such as 1 day. By comparing the data, the presence of duplicate sales records in each interval data set is determined, thereby generating corresponding duplicate data analysis signals or single data analysis signals.

[0064] Step 4: Analyze the obtained repeated data analysis signals and single data analysis signals separately. The specific method of analyzing the repeated data analysis signals is as follows:

[0065] Get the corresponding interval data set and get the duplicate data in it. At the same time, label the duplicate data in the order of appearance. Then replace the duplicate data according to the label. The specific replacement method is:

[0066] Duplicate data is obtained and labeled as n, where n = 1, 2, ..., m, where m represents the type of duplicate data. Then, the duplicate data labeled n = 1 is used as an example for replacement analysis. All duplicate data with n = 1 are obtained, and the corresponding order of occurrence is determined. Then, the duplicate data is replaced with 1c in sequence, where c = 1, 2, ..., j, where j represents the number of duplicate data of the same type. This process is repeated for all duplicate data to obtain nc, generating replacement information.

[0067] Then, the remaining data is converted into binary to obtain the converted data. At the same time, a replacement template is generated with four binary numbers 0 and 1, and 16 groups of replacement templates are obtained according to different combinations of 0 and 1. Specific replacement templates are, for example, 0000, 0001, 1111, 1110, and so on to obtain 16 groups of replacement templates. The replacement templates are sorted according to the priority of 0 and 1. Here, it is assumed that the priority of 0 is higher than that of 1, and when sorting, the sorting position is before 1, such as Figure 2 As shown, a replacement table is generated, and each group of replacement templates corresponds to a group of English characters, and the English characters here are represented as corresponding replacement forms;

[0068] Match the converted data with the replacement table, and replace according to the priority in the table when matching. If there are identical replacement templates, replace them according to the priority in the replacement table, and combine the corresponding English characters with the remaining converted data to obtain recombined data. Repeat this process for all converted data.

[0069] The single data analysis signal is analyzed. The specific analysis method is the same as that of the repeated data analysis signal. The difference from the repeated data analysis signal is that when analyzing the single data analysis signal, there is no need to analyze the existing repeated data. Recombined data is obtained after analysis.

[0070] Step 5: Perform storage analysis on the reorganized data to obtain the corresponding storage nodes. The storage nodes here represent the storage nodes corresponding to the private data and regular data respectively. At the same time, obtain the network speed corresponding to the storage node within time t, and calculate the average network speed. Then, divide the reorganized data based on the average network speed and store it at the same time to generate storage information.

[0071] Example 2

[0072] See also Figure 3 The present application provides a big data management system, which includes a data acquisition unit, a data processing and classification unit, a data encryption and analysis unit, and a data storage and analysis unit, and combines Figure 3 It can be known that the functional units are electrically connected in a unidirectional manner.

[0073] A data acquisition unit, which is used to acquire big data and transmit it to a data processing and classification unit;

[0074] A data processing and classification unit, which is used to perform denoising and missing value processing on the acquired big data to obtain preprocessed data, perform standardization and normalization processing, and perform data integration to obtain integrated data. The specific processing method is the same as the processing method of step one in Example 1. The integrated data is then classified to obtain private data and regular data, and different access rights are set. At the same time, the storage nodes are evaluated from three aspects: reading speed, available storage capacity, and read error rate, and corresponding values ​​are obtained. The sum of the three values ​​is calculated to obtain a storage node performance value. The storage nodes corresponding to the two groups of performance values ​​with the largest values ​​are selected to encrypt and store the private data and regular data. The selected storage nodes are transmitted to the data encryption analysis unit. The specific processing method is the same as the processing process of step two in Example 1.

[0075] A data encryption analysis unit is used to encrypt private data and regular data, divide the data into interval data sets according to the data time, and generate duplicate data analysis signals and single data analysis signals based on the presence of duplicate data, and analyze the two separately. The specific processing method is similar to the processing process of step 3 in embodiment 1;

[0076] Analyze the repeated data analysis signal, obtain repeated data and label them, replace the repeated data according to the repeated data labels, generate replacement information, then perform binary conversion on the remaining data to obtain converted data, and generate 16 groups of replacement templates based on the four binary numbers 0 and 1. Match and replace the replacement templates with the converted data, and combine them with the remaining converted data to obtain reconstructed data. Similarly, analyze the single data analysis signal to generate reconstructed data, and the specific processing method is the same as the processing process of step 4 in embodiment 1. Then, transmit the reconstructed data to the data storage and analysis unit;

[0077] The data storage and analysis unit is used to perform storage analysis on the reorganized data, calculate the average network speed of the storage nodes within a time period, and divide the reorganized data equally based on the average network speed, and store and generate storage information at the same time. The specific processing method is the same as the processing process of step five in embodiment one.

[0078] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0079] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A big data management method, characterized in that: The method specifically comprises the following steps: After removing noise from the acquired big data, processing missing values, standardizing and normalizing them, data integration is performed to obtain integrated data; The integrated data is divided into private data and regular data. The storage nodes are evaluated based on three aspects: reading speed, available storage capacity, and reading error rate. The corresponding evaluation values ​​are assigned and the performance value is calculated by summing the three. Select the two groups of storage nodes with the highest performance values, encrypt and analyze the private data and regular data, divide the data by time, and generate duplicate data or single data analysis signals; Analyze the repeated data analysis signal, label the repeated data and replace them to generate replacement information, convert the remaining data into binary, use four binary numbers to generate 16 groups of replacement templates, match and replace them with the converted data, and combine them to obtain recombined data. Do the same process for the single data analysis signal; Calculate the average network speed of the storage nodes, divide and store the reorganized data accordingly, and generate storage information.

2. The big data management method according to claim 1, characterized in that: The specific method for evaluating the storage node read speed is: Record the read speed data of the storage node in time period t, and set the assignment interval based on historical data analysis: if the average read speed is ≥ 100MB / s, assign a value of 3; if the average read speed is ≤ 50MB / s, assign a value of 3. If the average speed is less than 100MB / s, assign a value of 2; if the average read speed is less than 50MB / s, assign a value of 1. Match the calculated average with the assigned value interval to determine the read speed assignment.

3. The big data management method according to claim 1, characterized in that: The specific method for evaluating the available storage capacity of a storage node is as follows: The available storage capacity is divided into value ranges based on past data: if the available capacity ratio is ≥80%, the value is 3; if the available capacity ratio is 50% or less and the ratio is less than 80%, the value is 2; if the ratio is less than 50%, the value is 1.

4. The big data management method according to claim 1, characterized in that: The specific method for evaluating the storage node read error rate is as follows: The error ratio of read operations per unit time is statistically analyzed to determine the assigned value: if the error rate is ≤ 1%, the value is 3; if the error rate is 1% < error rate ≤ 5%, the value is 2; if the error rate is > 5%, the value is 1. For example, if 20 errors occur in 1000 read operations, the error rate is 2%, and the corresponding value is 2.

5. The big data management method according to claim 1, characterized in that: The specific method of generating repeated data or single data analysis signals is: Sort the private data by data time, using time T as the partitioning standard to obtain interval data sets, and label them as a, where a = 1, 2, ..., b, and b is the number of interval data sets. Check whether there is duplicate data in each interval data set. If it exists, a repeat data analysis signal is generated; If not present, a single data analysis signal is generated; The same analysis is performed on conventional data and the same analysis signals are generated.

6. The big data management method according to claim 1, characterized in that: The specific method of labeling and replacing duplicate data to generate replacement information is as follows: Extract duplicate data from the interval data set, label them in order of appearance, mark different types of duplicate data as n, where n = 1, 2, ..., m, where m is the number of duplicate data types, obtain all duplicate data of that type, replace them with 1c in order of appearance, where c = 1, 2, ..., j, where j is the number of duplicate data of the same type, process all types of duplicate data according to this rule, and generate replacement information.

7. The big data management method according to claim 1, characterized in that: The specific method of obtaining the reorganized data is: Convert the remaining data into conversion data using binary format, generate 16 groups of replacement templates using four binary numbers 0 and 1, sort them according to the rule that 0 takes precedence over 1, generate a replacement table, match a group of English characters for each group of templates, and match the conversion data with the replacement table according to priority; If the same template is encountered, it is also replaced according to priority, and the replaced English characters are combined with the remaining conversion data to obtain reorganized data, completing the matching and replacement of all conversion data.

8. The big data management method according to claim 1, characterized in that: The specific method of generating the storage information is as follows: Obtain the storage nodes corresponding to the private data and regular data, collect the network speed of these nodes within time t, and calculate the average speed. Use the average as the standard to split and reorganize the data, and then store it to generate storage information.

9. A big data management system for executing the big data management method according to any one of claims 1 to 8, characterized in that: include: A data acquisition unit, which is used to acquire big data and transmit it to a data processing and classification unit; A data processing and classification unit, which is used to perform denoising and missing value processing on the acquired big data to obtain preprocessed data, perform standardization and normalization processing, and perform data integration to obtain integrated data. The integrated data is then classified into private data and regular data, and different access rights are set. Storage nodes are evaluated based on three aspects: reading speed, available storage capacity, and reading error rate, and corresponding values ​​are obtained. The sum of the three values ​​is calculated to obtain the storage node performance value. The storage nodes corresponding to the two groups of performance values ​​with the largest values ​​are selected to encrypt and perform storage analysis on the private data and regular data, and the selected storage nodes are transmitted to the data encryption and analysis unit; The data encryption analysis unit is used to encrypt private data and regular data, divide the data into interval data sets according to the data time, and generate duplicate data analysis signals and single data analysis signals based on the existence of duplicate data, and analyze the two separately; Analyze the repeated data analysis signal, obtain the repeated data and label it, replace the repeated data according to the repeated data label, generate replacement information, then perform binary conversion on the remaining data to obtain converted data, and generate 16 groups of replacement templates based on the four binary numbers 0 and 1. Match and replace the replacement templates with the converted data, and combine them with the remaining converted data to obtain reconstructed data. Similarly, analyze the single data analysis signal to generate reconstructed data, and then transmit the reconstructed data to the data storage and analysis unit; The data storage and analysis unit is used to perform storage analysis on the reorganized data, calculate the average network speed of the storage nodes within a time period, and use it as a standard to evenly divide the reorganized data, while storing and generating storage information.

Images