A data processing method, device, storage system and computer readable storage medium

By cleaning and classifying data, the problem of impact on accuracy and accuracy in data processing is solved, efficient classification and secure storage of data are achieved, and the overall effect of data processing and storage is improved.

CN117874543BActive Publication Date: 2024-12-20XI AN SHANGDING ENERGY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410278304.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-12
Publication Date
2024-12-20
Estimated Expiration
2044-03-12

AI Technical Summary

Technical Problem

In the prior art, the accuracy and accuracy of data are affected due to large capacity during processing, and the storage is not classified in detail, resulting in poor data processing effect.

Method used

By initially cleaning and deletion of duplicate and abnormal data, similarity analysis is performed to convert the data into standard vectors for classification, and deduplication and encryption are performed during storage to ensure data quality and security.

Benefits of technology

It improves the accuracy and efficiency of data processing, ensures the overall quality and security of data, avoids storage chaos, and improves the overall effect of data analysis and storage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117874543B_ABST
    Figure CN117874543B_ABST
Patent Text Reader

Abstract

The present invention discloses a data processing method, device, storage system and computer-readable storage medium. The present invention relates to the field of data processing technology and solves the problem that different data are not classified in detail, which causes some irrelevant data to be stored in the same storage space, resulting in the stored data being relatively chaotic. The present invention performs deduplication processing on the data and then removes the internal abnormal data. For different standardized data, similarity analysis is performed on them, and the different standardized data are converted into standard vectors. The vector features between different vectors are analyzed to determine the specific features of the corresponding standardized data, so that the different standardized data can be classified and distinguished accordingly, and the data with similar features are stored in one place, which is convenient for subsequent analysis and sorting of the data, and the processing efficiency is better and the overall storage effect is better.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, storage system and computer-readable storage medium. Background Art

[0002] First, data collection is carried out, which involves collecting data from different data sources and methods to prepare for subsequent processing, and then data preprocessing: This step is the key to ensuring data quality, mainly including the following aspects:

[0003] Data cleaning: “cleaning” the data by filling in missing values, smoothing noisy data, removing outliers, and resolving data inconsistencies;

[0004] Data integration: merging data from multiple sources into a single consistent set of data;

[0005] Data reduction: reduce the amount of data while maintaining data integrity to improve processing efficiency;

[0006] Data transformation: converting data into a format suitable for analysis, such as digitizing categorical data;

[0007] Data storage: Store the processed data in appropriate media to facilitate subsequent access and analysis.

[0008] The application with publication number CN114840356B provides a data processing method, a data processing system and related devices, which relate to the field of data processing, including: obtaining shared data of at least a preset length; loading the shared data into a shared area of ​​a multidimensional data space; establishing a multidimensional data mapping of the shared data in the shared area so that each process can construct a shared data access linked list for the multidimensional data mapping; monitoring each process's update request for the multidimensional data mapping, and generating a change preprocessing queue within the process; determining whether the update request can be completed by changing the coordinate mapping; if so, completing the update request by changing the shared data access linked list; if not, rejecting the update request, creating a new local variable within the process, and temporarily storing the update request in the update queue. The present application can operate simultaneously based on different dimensions, avoid conflicts in the occupation of shared data, and improve the encryption strength and logical processing capabilities of the data.

[0009] During the data processing process, the data occupies a large capacity, which will cause the accuracy of the corresponding data and the accuracy of the results to be affected accordingly. When the data is stored, different data are not classified in detail, which will cause some unrelated data to be stored in the same storage space, resulting in confusion between the stored data and failure to achieve good data processing effects. Summary of the invention

[0010] In view of the deficiencies in the prior art, the present invention provides a data processing method, device, storage system and computer-readable storage medium, which solves the problem that the lack of detailed classification between different data will cause some irrelevant data to be stored in the same storage space location, resulting in relatively chaotic stored data.

[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions: a data processing method, comprising the following steps:

[0012] S1. Confirm the processed data of this stage and perform preliminary cleaning, give priority to deleting duplicate data within the processed data, then lock the abnormal data, and delete the abnormal data within the processed data, so as to obtain standardized data through processing data, including:

[0013] The data to be processed in this stage is determined, and the identical data is found from the determined data, and the duplicate data is deleted, and one set of data is retained. After the processing is completed, the data to be processed in the second stage is confirmed;

[0014] Arrange the result parameters of the second-stage processing data from small to large to generate a set of parameter sequences, and divide the parameter sequence into four equal parts according to the number of result parameters. If the four equal parts cannot be completed, keep the number of result parameters in the middle two equal parts the same;

[0015] Get the first parameter of the second part and mark it as SS, then confirm the last parameter of the third part and mark it as MS, then confirm the difference CZ between MS and SS: CZ=MS-SS, and determine a set of value intervals based on the determined difference CZ: [CZ-1.5×CZ, CZ+1.5×CZ], mark the data in this parameter sequence that does not belong to this value interval as abnormal data, and remove this abnormal data from the processed data, so as to obtain standardized data;

[0016] S2. Perform similarity analysis on standardized data at different stages, convert different standardized data into standard vectors, and then determine the corresponding features of the corresponding standardized data by analyzing different standard vectors of different standardized data, so as to classify the standardized data at different stages and store the standardized data belonging to the same classification in the same space, including:

[0017] According to the practical experience of the operators, three sets of different characteristic values ​​are extracted from the standardized data, and the characteristic values ​​belong to the relevant attribute data of the standardized data;

[0018] According to the different eigenvalues ​​corresponding to the different standardized data, the corresponding spatial points are determined in the three-dimensional coordinate system in the selected order, and then the standard vector of the standard data is generated from the origin to the location of the spatial point;

[0019] Determine a set of baselines in the three-dimensional coordinate system. The baselines are located between the X-axis, Y-axis, and Z-axis, and the included angles between the baselines and the X-axis, Y-axis, and Z-axis are consistent. Determine the included angle A between the standard vector and the baselines. Then, vectorize the standardized data of other stages and determine the included angle A between the baselines. i , where i represents the standardized data at different stages;

[0020] According to the determined angle A i and a preset angle interval, dividing the standard vectors belonging to the same angle interval into vectors of the same interval, wherein the angle interval is a preset interval;

[0021] Then classify the standard vector lengths of vectors in the same interval, and construct several groups of length classification intervals according to the preset value Ys: (0, Ys], (Ys, 2Ys], ..., ((n-1)Ys, nYs], where Ys is the preset value, and n represents the number of corresponding length classification intervals. According to the lengths of different standard vectors, several standard vectors of vectors in the same interval are divided into several different length classification intervals. According to the division results, the standardized data corresponding to the vectors in the same interval are stored in the same main space, and then the main space is divided into n sub-spaces, and then the standardized data corresponding to the different length classification intervals are sequentially stored in the corresponding sub-spaces;

[0022] S3. For the standardized data stored in the same space, firstly perform unified data deduplication processing, replace the duplicate data in this space with duplicate markers, then encrypt the standardized data in the same storage space, determine the encryption logic, and retain it in the corresponding storage space. The specific methods of unified data deduplication processing include:

[0023] First, confirm the duplicate data of several standardized data in the same space, and replace the confirmed duplicate data with corresponding duplicate markers. The expression of the duplicate markers is: C k , where k = 1, 2, ..., m, m represents the number of repeated data;

[0024] Using repeated marker C k Replace several duplicate data appearing in the standardized data in sequence, and after the replacement is completed, generate a duplicate data mark replacement table, and mark the replaced main space as the space to be encrypted;

[0025] The specific methods for encrypting the encrypted space include:

[0026] First, confirm the encryption key of the primary space corresponding to the secondary space, and mark the encryption key as JM;

[0027] Then, according to the time sequence of the subspaces in the corresponding main space during the construction process, the subspaces are initially sorted, and the data capacity in different subspaces is calibrated as R according to the sorting method. t , where t=1, 2, ..., g, where g represents the total number of corresponding subspaces. When t is 1, it means that this space is ranked first in the main space.

[0028] Using |R1-R2|+|R2-R3|+……+|R g-1 -R g |=HB to obtain the combined value HB of the capacity differences of several sub-spaces;

[0029] The intermediate value ZZ is obtained by combining the value and the encryption key (HB-JM), and a logical formula is set in the main space. The logical formula is: Y=HB-ZZ, and the encryption key JM of the main space is deleted. When the capacity value of the secondary space in the main space changes, the intermediate value ZZ will change accordingly.

[0030] Preferably, a data processing device comprises:

[0031] At least three groups of processors:

[0032] The first processor preferentially deletes duplicate data in the processed data, then locks the abnormal data, and deletes the abnormal data in the processed data, thereby obtaining standardized data by processing the data;

[0033] The second processor performs similarity analysis on the standardized data at different stages, converts the different standardized data into standard vectors, and then determines the corresponding features of the corresponding standardized data by analyzing the different standard vectors of the different standardized data, thereby classifying the standardized data at different stages, and storing the standardized data belonging to the same classification in the same space;

[0034] The third processor first performs unified data deduplication processing on the standardized data stored in the same space, replaces the duplicate data in this space with duplicate markers, and then encrypts the standardized data in the same storage space, determines the encryption logic formula, and retains it in the corresponding storage space.

[0035] Preferably, a data storage system stores the generated standardized data, and according to the specific classification of the data, stores the standardized data of different classifications in different storage space locations, wherein the storage space includes a plurality of main spaces, and a main space includes a plurality of secondary spaces;

[0036] Allows external operators to set keys, set conversion logic based on the set keys and specific stored data, and delete internal encryption keys;

[0037] Each main space is a single set of computer-readable storage media, and its storage medium is a solid-state readable disk for storing standardized data. The division of its internal space is confirmed and divided by the storage system.

[0038] The present invention provides a data processing method, device, storage system and computer-readable storage medium. Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention removes internal abnormal data after deduplication processing to ensure the overall data quality of the data to be processed. Subsequently, for different standardized data, similarity analysis is performed on them, and different standardized data are converted into standard vectors. The vector features between different vectors are analyzed to determine the specific features of the corresponding standardized data, so that different standardized data can be classified and distinguished accordingly, and data with similar features are stored in one place, which is convenient for subsequent analysis and sorting of data, and the processing efficiency is better and the overall storage effect is better;

[0040] Subsequently, for the corresponding data that has been stored, the set encryption key is deleted, and based on the proposed encryption key, the encryption logic of the corresponding stored data is determined, the inherent encryption logic is retained, and the encryption key is deleted. It is also difficult for external personnel to steal data by stealing the encryption key, thereby improving the security and privacy of the data storage process and the overall storage effect of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the principle framework of the present invention;

[0042] Figure 2 This is a schematic diagram for determining the vector angle of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1

[0045] See also Figure 1 , the present application provides a data processing method, comprising the following steps:

[0046] S1. Confirm the processed data of this stage and perform preliminary cleaning, give priority to deleting duplicate data in the processed data, then lock the abnormal data, and delete the abnormal data in the processed data, so as to obtain standardized data by processing the data. Specifically, for the data that appears, there are corresponding duplicate data and abnormal data in the data. Therefore, in order to ensure the quality of the processed data, it is necessary to delete the corresponding duplicate data in the data, and also remove the corresponding abnormal data to maintain the quality of the corresponding data. Among them, the specific way to obtain standardized data includes:

[0047] The data to be processed in this stage is determined, which is generally determined according to a cycle or time limit. At a certain period of time, a group of data to be processed is locked, and the same data is found from the determined data, and duplicate data is deleted, and one group of data is retained. After the processing is completed, the data for the second stage processing is confirmed;

[0048] Arrange the result parameters of the second-stage processed data from small to large (specifically, a single set of data includes the relevant data and result data needed in the process. Based on the specific logical relationship within the data, the result data, that is, the result parameters, can be directly locked), generate a set of parameter sequences, and divide this parameter sequence into four equal parts according to the number of result parameters. If the four equal parts cannot be completed, keep the number of result parameters in the middle two equal parts the same (for example, assuming that there are 10 result parameters, namely 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, then the four equal parts divided according to the above method are: (1, 2), (3, 4, 5), (6, 7, 8), (9, 10));

[0049] Get the first parameter of the second equal part, mark it as SS, then confirm the last parameter of the third equal part, mark it as MS, then confirm the difference CZ between MS and SS: CZ=MS-SS, determine a set of value intervals based on the determined difference CZ: [CZ-1.5×CZ, CZ+1.5×CZ], mark the data in this parameter sequence that does not belong to this value interval as abnormal data, and remove this abnormal data from the processed data to obtain standardized data. Specifically, the reason why this method is used here to determine this value interval is that in the process of determining abnormal data, the box plot method is used here to identify the corresponding abnormal data and remove them;

[0050] S2. Combination Figure 2, for standardized data at different stages, similarity analysis is performed, different standardized data are converted into standard vectors, and then the corresponding features of the corresponding standardized data are determined by analyzing the different standard vectors of different standardized data (specifically, different standardized data have specified feature parameters set internally, and different standardized data can be classified by manually determining the corresponding classification of their feature parameters), so as to classify the standardized data at different stages, and store the standardized data belonging to the same classification in the same space, wherein the sub-steps of data classification include:

[0051] Based on the practical experience of the operator, three different sets of characteristic values ​​are extracted from the standardized data, and the characteristic values ​​belong to the relevant attribute data of the standardized data (they can be some parameters closely related to the data, such as data capacity and data reading rate, and the corresponding values ​​to be extracted are determined in advance by the operator);

[0052] According to the different eigenvalues ​​corresponding to the different standardized data, the corresponding spatial points are determined in the three-dimensional coordinate system in the selected order, and then the standard vector of the standard data is generated from the origin to the location of the spatial point;

[0053] Determine a set of baselines in the three-dimensional coordinate system. The baselines are located between the X-axis, Y-axis, and Z-axis, and the included angles between the baselines and the X-axis, Y-axis, and Z-axis are consistent. Determine the included angle A between the standard vector and the baselines. Then, vectorize the standardized data of other stages and determine the included angle A between the baselines. i , where i represents the standardized data at different stages;

[0054] According to the determined angle A i and a preset angle interval, dividing the standard vectors belonging to the same angle interval into vectors of the same interval, wherein the angle interval is a preset interval, and its specific value is prepared in advance by the operator based on experience;

[0055] Then classify the standard vector lengths of vectors in the same interval, and construct several groups of length classification intervals according to the preset value Ys: (0, Ys], (Ys, 2Ys], ..., ((n-1)Ys, nYs], where Ys is a preset value, and its specific value is determined by the operator based on experience, and n represents the number of corresponding length classification intervals. According to the lengths of different standard vectors, several standard vectors of vectors in the same interval are divided into several different length classification intervals. According to the division results, the standardized data corresponding to the vectors in the same interval are stored in the same main space, and then the main space is divided into n sub-spaces, and then the standardized data corresponding to the different length classification intervals are sequentially stored in the corresponding sub-spaces;

[0056] Specifically, in order to ensure that the data that needs to be processed can be adequately managed during subsequent analysis and storage, it is necessary to perform similarity analysis on the data to determine the specific classification of the corresponding data, so that several different standardized data can be processed in different ways, the accuracy of subsequent data processing can be improved, and data comparison can be facilitated during the subsequent processing, so that data of the same classification can be stored in the same space, and data features can be confirmed and classified one by one.

[0057] S3. For the standardized data stored in the same space, firstly perform unified data deduplication processing, replace the duplicate data in this space with duplicate markers, so as to reduce the storage capacity of this storage space, then encrypt the standardized data in the same storage space, determine the encryption logic, and retain it in the corresponding storage space;

[0058] Among them, the specific methods of unified deduplication processing of data include:

[0059] First, confirm the duplicate data of several standardized data in the same space, and replace the confirmed duplicate data with corresponding duplicate markers. The expression of the duplicate markers is: C k , where k = 1, 2, ..., m, m represents the number of repeated data;

[0060] Using repeated marker C k Several duplicate data appearing in the standardized data are replaced in sequence, and after the replacement is completed, a duplicate data mark replacement table is generated, and the same space after replacement is marked as the space to be encrypted. Specifically, the deduplication processing performed in step S1 is performed on the same standard data, and the duplicate data replacement performed in this step S3 is performed on several standard data, that is, there may be multiple duplicate data in several standardized data, and it is necessary to use the corresponding duplicate mark to replace the duplicate data, so as to reduce the space capacity of the corresponding secondary space and achieve a better space reduction effect;

[0061] Among them, the specific methods of encrypting the encrypted space include:

[0062] First, confirm the encryption key of the primary space corresponding to the secondary space, where the encryption key is prepared in advance by the operator based on experience, and the encryption key is marked as JM;

[0063] Then, according to the time sequence of the subspaces in the corresponding main space during the construction process, the subspaces are initially sorted, and the data capacity in different subspaces is calibrated as R according to the sorting method. t , where t=1, 2, ..., g, where g represents the total number of corresponding subspaces. When t is 1, it means that this space is ranked first in the main space.

[0064] Using |R1-R2|+|R2-R3|+……+|R g-1 -R g |=HB to obtain the combined value HB of the capacity differences of several sub-spaces;

[0065] The intermediate value ZZ is obtained by combining the value and the encryption key (HB-JM), and a logical formula is set in the main space. The logical formula is: Y=HB-ZZ, and the encryption key JM of the main space is deleted. When the capacity value of the secondary space in the main space changes, the intermediate value ZZ changes accordingly. In this formula, HB is only a symbol and does not reflect any number.

[0066] Specifically, later, when the data stored in the main space is encrypted, it needs to be set according to the specific parameters of the corresponding main space. The data capacity stored in each sub-space in the main space is different. Then, based on the data capacity of different sub-spaces, the combined value between the capacity differences can be determined. Subsequently, the encryption key stored in the main space is deleted:

[0067] As a further embodiment of the present application:

[0068] When an external operator needs to extract data from this main space, he / she enters the corresponding encryption key, and the system receives the encryption key and marks it as a key to be processed;

[0069] Based on the capacity parameters of each subspace, determine the value of the corresponding HB in the corresponding logical formula, and then based on the set logical formula, determine whether the key to be processed is the same as the result generated by the logical formula. If the results are exactly the same, then directly open this space for external personnel to search or review the data. If the results are different, it means that the input encryption key is inaccurate.

[0070] Embodiment 2

[0071] A data processing device, comprising:

[0072] At least three groups of processors:

[0073] The first processor preferentially deletes duplicate data in the processed data, then locks the abnormal data, and deletes the abnormal data in the processed data, thereby obtaining standardized data by processing the data;

[0074] The second processor performs similarity analysis on the standardized data at different stages, converts the different standardized data into standard vectors, and then determines the corresponding features of the corresponding standardized data by analyzing the different standard vectors of the different standardized data, thereby classifying the standardized data at different stages, and storing the standardized data belonging to the same classification in the same space;

[0075] The third processor first performs unified data deduplication processing on the standardized data stored in the same space, replaces the duplicate data in this space with duplicate markers, and then encrypts the standardized data in the same storage space, determines the encryption logic formula, and retains it in the corresponding storage space.

[0076] A storage system stores the standardized data processed by the processor, and stores standardized data of different classifications in different storage space locations according to specific classifications of the data, wherein the storage space includes a plurality of main spaces, and a main space includes a plurality of secondary spaces;

[0077] Allows external operators to set keys, set conversion logic based on the set keys and specific stored data, and delete internal encryption keys;

[0078] Each main space is a single set of computer-readable storage media, and its storage medium is a solid-state readable disk for storing standardized data. The division of its internal space is confirmed and divided by the storage system.

[0079] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0080] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A data processing method, characterized in that: The following steps are involved: S1. Confirm the processed data of this stage and perform preliminary cleaning, give priority to deleting duplicate data within the processed data, then lock the abnormal data, and delete the abnormal data within the processed data, so as to obtain standardized data through processing data; S2. Perform similarity analysis on standardized data at different stages, convert different standardized data into standard vectors, and then determine the corresponding features of the corresponding standardized data by analyzing different standard vectors of different standardized data, so as to classify the standardized data at different stages and store the standardized data belonging to the same classification in the same space, including: Based on the practical experience of the operators, three sets of different characteristic values ​​are extracted from the standardized data, and the characteristic values ​​belong to the relevant attribute data of the standardized data; According to the different eigenvalues ​​corresponding to the different standardized data, the corresponding spatial points are determined in the three-dimensional coordinate system in the selected order, and then the standard vector of the standard data is generated from the origin to the location of the spatial point; Determine a set of baselines in the three-dimensional coordinate system. The baselines are located between the X-axis, Y-axis, and Z-axis, and the included angles between the baselines and the X-axis, Y-axis, and Z-axis are consistent. Determine the included angle A between the standard vector and the baselines. Then, vectorize the standardized data of other stages and determine the included angle A between the baselines. i , where i represents the standardized data at different stages; According to the determined angle A i and a preset angle interval, dividing the standard vectors belonging to the same angle interval into vectors of the same interval, wherein the angle interval is a preset interval; Then classify the standard vector lengths of vectors in the same interval, and construct several groups of length classification intervals according to the preset value Ys: (0, Ys], (Ys, 2Ys], ..., ((n-1)Ys, nYs], where Ys is the preset value, and n represents the number of corresponding length classification intervals. According to the lengths of different standard vectors, several standard vectors of vectors in the same interval are divided into several different length classification intervals. According to the division results, the standardized data corresponding to the vectors in the same interval are stored in the same main space, and then the main space is divided into n sub-spaces, and then the standardized data corresponding to the different length classification intervals are sequentially stored in the corresponding sub-spaces; S3. For the standardized data stored in the same space, firstly perform unified data deduplication processing, replace the duplicate data in this space with duplicate markers, then encrypt the standardized data in the same storage space, determine the encryption logic, and retain it in the corresponding storage space. The specific methods of unified data deduplication processing include: First, confirm the duplicate data of several standardized data in the same space, and replace the confirmed duplicate data with corresponding duplicate markers. The expression of the duplicate markers is: C k , where k = 1, 2, ..., m, m represents the number of repeated data; Using repeated marker C k The multiple duplicate data appearing in the standardized data are replaced in sequence, and after the replacement is completed, a duplicate data mark replacement table is generated, and the replaced main space is marked as the space to be encrypted; the specific method of encrypting the space to be encrypted includes: First, confirm the encryption key of the primary space corresponding to the secondary space, and mark the encryption key as JM; Then, according to the time sequence of the subspaces in the corresponding main space during the construction process, the subspaces are initially sorted, and the data capacity in different subspaces is calibrated as R according to the sorting method. t , where t=1, 2, ..., g, where g represents the total number of corresponding subspaces. When t is 1, it means that this space is ranked first in the main space. Using |R1-R2|+|R2-R3|+……+|R g-1 -R g |=HB to obtain the combined value HB of the capacity differences of several sub-spaces; The intermediate value ZZ is obtained by combining the value and the encryption key (HB-JM), and a logical formula is set in the main space. The logical formula is: Y=HB-ZZ, and the encryption key JM of the main space is deleted. When the capacity value of the secondary space in the main space changes, the intermediate value ZZ will change accordingly.

2. A data processing method according to claim 1, characterized in that: In step S1, the specific sub-steps of obtaining standardized data by processing data include: The data to be processed in this stage is determined, and the identical data is found from the determined data, and the duplicate data is deleted, and one set of data is retained. After the processing is completed, the data to be processed in the second stage is confirmed; Arrange the result parameters of the second-stage processing data from small to large to generate a set of parameter sequences, and divide the parameter sequence into four equal parts according to the number of result parameters. If the four equal parts cannot be completed, keep the number of result parameters in the middle two equal parts the same; Get the first parameter of the second portion and mark it as SS, then confirm the last parameter of the third portion and mark it as MS, then confirm the difference CZ between MS and SS: CZ=MS-SS, and determine a set of numerical intervals based on the determined difference CZ: [CZ-1.5×CZ, CZ+1.5×CZ], mark the data in this parameter sequence that does not belong to this numerical interval as abnormal data, and eliminate this abnormal data from the processed data to obtain standardized data.

3. A data processing device, which operates based on the processing method according to any one of claims 1 to 2, characterized in that: include: At least three groups of processors: The first processor preferentially deletes duplicate data in the processed data, then locks the abnormal data, and deletes the abnormal data in the processed data, thereby obtaining standardized data by processing the data; The second processor performs similarity analysis on the standardized data at different stages, converts the different standardized data into standard vectors, and then determines the corresponding features of the corresponding standardized data by analyzing the different standard vectors of the different standardized data, thereby classifying the standardized data at different stages, and storing the standardized data belonging to the same classification in the same space; The third processor first performs unified data deduplication processing on the standardized data stored in the same space, replaces the duplicate data in this space with duplicate markers, and then encrypts the standardized data in the same storage space, determines the encryption logic formula, and retains it in the corresponding storage space.

4. A data storage system, which stores standardized data generated by the processing method according to any one of claims 1 to 2, characterized in that: The storage system stores the generated standardized data, and according to the specific classification of the data, stores the standardized data of different classifications in different storage space locations, wherein the storage space includes a plurality of main spaces, and a main space includes a plurality of secondary spaces; Allows external operators to set keys, set conversion logic based on the set keys and specific stored data, and delete internal encryption keys; Each main space is a single set of computer-readable storage media, and its storage medium is a solid-state readable disk for storing standardized data. The division of its internal space is confirmed and divided by the storage system.