Self-adaptive data preprocessing method and system

Abstract

The invention discloses a self-adaptive data preprocessing method and system. The self-adaptive data preprocessing method comprises the step of performing extraction operation from a system database. According to the method, firstly, the data is extracted to obtain a plurality of pieces of data to be processed, and then the plurality of pieces of data to be processed are subjected to exception analysis operation, exception missing value supplementation, feature extraction, extracted feature recognition and data set category judgment according to the data type judgment model, so that the format of the data can be unified; the data is intelligently identified and classified, and the data is summarized and fused into the data of the corresponding classification, so that the problems of synonymy different names, same name different codes and the like existing in data fusion are solved, convenience is brought to data fusion and convergence, the classification error is small, summarization and fusion of different types of data are avoided, and finally risk analysis is performed on the data. According to the method and the system, the data does not contain viruses, the security of the data is ensured, in addition, privacy analysis is performed on the data, and the data with relatively high privacy is encrypted.

Description

Technical Field

[0001] The present invention relates to the technical field of data preprocessing, and in particular to an adaptive data preprocessing method and system. Background Art

[0002] Data analysis refers to the use of appropriate statistical analysis methods to analyze a large amount of collected data, summarize, understand and digest them, in order to maximize the function of the data and give full play to the role of the data. Data analysis is the process of detailed study and summary of data in order to extract useful information and form conclusions. The mathematical foundation of data analysis was established in the early 20th century, but it was not until the advent of computers that practical operations became possible and data analysis was promoted. Data analysis is the product of the combination of mathematics and computer science.

[0003] At present, there is an increasing amount of data information, most of which are disorganized. Manual sorting requires a lot of time and wastes manpower and material resources. In addition, the existing data processing and analysis system lacks matching degree for data processing, and its processing effect needs to be improved. It is also not easy to discover potentially risky data, resulting in low system security. Summary of the invention

[0004] In order to solve the above technical problems, an adaptive data preprocessing method and system are provided. The technical solution solves the problem that various data information proposed in the above background technology is increasing, most of which are disorganized, and manual sorting requires a lot of time, wasting manpower and material resources, and the existing data processing and analysis system lacks matching degree for data processing, and its processing effect needs to be improved. It is also not easy to discover potentially risky data, resulting in low system security.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] An adaptive data preprocessing method, comprising:

[0007] Perform an extraction operation from the system database to obtain a number of data to be processed in the database;

[0008] Perform abnormal analysis operations on several data to be processed, and replace the abnormal data to obtain a complete data set;

[0009] Extract features from several complete data groups to be processed, identify the extracted features, perform data group category analysis, and obtain several groups of the same data type;

[0010] Conduct risk analysis on several groups of the same data type to determine the security of the data;

[0011] Analyze and process data based on data security to obtain data privacy;

[0012] Choose whether to encrypt the data according to the privacy of the data to obtain encrypted data.

[0013] Preferably, performing an abnormal analysis operation on a plurality of data to be processed and replacing the abnormal data to obtain complete data specifically comprises the following steps:

[0014] Detect missing data in several pieces of data to be processed, and use the non-missing data as the training set;

[0015] Train the prediction model based on the data in the training set as the training environment;

[0016] Extract the features of the part where the data is missing, use the prediction model to predict the alternative data of the part where the data is missing, and use the alternative data to compensate for the part where the data is missing;

[0017] Make a scatter plot corresponding to a number of data to be processed, and obtain the upper quartile and the lower quartile in the scatter plot, the upper quartile is the data at one quarter of the position from the top to the bottom of the scatter plot, and the lower quartile is the data at one quarter of the position from the bottom to the top of the scatter plot;

[0018] Calculate the absolute value of the difference between the upper quartile and the lower quartile to obtain the benchmark difference;

[0019] The sum of the preset multiples of the benchmark difference and the upper quartile is the upper edge value, and the difference between the lower quartile and the preset multiples of the benchmark difference is the lower edge value;

[0020] Data outside the range between the upper edge value and the lower edge value are abnormal data;

[0021] Extract the features of abnormal data, use the prediction model to predict alternative data for the abnormal data, use the alternative data to fill in the abnormal data, and obtain complete data.

[0022] Preferably, the feature extraction of several complete data sets to be processed specifically comprises the following steps:

[0023] Calculate the average value of the data in the complete data set to be processed, and calculate the variance of the data in the complete data set to be processed according to the variance formula;

[0024] Setting the variance as the central tendency of the complete data set to be processed, and assigning a trend identification code to the central tendency;

[0025] Pairing each data in the complete data group to be processed with the remaining data in the complete data group to be processed to obtain at least one paired group;

[0026] Calculate the absolute value of the difference between two data in each paired group of the complete data group to be processed, and accumulate the absolute values ​​of the difference between two data in all paired groups to obtain a discrete value;

[0027] The discrete value is set as the discrete degree of each set of data to be fused, and a discrete identification code is assigned to the discrete degree;

[0028] The variance formula is as follows:

[0029]

[0030] Among them, D is the variance, n is the number of data in the data group, xi is the value of the data, and a is the average value of the data in the data group.

[0031] Preferably, the step of identifying the extracted features specifically comprises the following steps:

[0032] Obtain the central tendency and dispersion degree of each set of data, and convert the central tendency and dispersion degree into binary representation;

[0033] Identify the data corresponding to the trend identification code as the central tendency according to the trend identification code of the central tendency;

[0034] According to the discrete identification code of the discrete degree, the data corresponding to the discrete identification code is identified as the discrete degree.

[0035] Preferably, the data group category analysis specifically includes the following steps:

[0036] Retrieve the historical data classification and discrimination standard 1 from the database, and obtain the range interval of the central tendency ratio and the range interval of the dispersion ratio in the classification and discrimination standard 1;

[0037] Calculate the ratio of the central tendency of the complete data group to be processed to the central tendency mean of the sample data group;

[0038] Calculate the ratio of the degree of dispersion of the identified complete data group to be processed to the mean value of the degree of dispersion of the sample data group;

[0039] Compare whether ratio 1 belongs to the range of central tendency ratio;

[0040] Compare the ratio 2 to see if it belongs to the range of the dispersion ratio;

[0041] If the first ratio belongs to the range interval of the central tendency ratio and the second ratio belongs to the range interval of the dispersion ratio, calling the classification in the classification discrimination group 1, setting the classification as the first classification, and classifying the data group into the first classification;

[0042] If the first ratio does not belong to the range of the central tendency ratio or the second ratio does not belong to the range of the dispersion ratio, the remaining classification discrimination groups in the data type determination model are called;

[0043] Find the historical data classification discrimination criterion 2 in the remaining classification discrimination groups, and the classification discrimination criterion 2 satisfies that the ratio 1 of the data group belongs to the range interval of the central tendency ratio of the classification discrimination group 2 and the ratio 2 of the data group belongs to the range interval of the dispersion ratio of the classification discrimination group 2, call the classification in the classification discrimination group 2, set the classification as the second classification, and classify the data group into the second classification.

[0044] Preferably, the obtaining of several groups of the same data type specifically comprises the following steps:

[0045] Acquire data in a complete data group to be processed, acquire a third category to which the data group belongs, and acquire a data type of the data group under the third category;

[0046] According to the data type of the data group under the third classification, the data in the data group is converted to the same data type.

[0047] Preferably, the risk analysis of several groups of the same data type to determine the security of the data specifically includes the following steps:

[0048] Communication protocol for obtaining virus database;

[0049] Establishing a communication connection with a virus database based on a communication protocol of the virus database;

[0050] Scan and judge several groups of the same data type according to the virus data stored in the virus database;

[0051] If some of the contents in several groups of the same data type overlap with the virus data stored in the virus database, it means that the data is virus data;

[0052] If the contents of several groups of the same data type do not overlap with the virus data stored in the virus database, the data is verified again to determine the security of the data.

[0053] Preferably, the secondary verification of the data to determine the security of the data specifically includes the following steps:

[0054] Obtaining historical metadata of the data, the metadata including creator information, modification date information, and file size information;

[0055] Extract data and obtain real-time metadata of the data;

[0056] Make judgments based on the historical metadata and real-time metadata of the data;

[0057] If the creator information, modification date information, file size information in the historical metadata are exactly the same as the creator information, modification date information, file size information in the real-time metadata, the output data is safe;

[0058] If any of the creator information, modification date information, file size information in the historical metadata is different from the creator information, modification date information, file size information in the real-time metadata, the output data is at risk.

[0059] Preferably, the step of analyzing and processing the data based on the security of the data, obtaining the privacy of the data, and selecting whether to encrypt the data according to the privacy of the data to obtain the encrypted data specifically includes the following steps:

[0060] Screening the data to obtain a private data set, wherein the private data set includes personal identity information, financial information, and health records;

[0061] Calculate and process the private data set and data to obtain the weight of the private information ratio;

[0062] Compare the weight of the proportion of private information with the set first threshold;

[0063] If the weight of the private information ratio is greater than or equal to the set first threshold, the output data privacy is high and the data needs to be encrypted;

[0064] If the weight of the private information ratio is less than the set first threshold, the output data privacy is low and the data does not need to be encrypted;

[0065] Match the data format to obtain the data to be encrypted;

[0066] An asymmetric encryption algorithm is used to encrypt the data to be encrypted, so as to obtain the encrypted data.

[0067] An adaptive data preprocessing system, comprising:

[0068] An extraction module, the extraction module is used to perform an extraction operation from a system database to obtain a number of data to be processed in the database;

[0069] An abnormality analysis module, which is used to perform abnormality analysis operations on a number of data to be processed, and replace the abnormal data to obtain a complete data set;

[0070] A category analysis and judgment module, which is used to extract features from several complete data groups to be processed, identify the extracted features, perform category analysis on the data groups, and obtain several groups of the same data type;

[0071] A risk analysis module, which is used to perform risk analysis on several groups of the same data type to determine the security of the data;

[0072] The encryption module is used to analyze and process the data based on the security of the data, obtain the privacy of the data, and choose whether to encrypt the data according to the privacy of the data to obtain encrypted data.

[0073] Compared with the prior art, the present invention provides an adaptive data preprocessing method and system, which has the following beneficial effects:

[0074] The present invention first performs an extraction operation on the data to obtain a number of data to be processed, then performs an abnormal analysis operation on the data to be processed, supplements the abnormal missing values, performs feature extraction, identifies the extracted features, and performs data group category judgment according to the data type judgment model. The format of the data can be unified, the data can be intelligently identified and classified, and the data can be aggregated and merged into the corresponding classified data, thereby solving the problems of synonymous different names and the same name but different codes in data fusion, bringing convenience to data fusion and aggregation, and its classification error is small, and different types of data will not be aggregated and merged. Finally, a risk analysis is performed on the data to ensure that there is no virus in the data, thereby ensuring the security of the data. In addition, the privacy analysis is performed on the data, and the data with higher privacy is encrypted, thereby further improving the security of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic diagram of the flow chart of the method for data preprocessing in the present invention;

[0076] Figure 2 It is a schematic flow chart of a method for extracting features from a plurality of complete data sets to be processed in the present invention;

[0077] Figure 3 It is a schematic diagram of the flow of the method for identifying the extracted features in the present invention;

[0078] Figure 4 A schematic flow chart of a method for performing risk analysis on several groups of the same data type to determine the security of the data in the present invention;

[0079] Figure 5 The present invention is a flowchart of a method for performing secondary verification on data to determine the security of the data. DETAILED DESCRIPTION

[0080] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0081] Please refer to Figure 1-Figure 5As shown, an adaptive data preprocessing method comprises:

[0082] Perform an extraction operation from the system database to obtain a number of data to be processed in the database;

[0083] Perform abnormal analysis operations on several data to be processed, and replace the abnormal data to obtain a complete data set;

[0084] Extract features from several complete data groups to be processed, identify the extracted features, perform data group category analysis, and obtain several groups of the same data type;

[0085] Conduct risk analysis on several groups of the same data type to determine the security of the data;

[0086] Analyze and process data based on data security to obtain data privacy;

[0087] Choose whether to encrypt the data according to the privacy of the data to obtain encrypted data.

[0088] Perform an abnormal analysis operation on a number of data to be processed, and replace the abnormal data. The steps to obtain complete data include:

[0089] Detect missing data in several pieces of data to be processed, and use the non-missing data as the training set;

[0090] Train the prediction model based on the data in the training set as the training environment;

[0091] Extract the features of the part where the data is missing, use the prediction model to predict the alternative data of the part where the data is missing, and use the alternative data to compensate for the part where the data is missing;

[0092] Make a scatter plot corresponding to a number of data to be processed, and obtain the upper quartile and the lower quartile in the scatter plot, the upper quartile is the data at one quarter of the position from the top to the bottom of the scatter plot, and the lower quartile is the data at one quarter of the position from the bottom to the top of the scatter plot;

[0093] Calculate the absolute value of the difference between the upper quartile and the lower quartile to obtain the benchmark difference;

[0094] The sum of the preset multiples of the benchmark difference and the upper quartile is the upper edge value, and the difference between the lower quartile and the preset multiples of the benchmark difference is the lower edge value;

[0095] Data outside the range between the upper edge value and the lower edge value are abnormal data;

[0096] Extract the features of abnormal data, use the prediction model to predict alternative data for the abnormal data, use the alternative data to fill in the abnormal data, and obtain complete data.

[0097] The feature extraction of several complete data sets to be processed specifically includes the following steps:

[0098] Calculate the average value of the data in the complete data set to be processed, and calculate the variance of the data in the complete data set to be processed according to the variance formula;

[0099] Set the variance as the central tendency of the complete data set to be processed, and assign a trend identification code to the central tendency;

[0100] Pairing each data in the complete data group to be processed with the remaining data in the complete data group to be processed to obtain at least one paired group;

[0101] Calculate the absolute value of the difference between two data in each paired group of the complete data group to be processed, and accumulate the absolute values ​​of the difference between two data in all paired groups to obtain a discrete value;

[0102] Set the discrete value as the discrete degree of each set of data to be fused, and assign a discrete identification code to the discrete degree;

[0103] The variance formula is as follows:

[0104]

[0105] Among them, D is the variance, n is the number of data in the data group, xi is the value of the data, and a is the average value of the data in the data group.

[0106] The identification of the extracted features specifically includes the following steps:

[0107] Obtain the central tendency and dispersion degree of each set of data, and convert the central tendency and dispersion degree into binary representation;

[0108] Identify the data corresponding to the trend identification code as the central tendency according to the trend identification code of the central tendency;

[0109] According to the discrete identification code of the discrete degree, the data corresponding to the discrete identification code is identified as the discrete degree.

[0110] The data group classification analysis specifically includes the following steps:

[0111] Retrieve the historical data classification and discrimination standard 1 from the database, and obtain the range interval of the central tendency ratio and the range interval of the dispersion ratio in the classification and discrimination standard 1;

[0112] Calculate the ratio of the central tendency of the complete data group to be processed to the central tendency mean of the sample data group;

[0113] Calculate the ratio of the degree of dispersion of the identified complete data group to be processed to the mean value of the degree of dispersion of the sample data group;

[0114] Compare whether ratio 1 belongs to the range of central tendency ratio;

[0115] Compare the ratio 2 to see if it belongs to the range of the dispersion ratio;

[0116] If the first ratio belongs to the range of the central tendency ratio and the second ratio belongs to the range of the dispersion ratio, the classification in the classification discrimination group 1 is called, the classification is set to the first classification, and the data group is classified into the first classification;

[0117] If the first ratio does not belong to the range of the central tendency ratio or the second ratio does not belong to the range of the dispersion ratio, the remaining classification discrimination groups in the data type determination model are called;

[0118] Find the historical data classification and discrimination criterion 2 in the remaining classification and discrimination groups. The classification and discrimination criterion 2 satisfies that the ratio 1 of the data group belongs to the range interval of the central tendency ratio of the classification and discrimination group 2 and the ratio 2 of the data group belongs to the range interval of the dispersion ratio of the classification and discrimination group 2, call the classification in the classification and discrimination group 2, set the classification to the second classification, and classify the data group into the second classification.

[0119] People in the technical field can understand that the basis for judgment is that the categories covered by the data group include all categories, and the fused data group must belong to one of the categories. When the data group to be fused belongs to one of the categories, the ratio of the central tendency of the data group to be fused to the central tendency mean of the sample data group must belong to the range of the central tendency ratio of the given category, and the ratio of the discrete degree of the fused data group to the discrete degree mean of the sample data group must belong to the range of the discrete degree ratio of the given category. Therefore, the type of the data group can be judged based on this.

[0120] Obtaining several groups of the same data type specifically includes the following steps:

[0121] Obtain data in the complete data group to be processed, obtain the third category to which the data group belongs, and obtain the data type of the data group under the third category;

[0122] According to the data type of the data group under the third classification, the data in the data group is converted to the same data type.

[0123] Conducting risk analysis on several groups of the same data type to determine the security of the data includes the following steps:

[0124] Communication protocol for obtaining virus database;

[0125] Establishing a communication connection with a virus database based on a communication protocol of the virus database;

[0126] Scan and judge several groups of the same data type according to the virus data stored in the virus database;

[0127] If some of the contents in several groups of the same data type overlap with the virus data stored in the virus database, it means that the data is virus data;

[0128] If the contents of several groups of the same data type do not overlap with the virus data stored in the virus database, the data is verified again to determine the security of the data.

[0129] The second verification of data to ensure data security includes the following steps:

[0130] Get historical metadata of the data, including creator information, modification date information, and file size information;

[0131] Extract data and obtain real-time metadata of the data;

[0132] Make judgments based on the historical metadata and real-time metadata of the data;

[0133] If the creator information, modification date information, file size information in the historical metadata are exactly the same as the creator information, modification date information, file size information in the real-time metadata, the output data is safe;

[0134] If any of the creator information, modification date information, file size information in the historical metadata is different from the creator information, modification date information, file size information in the real-time metadata, the output data is at risk.

[0135] Analyze and process the data based on the security of the data, obtain the privacy of the data, and choose whether to encrypt the data according to the privacy of the data. The encrypted data specifically includes the following steps:

[0136] Screen the data to obtain private data sets, including personal identity information, financial information, and health records;

[0137] Calculate and process the private data set and data to obtain the weight of the private information ratio;

[0138] Compare the weight of the proportion of private information with the set first threshold;

[0139] If the weight of the private information ratio is greater than or equal to the set first threshold, the output data privacy is high and the data needs to be encrypted;

[0140] If the weight of the private information ratio is less than the set first threshold, the output data privacy is low and the data does not need to be encrypted;

[0141] Match the data format to obtain the data to be encrypted;

[0142] An asymmetric encryption algorithm is used to encrypt the data to be encrypted, so as to obtain the encrypted data.

[0143] An adaptive data preprocessing system, comprising:

[0144] The extraction module is used to perform extraction operations from the system database to obtain a number of data to be processed in the database;

[0145] The abnormality analysis module is used to perform abnormality analysis operations on a number of data to be processed, and replace the abnormal data to obtain a complete data set;

[0146] The category analysis and judgment module is used to extract features from several complete data groups to be processed, identify the extracted features, perform category analysis on the data groups, and obtain several groups of the same data type;

[0147] Risk analysis module, which is used to perform risk analysis on several groups of the same data type to determine the security of the data;

[0148] The encryption module is used to analyze and process the data based on the security of the data, obtain the privacy of the data, and choose whether to encrypt the data according to the privacy of the data to obtain encrypted data.

[0149] To summarize, the present invention first performs an extraction operation on the data to obtain a number of data to be processed, then performs an exception analysis operation on the data to be processed, supplements the abnormal missing values, performs feature extraction, identifies the extracted features, and performs data group category judgment according to the data type judgment model. It can unify the format of the data, intelligently identify and classify the data, and aggregate and fuse the data into the corresponding classified data, thereby solving the problems of synonymous different names and the same name but different codes in data fusion, bringing convenience to data fusion and aggregation, and its classification error is small, and different types of data will not be aggregated and fused. Finally, a risk analysis is performed on the data to ensure that there is no virus in the data, thereby ensuring the security of the data. In addition, the privacy analysis is performed on the data, and the data with higher privacy is encrypted, thereby further improving the security of the data.

[0150] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. An adaptive data preprocessing method, characterized in that: include: Perform an extraction operation from the system database to obtain a number of data to be processed in the database; Perform abnormal analysis operations on several data to be processed, and replace the abnormal data to obtain a complete data set; Extract features from several complete data groups to be processed, identify the extracted features, perform data group category analysis, and obtain several groups of the same data type; Conduct risk analysis on several groups of the same data type to determine the security of the data; Analyze and process data based on data security to obtain data privacy; Choose whether to encrypt the data according to the privacy of the data to obtain encrypted data.

2. The adaptive data preprocessing method according to claim 1, characterized in that: The method of performing an abnormal analysis operation on a plurality of data to be processed and replacing the abnormal data to obtain complete data specifically includes the following steps: Detect missing data in several pieces of data to be processed, and use the non-missing data as the training set; Train the prediction model based on the data in the training set as the training environment; Extract the features of the part where the data is missing, use the prediction model to predict the alternative data of the part where the data is missing, and use the alternative data to compensate for the part where the data is missing; Make a scatter plot corresponding to a number of data to be processed, and obtain the upper quartile and the lower quartile in the scatter plot, the upper quartile is the data at one quarter of the position from the top to the bottom of the scatter plot, and the lower quartile is the data at one quarter of the position from the bottom to the top of the scatter plot; Calculate the absolute value of the difference between the upper quartile and the lower quartile to obtain the benchmark difference; The sum of the preset multiples of the benchmark difference and the upper quartile is the upper edge value, and the difference between the lower quartile and the preset multiples of the benchmark difference is the lower edge value; Data outside the range between the upper edge value and the lower edge value are abnormal data; Extract the features of abnormal data, use the prediction model to predict alternative data for the abnormal data, use the alternative data to fill in the abnormal data, and obtain complete data.

3. The adaptive data preprocessing method according to claim 1, characterized in that: The feature extraction of several complete data sets to be processed specifically includes the following steps: Calculate the average value of the data in the complete data set to be processed, and calculate the variance of the data in the complete data set to be processed according to the variance formula; Setting the variance as the central tendency of the complete data set to be processed, and assigning a trend identification code to the central tendency; Pairing each data in the complete data group to be processed with the remaining data in the complete data group to be processed to obtain at least one paired group; Calculate the absolute value of the difference between two data in each paired group of the complete data group to be processed, and accumulate the absolute values ​​of the difference between two data in all paired groups to obtain a discrete value; The discrete value is set as the discrete degree of each set of data to be fused, and a discrete identification code is assigned to the discrete degree; The variance formula is as follows: Among them, D is the variance, n is the number of data in the data group, xi is the value of the data, and a is the average value of the data in the data group.

4. The adaptive data preprocessing method according to claim 1, characterized in that: The identification of the extracted features specifically comprises the following steps: Obtain the central tendency and dispersion degree of each set of data, and convert the central tendency and dispersion degree into binary representation; Identify the data corresponding to the trend identification code as the central tendency according to the trend identification code of the central tendency; According to the discrete identification code of the discrete degree, the data corresponding to the discrete identification code is identified as the discrete degree.

5. The adaptive data preprocessing method according to claim 1, characterized in that: The data group category analysis specifically includes the following steps: Retrieve the historical data classification and discrimination standard 1 from the database, and obtain the range interval of the central tendency ratio and the range interval of the dispersion ratio in the classification and discrimination standard 1; Calculate the ratio of the central tendency of the complete data group to be processed to the central tendency mean of the sample data group; Calculate the ratio of the degree of dispersion of the identified complete data group to be processed to the mean value of the degree of dispersion of the sample data group; Compare whether ratio 1 belongs to the range of central tendency ratio; Compare the ratio 2 to see if it belongs to the range of the dispersion ratio; If the first ratio belongs to the range interval of the central tendency ratio and the second ratio belongs to the range interval of the dispersion ratio, calling the classification in the classification discrimination group 1, setting the classification as the first classification, and classifying the data group into the first classification; If the first ratio does not belong to the range of the central tendency ratio or the second ratio does not belong to the range of the dispersion ratio, the remaining classification discrimination groups in the data type determination model are called; Find the historical data classification discrimination criterion 2 in the remaining classification discrimination groups, and the classification discrimination criterion 2 satisfies that the ratio 1 of the data group belongs to the range interval of the central tendency ratio of the classification discrimination group 2 and the ratio 2 of the data group belongs to the range interval of the dispersion ratio of the classification discrimination group 2, call the classification in the classification discrimination group 2, set the classification as the second classification, and classify the data group into the second classification.

6. The adaptive data preprocessing method according to claim 1, characterized in that: The obtaining of several groups of the same data type specifically comprises the following steps: Acquire data in a complete data group to be processed, acquire a third category to which the data group belongs, and acquire a data type of the data group under the third category; According to the data type of the data group under the third classification, the data in the data group is converted to the same data type.

7. The adaptive data preprocessing method according to claim 1, characterized in that: The risk analysis of several groups of the same data type to determine the security of the data specifically includes the following steps: Communication protocol for obtaining virus database; Establishing a communication connection with a virus database based on a communication protocol of the virus database; Scan and judge several groups of the same data type according to the virus data stored in the virus database; If some of the contents in several groups of the same data type overlap with the virus data stored in the virus database, it means that the data is virus data; The contents of several groups of the same data type do not overlap with the virus data stored in the virus database, and the data is verified twice to ensure its security.

8. The adaptive data preprocessing method according to claim 1, characterized in that: The secondary verification of data to determine the security of the data specifically includes the following steps: Obtaining historical metadata of the data, the metadata including creator information, modification date information, and file size information; Extract data and obtain real-time metadata of the data; Make judgments based on the historical metadata and real-time metadata of the data; If the creator information, modification date information, file size information in the historical metadata are exactly the same as the creator information, modification date information, file size information in the real-time metadata, the output data is safe; If any of the creator information, modification date information, file size information in the historical metadata is different from the creator information, modification date information, file size information in the real-time metadata, the output data is at risk.

9. The adaptive data preprocessing method according to claim 1, characterized in that: The step of analyzing and processing the data based on the security of the data, obtaining the privacy of the data, and selecting whether to encrypt the data according to the privacy of the data to obtain the encrypted data specifically includes the following steps: Screening the data to obtain a private data set, wherein the private data set includes personal identity information, financial information, and health records; Calculate and process the private data set and data to obtain the weight of the private information ratio; Compare the weight of the proportion of private information with the set first threshold; If the weight of the private information ratio is greater than or equal to the set first threshold, the output data privacy is high and the data needs to be encrypted; If the weight of the private information ratio is less than the set first threshold, the output data privacy is low and the data does not need to be encrypted; Match the data format to obtain the data to be encrypted; An asymmetric encryption algorithm is used to encrypt the data to be encrypted, so as to obtain the encrypted data.

10. An adaptive data preprocessing system, used to implement an adaptive data preprocessing method according to claims 1-9, characterized in that: include: An extraction module, the extraction module is used to perform an extraction operation from a system database to obtain a number of data to be processed in the database; An abnormality analysis module, which is used to perform abnormality analysis operations on a number of data to be processed, and replace the abnormal data to obtain a complete data set; A category analysis and judgment module, which is used to extract features from several complete data groups to be processed, identify the extracted features, perform category analysis on the data groups, and obtain several groups of the same data type; A risk analysis module, which is used to perform risk analysis on several groups of the same data type to determine the security of the data; The encryption module is used to analyze and process the data based on the security of the data, obtain the privacy of the data, and choose whether to encrypt the data according to the privacy of the data to obtain encrypted data.

Images