A data cleaning method and system based on data security sandbox

Abstract

The present invention provides a data cleaning method and system based on a data security sandbox, and relates to the field of data cleaning. Technical solution: extract and process data in a data security sandbox to obtain original data, further process to obtain a feature vector set, and classify to obtain several vector groups; resample the original data and divide it into a training set and a test set, introduce a feature difference weighting mechanism to stack and combine multiple enhanced feature encoders to build an initial model, train and test the initial model based on the training set and the test set to obtain a SEFE model; use the SEFE model to denoise the feature vector set and then perform outlier detection to obtain outliers, and mark outliers that exceed the judgment standard as abnormal data; calculate the difference between the distribution of abnormal data and the distribution of normal data after clustering, rate the abnormal data with a difference exceeding a preset threshold as dirty data, and extract the dirty data for data management. The present invention improves the efficiency, speed and screening capabilities of data cleaning for a data security sandbox.

Description

Technical Field

[0001] The present invention relates to the field of data cleaning, and in particular to a data cleaning method and system based on a data security sandbox. Background Art

[0002] The data security sandbox is based on the premise of not leaking the original data and security privacy of all parties. It realizes the value of data fusion by jointly calculating and analyzing data through the encrypted collaboration mechanism. The data sandbox separates the visible data information from the invisible computing part to achieve "data is available but invisible", thereby eliminating the concerns of all data participants about data security and privacy leakage, and solving the dilemma of "data island".

[0003] Data cleaning, also known as data cleaning or data washing, is the basis of data quality management. Data cleaning is performed on the original data of the data security sandbox. Its purpose is to detect, analyze and correct the noise, missing, duplicate, conflict and wrong values ​​in the data, improve data quality, and avoid the influence of dirty data on the correctness of the sandbox data analysis results. The identification of dirty data can be achieved through algorithms, rules or user marking methods.

[0004] However, the existing technology still has problems: first, the model calculation overhead of the existing technology is large, and the data cleaning and processing efficiency is low; second, the degree of automation in data processing is low, the response to dynamically changing data patterns is slow, and the ability to detect and review new and unknown abnormal data is insufficient.

[0005] Therefore, it is urgent to develop a solution to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a data cleaning method and system based on a data security sandbox, which improves the efficiency and speed of data cleaning for the data security sandbox and improves the ability to screen irrelevant abnormal data in the data security sandbox.

[0007] The present invention provides a data cleaning method and system based on a data security sandbox, which adopts the following technical solutions:

[0008] In a first aspect, a data cleaning method based on a data security sandbox specifically includes:

[0009] Extracting and preprocessing the data in the data security sandbox to obtain the original data, extracting features and polynomially transforming the original data to obtain a feature vector set, calculating the correlation score of the feature vector set and classifying it to obtain a number of vector groups;

[0010] The original data is resampled and divided into a training set and a test set. A feature difference weighting mechanism is introduced to stack and combine multiple enhanced feature encoders to build an initial model. The initial model is trained and tested based on the training set and the test set to obtain the SEFE model.

[0011] Based on the SEFE model, the feature vector set is denoised and then outlier detection is performed to obtain outliers, a judgment standard for abnormal data is set, and data features whose outliers exceed the judgment standard are marked as abnormal data, otherwise they are marked as normal data;

[0012] Normal data and abnormal data are clustered based on the vector group as the initial clustering center, the difference between the distribution of abnormal data and the distribution of normal data is calculated, the abnormal data whose difference exceeds a preset threshold is rated as dirty data, and the dirty data is extracted to an isolation area for data management.

[0013] The beneficial effect of a data cleaning method based on a data security sandbox provided by the present invention is that the present invention cleans dirty data for the original data of the data security sandbox, including denoising of noise values ​​and detection of outliers, and introduces a feature difference weighting mechanism in the SEFE network structure to adaptively allocate weights to each layer, highlight important features, improve the effect of feature learning and the model's ability to screen irrelevant abnormal data, and more effectively utilize normal data by judging and grading abnormal data, reducing the misjudgment rate of abnormal data, while managing dirty data in the isolation area. This method ensures the effective isolation of dirty data during the data cleaning process to prevent it from contaminating normal data and analysis results.

[0014] Optionally, the correlation score of each feature in the feature vector set is calculated and classified, and the process of calculating the correlation score includes:

[0015] Calculate the feature distance score of the enhanced feature vector, and the calculation formula is as follows:

[0016] ;

[0017] in, For all The maximum value in For all The minimum value in ; For the The characteristic value of the sample, To enhance the sum of distances between feature vectors and other features;

[0018] Calculate the association score between the enhanced feature vector and the target variable. The calculation formula is as follows:

[0019] ;

[0020] in, and are the characteristic value and target variable value of the i-th sample, respectively. and is the corresponding mean;

[0021] The correlation score is calculated by combining the feature distance score and the association score, and the calculation formula is as follows:

[0022] ;

[0023] in, , is the weight parameter, is the feature distance score, is the relevance score.

[0024] Optionally, classifying the feature vector set based on the feature with the highest relevance score comprises:

[0025] Each time, the feature with the highest correlation score in the feature vector set is selected for classification. After a set of classifications is obtained, the next feature with the corresponding score is selected for classification, and the feature vector set is divided into several non-overlapping vector groups.

[0026] Optionally, when the initial model is trained and tested based on the training set and the test set to obtain the SEFE model, it includes:

[0027] Resample the original data and divide it into control data and training data, add Gaussian distribution noise to the training data to obtain noise data and divide it into training set and test set;

[0028] The initial model is trained based on the training set, a reconstruction loss function is constructed based on the control data and the training prediction value, the reconstruction loss function is minimized based on the gradient descent method, and the weights and parameters of the initial model are iterated by back propagation until the reconstruction loss function tends to be stable. The training is terminated, and the trained model is tuned based on the test set to obtain the SEFE model.

[0029] Optionally, when introducing a feature difference weighting mechanism to stack and combine multiple enhanced feature encoders to build an initial model, it includes:

[0030] A feature difference weighting mechanism is introduced in each enhanced feature encoder. Each layer of enhanced feature encoder calculates the feature difference weighted weight as the importance indicator of the input of the next layer, forming a transmission and aggregation mechanism of the feature difference weighted weight to obtain the initial model.

[0031] Optionally, set abnormal data judgment criteria, including:

[0032] The calculation formula for the abnormal data judgment standard is as follows:

[0033] ;

[0034] In the formula, is the outlier value of the data; is the standard error of the SEFE model prediction, is the data to be determined, For data The residual error.

[0035] Optionally, clustering normal data and abnormal data based on the vector group as an initial clustering center includes:

[0036] Determine the number of initial clusters based on the vector group, randomly select the initial centroid, calculate the distance between the data point and the centroid, assign each data point to the cluster corresponding to the nearest centroid, and calculate the average value of all data points in each cluster as the new centroid;

[0037] Repeat the above steps until the centroid no longer changes or the maximum number of iterations is reached, and a clustering result containing normal data and abnormal data is obtained.

[0038] Optionally, extract dirty data to the isolation area for data management, including:

[0039] Create an independent virtual isolation zone to store and process dirty data;

[0040] Or, create a separate database to store dirty data and set its access permissions to administrators and system processes;

[0041] Alternatively, add field tags to dirty data, filter all data records of marked dirty data based on SQL query statements, and move the filtered dirty data from the original storage location to the isolation area based on scripts for management.

[0042] In the second aspect, a data cleaning system based on a data security sandbox specifically includes:

[0043] A significant difference feature extraction module is used to extract and preprocess the data in the data security sandbox to obtain the original data, extract features and perform polynomial transformation on the original data to obtain a feature vector set;

[0044] A significant difference feature classification module is used to calculate the correlation score of the feature vector set and classify it to obtain a plurality of vector groups;

[0045] SEFE model training module, which is used to resample the original data and divide it into training set and test set, introduce feature difference weighting mechanism to stack and combine multiple enhanced feature encoders to build an initial model, and train and test the initial model based on the training set and test set to obtain the SEFE model;

[0046] An abnormal data determination module, based on the SEFE model, performs abnormal value detection on the feature vector set after denoising to obtain abnormal values, sets abnormal data determination criteria, and marks abnormal values ​​exceeding the determination criteria as abnormal data, otherwise marks them as normal data;

[0047] An abnormality degree grading module, used to cluster normal data and abnormal data using the vector group as an initial clustering center, calculate the difference between the distribution of abnormal data and the distribution of normal data, and rate the abnormal data whose difference exceeds a preset threshold as dirty data;

[0048] The abnormal data processing module is used to extract dirty data to the isolation area for data management.

[0049] Optionally, the network structure of the SEFE model includes:

[0050] Data preprocessing network, used to add random small perturbations to the feature vector set;

[0051] Compression network, used to reduce the dimension of the preprocessed high-dimensional data into low-dimensional vectors;

[0052] Weighted hidden network, which is used to calculate the feature difference weights of low-dimensional vectors and apply them to the hidden layer feature representation to obtain the weighted hidden representation;

[0053] The reconstruction network is used to decode the weighted hidden representation, restore the low-dimensional data to the initial dimension, reconstruct the predicted value and calculate the outlier value.

[0054] The beneficial effects of the second aspect can refer to the description of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A block diagram of a data cleaning system based on a data security sandbox provided by the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0057] The embodiment of the present invention provides a data cleaning method based on a data security sandbox, and the specific steps include:

[0058] S1. Extract and preprocess the data in the data security sandbox to obtain original data, extract features and perform polynomial transformation on the original data to obtain a feature vector set, calculate the relevance score of each feature in the feature vector set, and classify the feature vector set based on the feature with the highest relevance score to obtain several vector groups;

[0059] S2, resample the original data and divide it into training set and test set, introduce feature difference weighting mechanism to stack and combine multiple enhanced feature encoders to build an initial model, train and test the initial model based on the training set and test set to obtain the SEFE model;

[0060] S3, performing outlier detection on the feature vector set after denoising based on the SEFE model to obtain outliers, setting an abnormal data judgment standard, marking outliers exceeding the judgment standard as abnormal data, otherwise marking them as normal data;

[0061] S4. Cluster the normal data and abnormal data based on the vector group as the initial clustering center, calculate the difference between the distribution of abnormal data and the distribution of normal data, rate the abnormal data whose distribution difference exceeds a preset threshold as dirty data, otherwise rate it as ordinary abnormal data, and extract the dirty data to the isolation area for data management.

[0062] In some embodiments, the specific implementation of step S1 includes:

[0063] S1-1, extract and preprocess the data in the data security sandbox to obtain the original data, extract features and perform polynomial transformation on the original data to obtain a feature vector set;

[0064] S1-2, calculating the correlation score of each feature in the feature vector set;

[0065] S1-3. Classify the feature vector set based on the feature with the highest correlation score to obtain several vector groups.

[0066] Specifically, when executing step S1-1, the data in the data security sandbox is normalized, deep feature extraction is performed based on the large language model, the normalized data is encoded using the large language model to obtain the original data, and in order to enhance the features, the original data is transformed through a polynomial transformation of the features to obtain a feature vector set.

[0067] Furthermore, the polynomial transformation formula of the feature is:

[0068] , ;

[0069] In the formula, is the coefficient, It is a feature exist The index of the item, is the total number of features, is the number of features in each polynomial term, a feature iterator .

[0070] In other embodiments, deep feature extraction is performed on the original data based on a large language model to obtain a feature vector containing rich deep feature information. The LLM can be guided by a small amount of contextual example prompts for learning. The prompt template is shown in Table 1.

[0071] Table 1 Large model prompt template table

[0072] ,

[0073] In some embodiments, when executing step S1-2, calculating the relevance score of each feature in the feature vector set includes:

[0074] Calculate feature distance scores for enhanced feature vectors;

[0075] Furthermore, the feature distance score calculation formula is as follows:

[0076] ;

[0077] In the formula, For all The maximum value in For all The minimum value in For the The characteristic value of the sample, To enhance the sum of distances between feature vectors and other features;

[0078] Calculate the association score between the enhanced feature vector and the target variable;

[0079] Furthermore, the calculation formula of the correlation score is as follows:

[0080] ;

[0081] In the formula, and They are The characteristic values ​​and target variable values ​​of samples, and is the corresponding mean;

[0082] Calculating a relevance score by combining the feature distance score and the association score;

[0083] Furthermore, the relevance score calculation formula is as follows:

[0084] ;

[0085] in, , is the weight parameter, is the feature distance score, is the relevance score.

[0086] Specifically, when executing step S1-3, the feature vector set is classified based on the feature with the highest correlation score, including:

[0087] Each time, the feature with the highest correlation score in the feature vector set is selected for classification. After a set of classifications is obtained, the next feature with the corresponding score is selected for classification, and the feature vector set is divided into several non-overlapping vector groups.

[0088] In fact, using a large language model classifier, the feature with the highest comprehensive score in the feature iterator is selected each time to train and classify the data. After obtaining a set of classifications, the corresponding features are deleted from the feature iterator, and the process is repeated until the feature iterator is empty, thereby dividing the data according to the most significant features and grouping them into several predefined non-overlapping categories.

[0089] In some embodiments, when executing step S2, it specifically includes:

[0090] S2-1, resample the original data and divide it into training set and test set;

[0091] S2-2, introduce feature difference weighting mechanism to stack and combine multiple enhanced feature encoders to build the initial model;

[0092] S2-3. Train and test the initial model based on the training set and test set to obtain the SEFE model.

[0093] Specifically, when executing step S2-1, the original data is resampled for normalization, and 70% of the data is selected as control data. In order to train the SEFE model, noise with different degrees of Gaussian distribution is added to the remaining 30% of the data, and the training set and the test set are divided into a certain proportion in the data with added noise.

[0094] Specifically, when executing step S2-2, a feature difference weighting mechanism is introduced to stack and combine multiple enhanced feature encoders to build an initial model, including:

[0095] A feature difference weighting mechanism is introduced in each enhanced feature encoder. Each layer of enhanced feature encoder calculates the feature difference weighted weight as the importance indicator of the input of the next layer, forming a transmission and aggregation mechanism of the feature difference weighted weight to obtain the initial model.

[0096] In fact, the SEFE model is composed of a stack of multiple EFE network structures (Enhanced Feature Encoder), each of which includes:

[0097] Data preprocessing network: Add white noise (i.e., random small-amplitude perturbations) to the normalized input data;

[0098] Data preprocessing network, used to add random small perturbations to the feature vector set;

[0099] Compression network, used to reduce the dimension of the preprocessed high-dimensional data into low-dimensional vectors;

[0100] Weighted hidden network, which is used to calculate the feature difference weights of low-dimensional vectors and apply them to the hidden layer feature representation to obtain the weighted hidden representation;

[0101] Reconstruction network: used to decode the weighted hidden representation, restore the low-dimensional data to the initial dimension, reconstruct and output.

[0102] Furthermore, the EFE network structure extracts data feature encoding through encoding-decoding method, automatically learns data features, extracts noise feature expression through denoising learning of noisy data, and repairs noisy data.

[0103] Specifically, the EFE network structure includes an input layer, a hidden layer, and an output layer, and its data processing flow includes:

[0104] set up for The EFE network structure inputs the data vector and compresses the high-dimensional input data vector through the compression network. Encoded into a low-dimensional vector That is, hidden space data;

[0105] Furthermore, the raw data from the input layer to the hidden layer The encoding formula is:

[0106] ;

[0107] in, is the dimension of the input data, To compress network parameters, is the activation function, for The parameter matrix is ​​the weight of the compressed network, For the dimension, for The bias vector of

[0108] A feature difference weighting mechanism is introduced in each layer of the EFE network structure to calculate the feature difference weight of each input feature;

[0109] Furthermore, the feature difference weight calculation formula is:

[0110] ;

[0111] in, It is an activation function whose output is between 0 and 1 and is used to adjust the importance of the feature; for The parameter matrix is ​​the weight of the feature difference weighting mechanism, is the bias vector of the feature difference weighting mechanism, is the feature difference vector;

[0112] Apply feature difference weights to hidden layer feature representation , and obtain the weighted hidden representation ;

[0113] Furthermore, the weighted hidden representation calculation formula is:

[0114] ;

[0115] in, is the element-wise product function;

[0116] EFE transforms the weighted hidden representation into Decoding, restoring low-dimensional data to the initial dimension, reconstructing and outputting;

[0117] Furthermore, the decoding formula from the hidden layer to the output layer is

[0118] ;

[0119] in For input data The output data vector reconstructed by EFE, is the activation function, To reconstruct the weights of the network, is the bias vector of the reconstruction network.

[0120] Specifically, when executing step S2-3, it includes:

[0121] The initial model is trained based on the training set, a reconstruction loss function is constructed based on the control data and the training prediction value, the reconstruction loss function is minimized based on the gradient descent method, and the weights and parameters of the initial model are iterated by back propagation until the reconstruction loss function tends to be stable. The training is terminated, and the trained model is tuned based on the test set to obtain the SEFE model.

[0122] In fact, during the training process, the SEFE model learns the intrinsic structure and characteristics of the data by minimizing the reconstruction loss function, which measures the error between the compressed and reconstructed data and the original data; its weights and biases are randomly initialized and iteratively updated through back-propagation during training.

[0123] Furthermore, the reconstruction loss function , that is, the calculation formula of the optimization objective function of the gradient descent algorithm is:

[0124] ;

[0125] in, is the total number of samples in the training set, is the original data, It is the output data vector reconstructed by EFE.

[0126] Furthermore, the calculation formula of the gradient descent method is:

[0127] ;

[0128] The parameter update formula is:

[0129] ;

[0130] In the formula, are the network parameters of the SEFE model, yes Relative to the parameters The gradient of is the learning rate.

[0131] In some embodiments, when executing step S3, it specifically includes:

[0132] S3-1, performing outlier detection on the feature vector set after denoising based on the SEFE model to obtain outliers;

[0133] S3-2. Set abnormal data judgment criteria, and mark abnormal values ​​exceeding the judgment criteria as abnormal data, otherwise mark them as normal data.

[0134] Specifically, when executing step S3-1, the compression structure of the SEFE model itself can remove noise during the encoding process to achieve a data cleaning effect.

[0135] Specifically, when executing step S3-2, the abnormal data determination criteria are set, including:

[0136] The calculation formula for the abnormal data judgment standard is as follows:

[0137] ;

[0138] In the formula, is the outlier value of the data; is the standard error of the SEFE model prediction, is the data to be determined, For data The residual error.

[0139] In fact, in the process of training the SEFE model, the reconstruction loss function of each training sample is Credited Arrays, , calculate the arithmetic mean root , residual error, and standard error.

[0140] Furthermore, when determining anomalies, if a data The residual error is , then the criterion for abnormal data is , then determine It is abnormal data containing large error values. Data with judgment values ​​exceeding the judgment standard are marked as abnormal data, otherwise they are marked as normal data.

[0141] Furthermore, the residual error calculation formula is:

[0142] ;

[0143] In the formula, for The sequence number in the array is Elements of is the arithmetic mean root;

[0144] Furthermore, the standard error calculation formula is:

[0145] ;

[0146] In the formula, for The total number of array elements. Other parameters are the same as above.

[0147] In some embodiments, when executing step S4, it specifically includes:

[0148] S4-1, clustering normal data and abnormal data based on the vector group as an initial clustering center;

[0149] S4-2, calculating the difference between the distribution of abnormal data and the distribution of normal data, and rating the abnormal data whose distribution difference exceeds a preset threshold as dirty data, otherwise it is rated as ordinary abnormal data;

[0150] S4-3. Extract dirty data to the isolation area for data management.

[0151] Specifically, when executing step S4-1, clustering normal data and abnormal data based on the vector group as the initial clustering center includes:

[0152] Determine the number of initial clusters based on the vector group, randomly select the initial centroid, calculate the distance between the data point and the centroid, assign each data point to the cluster corresponding to the nearest centroid, and calculate the average value of all data points in each cluster as the new centroid;

[0153] Repeat the above steps until the centroid no longer changes or the maximum number of iterations is reached, and a clustering result containing normal data and abnormal data is obtained.

[0154] In fact, we first determine the number of initial clusters as , the final clustering results include Normal data, and Class abnormal data, where The number of vector groups obtained by performing large model classification in step S1-3.

[0155] Specifically, when executing step S4-2, calculate The distribution of abnormal data The difference of normal data distribution, the calculation formula of distribution difference is:

[0156] ;

[0157] in, is the distribution difference, For normal data at point The probability density of For abnormal data at point The probability density of .

[0158] Furthermore, the preset threshold for classification is

[0159] ;

[0160] in is the classification threshold, is the average value of the distribution difference, is the standard deviation of the distribution difference, is the adjustment factor.

[0161] Specifically, when executing step S4-3, extracting dirty data to the isolation area for data management includes:

[0162] Create an independent virtual isolation zone to store and process dirty data;

[0163] Or, create a separate database to store dirty data and set its access permissions to administrators and system processes;

[0164] Alternatively, add field tags to dirty data, filter all data records of marked dirty data based on SQL query statements, and move the filtered dirty data from the original storage location to the isolation area based on scripts for management.

[0165] In fact, data that is judged to be dirty will be isolated or discarded, while ordinary abnormal data and normal data can be directly analyzed in the data security sandbox according to the requirements of the data users.

[0166] Applied to actual scenarios, isolation can be to create an independent virtual isolation area in the data security sandbox to store and process dirty data; it can be an independent database or folder specifically used to store data records that are judged to be dirty data, set strict access control policies, and only specific managers and system processes can access the isolation area to ensure that dirty data will not be misused or leaked; it can be to mark data points that are judged to be dirty data. For example, you can add a field such as "status": "dirty" to the data record, use scripts or query statements to filter out all data records marked as dirty data, write scripts or use automation tools to move the filtered dirty data from the original storage location to the isolation area, and complete the isolation operation.

[0167] Furthermore, data management in the isolation area includes log recording and regular cleanup, recording isolation operations of dirty data, including information such as time, operator and data identification, to facilitate subsequent tracking and auditing; regularly checking the data in the isolation area and choosing whether to permanently delete, archive or repair it according to the policy.

[0168] See also Figure 1 The embodiment of the present invention also provides a data cleaning system based on a data security sandbox, which specifically includes:

[0169] The significant difference feature extraction module extracts and preprocesses the data in the data security sandbox to obtain the original data, extracts features and polynomially transforms the original data to obtain a feature vector set;

[0170] A significant difference feature classification module calculates the correlation score of each feature in the feature vector set, and classifies the feature vector set based on the feature with the highest correlation score to obtain a plurality of vector groups;

[0171] SEFE model training module, resamples the original data and divides it into training set and test set, introduces feature difference weighting mechanism to stack and combine multiple enhanced feature encoders to build the initial model, and trains and tests the initial model based on the training set and test set to obtain the SEFE model;

[0172] An abnormal data determination module, based on the SEFE model, performs abnormal value detection on the feature vector set after denoising to obtain abnormal values, sets abnormal data determination criteria, and marks abnormal values ​​exceeding the determination criteria as abnormal data, otherwise marks them as normal data;

[0173] An abnormality degree grading module, clustering normal data and abnormal data based on the vector group as the initial clustering center, calculating the difference between the distribution of abnormal data and the distribution of normal data, and rating the abnormal data whose distribution difference exceeds a preset threshold as dirty data, otherwise it is rated as ordinary abnormal data;

[0174] The abnormal data processing module extracts dirty data to the isolation area for data management.

[0175] Specifically, the network structure of the SEFE model includes:

[0176] Data preprocessing network, used to add random small perturbations to the feature vector set;

[0177] Compression network, used to reduce the dimension of the preprocessed high-dimensional data into low-dimensional vectors;

[0178] Weighted hidden network, which is used to calculate the feature difference weights of low-dimensional vectors and apply them to the hidden layer feature representation to obtain the weighted hidden representation;

[0179] The reconstruction network is used to decode the weighted hidden representation, restore the low-dimensional data to the initial dimension, reconstruct the predicted value and calculate the outlier value.

[0180] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A data cleaning method based on a data security sandbox, characterized in that: include: Extracting and preprocessing the data in the data security sandbox to obtain the original data, extracting features and polynomially transforming the original data to obtain a feature vector set, calculating the correlation score of the feature vector set and classifying it to obtain a number of vector groups; The original data is resampled and divided into a training set and a test set. A feature difference weighting mechanism is introduced in each enhanced feature encoder. Each layer of enhanced feature encoder calculates the feature difference weighting as the importance index of the input of the next layer, forming a transmission and aggregation mechanism of the feature difference weighting to obtain the initial model. The initial model is trained and tested based on the training set and the test set to obtain the SEFE model. Based on the SEFE model, the feature vector set is denoised and then outlier detection is performed to obtain outliers, a judgment standard for abnormal data is set, and data features whose outliers exceed the judgment standard are marked as abnormal data, otherwise they are marked as normal data; Normal data and abnormal data are clustered based on the vector group as the initial clustering center, the difference between the distribution of abnormal data and the distribution of normal data is calculated, the abnormal data whose difference exceeds a preset threshold is rated as dirty data, and the dirty data is extracted to an isolation area for data management.

2. The data cleaning method based on data security sandbox according to claim 1 is characterized in that: The relevance score of the feature vector set is calculated and classified. The process of calculating the relevance score includes: Calculate the feature distance score of the enhanced feature vector, and the calculation formula is as follows: ； In the formula, For all The maximum value in For all The minimum value in To enhance the feature vector, To enhance the sum of distances between feature vectors and other features; Calculate the association score between the enhanced feature vector and the target variable. The calculation formula is as follows: ； in, and are the characteristic value and target variable value of the i-th sample respectively. and is the corresponding mean; The correlation score is calculated by combining the feature distance score and the association score, and the calculation formula is as follows: ； in, , is the weight parameter, is the feature distance score, is the relevance score.

3. The data cleaning method based on data security sandbox according to claim 1 is characterized in that: The relevance score of the feature vector set is calculated and classified, and the classification process includes: Each time, the feature with the highest correlation score in the feature vector set is selected for classification. After a set of classifications is obtained, the next feature with the corresponding score is selected for classification. The steps are repeated to divide the feature vector set into several non-overlapping vector groups.

4. The data cleaning method based on data security sandbox according to claim 1 is characterized in that: When the initial model is trained and tested based on the training set and the test set to obtain the SEFE model, it includes: Resample the original data and divide it into control data and training data, add Gaussian distribution noise to the training data to obtain noise data and divide it into training set and test set; The initial model is trained based on the training set, a reconstruction loss function is constructed based on the control data and the training prediction value, the reconstruction loss function is minimized based on the gradient descent method, and the weights and parameters of the initial model are iterated by back propagation until the reconstruction loss function tends to be stable. The training is terminated, and the trained model is tuned based on the test set to obtain the SEFE model.

5. The data cleaning method based on data security sandbox according to claim 4 is characterized in that: Set abnormal data judgment criteria, including: The calculation formula for the abnormal data judgment standard is as follows: ； In the formula, is the outlier value of the data, is the standard error of the SEFE model prediction, is the data to be determined, For data The residual error.

6. The data cleaning method based on data security sandbox according to claim 1 is characterized in that: When clustering normal data and abnormal data based on the vector group as the initial clustering center, it includes: Determine the number of initial clusters based on the vector group, randomly select the initial centroid, calculate the distance between the data point and the centroid, assign each data point to the cluster corresponding to the nearest centroid, and calculate the average value of all data points in each cluster as the new centroid; Repeat the above steps until the centroid no longer changes or the maximum number of iterations is reached, and a clustering result containing normal data and abnormal data is obtained.

7. The data cleaning method based on data security sandbox according to claim 1 is characterized in that: Extract dirty data to the isolation area for data management, including: Create an independent virtual isolation zone to store and process dirty data; Or, create a separate database to store dirty data and set its access permissions to administrators and system processes; Alternatively, add field tags to dirty data, filter all data records of marked dirty data based on SQL query statements, and move the filtered dirty data from the original storage location to the isolation area based on scripts for management.

8. A data cleaning system based on data security sandbox, characterized in that: include: A significant difference feature extraction module is used to extract and preprocess the data in the data security sandbox to obtain the original data, extract features and perform polynomial transformation on the original data to obtain a feature vector set; A significant difference feature classification module is used to calculate the correlation score of the feature vector set and classify it to obtain a plurality of vector groups; SEFE model training module, used to resample the original data and divide it into training set and test set, introduce feature difference weighting mechanism in each enhanced feature encoder, each layer of enhanced feature encoder calculates the feature difference weighting weight as the importance index of the input of the next layer, forms the transmission and aggregation mechanism of feature difference weighting weight to obtain the initial model, and trains and tests the initial model based on the training set and test set to obtain the SEFE model; An abnormal data determination module, based on the SEFE model, performs abnormal value detection on the feature vector set after denoising to obtain abnormal values, sets abnormal data determination criteria, and marks abnormal values ​​exceeding the determination criteria as abnormal data, otherwise marks them as normal data; An abnormality degree grading module, used to cluster normal data and abnormal data using the vector group as an initial clustering center, calculate the difference between the distribution of abnormal data and the distribution of normal data, and rate the abnormal data whose difference exceeds a preset threshold as dirty data; The abnormal data processing module is used to extract dirty data to the isolation area for data management.

9. The data cleaning system based on data security sandbox according to claim 8, characterized in that: The network structure of the SEFE model includes: Data preprocessing network, used to add random small perturbations to the feature vector set; Compression network, used to reduce the dimension of the preprocessed high-dimensional data into low-dimensional vectors; Weighted hidden network, which is used to calculate the feature difference weights of low-dimensional vectors and apply them to the hidden layer feature representation to obtain the weighted hidden representation; The reconstruction network is used to decode the weighted hidden representation, restore the low-dimensional data to the initial dimension, reconstruct the predicted value and calculate the outlier value.

Images