Data cleaning method based on big data analysis

Abstract

The invention relates to the technical field of data processing, in particular to a data cleaning method based on big data analysis, and the method comprises the steps: obtaining core data according to the low-dimensional representation of to-be-cleaned data, and training a neural network through the core data; calculating the training effectiveness of each piece of core data and the loss sensitivity of each dimension in the low-dimensional representation; calculating the selection probability of each to-be-cleaned data except the core data, and taking the to-be-cleaned data of which the selection probability is greater than a selection threshold as new core data; and training a neural network by using the new core data, and iteratively acquiring the new core data until the new core data cannot be acquired or the to-be-cleaned data except the core data does not exist, thereby completing data cleaning. Through the technical scheme of the invention, the training effect of each piece of to-be-cleaned data can be accurately evaluated, and the accuracy of data cleaning is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data cleaning method based on big data analysis. Background Art

[0002] Neural network, also known as artificial neural network (ANN), is a mathematical model or computational model that imitates the structure and function of biological neural network. Types of neural networks include convolutional neural network, recurrent neural network and fully connected neural network, which are applied to many fields such as image recognition, text recognition, information security, etc. In order to ensure the accuracy of the output results of the neural network, when using training data to train the neural network, the training data needs to be cleaned to prevent the training data with poor training effect from participating in the training process of the neural network, resulting in poor training effect.

[0003] At present, a patent application document with publication number CN113762519A discloses a data cleaning method, device and equipment, the method comprising: obtaining a data set, the data set comprising multiple initial training image data; determining a score value of each initial training image data according to feature information of each initial training image data in the data set, the score value being used to represent the training effect of the initial training image data; wherein, when the feature information represents that the training effect of the initial training data is good, the score value of the initial training data is high, and when the feature information represents that the training effect of the initial training data is poor, the score value of the initial training data is low; selecting target training image data from the data set according to the score value of each initial training image data; wherein the initial training data with a high score value is used as the target training data; performing data cleaning according to the target training image data, wherein the feature information comprises application scenarios and / or data quality.

[0004] The above method evaluates the score of the initial training image data from two aspects: application scenario and data quality. The score can represent the training effect of the initial training image data. However, the training process of the neural network is complicated and requires multiple iterations. It is impossible to accurately evaluate the training effect of the initial training image data based only on the application scenario and data quality, resulting in inaccurate data cleaning. Summary of the invention

[0005] In order to solve the technical problem of inaccurate data cleaning, the present application provides a data cleaning method based on big data analysis, which can accurately evaluate the training effect of each data to be cleaned and improve the accuracy of data cleaning.

[0006] In a first aspect, the present application provides a data cleaning method based on big data analysis, the cleaning method comprising: obtaining core data based on a low-dimensional representation of the data to be cleaned, training a neural network using the core data, wherein the neighborhood density of the core data is greater than a density threshold; calculating the training effectiveness of each core data and the loss sensitivity of each dimension in the low-dimensional representation; calculating the selection probability of each data to be cleaned other than the core data, and taking the data to be cleaned whose selection probability is greater than the selection threshold as the new core data, and the data to be cleaned The probability of selection for: , Data to be cleaned The number of core data, Data to be cleaned Core data The effectiveness of training, is the number of dimensions of the low-dimensional representation, For Dimension The loss sensitivity of Data to be cleaned Medium Dimension The numerical value of For core data Medium Dimension The numerical value of is an exponential function; the neural network is trained using the new core data, and the new core data is iteratively obtained until no new core data can be obtained, or there is no data to be cleaned other than the core data, and the data cleaning is completed.

[0007] First, a dimensionality reduction algorithm is used to obtain a low-dimensional representation of the data to be cleaned. Among all low-dimensional representations, the data to be cleaned whose neighborhood density is greater than the density threshold is used as the core data, and the neural network is trained in the first stage; the training effectiveness of each core data and the loss sensitivity of each dimension in the low-dimensional representation are calculated; the selection probability of each data to be cleaned other than the core data is calculated, and the selection probability can reflect the training effectiveness of the data to be cleaned. The greater the selection probability, the greater the training effectiveness of the data to be cleaned. The data to be cleaned whose selection probability is greater than the selection threshold is used as the new core data, and the neural network is trained in another stage using the new core data; new core data are iteratively obtained until new core data cannot be obtained, or there is no data to be cleaned other than the core data. At this time, at different training stages of the neural network, the training effect of each data to be cleaned is accurately evaluated, and the neural network is trained using the data to be cleaned with good training effects through an iterative method to complete data cleaning and improve the accuracy of data cleaning.

[0008] Preferably, before obtaining the core data according to the low-dimensional representation of the data to be cleaned, the cleaning method further comprises: performing dimensionality reduction processing on the data to be cleaned by using a dimensionality reduction algorithm to obtain the low-dimensional representation of the data to be cleaned.

[0009] Preferably, the dimensionality reduction algorithm is a PCA algorithm or an autoencoding network.

[0010] Preferably, obtaining the core data based on the low-dimensional representation of the data to be cleaned includes: counting the number of other data to be cleaned within the neighborhood range to obtain the neighborhood density of each data to be cleaned; taking the data to be cleaned whose neighborhood density is greater than the density threshold as the core data, wherein the neighborhood range of the data to be cleaned is a range centered on the low-dimensional representation of the data to be cleaned and whose distance is less than or equal to a preset distance.

[0011] When the neighborhood density is greater than the density threshold, it means that there are more data around the data to be cleaned, and the data to be cleaned is local high-density data. The greater the possibility that the data to be cleaned is normal data, the more likely it is that the data to be cleaned is normal data. Therefore, the data to be cleaned whose neighborhood density is greater than the density threshold is taken as the core data to participate in the training process of the neural network.

[0012] Preferably, training the neural network using the core data includes: updating the model parameters of the neural network on the core data using a gradient descent method, and the updating process of the neural network model parameters is: , are the neural network model parameters before updating, is the preset learning rate, For core data The gradient value of are the updated neural network model parameters.

[0013] Preferably, the core data The effectiveness of training for: , and Core data and core data The gradient value of is the gradient threshold, The number of core data.

[0014] In the process of updating the neural network model parameters, the core data The larger the gradient value, the greater the update degree of the neural network, and the core data The better the training effectiveness, the more accurate the training effectiveness can be. Therefore, the normalized gradient value can be used to reflect the training effectiveness of each core data and achieve accurate quantification of the training effectiveness.

[0015] Preferably, the method for obtaining the loss sensitivity of dimension 𝑗 comprises: calculating the loss function value of the core data, and calculating the dimension in the low-dimensional representation of the core data After adding random disturbance, the loss change of the core data is calculated; the variance of the loss change of each core data is used as the dimension sensitivity to loss.

[0016] Dimensions When the loss sensitivity of tends to 0, it means that the dimension in the low-dimensional representation The numerical change of will not cause the change of the loss function value, and the loss function value can reflect the accuracy of the output results of the neural network. Therefore, the dimension When the loss sensitivity of tends to 0, it means that the dimension It will not affect the accuracy of the neural network output results, and the dimension The smaller the impact on the neural network training process, the more accurate the quantization dimension The degree of influence on the neural network training process.

[0017] Preferably, the dimension in the low-dimensional representation of the core data Adding random perturbations includes: Setting the perturbation baseline ; For core data Medium Dimension The process of adding random perturbations is: , For core data Medium Dimension The numerical value of is a random number between 0 and 1. To add random perturbations to the core data Medium Dimension The numerical value of .

[0018] Preferably, the data to be cleaned The method for obtaining the core data includes: calculating any core data and the data to be cleaned In response to the Euclidean distance being less than the preset distance, the core data is the data to be cleaned. The core data of the.

[0019] The technical solution of this application has the following beneficial technical effects: First, a dimensionality reduction algorithm is used to obtain a low-dimensional representation of the data to be cleaned. Among all low-dimensional representations, the data to be cleaned whose neighborhood density is greater than the density threshold is used as the core data, and the neural network is trained in the first stage; the training effectiveness of each core data and the loss sensitivity of each dimension in the low-dimensional representation are calculated; the selection probability of each data to be cleaned other than the core data is calculated, and the selection probability can reflect the training effectiveness of the data to be cleaned. The greater the selection probability, the greater the training effectiveness of the data to be cleaned. The data to be cleaned whose selection probability is greater than the selection threshold is used as the new core data, and the neural network is trained in another stage using the new core data; new core data are iteratively obtained until new core data cannot be obtained, or there is no data to be cleaned other than the core data. At this time, at different training stages of the neural network, the training effect of each data to be cleaned is accurately evaluated, and the neural network is trained using the data to be cleaned with good training effects through an iterative method to complete data cleaning and improve the accuracy of data cleaning.

[0020] Furthermore, the calculation formula for the selection probability Can be regarded as data to be cleaned With Core Data The smaller the value, the more data to be cleaned. Off-core data The closer the data is, the more data needs to be cleaned. The closer the training effectiveness is to the core data The training effectiveness, that is, the data to be cleaned The probability of selection Can reflect the data to be cleaned The training effectiveness, data to be cleaned The greater the training effectiveness, the greater the selection probability The larger the value, the more accurate the evaluation of the training effect of each data to be cleaned. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 It is a flow chart of a data cleaning method based on big data analysis according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0023] It should be understood that when the terms "first", "second", etc. are used in the claims, specification and drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0024] According to the first aspect of the present application, the present application provides a data cleaning method based on big data analysis. Figure 1 1 is a flow chart of a data cleaning method based on big data analysis according to an embodiment of the present application. Figure 1 As shown, the data cleaning method based on big data analysis includes steps S101 to S104, which are described in detail below.

[0025] S101, obtaining core data according to the low-dimensional representation of the data to be cleaned, and training a neural network using the core data, wherein the neighborhood density of the core data is greater than a density threshold.

[0026] In one embodiment, the data to be cleaned is training data of a neural network, which is used to train the neural network; when the neural network is a convolutional neural network, the data to be cleaned is a plurality of image data; when the neural network is a recurrent neural network, the data to be cleaned is a plurality of time series data.

[0027] The dimensionality reduction algorithm is used to reduce the dimensionality of the data to be cleaned to obtain a low-dimensional representation of the data to be cleaned, and the dimensionality reduction algorithm is a PCA algorithm or an autoencoder network. The number of dimensions in the low-dimensional representation can be preset. In the embodiment of the present application, the number of dimensions in the low-dimensional representation is set to 5.

[0028] In one embodiment, obtaining core data based on the low-dimensional representation of the data to be cleaned includes: counting the number of other data to be cleaned within the neighborhood range to obtain the neighborhood density of each data to be cleaned; taking the data to be cleaned whose neighborhood density is greater than the density threshold as the core data, wherein the neighborhood range of the data to be cleaned is a range centered on the low-dimensional representation of the data to be cleaned and whose distance is less than or equal to a preset distance.

[0029] The preset distance is 2, which means the data to be cleaned The neighborhood range is the data to be cleaned The low-dimensional representation of is the center, and the distance is less than or equal to 2. The distance is the Euclidean distance of the low-dimensional representation between the data to be cleaned.

[0030] Among them, the value of the density threshold is 5. When the neighborhood density is greater than the density threshold, it means that there are more data around the data to be cleaned, and the data to be cleaned is local high-density data. The greater the possibility that the data to be cleaned is normal data, the data to be cleaned with a neighborhood density greater than the density threshold is taken as the core data to participate in the training process of the neural network.

[0031] In one embodiment, after the core data is acquired, a neural network is trained on the core data using a gradient descent method to update the model parameters of the neural network.

[0032] S102, calculating the training effectiveness of each core data and the loss sensitivity of each dimension in the low-dimensional representation.

[0033] In one embodiment, after the first stage of training of the neural network is completed using the core data, the training effectiveness of each core data is calculated, and the training effectiveness is used to determine the degree of updating of the neural network by the core data.

[0034] In the process of neural network training, the gradient descent method is used to update the model parameters of the neural network. The updating process of the neural network model parameters is: , are the neural network model parameters before updating, is the preset learning rate, For core data The gradient value of is the updated neural network model parameter. From the above update process, we can see that the core data The larger the gradient value, the greater the update degree of the neural network, and the core data The better the training effectiveness, the better the training effectiveness of each core data can be reflected by the normalized gradient value. The preset learning rate is 0.05.

[0035] Specifically, core data The effectiveness of training for: , and Core data and core data The gradient value of is the gradient threshold, is the number of core data. The value of the gradient threshold is 1.

[0036] In one embodiment, a method for obtaining the loss sensitivity of dimension 𝑗 includes: calculating the loss function value of the core data, and calculating the dimension in the low-dimensional representation of the core data After adding random disturbance, the loss change of the core data is calculated; the variance of the loss change of each core data is used as the dimension sensitivity to loss.

[0037] Among them, the dimension in the low-dimensional representation of the core data Adding random perturbations includes: Setting the perturbation baseline ; For core data Medium Dimension The process of adding random perturbations is: , For core data Medium Dimension The numerical value of is a random number between 0 and 1. To add random perturbations to the core data Medium Dimension The value of the disturbance benchmark The value of is 1.

[0038] Understandably, the dimension When the loss sensitivity of tends to 0, it means that the dimension in the low-dimensional representation The numerical change of will not cause the change of the loss function value, and the loss function value can reflect the accuracy of the output results of the neural network. Therefore, the dimension When the loss sensitivity of tends to 0, it means that the dimension It will not affect the accuracy of the neural network output results, and the dimension The smaller the impact on the neural network training process.

[0039] In this way, after completing the first stage of neural network training using core data, the training effectiveness of each core data is accurately quantified, and the loss sensitivity of each dimension in the low-dimensional representation is quantified based on the loss function value of each core data. Among them, the training effectiveness is used to judge the degree of update of the core data on the neural network, and the loss sensitivity is used to reflect the influence of each dimension in the low-dimensional representation on the neural network training process.

[0040] S103, calculating the selection probability of each to-be-cleaned data other than the core data, and taking the to-be-cleaned data with a selection probability greater than a selection threshold as new core data.

[0041] In one embodiment, the core data is normal data, and after completing the first stage of training of the neural network based on the core data, the training effectiveness of each core data and the loss sensitivity of each dimension in the low-dimensional representation are accurately quantified.

[0042] At this time, in addition to the core data, there are still a large number of data to be cleaned. The selection probability of each data to be cleaned other than the core data is calculated. The larger the selection probability, the more conducive the data to be cleaned is to the training of the neural network. Therefore, the data to be cleaned with a selection probability greater than the selection threshold is used as the new core data for the training of the neural network. The selection threshold is 0.5.

[0043] Specifically, the data to be cleaned The probability of selection for: , Data to be cleaned The number of core data, Data to be cleaned Core data The effectiveness of training, is the number of dimensions of the low-dimensional representation, For Dimension The loss sensitivity of Data to be cleaned Medium Dimension The numerical value of For core data Medium Dimension The numerical value of is an exponential function.

[0044] Among them, the data to be cleaned The method for obtaining the core data includes: calculating any core data and the data to be cleaned In response to the Euclidean distance being less than the preset distance, the core data is the data to be cleaned. The core data of the.

[0045] It can be understood that the preset distance is the preset distance of the neighborhood range in step S101, that is, if the data to be cleaned Falling into core data Within the neighborhood of Data to be cleaned The core data of the. Can be regarded as data to be cleaned With Core Data The smaller the value, the more data to be cleaned. Off-core data The closer the data is, the more data needs to be cleaned. The closer the training effectiveness is to the core data The training effectiveness, that is, the data to be cleaned The probability of selection Can reflect the data to be cleaned The training effectiveness, data to be cleaned The greater the training effectiveness, the greater the selection probability The bigger.

[0046] It should be noted that the data to be cleaned The number of core data is 0, 1 or more; when the data to be cleaned When the number of core data is 0, it indicates data to be cleaned Does not belong to any core data, data to be cleaned The probability of selection is 0.

[0047] In this way, the training effectiveness of the core data is weighted and summed according to the weighted distance between the data to be cleaned and the core data, and the selection probability of each data to be cleaned other than the core data is obtained. The selection probability can reflect the training effectiveness of the data to be cleaned, and the data to be cleaned that is effective for the neural network training is selected as the new core data according to the selection probability.

[0048] S104, using the new core data to train the neural network, iteratively acquiring new core data, until no new core data can be acquired, or there is no data to be cleaned other than the core data, and data cleaning is completed.

[0049] In one embodiment, the neural network is trained using new core data to complete another stage of neural network training. At this time, the model parameters of the neural network are different from the model parameters of the neural network that completed the first stage of training in step S102. Steps S102 to S103 are repeated to screen the remaining data to be cleaned again to obtain new core data for training the neural network.

[0050] Iteratively obtain new core data until no new core data can be obtained or there is no data to be cleaned other than the core data, and data cleaning is completed.

[0051] The above introduces the technical principles and implementation details of a data cleaning method based on big data analysis of the present application through specific embodiments. First, a dimensionality reduction algorithm is used to obtain a low-dimensional representation of the data to be cleaned. Among all low-dimensional representations, the data to be cleaned with a neighborhood density greater than the density threshold is used as the core data to train the neural network in the first stage; the training effectiveness of each core data and the loss sensitivity of each dimension in the low-dimensional representation are calculated; the selection probability of each data to be cleaned other than the core data is calculated, and the selection probability can reflect the training effectiveness of the data to be cleaned. The greater the selection probability, the greater the training effectiveness of the data to be cleaned. The data to be cleaned with a selection probability greater than the selection threshold is used as the new core data, and the new core data is used to train the neural network for another stage; iteratively obtain new core data until no new core data can be obtained, or there is no data to be cleaned other than the core data. At this time, in different training stages of the neural network, the training effect of each data to be cleaned is accurately evaluated, and the neural network is trained using the data to be cleaned with good training effects through an iterative method to complete data cleaning and improve the accuracy of data cleaning.

[0052] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A data cleaning method based on big data analysis, characterized in that: The cleaning method comprises: Acquire core data based on low-dimensional representation of the data to be cleaned, and train a neural network using the core data, wherein a neighborhood density of the core data is greater than a density threshold; Calculate the training effectiveness of each core data and the loss sensitivity of each dimension in the low-dimensional representation; Calculate the selection probability of each data to be cleaned outside the core data, and take the data to be cleaned with a selection probability greater than the selection threshold as the new core data. The probability of selection for: , Data to be cleaned The number of core data. Data to be cleaned Core data The effectiveness of training, is the number of dimensions of the low-dimensional representation, For Dimension The loss sensitivity of Data to be cleaned Medium Dimension The numerical value of For core data Medium Dimension The numerical value of is an exponential function; The neural network is trained using the new core data, and new core data is iteratively acquired until no new core data can be acquired or there is no data to be cleaned other than the core data, thus completing data cleaning.

2. The data cleaning method based on big data analysis according to claim 1 is characterized in that: Before obtaining the core data according to the low-dimensional representation of the data to be cleaned, the cleaning method further includes: The dimensionality reduction algorithm is used to reduce the dimensionality of the data to be cleaned to obtain a low-dimensional representation of the data to be cleaned.

3. The data cleaning method based on big data analysis according to claim 2 is characterized in that: The dimension reduction algorithm is a PCA algorithm or an autoencoding network.

4. The data cleaning method based on big data analysis according to claim 1 is characterized in that: The obtaining of core data based on the low-dimensional representation of the data to be cleaned includes: Count the number of other data to be cleaned within the neighborhood to obtain the neighborhood density of each data to be cleaned; The data to be cleaned whose neighborhood density is greater than the density threshold is taken as the core data, wherein the neighborhood range of the data to be cleaned is a range centered on the low-dimensional representation of the data to be cleaned and whose distance is less than or equal to the preset distance.

5. The data cleaning method based on big data analysis according to claim 1 is characterized in that: Training a neural network using Core Data includes: The gradient descent method is used to update the model parameters of the neural network on the core data. The updating process of the neural network model parameters is: , are the neural network model parameters before updating, is the preset learning rate, For core data The gradient value of are the updated neural network model parameters.

6. The data cleaning method based on big data analysis according to claim 1 is characterized in that: Core Data The effectiveness of training for: , and Core data and Core Data The gradient value of is the gradient threshold, The number of core data.

7. The data cleaning method based on big data analysis according to claim 1 is characterized in that: The methods for obtaining the loss sensitivity of dimension 𝑗 include: Calculate the loss function value of the core data and the dimension in the low-dimensional representation of the core data After adding random disturbance, calculating the loss change of the core data; The variance of each core data loss change is used as the dimension sensitivity to loss.

8. The data cleaning method based on big data analysis according to claim 7 is characterized in that: The dimension of the low-dimensional representation of the core data Adding random perturbations includes: Setting the Perturbation Baseline ; For core data Medium Dimension The process of adding random perturbations is: , For core data Medium Dimension The numerical value of is a random number between 0 and 1. To add random perturbations to the core data Medium Dimension The numerical value of .

9. The data cleaning method based on big data analysis according to claim 1, characterized in that: Data to be cleaned The methods for obtaining the core data include: Calculate any core data and data to be cleaned In response to the Euclidean distance being less than the preset distance, the core data is the data to be cleaned. The core data of the.

Images