Industrial data cleaning method

Through the preprocessing, missing value processing and noise filtering detection of industrial data cleaning methods, the problem of difficult data cleaning algorithms in the big data era is solved, efficient and accurate cleaning of data is achieved, and the quality and consistency of data is ensured. Data cleaning requirements in multiple fields and scales.

CN118916613APending Publication Date: 2024-11-08QISHENG (LIAONING) IND GRP CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410970262.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing data cleaning algorithms are difficult to effectively process various types of data in the era of big data, and cannot meet the needs of data growth and the diversity of error forms, affecting the data analysis results.

Method used

An industrial data cleaning method is adopted, including preprocessing, missing value processing, noise filtering detection and data cleaning result inspection, and through regression model, maximum likelihood estimation, outlier analysis, wavelet denoising and repeated data detection, ensure that Accuracy and consistency of data.

Benefits of technology

It realizes efficient and accurate detection and repair of errors, missing and outliers in the data, ensures the quality of the cleaned data, is suitable for a variety of data formats and sources, supports data cleaning needs in different fields and scales, and enhances the credibility of the data. and reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916613A_ABST
    Figure CN118916613A_ABST
Patent Text Reader

Abstract

The invention discloses an industrial data cleaning method, and relates to the technical field of data processing, and the method comprises the following steps: S1, carrying out the preprocessing of to-be-cleaned data; s2, missing value processing is carried out on the preprocessed data, the data have a linear relation, and a regression model is established to predict missing values; if the data do not have a linear relationship, calculating a missing value by using maximum likelihood estimation; s3, noise filtering detection; s4, checking a data cleaning result, verifying whether the cleaned data meets an expected requirement and does not meet a task requirement, modifying a cleaning rule or model, performing data cleaning again, and checking and evaluating the result again; and outputting the cleaned data when the requirements are met. By the adoption of the industrial data cleaning method, various types of industrial data can be effectively cleaned, the method is suitable for multiple fields and application scenes, a user can obtain a high-quality data set, and a reliable basis is provided for subsequent data analysis and decision making.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an industrial data cleaning method. Background Art

[0002] With the rapid development of Internet technology, the amount of data has exploded, and all industries have accumulated massive amounts of data. The unification and integrity of data information has played an increasingly important role in the daily management of companies. The management, analysis and prediction of big data information have become an important goal of the company's digital transformation. Only reliable and accurate data can accurately reflect the actual situation and effectively support organizational decision-making. However, dirty data is ubiquitous in the real world. Incorrect or inconsistent data will seriously affect the results of data analysis, thus producing negative effects. The importance of data cleaning is thus highlighted - it can improve the credibility and accuracy of data, so the research on data cleaning is crucial. With the changes of the times, the forms of erroneous data have changed in various ways, and the growth of data volume has also put forward new requirements for the design of data cleaning algorithms. Many traditional data cleaning algorithms can no longer meet the needs of the big data era.

[0003] Therefore, it is necessary to provide an industrial data cleaning method to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide an industrial data cleaning method that can effectively clean various types of industrial data and is applicable to multiple fields and application scenarios. Users can obtain high-quality data sets and provide a reliable basis for subsequent data analysis and decision-making.

[0005] To achieve the above object, the present invention provides an industrial data cleaning method, comprising the following steps:

[0006] S1: Preprocess the data to be cleaned;

[0007] S2: The preprocessed data is processed for missing values. If there is a linear relationship between the data, a regression model is established to predict the missing values. If there is no linear relationship between the data, the maximum likelihood estimation is used to calculate the missing values.

[0008] S3: noise filtering detection;

[0009] S4: Verify the data cleaning results to see if the cleaned data meets the expected requirements. If it does not meet the task requirements, modify the cleaning rules or model, re-clean the data, and re-verify and evaluate the results. If it meets the requirements, output the cleaned data.

[0010] Preferably, in step S1, the preprocessing method includes data standardization, data format conversion, data conversion and mapping, and data sampling and segmentation; and also includes removing data input errors and correcting data record errors through external functions and external source files;

[0011] Data unification, using unified abbreviations, and standardizing all data based on the correspondence between abbreviations and full names;

[0012] Data format conversion: converting data with different representation formats, converting inconsistent data into consistent data, and converting a data table into multiple data tables with different structures;

[0013] Data sampling and segmentation: For large-scale data sets, representative samples can be obtained through sampling techniques, or the data sets can be segmented into small batches for processing;

[0014] Data conversion and mapping: For unstructured or chaotic data, conversion and mapping are performed to convert it into structured data.

[0015] Preferably, in step S2, the regression model is established using known data:

[0016] Y=Xβ+ε1

[0017] Among them, Y represents the known value, X represents the selected independent variable of Y, β represents the estimated parameter, and ε1 represents the error;

[0018] Using the least squares method to estimate Then Substitute the missing value model to get a set of interpolation values:

[0019] Z=Wβ+ε2

[0020] Among them, Z represents missing values, W represents the independent variable of missing values, and ε2 represents error;

[0021] Preferably, in step S2, the maximum likelihood estimation MLE is performed using the expectation-maximization algorithm EM:

[0022] The complete data set Y is represented as: Y = (X, Z);

[0023] Among them, X is an incomplete data set, X={x1,x2,x3,.....x n};

[0024] Z for Z i The collection of Z i is an implicit variable, Z i ∈{1,2,......,M}, M is a given finite integer, Y={(x1,z1),......,(xn ,z n )}, then the likelihood function of the complete data set Y is:

[0025]

[0026] Where Z = {z1,z2,.....z n};

[0027] S21: Auxiliary function Q(θ,θ (i-1) )=E(log(L(θ|X,Z)), where, f(Z|X,θ (i-1) ) is X and the parameter estimate θ of the previous iteration (i-1) The conditional probability density function of the latent variable Z under the condition, the parameters θ and θ (i-1) is the initial value and the last iteration value, then:

[0028] Q(θ,θ (i-1) )=E(log(L(θ|X,Z))=∫ z log(L(θ|X,Z)f(Z|X,θ (i-1) )dz

[0029] S22: Solving Q * , so that Q(θ * ,θ (i-1) ) to obtain the maximum value:

[0030] θ * = argmaxQ(θ * ,θ (i-1) ).

[0031] Preferably, in step S3, the noise detection is a cleaning method oriented to structure conflicts and a method oriented to attribute value detection;

[0032] The cleaning method for structural conflicts is manual detection and cleaning or function dependency method;

[0033] The attribute value detection methods are outlier analysis, wavelet denoising and duplicate data detection.

[0034] Preferably, outlier analysis uses outlier clustering to detect outliers; wavelet denoising finds the best approximation to the original signal according to a measurement standard to distinguish the original signal from the noise signal; and duplicate data detection specifically includes the following processes:

[0035] S31: Select a sorting keyword, select a key field or attribute value string in the record as a keyword for record sorting;

[0036] S32: record sorting, sorting the entire data set according to the selected sorting keyword, similar duplicate records are close in position, preparing for the next step of duplicate detection;

[0037] S33: Similar duplicate record detection, the decision value of the sub-model is counted as the category label, each category label is one vote, and the category with the highest vote is selected by relative majority voting method as the final conclusion output.

[0038] Preferably, the outlier factor calculation formula in outlier cluster detection is as follows:

[0039]

[0040] u ij represents the degree of membership of data x, and c is the number of clusters;

[0041] The outlier factor of a data point x is expressed as the ratio of the number of points whose weighted distance wd(x,y) is greater than the parameter dis to the size of the data set:

[0042]

[0043] Among them, U represents the set of data point objects, y represents other objects in the set U except the data point objects, and dis represents the threshold of the data, which is set by experience or analysis.

[0044] Preferably, wavelet denoising finds the best mapping from the actual signal space to the wavelet function space, and restores the best original signal;

[0045] Assume that the noisy signal F(k) = f(k) + n(k), k = 0, 1, 2, ..., N-1, where f(k) is the original signal and n(k) is the signal that obeys N(0, σ 2 ), and make discrete wavelet transform on the noisy signal F(k), we get:

[0046] W j,k =W f (j,k)+W n (j,k)(j=1,2,…,M,k=1,2,…,N)

[0047] Where W j,k , W f (j, k) and Wn(j, k) are the wavelet coefficients of the noisy signal, the original signal and the noise signal at the jth layer, respectively. M is the maximum number of decomposition layers, and N is the total number of signals. After performing discrete wavelet transform on F(k), the wavelet coefficient W is obtained. j,k It consists of two parts: the wavelet coefficient W corresponding to the original signal f(k) f (j,k) and the wavelet coefficient W corresponding to the noise signal n(k) n(j, k), the useful information in the image is expressed as a low-frequency signal, while the noise signal is expressed as a high-frequency signal. After wavelet decomposition, the wavelet coefficient of the signal is greater than the wavelet coefficient of the noise. A threshold is specified. The threshold is estimated using a fixed threshold or based on statistical characteristics and noise models. When W j,k When it is less than the threshold, it is considered that W j,k It is mainly caused by noise, so it is set to zero; when W j,k When it is greater than the threshold, it is considered that W j,k It is mainly caused by the signal, which is retained or shrunk to achieve signal-to-noise separation.

[0048] Preferably, in step S4, the recall rate of data cleaning is used as the evaluation index of the experimental results, and the calculation method is as follows:

[0049]

[0050] Where: λ represents the recall rate of the cleaning result, N represents the total number of experimental data, n1 represents the amount of abnormal data identified, and n2 represents the amount of data successfully cleaned; the standard for successful cleaning is set as the scalability of the data is greater than 1.0.

[0051] Preferably, the data cleaning method is implemented based on a computer or a cloud platform, including hardware components and software components. The hardware components include a processor, a storage device, and an input and output device; the software components include a data preprocessing module, a missing value processing module, an outlier detection module, and a data output module.

[0052] Therefore, the present invention adopts the above industrial data cleaning method, which has the following beneficial effects:

[0053] (1) The present invention uses predefined rules and models for data cleaning, which can efficiently and accurately detect and repair errors, omissions and outliers in the data.

[0054] (2) The present invention has strong scalability, supports multiple data formats and data sources, and is suitable for data cleaning needs in different fields and scales.

[0055] (3) The data consistency of the present invention ensures that the cleaned data has good logical relationship and overall integrity through the consistency verification module.

[0056] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of an industrial data cleaning method of the present invention;

[0058] Figure 2is a classification diagram of data quality problems in an embodiment of the present invention;

[0059] Figure 3 It is a similar duplicate record cleaning model in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0061] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0062] The words "include" or "comprises" and the like used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms "inside", "outside", "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly specified and limited, the terms "attachment" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0063] like Figure 2 As shown, data quality refers to the accuracy, timeliness, consistency and completeness of data.

[0064] The degree to which these indicators are met in the information system. Data quality issues are usually divided into four categories: single data source model layer issues, single data source instance layer issues, multiple data source model layer issues and multiple data source instance layer issues.

[0065] like Figure 1 As shown, the present invention provides an industrial data cleaning method, comprising the following steps: S1: preprocessing the data to be cleaned; in step S1, the preprocessing method includes data standardization, data format conversion, data conversion and mapping, and data sampling and segmentation; and also includes removing data input errors and correcting data record errors through external functions and external source files;

[0066] Data unification, using unified abbreviations, and standardizing all data based on the correspondence between abbreviations and full names;

[0067] Data format conversion: converting data with different representation formats, converting inconsistent data into consistent data, and converting a data table into multiple data tables with different structures;

[0068] Data sampling and segmentation: For large-scale data sets, representative samples can be obtained through sampling techniques, or the data sets can be segmented into small batches for processing;

[0069] Data conversion and mapping: For unstructured or chaotic data, conversion and mapping are performed to convert it into structured data.

[0070] S2: The preprocessed data is processed for missing values. If there is a linear relationship between the data, a regression model is established to predict the missing values. If there is no linear relationship between the data, the maximum likelihood estimation is used to calculate the missing values. In step S2, the regression model is usually expressed as follows:

[0071]

[0072] Among them, Y is a known value, Z is a missing value, X and W are the independent variables of Y and Z respectively, β is the estimated parameter, and ε1 and ε2 are error terms respectively.

[0073] Regression models use known data to build models:

[0074] Y=Xβ+ε1

[0075] Among them, Y represents the known value, X represents the selected independent variable of Y, β represents the estimated parameter, and ε1 represents the error;

[0076] Using the least squares method to estimate Then Substitute the missing value model to get a set of interpolation values:

[0077] Z=Wβ+ε2

[0078] Among them, Z represents the missing value, W represents the independent variable of the missing value, that is, the attribute value related to the missing value, and ε2 represents the error;

[0079] If the independent variable is a qualitative variable, the dummy variable processing method is adopted; if the dependent variable is a qualitative variable, the Logit transformation can be used to perform Logistic linear regression.

[0080] In step S2, the maximum likelihood estimation MLE is performed using the expectation-maximization algorithm EM:

[0081] The complete data set Y is represented as: Y = (X, Z);

[0082] Among them, X is an incomplete data set, X={x1,x2,x3,.....x n};

[0083] Z for Z i The collection of Z i is an implicit variable, Z i ∈{1,2,......,M}, M is a given finite integer, Y={(x1,z1),......,(x n ,z n )}, then the likelihood function of the complete data set Y is:

[0084]

[0085] Where Z = {z1,z2,.....z n};

[0086] S21: Auxiliary function Q(θ,θ (i-1) )=E(log(L(θ|X,Z)), where, f(Z|X,θ (i-1) ) is X and the parameter estimate θ of the previous iteration (i-1) The conditional probability density function of the latent variable Z under the condition, the parameters θ and θ (i-1) is the initial value and the last iteration value, then:

[0087] Q(θ,θ (i-1) )=E(log(L(θ|X,Z))=∫ z log(L(θ|X,Z)f(Z|X,θ (i-1) )dz

[0088] S22: Solving Q * , so that Q(θ * ,θ (i-1) ) to obtain the maximum value:

[0089] θ * = argmaxQ(θ * ,θ (i-1) ).

[0090] S3: Noise filtering detection; In step S3, noise detection is a cleaning method oriented to structure conflicts and a method oriented to attribute value detection;

[0091] The cleaning methods for structural conflicts are manual detection and cleaning or functional dependency methods; in order to resolve dependency conflicts in structural conflicts, the functional dependency method is adopted.

[0092] The methods for attribute value detection are outlier analysis, wavelet denoising and duplicate data detection. The detection of attribute values ​​is mainly attribute error values ​​and null values. The methods used to detect attribute error values ​​include statistical methods, clustering methods, etc. The methods used for null value detection are mainly manual methods, representative function value filling methods, etc.

[0093] Outlier analysis uses outlier clustering to detect outliers; a set of data objects generated by clustering is called a cluster. Data objects in the same cluster are more similar, while data objects in other clusters are less similar (distance is often used to measure similarity). Values ​​that fall outside the cluster set are called outliers, which are deleted to achieve denoising. The outlier factor is used to characterize the degree of outliers.

[0094] The calculation formula of outlier factor in outlier cluster detection is as follows:

[0095]

[0096] u ij represents the degree of membership of data x, and c is the number of clusters;

[0097] The outlier factor of a data point x is expressed as the ratio of the number of points whose weighted distance wd(x,y) is greater than the parameter dis to the size of the data set:

[0098]

[0099] Among them, U represents the set of data point objects, y represents other objects in the set U except the data point objects, and dis represents the threshold of the data, which is set by experience or analysis.

[0100] Wavelet denoising finds the best approximation to the original signal based on the measurement criteria and distinguishes the original signal from the noise signal; similar duplicate record cleaning refers to the process of deleting records that constitute similar duplicates and retaining only one record. The most common situations of similar duplicate records are homophones, inconsistent formats, and addresses composed of abbreviations. The sort-merge method is adopted, and its basic idea is: select a sorting keyword, sort the records according to the keyword, and then compare the adjacent records to determine whether they are similar duplicate records. For example, the SNM algorithm uses a sliding window, which can reduce the number of comparisons and greatly improve the detection efficiency. Duplicate data detection specifically includes the following processes:

[0101] S31: Select a sorting keyword, select a key field or attribute value string in the record as a keyword for record sorting;

[0102] S32: record sorting, sorting the entire data set according to the selected sorting keyword, similar duplicate records are close in position, preparing for the next step of duplicate detection;

[0103] S33: Similar duplicate record detection, the decision value of the sub-model is counted as the category label, each category label is one vote, and the category with the highest vote is selected by relative majority voting method as the final conclusion output.

[0104] A fixed window of size W is slid over the sorted data set, and the first record in the window is compared with the remaining W-1 records. During the comparison process, the field similarity is generally calculated using a similarity comparison algorithm, and then the record similarity is obtained by weighted average using the field weights. The record similarity is compared with the record similarity threshold to determine whether the two records are similar duplicate records. After the comparison is completed, the first record is taken out of the window, and the W+1th record enters the window. At this time, the first record in the window is compared with the other W-1 records. This process is repeated until the last record in the data set is compared. Finally, all similar duplicate records in the data set are obtained.

[0105] If the amount of data is huge, you can use machine automatic cleaning. If the amount of data is small, you can also use manual cleaning. In addition, you can also use a combination of machine cleaning and manual cleaning. The cleaning model that combines machine cleaning and manual cleaning is as follows: Figure 3 shown.

[0106] Wavelet denoising finds the best mapping from the actual signal space to the wavelet function space and restores the best original signal;

[0107] Assume that the noisy signal F(k) = f(k) + n(k), k = 0, 1, 2, ..., N-1, where f(k) is the original signal and n(k) is the signal that obeys N(0, σ 2 ), and make discrete wavelet transform on the noisy signal F(k), we get:

[0108] W j,k =W f (j,k)+W n (j,k)(j=1,2,…,M,k=1,2,…,N)

[0109] Where W j,k , W f (j, k) and Wn(j, k) are the wavelet coefficients of the noisy signal, the original signal and the noise signal at the jth layer, respectively. M is the maximum number of decomposition layers, and N is the total number of signals. After performing discrete wavelet transform on F(k), the wavelet coefficient W is obtained. j,k It consists of two parts: the wavelet coefficient W corresponding to the original signal f(k) f (j,k) and the wavelet coefficient W corresponding to the noise signal n(k) n(j, k), the useful information in the image is expressed as a low-frequency signal, while the noise signal is expressed as a high-frequency signal. After wavelet decomposition, the wavelet coefficient of the signal is greater than the wavelet coefficient of the noise. A threshold is specified. The threshold is estimated using a fixed threshold or based on statistical characteristics and noise models. When W j,k When it is less than the threshold, it is considered that W j,k It is mainly caused by noise, so it is set to zero; when W j,k When it is greater than the threshold, it is considered that W j,k It is mainly caused by the signal, which is retained or shrunk to achieve signal-to-noise separation.

[0110] S4: Check the data cleaning results to verify whether the cleaned data meets the expected requirements. If it does not meet the task requirements, modify the cleaning rules or models, re-clean the data, and re-check and evaluate the results; if it meets the requirements, output the cleaned data. In step S4, the recall rate of data cleaning is used as the evaluation index of the experimental results, and the calculation method is as follows:

[0111]

[0112] In the formula: λ represents the recall rate of the cleaning result, N represents the total number of experimental data, n1 represents the amount of abnormal data identified, and n2 represents the amount of data successfully cleaned; the standard for successful cleaning is set as the scalability of the data is greater than 1.0. The recall rate can effectively evaluate the quality of abnormal data cleaning. The higher the value of the calculated recall rate result, the more abnormal data the method identifies and cleans, and the more sufficient the cleaning of abnormal data in the database is. The calculation result of the recall rate is positively correlated with the reliability of the abnormal data cleaning method. The higher the recall rate, the higher the reliability of the method, and the higher the practical application value.

[0113] The data cleaning method is implemented based on a computer or cloud platform, and includes hardware components and software components. The hardware components include processors, storage devices, and input and output devices; the software components include data preprocessing modules, missing value processing modules, outlier detection modules, and data output modules.

[0114] Therefore, the present invention adopts the above-mentioned industrial data cleaning method, which can effectively fill missing values, correct erroneous values, process data deviations, clean various types of industrial data, improve data quality, and is suitable for multiple fields and application scenarios. It ensures that the final data set is available for use, prevents the analysis results from affecting the user's judgment due to data errors or inaccuracies, and provides a reliable basis for subsequent data analysis and decision-making.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An industrial data cleaning method, characterized in that: The following steps are involved: S1: Preprocess the data to be cleaned; S2: The preprocessed data is processed for missing values. If there is a linear relationship between the data, a regression model is established to predict the missing values. If there is no linear relationship between the data, the maximum likelihood estimation is used to calculate the missing values. S3: noise filtering detection; S4: Data cleaning result inspection, verify whether the cleaned data meets the expected requirements. If it does not meet the task requirements, modify the cleaning rules or models, re-clean the data, and re-inspect and evaluate the results; Output the cleaned data in accordance with the requirements.

2. The industrial data cleaning method according to claim 1, characterized in that: In step S1, the preprocessing method includes data standardization, data format conversion, data conversion and mapping, and data sampling and segmentation; it also includes removing data input errors and correcting data record errors through external functions and external source files; Data unification, using unified abbreviations, and standardizing all data based on the correspondence between abbreviations and full names; Data format conversion: converting data with different representation formats, converting inconsistent data into consistent data, and converting a data table into multiple data tables with different structures; Data sampling and segmentation: For large-scale data sets, representative samples can be obtained through sampling techniques, or the data sets can be segmented into small batches for processing; Data conversion and mapping: For unstructured or chaotic data, conversion and mapping are performed to convert it into structured data.

3. The industrial data cleaning method according to claim 2, characterized in that: In step S2, the regression model uses known data to build a model: Y=Xβ+ε1 Among them, Y represents the known value, X represents the selected independent variable of Y, β represents the estimated parameter, and ε1 represents the error; Using the least squares method to estimate Then Substitute into the missing value model to obtain a set of interpolation values ​​for missing attribute values: Z=Wβ+ε2 Among them, Z represents missing values, W represents the independent variable of missing values, and ε2 represents the error.

4. The industrial data cleaning method according to claim 3, characterized in that: In step S2, the maximum likelihood estimation MLE is performed using the expectation-maximization algorithm EM: The complete data set Y is represented as: Y = (X, Z); Among them, X is an incomplete data set, X={x1,x2,x3,.....x n }; Z for Z i The collection of Z i is an implicit variable, Z i ∈{1,2,......,M}, M is a given finite integer, Y={(x1,z1),......,(x n ,z n )}, then the likelihood function of the complete data set Y is: Where Z = {z1,z2,.....z n }; S21: Auxiliary function Q(θ,θ (i-1) )=E(log(L(θ|X,Z)), where, f(Z|X,θ (i-1) ) is X and the parameter estimate θ of the previous iteration (i-1) The conditional probability density function of the latent variable Z under the condition, the parameters θ and θ (i-1) is the initial value and the last iteration value, then: Q(θ,θ (i-1) )=E(log(L(θ|X,Z))=∫ z log(L(θ|X,Z)f(Z|X,θ (i-1) )dz S22: Solving Q * , so that Q(θ * ,θ (i-1) ) to obtain the maximum value: i * =argmaxQ(θ * ,i (i-1) )。 5. The industrial data cleaning method according to claim 4, characterized in that: In step S3, noise detection is a method for cleaning structure conflicts and a method for detecting attribute values; The cleaning method for structural conflicts is manual detection and cleaning or function dependency method; The attribute value detection methods are outlier analysis, wavelet denoising and duplicate data detection.

6. The industrial data cleaning method according to claim 5, characterized in that: Outlier analysis uses outlier clustering to detect outliers; wavelet denoising finds the best approximation to the original signal based on the measurement criteria to distinguish the original signal from the noise signal; duplicate data detection specifically includes the following processes: S31: Select a sorting keyword, select a key field or attribute value string in the record as a keyword for record sorting; S32: record sorting, sorting the entire data set according to the selected sorting keyword, similar duplicate records are close in position, preparing for the next step of duplicate detection; S33: Similar duplicate record detection, the decision value of the sub-model is counted as the category label, each category label is one vote, and the category with the highest vote is selected by relative majority voting method as the final conclusion output.

7. An industrial data cleaning method according to claim 6, characterized in that: The calculation formula of outlier factor in outlier cluster detection is as follows: u ij represents the degree of membership of data x, and c is the number of clusters; The outlier factor of a data point x is expressed as the weighted distance w d The ratio of the number of points (x,y) greater than the parameter dis to the size of the data set: Among them, U represents the set of data point objects, y represents other objects in the set U except the data point objects, and dis represents the threshold of the data, which is set by experience or analysis.

8. An industrial data cleaning method according to claim 7, characterized in that: Wavelet denoising finds the best mapping from the actual signal space to the wavelet function space and restores the best original signal; Assume that the noisy signal F(k) = f(k) + n(k), k = 0, 1, 2, ..., N-1, where f(k) is the original signal and n(k) is the signal that obeys N(0, σ 2 ), and make discrete wavelet transform on the noisy signal F(k), we get: W j,k =W f (j,k)+W n (j,k)(j=1,2,…,M,k=1,2,…,N) Where W j,k , W f (j, k) and Wn(j, k) are the wavelet coefficients of the noisy signal, the original signal and the noise signal at the jth layer, respectively. M is the maximum number of decomposition layers, and N is the total number of signals. After performing discrete wavelet transform on F(k), the wavelet coefficient W is obtained. j,k It consists of two parts: the wavelet coefficient W corresponding to the original signal f(k) f (j,k) and the wavelet coefficient W corresponding to the noise signal n(k) n (j, k), the useful information in the image is expressed as a low-frequency signal, while the noise signal is expressed as a high-frequency signal. After wavelet decomposition, the wavelet coefficient of the signal is greater than the wavelet coefficient of the noise. The threshold is specified. The threshold is estimated using a fixed threshold or based on statistical characteristics and noise models. When W j,k When it is less than the threshold, it is considered that W j,k It is mainly caused by noise, so it is set to zero; when W j,k When it is greater than the threshold, it is considered that W j,k It is mainly caused by the signal, which is retained or shrunk to achieve signal-to-noise separation.

9. The industrial data cleaning method according to claim 8, characterized in that: In step S4, the recall rate of data cleaning is used as the evaluation index of the experimental results, and the calculation method is as follows: Where: λ represents the recall rate of the cleaning result, N represents the total number of experimental data, n1 represents the amount of abnormal data identified, and n2 represents the amount of data successfully cleaned; the standard for successful cleaning is set as the scalability of the data is greater than 1.

0.

10. An industrial data cleaning method according to claim 9, characterized in that: The data cleaning method is implemented based on a computer or cloud platform, and includes hardware components and software components. The hardware components include processors, storage devices, and input and output devices; the software components include data preprocessing modules, missing value processing modules, outlier detection modules, and data output modules.

Citation Information

Cited By

  • Data processing method, device and equipment for hot galvanizing production line

    CN121682270A