Iterative Cleaning Method for Transformer Oil Chromatographic Data

Abstract

An iterative cleaning method for transformer oil chromatogram data belongs to the technical field of data cleaning. The present invention is aimed at the problem that inferior data in transformer oil chromatogram data cannot be effectively identified and cleaned. It includes: dividing the data in the original data set into a rule-violating data set and a rule-compliant data set; using the rule-compliant data set to pre-train a classifier; using the classifier to calculate the violation score of the rule-violating data and select the data to be repaired; after repairing the data to be repaired, using the repaired data to retrain the classifier and re-update the model parameters of the classifier, iterating the above "selection-repair-update" process to improve the effect of the classifier; finally using the trained classifier to predict the rule-violating operation data in the actual operation data to obtain the cleaned data. The present invention is used for cleaning oil chromatogram data.

Description

Technical Field

[0001] The invention relates to an iterative cleaning method for transformer oil chromatographic data, belonging to the technical field of data cleaning. Background Art

[0002] At present, the research on poor quality data in time series often focuses on data cleaning based on statistical characteristics of the data and data cleaning based on prior knowledge such as rule dependency.

[0003] The data cleaning method based on the statistical characteristics of data calculates the statistics and statistical indicators of the time series data itself according to the distribution of the currently known sequence, and then clusters the data with close similarity coefficients through clustering and other methods to achieve the cleaning of poor quality data. For example, ActiveClean proposed by Sanjay et al. selects the data samples to be cleaned by judging the possibility of the data becoming poor quality data in the corresponding model. In recent years, the deep learning method of using autoencoders to select poor quality data is proposed to convert the data into a low-dimensional space, and reconstruct it through a decoder to propose the features in the data. The data that can be reconstructed well is regarded as correct data, while the data with problems in the reconstruction is regarded as poor quality data. Kim-HungLe et al. use the method of calculating the violation score of the data to clean the data, including combining the inverse nearest neighbor (INN) algorithm to calculate the three types of violation scores of the data - amplitude score, correlation score and variance score, and construct a decision tree, and finally use the correlation coefficient on the decision tree for clustering, and then achieve data cleaning through manual repair according to the clustering results.

[0004] The data cleaning method based on rule dependency is to determine the rules based on prior knowledge or infer the rules from the cleaned data through learning, and then rely on the determined rules to clean the remaining data, which reduces the cost of manual participation. For example, Manel Charfi, Yann Gripay and others divided the data into different granularities in time and space, and used different granularity constraints to process the data of different time and space granularities accordingly, so as to achieve more sophisticated data cleaning. Fan Ju and others proposed the concept of data preparation with people in the loop, and summarized the methods and tasks of manual participation in the preparation process of data extraction, labeling, integration, cleaning, etc. Compared with the automated repair algorithm, manual repair has the advantages of high repair accuracy, strong reliability, and good data repair effect in specific fields, but it also has the problem of high repair cost.

[0005] The current data cleaning models are faced with the following two main challenges: first, there are a large number of low-quality data and the causes of errors are complex. If all of this data is cleaned, the time cost of cleaning will be too high. Therefore, how to select low-quality, high-violation score data from low-quality data sets for data cleaning and reduce the time and space costs required for data cleaning is a major challenge currently faced; second, how to select a highly efficient model in data cleaning, which is also an important challenge to reduce training costs.

[0006] A series of data points arranged in the order of data generation time is called a time series. A multivariate time series refers to a time series that contains multiple univariate time series as components. Time series are widely used in industries such as financial services, climate, hydrology and water conservancy, signal analysis, industrial production and manufacturing. The high costs and risks caused by the inaccuracy, contradictions and inconsistencies in the timing of time series data have always been the focus of enterprises and governments. However, due to errors in the collection process and other situations, time series data has various errors.

[0007] Transformer oil chromatographic data has the characteristics of large volume, continuous sampling, low value density, and strong dynamics. Due to the certain correlation between the time series of oil chromatographic data, the complex correlation relationship makes it difficult to directly model and distinguish between poor quality data and correct data. There are widespread errors such as missing values, over-limit values, zero values, and non-over-limit mutation values ​​in transformer oil chromatographic data. If these abnormal and erroneous data cannot be effectively cleaned in time, using these erroneous data for analysis and decision-making will lead to hidden safety hazards in the production process and may cause collateral losses to power equipment. If the poor quality oil chromatographic data is simply discarded, the integrity of the relationship between the oil chromatographic data will be destroyed, and the value of the data in the time series will be reduced. In order to avoid the corresponding problems caused by poor quality data, it is necessary to identify the transformer oil chromatographic data, clean the poor quality data into clean data, and repair the inconsistency in the data. Summary of the invention

[0008] Aiming at the problem that poor quality data in transformer oil chromatographic data cannot be effectively identified and cleaned, the present invention provides an iterative cleaning method for transformer oil chromatographic data.

[0009] An iterative cleaning method for transformer oil chromatographic data of the present invention comprises:

[0010] Step 1: Obtain the original data set of transformer oil chromatogram, and use the rule-dependent detection method to divide the data in the original data set into rule-violating data sets X vio And the rule data set X acc ;

[0011] Step 2: Use the rule-compliant dataset Xacc Pre-train the classifier;

[0012] Step 3: Set the violation rule dataset X vio The data in is input into the classifier to obtain the data prediction value corresponding to the rule violation data, the violation score of the rule violation data is calculated according to the data prediction value and the rule violation data, and then the rule violation data is sorted in descending order according to the violation score; according to the violation score, the rule violation data that meets the score threshold is selected as the data to be repaired;

[0013] Step 4: Iterate and repair the data to be repaired:

[0014] In each round of iteration, the current batch of data to be repaired is manually repaired to obtain the current batch of repaired data; the classifier is retrained using the current batch of repaired data;

[0015] At the same time, the average gradient of the repaired data of the current batch is used to update the model parameters θ of the classifier;

[0016] Then, the next round of iteration is carried out, and steps 3 and 4 are repeated until the comparison result between the predicted value output by the updated classifier and the corresponding true value meets the end condition, and the iteration process ends; the final classifier is obtained;

[0017] Step 5: Obtain the actual operation data of the transformer oil chromatogram, and select the rule-violating operation data from the actual operation data based on the rule-dependent detection method, input the rule-violating operation data into the final classifier for prediction, and obtain the cleaned data of the rule-violating operation data.

[0018] According to the iterative cleaning method of transformer oil chromatographic data of the present invention, in step 1, the detection method based on rule dependence is:

[0019] The data at the same time point in the multivariate time series of the original data set are grouped into a tuple. If the tuple data includes data X and data Y, and data X satisfies rule Z and data Y exists, then the current tuple is regarded as rule-compliant data, otherwise it is regarded as rule-violating data.

[0020] According to the iterative cleaning method for transformer oil chromatogram data of the present invention, the violation score of the rule-violating data in step 3 is Score:

[0021] Score=Score sin +Score sinmul +Score mul ,

[0022] Where Score sin Score is the single dimension violation score of the rule violation data. sinmulScore is a comprehensive violation score for different dimensions of rule violation data. mul It is the violation score combining the ratio method between different dimensions of different violation rule data.

[0023] According to the iterative cleaning method of transformer oil chromatographic data of the present invention, the single dimension violation score Score sin The calculation method is:

[0024] Score sin =|xX avg | / |X max -X min |;

[0025] Where x is the single dimension data of the rule violation data, X avg is the average value of all single-dimensional data that violate the rule, X max is the maximum value of a single dimension of the data that violates the rule, X min The minimum value of a single dimension of data that violates the rule.

[0026] According to the iterative cleaning method of transformer oil chromatographic data of the present invention, the comprehensive violation scores of different dimensions of the rule-violating data are calculated. sinmul The calculation method is:

[0027]

[0028] Where P is the rule violation data, X predict is the predicted value of P.

[0029] According to the iterative cleaning method of transformer oil chromatographic data of the present invention, the violation scores Score of different dimensions of different violation rule data combined with the ratio method are calculated. mul The calculation method is:

[0030] After adding up the ratios of the rule violation data of different dimensions in the same multi-tuple and its predicted value, the average value is calculated as the violation score of the ratio method between different dimensions of different rule violation data. mul .

[0031] According to the iterative cleaning method for transformer oil chromatographic data of the present invention, the classifier adopts a small batch gradient descent classification model.

[0032] According to the iterative cleaning method for transformer oil chromatographic data of the present invention, in step 4, the method for obtaining the average gradient of the current batch of data to be repaired is:

[0033] For each multi-tuple of data to be repaired, the comprehensive gradients of all dimensions of the data are calculated separately, and then the average gradient of the current batch of data to be repaired is calculated based on all comprehensive gradients.

[0034] According to the iterative cleaning method of transformer oil chromatogram data of the present invention, for rule-violating data P, assuming that P includes first dimension data P1 and second dimension data P2, the method of manually modifying includes:

[0035] If P1 satisfies rule Z, but the corresponding data P2 does not satisfy rule Z, then the data P2 is repaired to make it satisfy rule Z.

[0036] According to the iterative cleaning method of transformer oil chromatogram data of the present invention, for rule-violating data P, assuming that P includes first dimension data P1 and second dimension data P2, the method of manually modifying includes:

[0037] If, under the condition that P2 satisfies rule Z, the corresponding data P1 does not satisfy rule Z, then the data P1 is repaired to make it satisfy rule Z.

[0038] Beneficial effects of the present invention: The method of the present invention combines manual participation to achieve the cleaning of transformer oil chromatographic data.

[0039] The method of the present invention combines rule dependency and original data sets, divides data into rule-violating data and rule-compliant data in the initialization phase, optimizes the selection of labeled data in the initialization phase, and enables the classifier to have higher accuracy in the initial phase.

[0040] The method of the present invention improves the representativeness of the violation score by optimizing the classifier model and the way of calculating the violation score, making the data with higher violation scores more likely to be low-quality data, thereby improving the accuracy of selecting low-quality data for manual labeling, so that the data can obtain a higher accuracy rate with less manual labeling workload.

[0041] The method of the present invention can achieve a high-quality low-quality data detection task with a smaller sample size, and can significantly reduce the number of annotations required for data cleaning, thereby reducing the burden of manual annotation.

[0042] The method of the present invention is used to clean transformer oil chromatographic data, which can ensure the integrity of the relationship between the oil chromatographic data, improve the value of its time series data, and thus ensure that the production link is more reliable and avoid losses to power equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of the iterative cleaning method of transformer oil chromatographic data of the present invention;

[0044] Figure 2 This is a diagram showing the effect of using the method of the present invention to clean the true value of H2 depth in transformer oil chromatogram data;

[0045] Figure 3 A comparison chart of the accuracy rates obtained based on the scale of the training set when the method of the present invention and other existing methods are used for data cleaning;

[0046] Figure 4 This is a comparison chart of the accuracy rates based on the number of iterations when the method of the present invention is used for data cleaning with other existing methods. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0049] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0050] Specific implementation method 1. Combination Figure 1 As shown, the present invention provides an iterative cleaning method for transformer oil chromatographic data, comprising:

[0051] Step 1: Obtain the original data set of transformer oil chromatogram, and use the rule-dependent detection method to divide the data in the original data set into rule-violating data sets X vio And the rule data set X acc ;

[0052] Step 2: Use the rule-compliant dataset X acc Pre-train the classifier;

[0053] Step 3: Set the violation rule dataset X vio The data in is input into the classifier to obtain the data prediction value corresponding to the rule violation data, the violation score of the rule violation data is calculated according to the data prediction value and the rule violation data, and then the rule violation data is sorted in descending order according to the violation score; according to the violation score, the rule violation data that meets the score threshold is selected as the data to be repaired;

[0054] Step 4: Iterate and repair the data to be repaired:

[0055] In each round of iteration, the current batch of data to be repaired is manually repaired to obtain the current batch of repaired data; the classifier is retrained using the current batch of repaired data;

[0056] At the same time, the average gradient of the repaired data of the current batch is used to update the model parameters θ of the classifier;

[0057] Then, the next round of iteration is carried out, and steps 3 and 4 are repeated until the comparison result between the predicted value output by the updated classifier and the corresponding true value meets the end condition, and the iteration process ends; the final classifier is obtained;

[0058] Step 5: Obtain the actual operation data of the transformer oil chromatogram, and select the rule-violating operation data from the actual operation data based on the rule-dependent detection method, input the rule-violating operation data into the final classifier for prediction, and obtain the cleaned data of the rule-violating operation data.

[0059] In this implementation, the end condition of the classifier update is:

[0060] (Predicted value - actual value) 2 ≤ε, where ε is the conditional threshold of the iteration end condition in step 4.

[0061] This implementation method calculates the violation score to find the data with a higher violation score. Figure 1 As shown in the figure, firstly, for the original data set, the data is preliminarily tested based on the existing rule dependency, and the original data set is divided into rule-violating data and rule-compliant data. The rule-compliant data will initialize the classifier model. Next, the violation score of the rule-violating data is calculated through the updated classifier, and the low-quality data with high violation scores is selected for manual modification. The modified data constitutes clean data and is then passed to the classifier to update the classifier model. Through repeated iterations, the accuracy of the classifier is improved, and then the accuracy of the true value prediction of the low-quality data is improved.

[0062] In this embodiment, the cleaned data does not need to be screened again for cleaning.

[0063] Furthermore, in step 1, the detection method based on rule dependency is:

[0064] The data at the same time point in the multivariate time series of the original data set are grouped into a tuple. If the tuple data includes data X and data Y, and data X satisfies rule Z, data Y must exist. In this case, the current tuple is regarded as rule-compliant data, otherwise, as rule-violating data.

[0065] Furthermore, the predicted value of the poor quality data output by the classifier is used as the basis for calculating the violation score; in each round of iteration, the predicted value of the poor quality data is used as the basis for calculating the violation score in the next step.

[0066] In order to select low-quality data for manual repair, it is necessary to calculate the violation scores corresponding to the data points, so as to find data with higher violation scores from the data points for manual repair.

[0067] The violation score of the rule-violating data in step 3 is Score:

[0068] Score=Score sin +Score sinmul +Score mul ,

[0069] Where Score sin Score is the single dimension violation score of the rule violation data. sinmul Score is a comprehensive violation score for different dimensions of rule violation data. mul It is the violation score combining the ratio method between different dimensions of different violation rule data.

[0070] Among them, a single dimension violates the score Score sin The calculation method is:

[0071] Score sin =|xX avg | / |X max -X min |;

[0072] Where x is the single dimension data of the rule violation data, X avg is the average value of all single-dimensional data that violate the rule, X max is the maximum value of a single dimension of the data that violates the rule, X min The minimum value of a single dimension of data that violates the rule.

[0073] Comprehensive violation scores of different dimensions of rule violation data Score sinmul The calculation method is:

[0074]

[0075] Where P is the rule violation data, X predict is the predicted value of P.

[0076] In violation of Score sinmul In the calculation, the initial value of the data is compared with its predicted value. If the gap is larger, it means that the violation score of the data is lower.

[0077] Violation score Score of different dimensions of different violation rule data combined with ratio method mul The calculation method is:

[0078] For multi-dimensional erroneous data, the data is judged to be erroneous by whether the ratio of different data in the same tuple exceeds the range. For multiple data with correlation and the corresponding predicted values, the average of their ratios is used as the violation score. That is, the ratios of different dimensions of the rule-violating data and their predicted values ​​in the same tuple are summed up, and the average value is calculated as the violation score of different dimensions of different rule-violating data combined with the ratio method. mul .

[0079] As an example, the classifier adopts a mini-batch gradient descent classification model.

[0080] Furthermore, in step 4, the average gradient of the current batch of data to be repaired is obtained as follows:

[0081] For each multi-tuple of data to be repaired, the comprehensive gradients of all dimensions of the data are calculated separately, and then the average gradient of the current batch of data to be repaired is calculated based on all comprehensive gradients. The average gradient is used to initialize the coefficient θ of the small batch gradient descent model.

[0082] As an example, for rule-violating data P, assuming that P includes first dimension data P1 and second dimension data P2, the method for manual modification includes:

[0083] If P1 satisfies rule Z, but the corresponding data P2 does not satisfy rule Z, then the data P2 is repaired to satisfy rule Z.

[0084] Alternatively, for rule-violating data P, assuming that P includes first dimension data P1 and second dimension data P2, the method for manual modification includes:

[0085] If, under the condition that P2 satisfies rule Z, the corresponding data P1 does not satisfy rule Z, then the data P1 is repaired to make it satisfy rule Z.

[0086] In the data repair step, this implementation is discussed in two repair situations:

[0087] For P1, Z-→P2, there are two cases: repairing P2 or P1. 1) Repairing P2, changing the properties of P2 so that P2 meets the constraints of rule Z. For data with multiple constraints, automated repair methods have certain difficulties, and manual repair can give the correct repair value. 2) Repairing P1. Normally, it is assumed that the data P1 involved in the dependency is correct, but if the data P1 violates too many rule dependencies, such as P1, Z1-→P2, P1, Z2-→P2, if the rules Z1 and Z2 corresponding to the data P1 are violated, then consider repairing P1.

[0088] In the repair phase, the data that needs to be repaired is usually continuous data within one or several periods of time. Manual repair requires not only indicating whether each piece of data is wrong, but also returning the correct result to the classifier. The result of manual repair is passed to the classifier model, and the model is updated using the mini-batch gradient descent method.

[0089] Figure 2 As shown, it can be seen that between the timestamp T0 and the timestamp T60, the oil chromatogram data has no obvious abnormal value, and the repair value is close to the data itself; for the mutation values ​​at the four single points of T62, T79, T110, and T138 and the oscillation values ​​between T156 and T180, after repairing with the method of the present invention, a more reasonable repair result is obtained. It can be seen that the method of the present invention has good applicability for cleaning oil chromatogram data.

[0090] Figure 3 and Figure 4 IDCHI is used to refer to the method of the present invention. The accuracy comparison of the two indicators of training set size and iteration rounds is tested respectively. From the comparison of the curves, it can be seen that the method of the present invention is superior to other existing methods. GBDT is a gradient boosting decision tree method, which adopts a multi-model integration strategy to fit the residual to reduce the deviation and variance of the model; the MLP method is a data cleaning method that combines neural networks and active learning methods.

[0091] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. An iterative cleaning method for transformer oil chromatographic data, characterized in that include, Step 1: Obtain the original data set of transformer oil chromatogram, and use the rule-dependent detection method to divide the data in the original data set into rule-violating data sets X vio And the rule data set X acc ; Step 2: Use the rule-compliant dataset X acc Pre-train the classifier; Step 3: Set the rule violation dataset X vio The data in is input into the classifier to obtain the data prediction value corresponding to the rule violation data, the violation score of the rule violation data is calculated according to the data prediction value and the rule violation data, and then the rule violation data is sorted in descending order according to the violation score; according to the violation score, the rule violation data that meets the score threshold is selected as the data to be repaired; Step 4: Iterate and repair the data to be repaired: In each round of iteration, the current batch of data to be repaired is manually repaired to obtain the current batch of repaired data; Retrain the classifier using the current batch of repaired data; At the same time, the average gradient of the repaired data of the current batch is used to update the model parameters θ of the classifier; Then proceed to the next round of iteration, repeating steps 3 and 4 until the comparison result between the predicted value output by the updated classifier and the corresponding true value meets the end condition, and the iteration process ends; Get the final classifier; Step 5: Obtain the actual operation data of the transformer oil chromatogram, and select the rule-violating operation data from the actual operation data based on the rule-dependent detection method, input the rule-violating operation data into the final classifier for prediction, and obtain the cleaned data of the rule-violating operation data.

2. The iterative cleaning method for transformer oil chromatographic data according to claim 1, characterized in that: In step 1, the detection method based on rule dependency is: The data at the same time point in the multivariate time series of the original data set are grouped into a tuple. If the tuple data includes data X and data Y, and data X satisfies rule Z and data Y exists, then the current tuple is regarded as rule-compliant data, otherwise it is regarded as rule-violating data.

3. The iterative cleaning method for transformer oil chromatographic data according to claim 2, characterized in that: The violation score of the rule-violating data in step 3 is Score: .Score=.Score sin +Score sinmul +Score mul , Where Score sin Score is the single dimension violation score of the rule violation data. sinmul Score is a comprehensive violation score for different dimensions of rule violation data. mul It is the violation score combining the ratio method between different dimensions of different violation rule data.

4. The iterative cleaning method for transformer oil chromatographic data according to claim 3, characterized in that: Single dimension violation score Score sin The calculation method is: Score sin =|x-X avg | / |X max -X min |; Where x is the single dimension data of the rule violation data, X avg is the average value of all single-dimensional data that violate the rule, X max is the maximum value of a single dimension of the data that violates the rule, X min The minimum value of a single dimension of data that violates the rule.

5. The iterative cleaning method for transformer oil chromatographic data according to claim 4, characterized in that: Comprehensive violation scores of different dimensions of rule violation data Score sinmul The calculation method is: Where User is the rule violation data, X predict The predicted value for the household.

6. The iterative cleaning method for transformer oil chromatographic data according to claim 5, characterized in that: Violation score Score of different dimensions of different violation rule data combined with ratio method mul The calculation method is: After adding up the ratios of the rule violation data of different dimensions in the same multi-tuple and its predicted value, the average value is calculated as the violation score of the ratio method between different dimensions of different rule violation data. mul .

7. The iterative cleaning method for transformer oil chromatographic data according to any one of claims 1 to 6, characterized in that: The classifier adopts a small batch gradient descent classification model.

8. The iterative cleaning method for transformer oil chromatographic data according to claim 7, characterized in that: In step 4, the average gradient of the current batch of data to be repaired is obtained as follows: For each multi-tuple of data to be repaired, the comprehensive gradients of all dimensions of the data are calculated separately, and then the average gradient of the current batch of data to be repaired is calculated based on all comprehensive gradients.

9. The iterative cleaning method for transformer oil chromatographic data according to claim 8, characterized in that: For rule-violating data P, assuming that P includes first dimension data P1 and second dimension data P2, the method for manual modification includes: If P1 satisfies rule Z, but the corresponding data P2 does not satisfy rule Z, then the data P2 is repaired to satisfy rule Z.

10. The iterative cleaning method for transformer oil chromatographic data according to claim 8, characterized in that: For rule-violating data P, assuming that P includes first dimension data P1 and second dimension data P2, the method for manual modification includes: If, under the condition that P2 satisfies rule Z, the corresponding data P1 does not satisfy rule Z, then the data P1 is repaired to make it satisfy rule Z.

Images