Monitoring data processing method, system and equipment of transformer and medium

Abstract

The invention provides a monitoring data processing method, system and equipment of a transformer and a medium, and belongs to the field of data processing, and the method comprises the following steps: obtaining an original data set of a sensor of a power transformer; using an RANSAC algorithm to iteratively select a random sample from the sensor original data set, fitting a data relation model through the random sample, and calculating a residual error of the data relation model; taking the residual error as the input of an improved multi-scale Sensen isolation algorithm IFore, carrying out random sampling on subsets with different sizes, constructing a plurality of isolated forests with different sample scales, and generating isolated trees with different scales in each isolated forest; calculating abnormal scores of the data points in each isolated tree on different scales, and averaging the abnormal scores on different scales to obtain a final abnormal score; and processing the data points according to the final abnormal score. According to the invention, the accuracy of abnormal data processing can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of data cleaning, and in particular relates to a transformer monitoring data processing method, system, equipment and medium. Background Art

[0002] Impact of unreliable data on power transformer condition monitoring: Due to the operating environment, sensor failure, unstable data transmission and other reasons, unreliable abnormal data is often generated in the power transformer monitoring system. These unreliable data will interfere with the analysis and detection results of the monitoring system, and thus affect the accurate identification of the transformer status and fault warning. Therefore, an effective data processing method is needed to identify and repair these unreliable data, so as to improve the quality and reliability of monitoring data.

[0003] In the analysis of dissolved gases in transformer oil, due to the complexity and diversity of data, traditional data processing methods often have difficulty in accurately identifying and processing abnormal data when dealing with data with complex noise interference and outliers, especially when the data set contains anomalies at multiple scales, which may occur in different time and frequency ranges. Existing methods cannot detect and identify anomalies at different scales when processing multi-scale abnormal data, and are therefore insufficient to comprehensively detect and process all anomalies, resulting in low accuracy when processing high-dimensional complex data. Summary of the invention

[0004] In order to overcome the disadvantage of low data processing accuracy in the above-mentioned prior art, the present invention provides a transformer monitoring data processing method, comprising the following steps:

[0005] Get the raw sensor data set of the power transformer;

[0006] Iteratively select random samples from the sensor raw data set using the RANSAC algorithm, fit the data relationship model through the random samples, and calculate the residual of the data relationship model;

[0007] The residual is used as the input of the improved multi-scale forest isolation algorithm IForest, and the improved multi-scale forest isolation algorithm IForest is used to randomly sample subsets of different sizes of the sensor original data set to construct multiple isolation forests with different sample sizes, and isolated trees are generated at different scales in each isolation forest; the anomaly scores of the data points in each isolation tree at different scales are calculated, and the anomaly scores at different scales are averaged to obtain the final anomaly score;

[0008] Data points are processed based on the final anomaly score.

[0009] Preferably, the method of constructing a plurality of isolation forests of different sample sizes and generating isolated trees at different scales in each isolation forest is specifically as follows: recursively partitioning the data by randomly selecting features and splitting points to obtain each isolated tree.

[0010] Preferably, the calculation of the anomaly scores of the data points in each isolated tree at different scales comprises the following steps:

[0011] Obtain multiple path lengths by traversing the number of edges from the data point to the root node in the isolation tree;

[0012] Get the average path length by passing the data point through all the trees in the isolation forest;

[0013] The anomaly score is calculated based on the average path length of the data points.

[0014] Preferably, before using the RANSAC algorithm to iteratively select random samples from the sensor raw data set, the method also includes deleting redundant data in the sensor raw data set, specifically: checking whether multiple entries of data in the sensor raw data set are the same at any given time, and if the data are exactly the same, it is considered to be redundant data and deleted from the data set.

[0015] Preferably, before using the RANSAC algorithm to iteratively select random samples from the sensor raw data set and after deleting redundant data, the method also includes processing missing values ​​in the sensor raw data set, specifically: checking the missing data in the sensor raw data set and filling the missing values ​​using a forward filling method.

[0016] The present invention also provides a transformer monitoring data processing system, comprising:

[0017] The data acquisition module is used to acquire the original data set of the sensor of the power transformer.

[0018] The model building module is used to iteratively select random samples from the sensor raw data set using the RANSAC algorithm, fit the data relationship model through the random samples, and calculate the residual of the data relationship model.

[0019] The anomaly score acquisition module is used to use the residual as the input of the improved multi-scale forest isolation algorithm IForest, use the improved multi-scale forest isolation algorithm IForest to randomly sample subsets of different sizes of the sensor original data set, construct multiple isolation forests of different sample sizes, and generate isolated trees at different scales in each isolation forest; calculate the anomaly scores of the data points in each isolation tree at different scales, and average the anomaly scores at different scales to obtain the final anomaly score.

[0020] Data cleaning module, used to process data points according to the final anomaly score.

[0021] The present invention also provides a computer device, comprising a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the transformer monitoring data processing method.

[0022] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the transformer monitoring data processing method.

[0023] The transformer monitoring data processing method provided by the present invention has the following beneficial effects:

[0024] The present invention uses the RANSAC algorithm to iteratively select random samples from the original data set of the sensor, and can fit the data relationship model through the random samples. By inputting the residual of the fitted data relationship model into the improved multi-scale isolation forest algorithm, a plurality of isolation forests of different sample sizes are constructed, and isolated trees are generated at different scales in each isolation forest, and the abnormal scores of the data points in each isolated tree at different scales are calculated, so as to effectively detect and identify abnormal data at different scales. This method overcomes the limitations of traditional methods in multi-scale anomaly detection and significantly improves the accuracy of abnormal data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 Flow chart of a transformer monitoring data processing method according to an embodiment of the present invention;

[0027] Figure 2 It is the flow chart of RANSAC algorithm;

[0028] Figure 3 The RANSAC fitting results show (the relationship between CH4 and H2);

[0029] Figure 4 The method proposed in the present invention is used to process the influence of abnormal data on H2 concentration. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0031] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the technical solutions of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0032] In addition, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified or limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In the description of the present invention, unless otherwise specified, "plurality" means two or more, which will not be described in detail here.

[0033] Example

[0034] The present invention provides a method for processing transformer monitoring data specifically for complex data sets, such as those from power transformers. The process ensures that untrustworthy and erroneous data are identified and corrected, improving the reliability of the data set for further analysis. The method is specifically as follows: Figure 1 As shown, the following steps are included:

[0035] Step 1: Obtain the raw sensor data set of the power transformer.

[0036] The data in the sensor raw data set of the power transformer of the present invention is mainly the oil-soluble gas data of the transformer, which is divided into two categories: normal data and abnormal data. Abnormal data can be further divided into two categories: unreliable data and fault data. Unreliable data refers to data generated due to short-term sensor failure, communication error, storage problem or other interference, which usually includes duplicate data, lost data and erroneous data; fault data is data generated due to equipment abnormality, interference or failure, reflecting the abnormal operating state of the power equipment. These data are crucial to determine the status of the equipment. In the process of transformer data cleaning, it is very important to distinguish between unreliable data and fault data. The focus is on detecting and correcting unreliable data to ensure the high quality and reliability of the data.

[0037] Step 2: Redundant data removal.

[0038] Figure 1 The first decision point in checks if the data at any given time is identical across multiple entries. If the data is exactly the same, it is considered redundant and is removed from the dataset. This step helps eliminate duplicate data that could affect your analysis.

[0039] Step 3: Missing value processing.

[0040] After removing redundant data, the next step is to detect and handle missing values ​​in the dataset. Missing values ​​are filled using the forward filling method, which assumes that the last known value can be a reasonable estimate of the missing data point. This method maintains the temporal continuity of the dataset.

[0041] Step 4: Use the RANSAC algorithm to iteratively select random samples from the sensor raw data set, fit the data relationship model through the random samples, and calculate the residual of the data relationship model.

[0042] The gas ratio method is based on thermodynamics and empirical observations, and uses three pairs (C2H2 / C2H4, CH4 / H2, C2H4 / C2H6) of ratios of five gases (CH4, C2H4, C2H6, C2H2, H2) to identify the type of fault. Behind this method is that during the operation of the power transformer, electrical stress and thermal stress cause the decomposition of insulating oil and solid insulating materials, producing gas. Different types of faults produce different types and concentrations of gas. For example, high temperature overheating produces more acetylene (C2H2), while partial discharge produces more hydrogen (H2). This method is widely used in fault diagnosis and condition monitoring of power transformers. By analyzing the dissolved gas in the transformer oil, various internal faults of the transformer can be identified. The present invention is analyzed and studied on the basis of this principle.

[0043] RANSAC is an iterative method for estimating the parameters of a mathematical model from a set of observations that contain outliers (i.e., erroneous data). Unlike the least squares method, RANSAC incorporates the concept of excluding unqualified data points, making it more effective and accurate in identifying results for data sets with some anomalies. The algorithm was first proposed by Fischler and Bolles in 1981 to solve the localization problem (LDP) in image processing.

[0044] The core idea of ​​RANSAC is to randomly select the minimum number of data points required to estimate the model parameters from a data set containing outliers. Then, the algorithm verifies how many data points fit the model. This process is repeated many times, and each iteration may generate a different model and corresponding set of inliers (data points that fit the model). Finally, the model with the largest number of inliers is selected as the best model.

[0045] RANSAC operates under several key assumptions: 1) Inliers: The dataset consists of inliers that can be described by the data relationship model; 2) Outliers: The dataset also contains outliers that do not fit the data relationship model, which may be caused by measurement errors, incorrect assumptions, noise, etc.; 3) Proportion: The number of inliers is greater than the number of outliers.

[0046] The main goal of RANSAC is to estimate the parameters of the data relationship model from a dataset with outliers. This is achieved by repeatedly random sampling and statistical verification to identify and select the best data relationship model. Figure 2 The iterative process of the RANSAC (Random Sample Consensus) algorithm, which is used to estimate model parameters from data containing outliers, is presented. The algorithm first randomly samples data points and calculates the initial model parameters. This initial model is then checked against the data to assess its accuracy. If the loss function that measures the model fit reaches a minimum, the inliers (data points that fit the model well) are updated. If not, the algorithm resamples the data and recalculates the model parameters. This process is repeated until the maximum number of iterations is reached or the best model parameters are determined. The final step involves calculating the model parameters using the optimal set of inliers, ensuring that the model accurately represents the underlying data while excluding outliers. RANSAC is a robust method for estimating model parameters from datasets with outliers. By properly tuning its parameters, RANSAC can be very effective in a variety of applications, providing reliable and accurate model estimates even in the presence of significant noise and anomalies.

[0047] After filling the missing values, the present invention applies the RANSAC algorithm to fit the data relationship model. RANSAC is particularly effective in estimating model parameters, even if the data contains outliers. The algorithm iteratively selects random samples from the data set, fits these samples, and calculates the residual, which represents the difference between the observed data point and the model prediction value. The specific process of fitting the data relationship model is as follows:

[0048] Model assumptions: For transformer gas data processing tasks, the model is usually assumed to be a linear relationship, as follows:

[0049] y=ax+b (1);

[0050] Among them, a is the slope, b is the intercept, and the goal of the RANSAC algorithm is to maximize the number of inliers. The definition of inliers satisfies the following conditions:

[0051] |y i -(ax i +b)|<t (2);

[0052] Among them, (x i ,y i ) is a data point, t is the distance threshold used to determine whether the data point conforms to the model. The threshold t reflects the noise level of the data and the expected proportion of outliers in the data. Setting a too high threshold may cause outliers to be misclassified as inliers, while setting a too low threshold may mistakenly exclude true inliers.

[0053] Residual calculation: RANSAC evaluates the effectiveness of the model through residual (the difference between the observed value and the predicted value). The residual calculation formula is as follows:

[0054] e i =|y i -(ax i +b)| (3);

[0055] Step 5: Use the residual as the input of the improved multi-scale forest isolation algorithm IForest. Use the improved multi-scale forest isolation algorithm IForest to randomly sample subsets of different sizes of the sensor original data set, build multiple isolation forests with different sample sizes, and generate isolated trees at different scales in each isolation forest; calculate the anomaly scores of the data points in each isolation tree at different scales, and average the anomaly scores at different scales to obtain the final anomaly score.

[0056] The Isolation Forest Algorithm is an unsupervised machine learning method specifically used for anomaly detection. It constructs multiple decision trees (i.e., isolation trees) and uses these trees to isolate data points and thus identify outliers. The core idea of ​​the algorithm is that abnormal data points are more likely to be isolated or quarantined than normal data points. The Isolation Forest Algorithm is highly efficient and is particularly suitable for processing high-dimensional and large-scale data.

[0057] The present invention uses the residuals calculated from the RANSAC fit as input to the improved IForest algorithm, which allows the algorithm to detect anomalies at different scales and provide a more comprehensive analysis of potential outliers. This step is critical for identifying data points that do not conform to expected patterns and may represent unreliable data or errors.

[0058] In the present invention, the isolation forest is a set of multiple isolated trees constructed by sampling a data set D (sensor original data set) multiple times, each isolated tree is constructed based on a subset of D, and the subset is generated by random sampling.

[0059] The key steps of the present invention for obtaining anomaly scores using the improved isolation forest algorithm are as follows:

[0060] 1) Build an isolation tree: Randomly select subsets from the data set and build an isolation tree for each subset. Each tree recursively divides the data by randomly selecting features and split points. This process can be described mathematically as follows:

[0061] a) Path length h(x): the number of edges traversed from the root node to the data point x in the isolation tree.

[0062] b) Average path length E(h(x)): For a dataset D, the average path length of a data point x through all the trees in the forest.

[0063] 2) Calculate the anomaly score: Calculate the anomaly score s(x,n) based on the average path length E(h(x)) of the data point x. The calculation formula for this score is:

[0064]

[0065] Where c(n) is a constant representing the average path length of an isolated tree, which is approximately:

[0066]

[0067] Wherein, H(i) is the i-th harmonic number, which is approximately H(i)=ln(i)+0.5772156649.

[0068] 3) Identify anomalies: Data points with shorter average path lengths (higher anomaly scores) are more likely to be anomalies. This process involves calculating the anomaly scores for all data points and identifying the data points with the highest scores as outliers.

[0069] 4) Abnormality processing: According to the abnormality score, identify and process abnormal gas data points, and ensure the reliability of data by repairing or removing abnormal gas data points.

[0070] The traditional isolation forest algorithm usually uses a single scale (fixed subset size) when constructing isolated trees. Although it is very efficient, it may not fully capture anomalies distributed at different scales. The improved isolation forest algorithm improves on this and constructs multiple isolation forests at different scales (different subset sizes) to capture anomalies at different scales. The method includes: randomly sampling at different subset sizes to construct multiple isolation forests; generating isolated trees at different scales in each forest; calculating the anomaly scores of each data point at different scales, and averaging these scores to obtain the final anomaly score. Table I shows the comparison of the two methods.

[0071] Table 1 Comparison of the characteristics of the traditional isolation forest algorithm and the improved isolation forest algorithm in the present invention

[0072]

[0073] Step 6: Process the data points based on the final anomaly score.

[0074] The present invention also provides a transformer monitoring data processing system, comprising:

[0075] The data acquisition module is used to acquire the original data set of the sensor of the power transformer.

[0076] The model building module is used to iteratively select random samples from the sensor raw data set using the RANSAC algorithm, fit the data relationship model through the random samples, and calculate the residual of the data relationship model.

[0077] The anomaly score acquisition module is used to use the residual as the input of the improved multi-scale forest isolation algorithm IForest, use the improved multi-scale forest isolation algorithm IForest to randomly sample subsets of different sizes of the sensor original data set, construct multiple isolation forests of different sample sizes, and generate isolated trees at different scales in each isolation forest; calculate the anomaly scores of the data points in each isolation tree at different scales, and average the anomaly scores at different scales to obtain the final anomaly score.

[0078] Data cleaning module, used to process data points according to the final anomaly score.

[0079] The present invention also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the transformer monitoring data processing method.

[0080] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor to execute the transformer monitoring data processing method.

[0081] In order to verify the effectiveness and accuracy of the method proposed in this invention, a case study is used to verify the performance of the data preprocessing method. Partial data sets of the IECTC10 database and the Chinese electric power industry standard DL / T7222014 case are sorted out for research.

[0082] like Figure 3 As shown, the RANSAC algorithm is able to effectively fit a model of the data even in the presence of outliers. This robustness to outliers is a key advantage of RANSAC as it allows accurate determination of underlying data trends in the presence of noise and errors. The residuals obtained from the RANSAC fitting process are then used as input to an improved isolation forest algorithm to detect additional anomalies.

[0083] To further evaluate the effectiveness of the improved isolation forest algorithm, a comparative analysis was performed with the traditional isolation forest and K-means clustering methods. The comparison focuses on the precision and recall of anomaly detection. As shown in Table 2, the proposed method shows excellent performance in both precision and recall, highlighting its effectiveness in accurately identifying and isolating anomalous data points.

[0084] Table 2 Comparison of the results of untrustworthy data identification using the improved isolation forest algorithm of the present invention and other traditional methods

[0085] Model Accuracy Recall K-Means Clustering Method 80.00％ 74.07％ Traditional Isolation Forest Algorithm 84.62％ 81.48％ Improved Isolation Forest Algorithm of the Present Invention 85.19％ 85.19％

[0086] In the anomaly detection process, the theoretical values ​​predicted by the RANSAC model are used to correct the unreliable data, while the faulty data is retained for further analysis. Figure 4 As shown in Figure 2, the cleaned data results show a significant reduction in noise and outliers, resulting in a more refined dataset. This clean dataset improves the reliability and accuracy of subsequent analysis and decision-making processes for transformer condition monitoring.

[0087] In summary, the proposed method of combining RANSAC with the improved multi-scale isolation forest algorithm not only improves the accuracy of transformer data anomaly detection, but also provides a robust method for cleaning unreliable data while retaining valuable fault information. The results prove the practicality and superiority of this method in actual scenarios.

[0088] The present invention uses a method combining RANSAC and improved IForest algorithm to preprocess the data set. This method uses gas data as model input and significantly improves the accuracy and efficiency of anomaly detection. The superior performance of this algorithm in processing transformer data is demonstrated through comparative analysis with traditional data processing methods.

[0089] The embodiments described above are only preferred specific implementation modes of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belong to the protection scope of the present invention.

Claims

1. A transformer monitoring data processing method, characterized in that: The steps include: Get the raw data set of sensors of power transformers; Iteratively select random samples from the sensor raw data set using the RANSAC algorithm, fit the data relationship model through the random samples, and calculate the residual of the data relationship model; The residual is used as the input of an improved multi-scale forest isolation algorithm IForest, and the improved multi-scale forest isolation algorithm IForest is used to randomly sample subsets of different sizes of the sensor original data set to construct multiple isolation forests with different sample sizes, and isolated trees are generated at different scales in each isolation forest; Calculate the anomaly scores of the data points in each isolated tree at different scales, and average the anomaly scores at different scales to get the final anomaly score; Data points are processed based on the final anomaly score.

2. The transformer monitoring data processing method according to claim 1, characterized in that: The method of constructing a plurality of isolation forests with different sample sizes and generating isolated trees with different scales in each isolation forest is specifically as follows: recursively dividing the data by randomly selecting features and splitting points to obtain each isolated tree.

3. The transformer monitoring data processing method according to claim 1, characterized in that: The calculation of the anomaly scores of the data points in each isolated tree at different scales includes the following steps: Obtain multiple path lengths by traversing the number of edges from the data point to the root node in the isolation tree; Get the average path length by passing the data point through all the trees in the isolation forest; The anomaly score is calculated based on the average path length of the data points.

4. The transformer monitoring data processing method according to claim 1, characterized in that: Before using the RANSAC algorithm to iteratively select random samples from the sensor raw data set, the method also includes deleting redundant data in the sensor raw data set, specifically: checking whether multiple entries of data in the sensor raw data set are the same at any given time. If the data are exactly the same, they are considered to be redundant data and deleted from the data set.

5. The transformer monitoring data processing method according to claim 1, characterized in that: Before using the RANSAC algorithm to iteratively select random samples from the sensor original data set and after deleting redundant data, the method also includes processing missing values ​​in the sensor original data set, specifically: checking the missing data in the sensor original data set and filling the missing values ​​using a forward filling method.

6. A transformer monitoring data processing system, characterized in that: include: The data acquisition module is used to acquire the original data set of the sensor of the power transformer. The model building module is used to iteratively select random samples from the sensor raw data set using the RANSAC algorithm, fit the data relationship model through the random samples, and calculate the residual of the data relationship model. The anomaly score acquisition module is used to use the residual as the input of the improved multi-scale forest isolation algorithm IForest, use the improved multi-scale forest isolation algorithm IForest to randomly sample subsets of different sizes of the sensor original data set, build multiple isolation forests of different sample sizes, and generate isolated trees at different scales in each isolation forest; The anomaly scores of the data points in each isolated tree at different scales are calculated and the anomaly scores at different scales are averaged to get the final anomaly score. The data cleaning module is used to process the data points according to the final anomaly score.

7. A computer device, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the transformer monitoring data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the transformer monitoring data processing method according to any one of claims 1 to 5.

Images