Missing abnormal data restoration method applied to electric power measurement system

Abstract

The invention discloses a missing abnormal data restoration method applied to an electric power measurement system, and the method comprises the following steps: 1, presetting a generation model based on a missing mask, and detecting missing abnormal data in the electric power measurement system through the generation model; step 2, carrying out normalization processing on the detected abnormal data to obtain data to be repaired; and step 3, reconstructing the to-be-repaired data to obtain image data, detecting the image data, dividing the image data into a plurality of grades, and repairing the image data of the plurality of grades step by step through the repair network to obtain repaired data. According to the invention, abnormal data detection is carried out through the preset generative model, and the intelligent and automatic level of detection is improved. The generation model can learn features from data distribution, abnormal data can be accurately identified, the possibility of human intervention and misjudgment is reduced, and meanwhile, the comprehensiveness and accuracy of restoration are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of anomaly detection, and in particular to a method for repairing missing abnormal data applied to an electric power measurement system. Background Art

[0002] The power measurement system is a system used to measure and monitor various power parameters in the power measurement system. It plays an important role in the power industry, and can obtain the operating status of the power grid and power equipment in real time, helping power companies to optimize power supply, reduce energy consumption, and improve power grid stability and security. The power measurement system can record various power parameter data for a long time for future analysis and mining. These data help power companies understand the operation trend of the power grid, conduct load forecasting, equipment maintenance, and plan new power facilities. However, due to terminal equipment failures, transmission channel interference or other interference factors, the existing sampling and communication systems inevitably have the problem of data loss. In the existing technology, these lost data cannot be detected well, nor can they be repaired well. Summary of the invention

[0003] The purpose of the present invention is to solve the problem of data loss in sampling and communication systems due to terminal equipment failure, transmission channel interference or other interference factors. It is necessary to overcome the defects of existing technologies that cannot detect these lost data well and cannot repair these data well, and provide a method for repairing missing abnormal data applied to power measurement systems.

[0004] The purpose of the present invention is achieved through the following technical solutions: A method for repairing missing abnormal data applied to an electric power measurement system comprises the following steps: Step 1, presetting a generation model based on a missing mask, and detecting missing abnormal data in a power measurement system through the generation model; Step 2: normalize the detected abnormal data to obtain the data to be repaired; Step 3, reconstructing the data to be repaired to obtain image data, detecting the image data, dividing the image data into several levels, and repairing the image data of several levels step by step through the repair network to obtain repaired data.

[0005] Preferably, the generation model based on the missing mask is a generation model of a binary missing mask, specifically: NMD = n γ Among them, X(x1, x2, ..., x n-1 , xn , ) represents normalized data, M(m1, m2, ..., m n-1 , m n ,) represents a binary missing mask, X`(x`1, x`2, ..., x` n-1 , x` n , ) indicates missing data, represents the Gaussian noise at the missing data location, γ∈(0,1) represents the data missing rate, NMD represents the number of missing data, and ⊙ represents the element scale multiplier.

[0006] As a preferred embodiment, the missing abnormal data repair method applied to the power measurement system further optimizes the generation model of the binary missing mask and defines the following variables: M=SS(γ,α,β) NMS=random([NMDα],[NMDβ]) Where SS(γ, α, β) represents the stratified sampling function, α∈(0, 1) represents the length distribution coefficient 1, β∈(0, 1) represents the length distribution coefficient 2, ALMS represents the average missing length, NMS represents the number of missing segments, and [] represents the rounding operator; NMD represents the number of missing data. An integer between NMDa and NMDβ is randomly generated as NMS. Then NMD is randomly divided into NMS segments. Then, M is also divided into NMS sub-segments according to the ratio of NMS segments. In each sub-segment of M, a missing segment is selected, and the bits in the sub-segment are set to 1, and the others are set to 0.

[0007] Preferably, in step 3, the method for reconstructing the data to be repaired to obtain image data is: constructing a data enhancement algorithm, reconstructing the one-dimensional data to be repaired into two-dimensional matrix data through the data enhancement algorithm, and treating the two-dimensional matrix data as image data.

[0008] Preferably, in step 3, the image data is divided into three levels, including slightly missing mask image data, moderately missing mask image data and severely missing mask image data.

[0009] Preferably, in step 3, the repair network is a repair network based on a context encoder.

[0010] Preferably, the restoration network of the context encoder only includes two convolutional layers for feature learning and two convolutional layers for image restoration, and does not include a platform layer, a fully connected layer, and a pooling layer. The filling mode is replaced from effective to the same, so that the height and width of the output tensor of each layer are equal to the input image. The input of the context encoder is image data, the missing mask is the filter of the output layer, and the output is a restored image.

[0011] Preferably, the repair network comprises three, corresponding to three levels of image data, the first repair network repairs slightly missing mask image data, the second repair network repairs moderately missing mask image data, and the third repair network repairs severely missing mask image data.

[0012] Preferably, the input of the second repair network is the output of the first repair network, the input of the third repair network is the output of the second repair network, the second repair network retains the data repaired under the mild deletion mask, and the third repair network retains the data repaired under the mild deletion mask and the data repaired under the moderate deletion mask.

[0013] The beneficial effects of the present invention are as follows: the present invention performs abnormal data detection through a preset generative model, thereby improving the intelligence and automation level of detection. The generative model can learn features from the distribution of data, accurately identify abnormal data, and reduce the possibility of human intervention and misjudgment. The abnormal data is normalized to make the data more unified and standardized. This can reduce the heterogeneity of the data and improve the accuracy and efficiency of subsequent repairs. The data to be repaired is converted into image data for processing, and the spatial relationship between the data can be better captured by using image processing technology. Electric power data usually has complex multidimensional features, and visualization can more intuitively display these features, and mature image repair technology can be applied to improve the data repair effect. The image data is processed in a hierarchical manner and repaired step by step through the repair network. This method can process abnormal data more finely, optimize step by step from simple to complex, avoid local errors caused by one-time repair, and improve the comprehensiveness and accuracy of the repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart according to an embodiment of the present application; Figure 2 is a schematic diagram of a fragment generated by a missing mask according to an embodiment of the present application; Figure 3 This is a schematic diagram of test results when n=200 according to an embodiment of the present application; Figure 4 is a schematic diagram of the impact of ALMS according to an embodiment of the present application; Figure 5 is a schematic diagram of a typical missing segment according to an embodiment of the present application; Figure 6 is a schematic diagram of another typical missing segment according to an embodiment of the present application; Figure 7 is a schematic diagram of visual analysis of an absolute difference sequence according to an embodiment of the present application; Figure 8 is a schematic diagram of the correlation between normal data and time according to an embodiment of the present application; Fig. 9 is a schematic diagram of data distribution in an ADS according to an embodiment of the present application; Fig.10 is a schematic diagram of normalized value distribution according to an embodiment of the present application; Fig.11 is a schematic diagram after the noise in the missing segment is replaced by 0 according to an embodiment of the present application; Fig.12 It is a schematic diagram of one quarter of 400*96 data points according to an embodiment of the present application; Fig.13 is a schematic diagram corresponding to cutting according to an embodiment of the present application; Fig.14 is a panoramic schematic diagram of original data according to an embodiment of the present application; Fig.15 This is a schematic diagram of a cutting width of 96 according to an embodiment of the present application; Fig.16 is a schematic diagram of square cutting according to an embodiment of the present application; Fig.17 is a schematic diagram of a matrix according to an embodiment of the present application; Fig.18 is a schematic diagram of repairing a network according to an embodiment of the present application; Fig.19 is a schematic diagram of three repair networks shared according to an embodiment of the present application; Fig. 20 is a schematic diagram of a repair network improved according to an embodiment of the present application; Fig.21 It is a schematic diagram of a test data set after restoration according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0016] Example: A method for repairing missing abnormal data applied to power measurement systems, such as Figure 1 As shown, the following steps are included: Step 1, presetting a generation model based on a missing mask, and detecting missing abnormal data in a power measurement system through the generation model; Step 2: normalize the detected abnormal data to obtain the data to be repaired; Step 3, reconstructing the data to be repaired to obtain image data, detecting the image data, dividing the image data into several levels, and repairing the image data of several levels step by step through the repair network to obtain repaired data.

[0017] Data loss caused by sampling and communication equipment failures mainly manifests as discrete missing points, continuous missing segments, or a combination of the two. When the sampling end contacts are loose or other interference occurs, discrete missing points may appear in the data waveform. In various links of transmission, data packet loss due to temporary communication failures will lead to continuous missing segments. Obviously, the situation of continuous missing segments is much more serious than that of discrete shortcomings.

[0018] In order to describe the actual situation when data is lost, this application first models the data loss process and designs a generation model for binary missing masks, specifically: NMD = nγ Among them, X(x1, x2, ..., x n-1 , x n , ) represents normalized data, M(m1, m2, ..., m n-1 , m n ,) represents a binary missing mask, X`(x`1, x`2, ..., x` n-1 , x` n , ) indicates missing data, represents the Gaussian noise at the missing data location, γ∈(0,1) represents the data missing rate, NMD represents the number of missing data, and ⊙ represents the element scale multiplier.

[0019] This application treats the missing data as the normal data minus the element-wise product of the binary missing mask and the normal data, and then adds the product of the mask and Gaussian noise. The data at each position of the binary missing mask is a binary code, where 1 indicates that the data at that position is missing and 0 indicates normal. In order to unify the two types of missing, discrete missing points are treated as special continuous missing segments of length 1.

[0020] At a given missing rate, if NMD missing points are randomly selected directly from the n bits of M, the probability of discrete missing will be much greater than the probability of continuous missing; if the present application first cuts the NMD missing points into several continuous fragments and then randomly places these fragments in M, collisions may occur, thereby reducing the actual missing rate γ.

[0021] To address the above issues, this application further proposes a missing mask generation model based on the layered sampling principle, and defines the following variables: M=SS(γ,α,β) NMS=random([NMDα],[NMDβ]) Where SS(γ, α, β) represents the stratified sampling function, α∈(0, 1) represents the length distribution coefficient 1, β∈(0, 1) represents the length distribution coefficient 2, ALMS represents the average missing length, NMS represents the number of missing segments, and [] represents the rounding operator; NMD represents the number of missing data. An integer between NMDa and NMDβ is randomly generated as NMS. Then NMD is randomly divided into NMS segments. Then M is also divided into NMS sub-segments according to the ratio of NMS segments. In each sub-segment of M, a missing segment is selected, and the bits in the sub-segment are set to 1 and the others are set to 0. This segment ensures that the distribution of γ and the missing segment is independent. The segment generated by the missing mask is as follows Figure 2 shown.

[0022] When n=200, the test results are as follows Figure 3 To show the effect of ALMS, this is also 2 / (α+β) in Figure 4 As shown in , the present application uses an additional variable k to control ALMS. It can be seen from the present application that: (1) the missing rate directly affects the total number of missing data in NMD; (2) the smaller (larger) the sum of α and β, the larger (smaller) the total number of missing segments NMS, and the longer (shorter) the average length of the missing segments ALMS, that is, the continuous (discrete) missing probability is better than the discrete (continuous) missing probability; (3) when the sum of α and β remains unchanged, the closer (farther) the distance between α and β, the smaller (larger) the variance of the missing segment length, that is, the more uniform (uneven) the distribution of the missing segment length. It can be seen from the results that the generation of the missing mask can be adjusted very flexibly. The overall missing rate of the measured data of the power measurement system is relatively low, of which continuous missing accounts for the main proportion. Therefore, continuous missing is also a type of missing that needs to be paid attention to in this application.

[0023] This application designs a missing data detection algorithm based on the characteristics of missing data. In actual work, the measurement value of the missing segment is usually not zero, but the noise is distributed around zero. This noise includes both normal noise of the sampling device and the transmission channel, and noise caused by transient faults, which are the cause of data loss. The signal-to-noise ratio between the normal data signal and the noise signal is defined in this application.

[0024] In order to simplify the model, the present application can reasonably assume that the noise of the missing segment follows a Gaussian distribution with a mean of zero and a variance smaller than that of normal data. The missing mask obtained in the present application can replace the data of the missing segment with Gaussian noise, whose mean is zero, and the variance, that is, the power of the noise signal, is significantly smaller than the power of the original signal. The present application temporarily sets it to 1% of the original signal power, without missing any data, that is, the SNR is 20dB.

[0025] In order to understand the noise characteristics of the missing segments, the present application first needs to detect or locate some missing segments for calculation, and the missing segments located at this stage must have high reliability, otherwise it will affect the subsequent detection performance.

[0026] The present application further observes the data set under the effect of the missing mask and finds that since the normal data is likely to be far away from zero, while the noise of the missing segment is distributed near zero, the curve will suddenly jump at the beginning or end of the missing segment. The present application calls these missing segments typical missing segments, which is also the case in most cases, such as Figure 5 and Figure 6 As shown. Conversely, the normal data before and after the missing segment are also likely to be distributed around zero, which results in an overlap between the ranges of the normal data and the missing data. In this case, the curve may have an unobvious jump at the beginning or end of the missing segment.

[0027] In addition to the sudden jump of the curve, another feature may not be so obvious. Under the condition of a small window, the normal data segment usually presents a regular curve and has a high linear correlation with time, while the missing data segment has a low linear correlation due to the noise characteristics. This may be an important factor when the application distinguishes missing data from normal data.

[0028] The definition of typical missing segments and atypical missing segments is qualitative rather than quantitative. It can be seen from the present application that the characteristics of typical missing segments are more obvious than the other and are easier to detect and locate. Therefore, the present application will detect some of them in advance as samples to calculate the noise of the missing segments.

[0029] For a typical missing segment, the present application can see that when the curve enters and leaves the missing segment, the variation between adjacent sampling points increases significantly. In order to visualize this phenomenon, the present application can perform a visual analysis by calculating the absolute difference sequence (ADS) defined in the following formula, such as Figure 7 shown.

[0030] ADS(X`)={a i =|x` i -x` i+1 |f;orx` i ∈X`,andi≤n-1} According to the previous analysis, another important feature is that the linear correlation coefficient between the missing segment and time is low, because the data in the missing segment is replaced by unknown noise, while the correlation between normal data and time is high under small windows, such as Figure 8 shown.

[0031] Therefore, the present application can find typical missing fragments by checking the sudden jumps and linear correlation with time in the absolute difference sequence. To do this, it is necessary to solve how to quantify the outliers in the absolute difference sequence (ADS). Take the actual load data of the power measurement system as an example. It is reasonable that the normal data in the difference sequence (DS) obeys the normal distribution.

[0032] After taking the absolute value of DS, the data distribution in ADS can be regarded as a semi-bell shape, such as Fig. 9 Therefore, if the application knows the mean μ and the standard deviation σ, the 3σ principle will enable the application to detect outliers, where values ​​greater than μ+3σ are likely to be outliers because of missing segments.

[0033] However, when the absolute difference sequence contains a large number of abnormally large values ​​due to sudden jumps in missing segments, the mean and standard deviation calculated directly on the sequence may be very different from the true mean and standard deviation of normal data in ADS, because the abnormal values ​​will contaminate the normal values. In order to avoid the contamination of data by ADS abnormal values ​​and improve the robustness of the algorithm, this application selects two other statistical robustness indicators in the following formula: median and mean-absolute deviation (MAD) as robust approximations of the true mean and standard deviation. MAD(X`)=median({|x`i-median(X`)|forx` i ∈x`}) μ`≈MAD(X`) σ`≈median(X`) Then the application marks the first point greater than μ′+kσ′ as a possible beginning of the missing part, and the second as a possible end, and then the application will check the linear correlation between n after this time, if n is less than n*a set of initial values ​​the application advances, this part will be labeled as a typical missing part, otherwise the application will abandon the first point and the second point mask the possible beginning of the missing part, find the third point greater than μ′+kσ′ as a possible end point, and then check the linear correlation again.

[0034] Repeating the previous algorithm flow, the present application will find all possible typical missing segments. It should be noted here that the linear correlation test requires at least two points. In order to ensure accuracy, the present application only tests line segments with a length greater than 5 points. More specifically, the present application only calculates the noise amplitude within segments greater than 5 points.

[0035] In order to evaluate the detector of this application, this application will refer to the confusion matrix in Table 1 and calculate the precision, recall and F1 score as shown in the following formula: Among them, TP represents true positive, FP represents false positive, TN represents true negative, FN represents false negative, P represents precision, R represents recall, and F1 represents F1 index.

[0036] Table 1 Confusion matrix in detection Missing Not missing Detected as missing TP FP Detected as not missing FN TN After detecting the missing mask on the original data, in order to improve the restoration effect, all data need to be normalized. This application considers normalizing the value to (0, 1).

[0037] Due to the contamination of missing segment noise, this application may not be able to calculate the true maximum and minimum values. This will result in significant gaps in the normalized value distribution, such as Fig.10 As shown in , the data values ​​cannot completely fill this normalized range. To avoid this problem, this application only calculates the maximum and minimum values ​​on the remaining normal data based on the detection results of , and first normalizes the remaining normal data to (0.01, 1). Then manually replace all noise in the missing segment with 0. The result is shown in Fig.11 As shown. There are no gaps in the distribution of values ​​anymore. A potential advantage of doing this is that after normalization, a value of 0 will only represent missing data and vice versa. Therefore, the following repair algorithm can easily know which are the missing data, just by reading the data values ​​as 0, without knowing the missing mask.

[0038] It should be noted that the local maximum and minimum values ​​used for normalization here may not be the global maximum and minimum values ​​of the normal data before the loss occurs. Because the true maximum and minimum values ​​may be blocked by the missing mask. Therefore, when this application normalizes those blocked data, the result may be outside the range (0, 1), such as Fig.11 However, since the data discussed in this application are load data with obvious periodicity, the local maximum and minimum values ​​will be very close to the global maximum and minimum values.

[0039] Abnormal data detection result verification: Since padding has no effect on the missing detection problem, this application will directly detect the missing mask on the load data. Here, this application considers two main factors in the detection process, namely the signal-to-noise ratio and the missed detection rate γ. This application tests the detection algorithm of this application on normalized data of (-500, 0), (-250, 250) and (0, 500), with a missing rate of 10% to 40% and a signal-to-noise ratio of 15dB to 40dB.

[0040] For the data normalized to the positive range (0, 500) and the negative range (-500, 0), when the signal-to-noise ratio is greater than 20dB, all precision, recall, and F1 are close to 100%, while for the data normalized to (-250, 250), if the same result is to be obtained, the signal-to-noise ratio should be greater than 30dB. This is because in this case, the noise on the missing segment will have a very high probability of overlapping with the normal data. Then, the application may not be able to obtain enough typical missing segments for noise calculation, which directly affects the detection performance.

[0041] When the SNR decreases from 20dB to 15dB, the detection results deteriorate significantly. The F1 of the data normalized in (-500, 0) and (0, 500) can drop to 0.7. The situation is even worse for the data normalized in (-250, 250), where the F1 may be lower than 0.6. In most cases, the data is always positive or negative, far away from zero, and the noise of the missing data is small enough, which guarantees a high SNR.

[0042] Another phenomenon is that in high signal-to-noise ratio areas, the missed detection rate has no effect on detection, while in low signal-to-noise ratio areas, the higher the missed detection rate, the better the detection effect. Further analysis of this application shows that when the signal-to-noise ratio is low, even when the missed detection rate is 0%, the proportion of false positive samples will increase significantly due to the overlap of normal data with the noise on the missing segment. Therefore, the higher the missed detection rate and the more positive samples, the fewer false positive samples will be to a certain extent.

[0043] In the step 3, the method of reconstructing the data to be repaired to obtain image data is: constructing a data enhancement algorithm, reconstructing the one-dimensional data to be repaired into two-dimensional matrix data through the data enhancement algorithm, and treating the two-dimensional matrix data as image data.

[0044] The load data of the power measurement system will change with the changes in social operation and production methods. Therefore, it will show significant multiple periodicity in units of days, weeks, quarters and years.

[0045] When repairing this type of data, it is not enough to refer only to the continuity between the directly adjacent normal data and the missing data (such as the data half an hour ago and half an hour later), which will lead to poor performance. Instead, the periodicity should be taken into account, which means that the application must also refer to the data in the adjacent period (such as the data at the same time but in the previous or later days and weeks), because these data are indirectly adjacent to some extent and share very similar patterns.

[0046] In terms of semantic strength, direct adjacency is stronger than indirect adjacency, but the two can be combined as a reference. For the data at the edge of the missing segment, due to the limitation of waveform continuity, the direct adjacent data is almost the most important reference for repair; but for the missing data in the missing segment, there is no direct adjacent data for reference, and the indirect adjacent data becomes a very important reference. However, traditional mathematical estimation can only perceive the data within a very limited range around the missing data, so it is impossible to make full use of the information of the indirect adjacent data. Therefore, the repair accuracy of the missing segment is low, especially for the long-term continuous missing segment. How to adaptively adjust the reliability ratio of direct adjacent data and indirect adjacent data is the key to improving the repair performance.

[0047] The one-dimensional structure of the measured data determines that the direct neighbor relationship between the data is visible, while the indirect neighbor relationship is invisible. In order to observe this invisible feature, special processing methods such as Fourier transform are usually required. This method can analyze the periodic characteristics of the data, but it also sacrifices the perception of direct neighbor relationships, and the occurrence of missing is not necessarily periodic, so it is difficult to repair missing data at certain positions from the spectrum.

[0048] This application reshapes one-dimensional data into a two-dimensional matrix, fundamentally solving the problem that the indirect adjacency of data is hidden by the data structure.

[0049] When the one-dimensional data is periodically truncated and reshaped into a two-dimensional matrix, the original direct adjacency data is still direct adjacency data and is distributed separately in the rows, while the original indirect adjacency data also becomes direct adjacency data and is distributed along the columns. If the present application visualizes this matrix with a grayscale image, the direct adjacency between rows and columns will form some regular textures on this image. In other words, the two-dimensional structure of the matrix makes the direct adjacency and indirect adjacency of the original one-dimensional data compatible with each other, respectively, in rows and columns.

[0050] Taking the load data recorded every 15 minutes (96 points per day) from 2014 to 2020 as an example, when selecting the cutting width, on the one hand, it can be artificially set to an empirical value, such as taking an integer multiple of a day or week as the cutting width; on the other hand, the application can also perform Fourier analysis on the data and select a suitable cutting width based on the distribution of harmonic components. The application considers the data of the first 400 days, that is, 400*96 data points, of which one quarter of the waveforms are as follows Fig.12 As shown; then the application sets the cutting width to 96, 100, 192 and 200 respectively, and the corresponding images are as follows Fig.13 shown.

[0051] It can be seen that when the cutting width is not an integer multiple of 96 (i.e. one day), due to the misalignment of the columns, the image will have horizontal and diagonal stripes, and the image semantics will be seriously confused. Only when the cutting width is an integer multiple of 96, the image will have only horizontal stripes, and the image texture will be relatively regular and clear; because the attenuation of semantic information in this image is proportional to the cutting width, 96, i.e. one day, is selected as the cutting width to ensure that the image is as clear as possible.

[0052] Since direct adjacency is stronger than indirect adjacency, the horizontal semantic strength of the constructed image will also be stronger than the vertical image, while the semantic strength of the real image in all directions is almost the same. Compared with the real scene image, the structural image will have obvious horizontal stripes and slight vertical stripes, because the change of adjacent data in the horizontal direction is smoother than that in the vertical direction. But at the same time, they are all approximately regular. Therefore, the present application proposes that such an image constructed from one-dimensional data can be regarded as a "generalized" image.

[0053] Based on the above analysis, when the present application reconstructs the one-dimensional data into a two-dimensional image, the problem of repairing the missing one-dimensional data is converted into the problem of reinforcing the two-dimensional image. In the following discussion, the present application will combine the deep learning method to solve the latter.

[0054] Since the original data totals 2289 days, for the convenience of display, it is truncated every 5 days to obtain a 457×480 panorama, as shown in Fig.14 shown.

[0055] It can be seen that: (1) The grayscale change in the horizontal direction is significantly slower than that in the vertical direction.

[0056] (2) There is an obvious white sine curve on the right side of the image. Since the white in the gray image represents the maximum value, this sine curve will represent the peak load at night, which is also highly consistent with the daily sunset time of the year.

[0057] (3) When the cutting width is 96, Fig.15As shown, the horizontal black and white stripes are regularly distributed along the rows with equal spacing. If the user zooms in, the user can find that the height of the white stripes is 5 and the height of the black stripes is 2. Obviously, those white and black stripes are weekdays and weekends, respectively.

[0058] However, these features are difficult to observe directly in a one-dimensional structure. Therefore, using a two-dimensional image to represent the one-dimensional load data of the power measurement system more intuitively shows the characteristics of the data. In actual repair work, the present application may not have enough context data around the missing data, so it is difficult to generate a huge matrix of 2291*96; at the same time, when repairing the missing data at a certain position, so much context data is not needed. Because the greater the distance, the greater the attenuation of semantic association (the attenuation in the vertical direction is faster than the attenuation in the horizontal direction). Data that is too far away may not provide any useful reference information for the current repair problem, so what the present application actually does is to divide the huge matrix into several smaller sub-matrices through a slicing window.

[0059] In this vein, a possible choice for the slicing window is a square with a side length equal to the cutting width, such as Fig.16 shown.

[0060] It is also important to note that: (1) the missing mask will be truncated and reconstructed into a matrix, divided into sub-matrices with the same shape as the data, such as Fig.17 As shown; Since the grayscale in a computer is usually 8 bits (an integer between 0 and 255), it is likely to cause irreversible precision loss when converting the original data value to grayscale. Therefore, what this application does is to save the data in a matrix and use it in the subsequent calculations. Only when this application wants to visualize the matrix, this application converts them to grayscale. This is also the reason why this application calls these matrices "generalized" images.

[0061] In the step 3, the image data is divided into three levels, including slightly missing mask image data, moderately missing mask image data and severely missing mask image data.

[0062] In step 3, the repair network is a repair network based on a context encoder.

[0063] When this application uses a two-dimensional image to represent one-dimensional data, the problem of repairing one-dimensional data becomes an image repair problem. This application will design a new network to repair the missing data in the constructed image.

[0064] First, the first five convolutional layers of AlexNet are used as encoders to extract the real-life image features with missing areas, and then the six deconvolutional layers are used as decoders to repair the missing areas from the learned features.

[0065] However, during the training process, the network needs to learn how to identify which area is the missing area, and only output the repaired image of the missing area in the final output, while the output of the normal area should be zero. This requires the network to be able to encode the location information of the feature in the feature learning stage. The feature learning in AlexNet is designed for object classification problems. Since the category of the object is independent of the position, the position of the feature is not considered. Due to the use of maximum pooling, effective convolution, and flat layers, the height and width of the tensor will be compressed, making the position information blurred. Therefore, it is almost impossible to trace back to the original position of the object in the deep layer.

[0066] For images constructed from one-dimensional data, since the semantic information is not evenly distributed in rows and columns as mentioned above, the original position of the feature is particularly important when extracting features. If the convolution and deconvolution network framework is used directly, the ambiguous position information of deep objects will further aggravate the contradictions in the network. In addition, since the constructed "generalized" images are usually low-dimensional and low-rank, the performance of the context encoder will be poor.

[0067] In fact, through detection, the mask of image missing can be known in advance, and the present application can inform the network instead of requiring it to know.

[0068] The restoration network of the context encoder only includes two convolutional layers for feature learning and two convolutional layers for image restoration. It does not include a platform layer, a fully connected layer, and a pooling layer. The filling mode is replaced from valid to the same so that the height and width of the output tensor of each layer are equal to the input image. The input of the context encoder is image data, the missing mask is the filter of the output layer, and the output is a restored image.

[0069] Specifically, this application proposes a context encoder-based restoration network: only convolutional layers are used, no flattening layers and fully connected layers are used, no pooling layers are used, and the padding mode is replaced from "valid" to "same", so that the height and width of the output tensor of each layer are equal to the input "generalized" image, that is, 114*114. Fig.18 shown.

[0070] Without flattening and fully connected layers, this application uses only two convolutional layers for feature learning. Therefore, the learned features will be localized to their original positions, which means that the network will encode a feature vector for each point in the input "generalized" image and place the feature vector at the same position of the original point in the input "generalized" image. This can maximize the retention of position information during feature learning. Then, two more convolutional layers are used to replace the deconvolutional layers for image restoration.

[0071] Here, instead of using the missing mask as an additional input to the first layer of the network, the missing mask is used as a filter in the final output layer. Only the output of the missing region is retained, which is then added to the input image to form the inpainted image. Note that the image is still a filled patch, so the core region is taken out as the final output of the network.

[0072] The loss function is as follows: Among them, ε is a very small value used to constrain the curvature of the nonlinear region, C = w-wln (1 + w / ε) is a constant that can smoothly connect the linear and nonlinear parts, and w is used to limit the range of the nonlinear part to the interval [-w, wJ. When the absolute value of x is less than w, the ln (x) function is used, and its gradient is 1 / x. This part can enhance the impact of small errors. In order to avoid over-compensation of small errors and large updates in the wrong direction, it is necessary to limit the gradient when dealing with small losses. Two parameters are used to control the gradient, namely w and ε. When the absolute value of x is greater than w, large errors can be quickly repaired.

[0073] Since the application does not change the height and width of each layer of tensors, and encodes feature vectors in the depth or channel of the tensor, it is easy to trace back and perceive the location information of each feature vector even in deeper layers. Although the size of the kernel is relatively small, it does not mean that the perception range of the feature vector is within a small area. Since the data in the "generalized" image here is folded, a small area in the image actually represents very extensive data in the original one-dimensional structure. This is also the reason why the improved network can be used for "generalized" images.

[0074] The repair network includes three, corresponding to three levels of image data, the first repair network repairs slightly missing mask image data, the second repair network repairs moderately missing mask image data, and the third repair network repairs severely missing mask image data.

[0075] Because the missing masks have been detected in advance and divided into different levels, the application can transfer the idea of ​​level to the model design of the application. Here, the application uses three cascaded repair networks (repair networks) instead of only one repair network to repair the corresponding three missing levels, such as Fig.19 The first repair network will only repair those slightly missing data. Then the second repair network will only repair the moderately missing data based on the output of the first repair network, and so on, and the same goes for the third repair network.

[0076] The input of the second repair network is the output of the first repair network, the input of the third repair network is the output of the second repair network, the second repair network retains the data repaired under the mild missing mask, and the third repair network retains the data repaired under the mild missing mask and the data repaired under the moderate missing mask.

[0077] In the application, it is found that although the task of the first repair network is to repair mild deletions, it will also repair moderate and severe deletions in some way, but with lower accuracy, which may be due to the spatial adjacency between mild, moderate and severe deletion areas. Although the repair results for moderate and severe deletions are not very accurate, they can provide very useful estimates for the following two repair networks. Therefore, this application further adjusts the previous filtering strategy to retain these useful estimates instead of discarding them. The final improved repair network is as follows Fig. 20 shown.

[0078] For the first repair network, the present application uses mild, moderate and severe missing masks as filters acting on the output of the first repair network. All normal data of the input remains unchanged in the output, while the first repair network only updates the values ​​of the mild, moderate and severe missing areas. The second repair network will use the output of the first repair network as input and use the moderate and severe missing masks as filters for its output. Similarly, the normal data in the input will be retained, and the data repaired under the mild missing mask will be retained, while the data under the moderate and severe missing masks will be updated. Similarly, the third repair network uses the output of the second repair network as input and uses the heavy mask as a filter. Therefore, only the data under the heavy mask will be updated this time. Finally, the present application removes the padding area and only retains the core area as the final output of the repair network.

[0079] Verification of abnormal data repair results: The repair network model and the improved repair network model were evaluated when the filling depth was 0 to 11 and the corresponding filling rate was 0% to 23%.

[0080] During the training process, this application sets the batch number to 20, the initial learning rate to 0.01, and the initial learning rate decreases exponentially over time. This application will stop training until the loss function no longer decreases. The final results are shown in Table 2, where the mean absolute error (MAE) is used as a supplementary indicator.

[0081] Table 2 (0.01, 1) Repair RMSE (MAE) of normalized data Although the number of parameters in the repair network is the same as that in the improved repair network, the improved repair network model has stronger deductive ability and performs significantly better than the original repair network model. The filling depth can further reduce the repair error. In addition, as mentioned above, too deep a filling depth may lead to performance degradation. The best choice is 9, corresponding to a filling rate of about 20%.

[0082] In order to more intuitively demonstrate the repair effect of the repair network, this application randomly selects some results on the test data set and visualizes them, such as Fig.21 shown.

[0083] The core area of ​​the output image is almost identical to the core area of ​​the ground truth, even for inputs with high dropout rates and long dropout segments. You can't even tell the difference between the core of the output and the ground truth. There are 5 white and 2 dark stripes in the output, corresponding to 3 weekdays and weekends. Another important feature is the white sinusoidal texture on the right half of the core, which is composed of daily peak loads. Again, the application can clearly identify this special texture in the output. This means that the application's improved repair network model is able to learn the characteristics of the load data well and can accurately predict the values ​​of the missing data. Only by zooming in on the output image can the application notice the subtle difference in texture relative to the ground truth, which is a tolerable error.

[0084] In the filling area outside the core area, since these areas are ignored when defining the loss function in this application, obvious black areas appear, such as Fig.21 The circles shown. Therefore, during the training process, the improved repair network allows higher errors to be output in these filled areas, making it easier and more effective for the loss function to converge. If there is no filled area, these errors will appear in the core area, which is why this application designs the filling patch technology.

[0085] To better illustrate the training process, the application then visualizes the convolution kernels in the first layer of the improved inpainting network. There are obvious horizontal stripes in the convolution kernels, which is consistent with the characteristics of the "generalized" image. The parameters inside these stripes are much lower than the outside, and the stripe handles are usually darker than other areas. This is because the missing data is distributed within these stripes. Since the noise in the missing data is just zeros, the application will not reference these zeros when inpainting. Therefore, the corresponding parameters are small.

[0086] The above are merely embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structural transformations made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for repairing missing abnormal data applied to an electric power measurement system, characterized in that: The following steps are involved: Step 1, presetting a generation model based on a missing mask, and detecting missing abnormal data in a power measurement system through the generation model; Step 2: normalize the detected abnormal data to obtain the data to be repaired; Step 3, reconstructing the data to be repaired to obtain image data, detecting the image data, dividing the image data into several levels, and repairing the image data of several levels step by step through the repair network to obtain repaired data.

2. The method for repairing missing abnormal data applied to a power measurement system according to claim 1, characterized in that: The generation model based on missing mask is a generation model of binary missing mask, specifically: NMD = nγ Among them, X(x1,x2,…,x n-1 ,x n ,) represents normalized data, M(m1,m2,...,m n-1 ,m n ,) represents a binary missing mask, X`(x`1,x`2,...,x` n-1 ,x` n ,) indicates missing data, represents the Gaussian noise at the missing data location, γ∈(0,1) represents the data missing rate, NMD represents the number of missing data, and ⊙ represents the element scale multiplier.

3. The method for repairing missing abnormal data applied to a power measurement system according to claim 2 is characterized in that: The generation model of binary missing masks is also optimized, and the following variables are defined: M=SS(γ,α,β) NMS=random([NMD α ],[NMD β ]) Where SS(γ,α,β) represents the stratified sampling function, α∈(0,1) represents the length distribution coefficient 1, β∈(0,1) represents the length distribution coefficient 2, ALMS represents the average missing length, NMS represents the number of missing segments, and [] represents the rounding operator; NMD represents the number of missing data. An integer between NMDα and NMDβ is randomly generated as NMS. Then NMD is randomly divided into NMS segments. Then, M is also divided into NMS sub-segments according to the ratio of NMS segments. In each sub-segment of M, a missing segment is selected, and the bits in the sub-segment are set to 1, and the others are set to 0.

4. The method for repairing missing abnormal data applied to an electric power measurement system according to claim 1, characterized in that: In the step 3, the method of reconstructing the data to be repaired to obtain image data is: constructing a data enhancement algorithm, reconstructing the one-dimensional data to be repaired into two-dimensional matrix data through the data enhancement algorithm, and treating the two-dimensional matrix data as image data.

5. The method for repairing missing abnormal data applied to a power measurement system according to claim 4 is characterized in that: In the step 3, the image data is divided into three levels, including slightly missing mask image data, moderately missing mask image data and severely missing mask image data.

6. The method for repairing missing abnormal data applied to a power measurement system according to claim 5, characterized in that: In step 3, the repair network is a repair network based on a context encoder.

7. A method for repairing missing abnormal data applied to an electric power measurement system according to claim 6, characterized in that: The repair network of the context encoder only includes two convolutional layers for feature learning and two convolutional layers for image repair. It does not include a platform layer, a fully connected layer, and a pooling layer. The filling mode is replaced from valid to the same so that the height and width of the output tensor of each layer are equal to the input image. The input of the context encoder is image data, the missing mask is the filter of the output layer, and the output is a repaired image.

8. The method for repairing missing abnormal data applied to a power measurement system according to claim 7 is characterized in that: The repair network includes three, corresponding to three levels of image data, the first repair network repairs slightly missing mask image data, the second repair network repairs moderately missing mask image data, and the third repair network repairs severely missing mask image data.

9. The method for repairing missing abnormal data applied to a power measurement system according to claim 8, characterized in that: The input of the second repair network is the output of the first repair network, the input of the third repair network is the output of the second repair network, the second repair network retains the data repaired under the mild missing mask, and the third repair network retains the data repaired under the mild missing mask and the data repaired under the moderate missing mask.

Images