A time series-based data cleaning method and related apparatus
By combining improved wavelet transform and variational mode decomposition with stacked noise reduction autoencoder, the problem of noise and missing values in data cleaning in power equipment condition monitoring is solved, achieving efficient and automated data cleaning and improving data quality and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN NARI LIABILITY OF STATE GRID ELECTRIC POWER RES INST
- Filing Date
- 2023-10-23
- Publication Date
- 2026-07-21
AI Technical Summary
Existing power equipment condition monitoring methods suffer from insufficient noise processing and incomplete handling of missing values during data cleaning, and require manual threshold setting, resulting in low data quality and utilization, as well as high computational complexity.
An improved wavelet transform and an improved variational mode decomposition method are combined with a stacked denoising autoencoder to perform frame processing on the signal, including denoising, abnormal frame detection, and missing value filling, thus automating the data cleaning process.
It improves the signal-to-noise ratio and robustness of the data, reduces computational complexity, achieves automated and efficient data cleaning, preserves the detailed features of the signal, and enhances data consistency.
Smart Images
Figure CN117370734B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of power equipment condition monitoring, and specifically relates to a data cleaning method and related apparatus based on time series. Background Technology
[0002] In the field of power equipment condition monitoring, due to the characteristics of power equipment condition information such as multi-source nature, information heterogeneity, and data redundancy, the ultra-high frequency (UHF) partial discharge data of gas-insulated switchgear (GIS) is often noisy and incomplete. Therefore, the raw monitoring parameter data cannot be directly used for diagnosis and should be cleaned first. Data cleaning is a key step in the preprocessing of power equipment condition assessment data, which helps to improve data quality, data utilization, and processing efficiency.
[0003] Existing data cleaning methods mainly include the following: using rules based on phase-to-phase voltage drop and time difference, applying density-based clustering and outlier removal principles to denoise transformer operation data; using an online data cleaning framework and methods that integrate similar user clustering and singular value thresholding theory to perform online cleaning of power marketing big data; and applying clustering to simplify parameter correlations and using the simplified data to train a time cyclic network model to achieve data cleaning. These methods are based on statistical clustering, requiring manual setting of thresholds and manual selection of parameters, which brings great inconvenience and subjectivity. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this application aims to propose a time-series-based data cleaning method and related apparatus. This method improves data quality through noise reduction, abnormal frame detection, and missing value imputation, thereby completing the data cleaning of GIS partial discharge UHF signals. This method boasts advantages such as high robustness, no need for manual operation, and low computational complexity, making it highly valuable for engineering applications.
[0005] To achieve this objective, the present invention provides a time-series-based data cleaning system, comprising an input module, a cleaning module, and an output module. The input module is used to input the data to be cleaned. The cleaning module cleans the data output by the input module based on a hybrid method of improved wavelet transform and improved variational mode decomposition to obtain cleaned data, and transmits the cleaned data to the output module, which outputs the cleaned data.
[0006] A time series-based data cleaning method includes the following steps: inputting data to be cleaned, cleaning the data based on a hybrid method of improved wavelet transform and improved variational mode decomposition, and outputting the cleaned data.
[0007] An electronic device includes a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform a time-series-based data cleaning method.
[0008] A computer storage medium storing a computer program that, when executed by a processor, implements the steps of a time-series-based data cleaning method.
[0009] The beneficial effects of this invention are as follows:
[0010] 1. The method of hybrid denoising using improved wavelet transform and improved variational mode decomposition has the following advantages compared with traditional linear filtering methods:
[0011] (1) By using variational mode decomposition, the signal can be denoised within the decomposed frequency band, which can remove noise in each frequency band more precisely, and the signal-to-noise ratio of the denoised signal is higher.
[0012] (2) Due to the multi-resolution characteristics of wavelet denoising, wavelet denoising methods can better characterize the details of the signal, which is beneficial to preserving the details of the signal.
[0013] (3) The diversity of wavelet bases provides multiple options for wavelet denoising, and the most suitable wavelet base can be selected according to the signal characteristics.
[0014] 2. By manipulating frames during the denoising process, rather than directly manipulating the signal, more details of the data can be preserved.
[0015] 3. In this patent, the stacked noise reduction autoencoder operates on each frame of the signal, rather than directly manipulating the signal. This gives the method higher robustness and accuracy.
[0016] 4. In the process of abnormal frame removal and missing point filling, frames are removed and missing points are filled with frames with higher confidence rates from the existing frames, so that the data cleaned by this method is more consistent.
[0017] 5. This method improves the threshold of wavelet signals, taking into account the signal-to-noise ratio and noise variance, and can remove noise more effectively.
[0018] 6. This method improves the variational mode decomposition process. By performing variational mode decomposition at different scales, more details can be preserved during the denoising process.
[0019] 7. This method has low computational complexity and is simple and fast to operate.
[0020] 8. This method is an adaptive method and does not require manual parameter setting. Attached Figure Description
[0021] Figure 1 This is a system structure diagram of the present invention;
[0022] Figure 2 This is a flowchart of the method of the present invention;
[0023] Figure 3 A flowchart illustrating the data denoising process of the hybrid method of wavelet transform and variational mode decomposition provided in this application embodiment;
[0024] Figure 4 A structural diagram of a stacked noise reduction autoencoder provided in an embodiment of this application;
[0025] Figure 5 A flowchart illustrating the specific implementation method of the stacked noise reduction autoencoder model network for abnormal frame detection and missing point filling of the signal in this application embodiment;
[0026] Figure 6 This is the interactive interface of the GIS partial discharge UHF signal data cleaning system provided in the embodiments of this application;
[0027] Among them, 1-input module, 2-cleaning module, 3-output module, 21-calculation module, 22-viewing module and 23-feedback module. Detailed Implementation
[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:
[0029] like Figure 1 As shown, a time series-based data cleaning system includes an input module 1, a cleaning module 2, and an output module 3. The input module 1 is used to input the data to be cleaned. The cleaning module 2 cleans the data to be cleaned output by the input module 1 based on a hybrid method of improved wavelet transform and improved variational mode decomposition to obtain cleaned data, and transmits the cleaned data to the output module 3. The output module 3 is used to output the cleaned data.
[0030] In the above technical solution, the cleaning module 2 further includes a calculation module 21, a viewing module 22, and a feedback module 23. The calculation module 21 is used to perform cleaning calculations on the data to be cleaned according to the steps. The viewing module 22 is used to view the results of the cleaning calculations on the data to be cleaned in real time. The feedback module 23 is used to provide feedback on the running log information of the cleaning calculations on the data to be cleaned.
[0031] In the above technical solutions, such as Figure 2 As shown, the cleaning calculation steps of the calculation module 21 are as follows:
[0032] Step 1: Obtain the data to be cleaned input by the input module 1;
[0033] Step 2: Perform data preprocessing on the data to be cleaned to obtain preprocessed data;
[0034] Step 3: Divide the preprocessed data into frames to obtain framed signals;
[0035] Step 4: Based on the hybrid method of improved wavelet transform and improved variational mode decomposition, perform data denoising on each frame of the framed signal obtained in Step 3 to obtain the denoised framed signal.
[0036] Step 5: Based on, for example Figure 4 The stacked denoising autoencoder shown performs time-series abnormal frame detection and missing point filling on a frame-by-frame basis on the denoised framed signal.
[0037] In the above technical solution, the data preprocessing in step 2 of the calculation module 21 uses cubic spline interpolation to fill in the missing values.
[0038] In the above technical solution, the user interface of the time-series-based data cleaning system is as follows: Figure 6 As shown.
[0039] In the above technical solution, the specific steps of the frame division method in step 3 of the calculation module 21 are as follows:
[0040] Step 31: Perform a Fourier transform on the preprocessed data to extract the peak frequency f of the preprocessed data. p ;
[0041] Step 32, calculate the frame duration using the formula:
[0042]
[0043] In the formula, T is the frame duration, f p Where A is the peak frequency and A is the signal window constant;
[0044] Step 33: Divide the preprocessed data into short-time signals of duration T to obtain framed signals;
[0045] Step 34: Window the frame signal of each frame;
[0046] Preferably, A is 100, and the signal window is a rectangular window.
[0047] The purpose of framing is to reduce computational complexity and increase the robustness of the algorithm by extracting features from each frame, since the original signal data is too large to process directly and the signal is unstable as a whole but stable locally.
[0048] In the above technical solutions, such as Figure 3 As shown, the data denoising step in step 4 of the calculation module 21 is as follows:
[0049] Step 41: Perform variational mode decomposition on the framed signal to obtain the first layered data. The number of decomposition layers is n, preferably 3 layers.
[0050] Step 42: Retain the first layer of data in the first layer of data, divide the remaining data in the first layer of data into two identical parts in time, perform variational mode decomposition on the two parts of data in the first layer of data respectively, and obtain two parts of second layer of data. The number of decomposition layers is m, and m is preferably 3 layers.
[0051] Step 43: Decompose the two sets of second-layer data obtained in step 42. The decomposition method is to retain the first layer data of the second-layer data, divide the remaining data of the second-layer data into two identical sets of data in time, and perform variational mode decomposition on the two sets of data of the second-layer data. The number of decomposition layers is l, preferably l is 3 layers, to obtain multiple sub-signals. In this way, a total of 15 sub-signals can be obtained.
[0052] Step 44: Perform multi-resolution analysis on the multiple sub-signals obtained in step 43 using wavelet decomposition, select wavelet basis functions and determine the required number of decomposition layers k, k is preferably 20, and perform wavelet decomposition on the multiple sub-signals to obtain a wavelet coefficient sequence composed of multiple wavelet sub-bands.
[0053] Step 45: For the multiple wavelet subbands, calculate their energy and signal-to-noise ratio, and selectively perform threshold denoising based on the thresholds of their energy and signal-to-noise ratio to obtain the denoised wavelet coefficient sequence.
[0054] Step 46: Perform inverse wavelet transform on the denoised wavelet coefficient sequence to reconstruct the reconstructed sub-signal;
[0055] Step 47: Add the reconstructed sub-signals together to obtain the denoised signal.
[0056] This invention improves the threshold calculation method of wavelet transform by incorporating the influence of signal-to-noise ratio during the threshold calculation process, enabling wavelet denoising to retain the original signal while better removing noise.
[0057] In the above technical solution, the wavelet basis function selected in step 44 of the calculation module 21 is the db series wavelet basis;
[0058] In step 45 of the calculation module 21, the thresholds for energy and signal-to-noise ratio are quantified using the following formula:
[0059]
[0060] In the formula: The threshold is represented by j; the number of decomposition layers is represented by j; and the length of the frame is represented by N. S represents the variance of the noise estimate. j This represents the signal-to-noise ratio of the j-th layer.
[0061] In the above technical solution, the stacked noise reduction autoencoder model network in step 5 of the computing module 21 is trained and learned using the gradient algorithm principle for its key parameters;
[0062] The specific implementation of the gradient algorithm principle is as follows: First, determine the weight parameters and network parameters of the first layer of the model. Then, based on the weight parameters and network parameters of the first layer, train and learn the parameters of the second layer of the model. Finally, iterate and loop until the parameters of the last layer of the model are determined.
[0063] The model network of the stacked noise reduction autoencoder is pre-trained using greedy layer-by-layer unsupervised training before model training. The model network uses a t-layer autoencoder, where t is preferably 3. During model training, a fine-tuning supervised method is used.
[0064] like Figure 5 As shown, the specific implementation method for abnormal frame detection and missing point filling in step 5 of the calculation module 21 is as follows:
[0065] Step 51: Classify the denoised frame signals obtained in step 4, select M frame signals as training frames, and use the remaining frame signals as test frames to train the stacked denoising autoencoder model network.
[0066] Step 52: The test frames are filtered through the stacked denoising autoencoder model network. Frames with abnormal data or missing data are removed from the dataset as abnormal frames, and the confidence of the removed test frames is obtained.
[0067] Step 53: Replace the abnormal frame with the test frame with the highest confidence level.
[0068] The method for obtaining the M frame signals is to manually or otherwise select data without data anomalies or missing data as training frames.
[0069] Preferably, M is 1000.
[0070] A time-series-based data cleaning method includes the following steps:
[0071] Input the data to be cleaned, clean the data based on a hybrid method of improved wavelet transform and improved variational mode decomposition, and output the cleaned data.
[0072] This invention improves data quality by employing denoising, abnormal frame detection, and missing value imputation to complete the data cleaning of GIS partial discharge UHF signals. First, the data is preprocessed; then, the signal is segmented into frames; denoising is performed using improved wavelet transform and improved variational mode decomposition; for online monitoring time series data, after segmenting the signal, a stacked denoising autoencoder is used to detect abnormal frames and imput missing values, thus completing the data cleaning process.
[0073] This invention improves the decomposition process of variational mode decomposition by drawing on the multi-scale features of wavelet transform and performing variational mode decomposition on the signal at multiple scales in each frame, thereby achieving a higher signal-to-noise ratio after denoising.
[0074] An electronic device includes a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform a time-series-based data cleaning method.
[0075] A computer storage medium storing a computer program that, when executed by a processor, implements the steps of a time-series-based data cleaning method.
[0076] The contents not described in detail in this specification are prior art known to those skilled in the art. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0077] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.
Claims
1. A data cleaning method based on time series, characterized in that: Includes the following steps: Step 1: Input the data to be cleaned, which is the UHF signal of partial discharge in GIS; Step 2: Perform data preprocessing on the data to be cleaned to obtain preprocessed data; Step 3: Divide the preprocessed data into frames to obtain framed signals; Step 4: Denoise each frame of the framed signal obtained in Step 3 using a hybrid method based on improved wavelet transform and improved variational mode decomposition to obtain the denoised framed signal. Step 5: Based on the stacked denoising autoencoder, perform time-series abnormal frame detection and missing point filling on the denoised framed signal frame by frame; The data denoising steps in step 4 are as follows: Step 41: Perform variational mode decomposition on the framed signal to obtain the first layered data, with the number of decomposition layers being n; Step 42: Retain the first layer of data in the first layer of data, divide the remaining data in the first layer of data into two equal parts in time, and perform variational mode decomposition on the two parts of the first layer of data to obtain two parts of second layer of data, with the number of decomposition layers being m. Step 43: Decompose the two sets of second-layer data obtained in step 42. The decomposition method is to retain the first layer data of the second-layer data, divide the remaining data of the second-layer data into two identical sets of data in time, and perform variational mode decomposition on the two sets of data of the second-layer data. The number of decomposition layers is l, and multiple sub-signals are obtained. Step 44: Perform multi-resolution analysis on the multiple sub-signals obtained in step 43 using wavelet decomposition, select wavelet basis functions and determine the required number of decomposition layers k, and perform wavelet decomposition on the multiple sub-signals to obtain a wavelet coefficient sequence composed of multiple wavelet sub-bands. Step 45: For the multiple wavelet subbands, calculate their energy and signal-to-noise ratio, and selectively perform threshold denoising based on the signal-to-noise ratio threshold to obtain the denoised wavelet coefficient sequence. Step 46: Perform inverse wavelet transform on the denoised wavelet coefficient sequence to reconstruct the reconstructed sub-signal; Step 47: Add the reconstructed sub-signals to obtain the denoised signal; The signal-to-noise ratio threshold in step 45 is quantified using the following formula: In the formula: The threshold is represented by j; the number of decomposition layers is represented by j; and the length of the frame is represented by N. S represents the variance of the noise estimate. j This represents the signal-to-noise ratio of the j-th layer.
2. The data cleaning method based on time series according to claim 1, characterized in that: In step 2, the data preprocessing uses cubic spline interpolation to fill in the missing values.
3. The data cleaning method based on time series according to claim 1, characterized in that: The specific steps of the frame segmentation method in step 3 are as follows: Step 31: Perform Fourier transform on the preprocessed data to extract the peak frequency fp of the preprocessed data; Step 32, calculate the frame duration using the formula: In the formula, T is the frame duration, fp is the peak frequency, and A is the signal window constant; Step 33: Divide the preprocessed data into short-time signals of duration T to obtain framed signals; Step 34: Window the frame signal of each frame.
4. The time-series-based data cleaning method according to claim 3, characterized in that: In step 44, the wavelet basis function is selected from the db series wavelet basis.
5. The data cleaning method based on time series according to claim 1, characterized in that: In step 5, the model network of the stacked denoising autoencoder is trained and learned using the gradient algorithm principle for its key parameters. The gradient algorithm principle is as follows: determine the weight parameters and network parameters of the first layer of the model, and use the weight parameters and network parameters of the first layer as a basis to train and learn the parameters of the second layer of the model, and iterate in this way until the parameters of the last layer of the model are determined. The model network of the stacked denoising autoencoder is pre-trained using greedy layer-by-layer unsupervised training. The model network uses a t-layer autoencoder, and the model training process employs a supervised fine-tuning method. The steps for abnormal frame detection and missing point filling in step 5 are as follows: Step 51: Classify the denoised frame signals obtained in step 4, select M frame signals as training frames, and use the remaining frame signals as test frames. Use the training frames to train the stacked denoising autoencoder model network. The M frame signals are obtained by manually or by other means to select data without data anomalies or missing data as training frames. Step 52: The test frames are filtered through the stacked denoising autoencoder model network. Frames with abnormal data or missing data are removed from the dataset as abnormal frames, and the confidence of the removed test frames is obtained. Step 53: Replace the abnormal frame with the test frame with the highest confidence level.
6. A data cleaning system based on the method of claim 1, characterized in that: It includes an input module (1), a cleaning module (2) and an output module (3). The input module (1) is used to input the data to be cleaned. The cleaning module (2) cleans the data to be cleaned input by the input module (1) based on a hybrid method of improved wavelet transform and improved variational mode decomposition, obtains the cleaned data, and transmits the cleaned data to the output module (3). The output module (3) is used to output the cleaned data.
7. The data cleaning system according to claim 6, characterized in that: The cleaning module (2) also includes a calculation module (21), a viewing module (22) and a feedback module (23). The calculation module (21) is used to perform cleaning calculations on the data to be cleaned. The viewing module (22) is used to view the cleaning calculation results of the data to be cleaned in real time. The feedback module (23) is used to provide feedback on the cleaning calculation operation log information of the data to be cleaned.
8. An electronic device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform a time-series-based data cleaning method as described in claim 1.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in claim 1.