Data processing method and device, computer equipment, storage medium and program product
By using a data processing model trained with a generative adversarial network, local morphological features of power time-series data can be identified and missing values can be filled in, thus solving the problem of outliers and missing values in the power system and improving the accuracy and quality of data processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-18
- Publication Date
- 2026-03-27
AI Technical Summary
Time series data in power systems often contain outliers and missing values, which affect the accuracy of subsequent analysis. Existing methods struggle to capture nonlinear time series dependencies.
A generative adversarial network is used to train the data processing model. The discriminator identifies local morphological features and outliers in the power time series data. The generator fills in missing values and corrects them based on the outlier identification results.
It improves the accuracy of outlier identification and the reliability of missing value filling, enhances the quality of power data, and supports the efficient operation and maintenance of smart grids.
Smart Images
Figure CN121743937A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and program product. Background Technology
[0002] Sensors and smart devices deployed in power systems continuously generate massive amounts of time-series data, such as voltage, current, and power. This data forms the basis for condition monitoring, load forecasting, and fault diagnosis. However, due to sensor failures, transmission interference, or storage errors, power time-series data often contains outliers and missing values, directly affecting the accuracy of subsequent analyses.
[0003] Traditional anomaly detection methods include those based on statistics or thresholds, such as linear interpolation or mean filling; however, statistical or threshold-based methods are poorly adapted to complex time-series patterns, while linear interpolation and mean filling struggle to capture nonlinear time-series dependencies. Therefore, these technologies suffer from low accuracy in processing time-series data. Summary of the Invention
[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of time-series data processing in response to the above-mentioned technical problems.
[0005] In a first aspect, this application provides a data processing method, including:
[0006] Acquire power time-series data from the power system;
[0007] Using power time-series data as input, a trained data processing model is invoked. Through the discriminator in the data processing model, local morphological features of the power time-series data are extracted, and outliers in the power time-series data are identified based on the local morphological features, thus obtaining outlier identification results. Among them, the local morphological features at least characterize the changing trend and waveform shape of the power time-series data.
[0008] The generator in the data processing model fills in missing values in the power time series data based on the context information of the power time series data, and obtains the filled power time series data.
[0009] Based on the outlier identification results, the outliers in the filled power time series data are corrected to obtain the target power time series data.
[0010] Secondly, this application also provides a data processing apparatus, comprising:
[0011] The data acquisition module is used to acquire power time-series data of the power system;
[0012] The first data processing module takes power time-series data as input, calls a trained data processing model, extracts local morphological features from the power time-series data using a discriminator in the data processing model, and identifies outliers in the power time-series data based on the local morphological features, obtaining outlier identification results; wherein, the local morphological features at least characterize the changing trend and waveform shape of the power time-series data; and through the generator in the data processing model, fills in missing values in the power time-series data based on the context information of the power time-series data, obtaining filled power time-series data.
[0013] The second data processing module is used to correct outliers in the filled power time series data based on the outlier identification results, so as to obtain the target power time series data.
[0014] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above-described data processing method embodiments.
[0015] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above-described data processing method embodiments.
[0016] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in any of the above-described data processing method embodiments.
[0017] The aforementioned data processing methods, apparatus, computer equipment, computer-readable storage media, and computer program products are pre-trained using generative adversarial networks (GANs) to capture the dynamic characteristics of power time-series data, resulting in a data processing model. In the actual data processing, firstly, power time-series data of the power system is acquired. Secondly, using the power time-series data as input, the trained data processing model is invoked. The discriminator within the data processing model analyzes the changing trends and waveform characteristics of the power time-series data, extracting local morphological features. Based on these local morphological features, outliers in the power time-series data are identified, yielding outlier identification results. Compared to traditional outlier detection methods, this approach can capture nonlinear temporal dependencies, improving the accuracy of outlier identification. Subsequently, the generator within the data processing model acquires the contextual information of the power time-series data, thereby imputing missing values to obtain the imputed power time-series data. Finally, based on the outlier identification results, the outliers in the imputed power time-series data are corrected to obtain the target power time-series data. This achieves integrated processing of outlier identification and missing value imputation, improving power data quality and providing reliable support for the efficient operation and maintenance of smart grids. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.
[0020] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;
[0021] Figure 3 This is a flowchart illustrating the data processing method in another embodiment;
[0022] Figure 4 This is a flowchart illustrating the data processing method in yet another embodiment;
[0023] Figure 5 This is a structural block diagram of a data processing device in one embodiment;
[0024] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0026] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.
[0027] Specifically, the operator can upload the collected power time-series data of the power system to the server 104 via terminal 102, and then send a data processing message to the server 104 via terminal 102. The server 104 obtains the power time-series data of the power system, and then, using the power time-series data as input, calls the trained data processing model. Through the discriminator in the data processing model, it extracts the local morphological features of the power time-series data, and identifies outliers in the power time-series data based on the local morphological features, thus obtaining the outlier identification result. Among them, the local morphological features at least characterize the changing trend and waveform shape of the power time-series data. Then, through the generator in the data processing model, it fills in the missing values of the power time-series data according to the context information of the power time-series data, thus obtaining the filled power time-series data. Finally, based on the outlier identification result, it corrects the outliers in the filled power time-series data to obtain the target power time-series data.
[0028] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0029] In one exemplary embodiment, such as Figure 2 As shown, a data processing method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps (hereinafter referred to as S): S100 to S400. Wherein:
[0030] S100: Acquire power time-series data of the power system.
[0031] Power time-series data can be time-series data of power equipment in a power system, including timestamps and corresponding measurement values of the power equipment. Power equipment includes, but is not limited to, various sensors, smart meters, power generation equipment, and power transmission and transformation equipment.
[0032] In practice, operators can acquire power time-series data from various sensors in the power system via acquisition terminals, acquire power time-series data such as current, voltage, power, and energy monitored by smart meters, and acquire power time-series data from different power devices through the SCADA (Supervisory Control and Data Acquisition) system. Subsequently, the data is uploaded to the server via the terminal so that the server can obtain the power time-series data of the power system.
[0033] S200 takes power time series data as input, calls a trained data processing model, extracts local morphological features of power time series data through the discriminator in the data processing model, and identifies outliers in power time series data based on local morphological features, and obtains outlier identification results; wherein, local morphological features at least characterize the changing trend and waveform shape of power time series data.
[0034] The trained data processing model is derived from a generative adversarial network trained on historical power time-series data collected within a historical time period. Outliers in power time-series data represent data points that significantly deviate from normal patterns or expected behavior; outliers can originate from measurement errors, equipment failures, etc.
[0035] The constructed generative adversarial network includes a discriminator and a generator. The generator, which may include, but is not limited to, variational autoencoders or autoregressive models, is used to learn the overall distribution of power time-series data and fill in missing values in the data. The discriminator is used to identify outliers in the power time-series data, and may include, but is not limited to, convolutional neural networks, recurrent neural networks, decision trees, or support vector machines.
[0036] In practice, the power time-series data can be standardized first. Specifically, the same mean μ and standard deviation σ as in the training phase can be used to perform Z-score standardization on the power time-series data X_test, resulting in the standardized power time-series data X_test_norm. Alternatively, the power time-series data can be standardized directly.
[0037] Subsequently, a fixed-length time window and step size can be used to perform sliding window segmentation on the standardized power time series data to obtain multiple overlapping or continuous sequences. For example, using the same fixed-length time window_size and step size as in the training phase, X_test_norm can be segmented into multiple overlapping or continuous window samples {W_1, W_2, ..., W_N}.
[0038] Subsequently, the sequence W_i for each time window is input into the trained discriminator. For each sequence W_i, the discriminator analyzes the changing trend and waveform shape of the sequence, extracting local morphological features. Then, based on the local morphological features, it predicts the probability that each data point in the sequence is an outlier, obtaining an outlier probability sequence A_i. Since there may be overlapping regions in the segmented sequences, for each time point t, the outlier probability of the data point at time point t in different windows is obtained, and the target outlier probability of the data point is determined by statistical methods. Specifically, the target outlier probability Score(t) is obtained by taking the average, median, or maximum value of the obtained multiple outlier probabilities. Then, for each time point t, the target outlier probability Score(t) at time point t is compared with a preset outlier probability threshold τ. If Score(t) > τ, the data point at time point t is determined to be an outlier; otherwise, the data point is determined to be a normal value. Based on the above judgment results, an outlier identification result is generated. Specifically, the outlier identification result is a binary label sequence L = [l_1, l_2,...], where l_t = 1 indicates that the data point is an outlier and l_t = 0 indicates that the data point is a normal value.
[0039] S300 uses the generator in the data processing model to fill in missing values in the power time series data based on the context information of the power time series data, and obtains the filled power time series data.
[0040] In practice, before inputting the power time-series data into the data processing model, the power time-series data X_missing can be pre-scanned to identify the locations of missing values. Then, a mask matrix M is constructed based on the locations of the missing values. M is a binary mask vector of the same length as X_missing, where M[t] = 1 indicates the presence of a data point at time t, and M[t] = 0 indicates the absence of a data point at time t. Subsequently, the missing values in the power time-series data can be temporarily filled using a target value, such as 0 or the global mean. Finally, the temporarily filled power time-series data X_temp is Z-score standardized to obtain the standardized power time-series data X_input.
[0041] Next, the power time series data X_input is divided into time windows: using the same window_size and stride as during training, X_input and the corresponding mask M are divided into multiple sequences {W_i} and masks {M_i}.
[0042] Each sequence W_i and its corresponding mask M_i are input into the trained generator, a forward propagation is performed, the context information of the sequence is obtained according to the attention mechanism in the generator, and the missing values are filled according to the mask to obtain power time series data containing multiple filled sequences X_filled_i.
[0043] In other implementations, after obtaining X_filled_i, X_filled_i and its corresponding mask can be input into the generator again to fill the missing data values and optimize the filling effect. This process is repeated a preset number of times, such as 3-5 times, to obtain the filled power time series data.
[0044] S400: Based on the outlier identification results, correct the outliers in the filled power time series data to obtain the target power time series data.
[0045] In practice, the filled power time series data X_filled_normalized can be converted back to the original physical dimensions: X_final = X_filled_normalized *σ + μ.
[0046] Subsequently, based on the outlier identification results, the locations of outliers in the transformed power time-series data can be determined, and then these outliers can be corrected. Correction methods may include, but are not limited to, replacing outliers with the median, mean, or mode (for categorical variables), or using interpolation techniques (such as linear interpolation) to estimate values. For example, replacing outliers with the median can reduce sensitivity to the mean.
[0047] In the aforementioned data processing method, a generative adversarial network (GAN) is pre-trained to capture the dynamic characteristics of power time-series data, resulting in a data processing model. In the actual data processing, firstly, power time-series data of the power system is acquired. Secondly, using the power time-series data as input, the trained data processing model is invoked. The discriminator within the model analyzes the changing trends and waveform characteristics of the power time-series data, extracting local morphological features. Based on these local morphological features, outliers are identified, yielding outlier identification results. Compared to traditional outlier detection methods, this approach captures nonlinear temporal dependencies, improving the accuracy of outlier identification. Subsequently, the generator within the data processing model acquires the contextual information of the power time-series data, thereby imputing missing values and obtaining the imputed power time-series data. Finally, based on the outlier identification results, the outliers in the imputed power time-series data are corrected to obtain the target power time-series data. This integrated processing of outlier identification and missing value imputation improves power data quality and provides reliable support for the efficient operation and maintenance of smart grids.
[0048] In one exemplary embodiment, such as Figure 3 As shown, the training steps of the data processing model include S501 to S504, wherein:
[0049] S501: Obtain historical power time-series data of the power system and generate training samples based on the historical power time-series data.
[0050] Among them, historical power time-series data refers to the time-series data of power equipment collected within a historical time period.
[0051] In practice, the implementation method for obtaining historical power time series data of the power system can refer to the implementation method for obtaining power time series data of the power system in the above embodiments, which will not be repeated here.
[0052] After acquiring historical power time-series data, preliminary data quality processing can be performed, such as outlier removal and data standardization. Outlier removal can be based on physical principles or the Laida criterion to identify and remove extreme outliers that clearly exceed reasonable ranges. These extreme outliers are characterized by data points caused by sensor malfunctions, communication errors, or recording mistakes. Specifically, for different types of historical power time-series data, outliers can be identified and removed based on pre-set standard numerical ranges for power time-series data, or outliers can be identified and removed using statistical distribution-based numerical anomaly detection methods.
[0053] Data standardization can include, but is not limited to, Min-max standardization and Z-score standardization. For example, Z-score standardization can be used to standardize historical power time-series data: x_normalized = (x - μ) / σ; where x is the original data point, μ is the mean of the historical power time-series data, and σ is the standard deviation of the historical power time-series data.
[0054] After preprocessing the historical power time-series data, the data is divided into fixed-length time windows to obtain training samples containing multiple time windows. Specifically, a fixed-length sliding window `window_size` is pre-set and slides across the standardized historical power time-series data with a preset stride. The subsequence [x_t, x_{t+1}, ..., x_{t+window_size-1}] extracted by each window constitutes an independent training sample.
[0055] Next, for each subsequence within a time window, missing data simulation and masking are performed. Specifically, first, a binary mask vector M of the same length is generated for each training sample in the sliding window, where 1 indicates the presence of a data point at that position, and 0 indicates the absence of a data point at that position. Then, the missing data pattern is simulated: some data points in the training samples are randomly marked as missing with probability p_miss, or a starting point can be randomly selected to simulate the absence of a continuous data segment. Finally, partial observation samples are constructed: the original training sample X is element-wise multiplied with the mask matrix M to obtain the input data X_incomplete = X⊙M for missing data simulation.
[0056] Then, for each time window, the training sample (X_incomplete, M, X) can be generated by combining the training sample X_incomplete after missing simulation processing, the mask vector M, and the original training sample X in a triplet structure, resulting in a training set containing multiple training samples.
[0057] S502, Discriminator training steps: Extract local morphological features of training samples and generated samples through the discriminator, generate true data prediction probability and outlier prediction probability based on the local morphological features, determine the first loss value based on the true data prediction probability and outlier prediction probability, and update the hyperparameters of the discriminator based on the first loss value.
[0058] In this embodiment, the discriminator includes a feature extraction layer and an output layer. The input of the discriminator may include training samples from the training set (denoted as real data sequence X_real) or sequence X_filled from the generated samples after repair by the generator.
[0059] The feature extraction layer consists of multiple stacked one-dimensional convolutional layers, followed by a LeakyReLU activation function and a Dropout layer. The convolutional kernels of the one-dimensional convolutional layers slide along the time dimension to extract local morphological features of the sequence. These local morphological features can include information such as the waveform shape, trend of change, and distribution of local extrema of the input data within the time window.
[0060] The output layer flattens the local morphological features extracted by the convolutional layer and outputs the true data prediction probability and outlier prediction probability through two parallel fully connected layers. The true data prediction probability is a scalar generated by the sigmoid activation function, and it represents the probability that the discriminator determines that the input sequence is a true data sequence. The outlier prediction probability consists of multiple data points, forming a vector of the same length as the input sequence. Each element is generated by the sigmoid function, and the outlier prediction probability represents the probability that the data at the predicted time point is an outlier.
[0061] The first loss value can be calculated by quantizing the discriminator's loss based on the predicted probabilities of real data and outliers, along with the input time-series data. These two losses are then fused together using summation or weighted summation to obtain the first loss value. The loss quantization method can refer to the loss function of a standard generative adversarial network. After obtaining the first loss value, the discriminator's parameters can be updated via backpropagation.
[0062] S503, Generator Training Steps: The training samples are encoded into a hidden state sequence by the generator. The hidden state sequence is reconstructed based on the context information of the hidden state sequence to obtain the generated samples. Based on the generated samples and the training samples, the second loss value is determined, and the hyperparameters of the generator are updated based on the second loss value.
[0063] In this embodiment, the generator consists of an encoder and a decoder. The encoder is a multi-layer bidirectional gated loop unit that reads the masked sequence X_incomplete and encodes it into a hidden state sequence H_enc. The encoder's final hidden state is transformed as the potential representation z of the sequence, and a random noise vector is introduced into z.
[0064] The decoder is a multi-layer bidirectional gated recurrent unit. Starting with a latent representation z and noise as initial states, the decoder progressively reconstructs the entire sequence. During reconstruction, at each decoding step t, the decoder passes through an attention layer, focusing on all hidden states H_enc of the encoder. It dynamically acquires the most relevant contextual information from the entire input sequence, calculates attention weights based on this contextual information, and obtains a context vector. This context vector is then fused with the current decoder hidden state to generate candidate values. A gated fusion layer is used to determine, based on the mask M, whether the output at the current time step depends on the candidate values generated by the decoder or on the original true values of the input, thus reconstructing the generated samples. Specifically, if the component representation data points in the mask vector at the current time step exist, the original input data points are retained; if the component representation data points are missing, the generated candidate values are used to fill the gaps. The bidirectional gated recurrent unit combines the characteristics of recurrent neural networks and gated recurrent units, thus possessing the ability to simultaneously focus on information from both past and future time steps of the input data, and the advantage of capturing the dependencies in time-series data.
[0065] After the decoder reconstructs the generated samples based on the hidden state, the generator's reconstruction loss can be quantized using the generated samples, mask vector, and original training samples. The generator's loss for generating samples can be quantized by predicting probabilities based on the real data of the generated samples. These two losses are then fused using summation or weighted summation to obtain a second loss value. The loss quantization method can refer to the loss function of a standard generative adversarial network. After obtaining the second loss value, the generator's parameters can be updated through backpropagation.
[0066] S504, alternately execute S502 and S503 until the preset training termination condition is met, and obtain the trained data processing model.
[0067] In this embodiment, the alternating iterations of S502 and S503 can involve fixing the generator parameters, performing a discriminator training step to update the discriminator parameters and minimize the first loss value, then fixing the discriminator parameters again and performing a generator training step to update the generator parameters. This alternating iteration continues until a preset training termination condition is met, the model converges, and a trained data processing model is obtained. The preset training termination condition may include, but is not limited to, terminating the data processing model training step when the first loss value is continuously less than a preset loss threshold within a preset number of iterations, and the second loss value is continuously less than a preset loss threshold within a preset number of iterations. Alternatively, it may terminate the data processing model training step when the number of alternating iterations reaches a preset maximum number of alternating iterations. It is understood that the training termination condition can be set according to data processing requirements, and this application does not limit it in this regard.
[0068] In one exemplary embodiment, such as Figure 4 As shown, based on the generated samples and training samples, the second loss value is determined, including S531 to S534, where:
[0069] S531, determine the first adversarial loss value based on the generated sample and the preset first adversarial loss function.
[0070] S532, determine the reconstruction loss value based on the training samples, generated samples, and the preset reconstruction loss function.
[0071] S533, determine the mask region loss value based on the training samples and the preset mask region loss function.
[0072] S534, determine the second loss value based on the first adversarial loss value, the reconstruction loss value, and the mask region loss value.
[0073] In specific implementation, for S531, the first adversarial loss value is obtained through the following first adversarial loss function:
[0074] L_adv_G= - E[log(D(G(X_incomplete, M)))]
[0075] The generator aims to make its generated samples G(...) score high in the discriminator D, i.e., to make D(G(...)) close to 1.
[0076] For S532, the preset reconstruction loss may include, but is not limited to, loss functions such as mean squared error, mean absolute error, or root mean square error, used to quantify the error between training samples and generated samples to obtain the reconstruction loss value. For example, the reconstruction loss value L_rec = MSE(X_real⊙M, G(X_incomplete, M)⊙M) is determined using mean squared error; where MSE is the mean squared error.
[0077] For S533, the preset mask region loss function may include, but is not limited to, loss functions such as mean squared error, mean absolute error, or root mean square error. These functions are used to quantify the error between training samples and generated samples within the mask region (representing the region with missing data), thus obtaining the mask region loss value. For example, the mean squared error is used to determine the mask region loss value: L_mask = MSE(X_real⊙(1-M), G(X_incomplete, M)⊙(1-M)).
[0078] For S534, the second loss value can be obtained by summing, weighted summing, or other methods of the first adversarial loss value, the reconstruction loss value, and the mask region loss value. For example, the second loss value L_G = L_adv_G + λ_rec * L_rec + λ_mask * L_mask. λ_rec and λ_mask can be hyperparameters that control the weights of each loss term.
[0079] In this embodiment, the generator's loss value is calculated using three dimensions of loss, which helps to improve the generator's robustness.
[0080] In an exemplary embodiment, a first loss value is determined based on the predicted probability of real data and the predicted probability of outliers, including steps S521 to S523, wherein:
[0081] S521, Determine the anomaly identification loss value based on the outlier prediction probability and the preset anomaly identification loss function.
[0082] S522, based on the predicted probability from real data and the preset second adversarial loss function, determine the second adversarial loss value.
[0083] S523, determine the first loss value based on the anomaly identification loss value and the second adversarial loss value.
[0084] In specific implementation, for S521, the preset anomaly recognition loss function may include, but is not limited to, the binary cross-entropy loss function or the least squares loss function, used to quantify the loss of the discriminator in recognizing outliers. For example, the binary cross-entropy loss function is used to determine the anomaly recognition loss value L_anom = BCEWithLogits(0, D_anom(X_real)); where BCE: binary cross-entropy, the discriminator outputs a low anomaly score.
[0085] For S522, the preset second adversarial loss function is as follows:
[0086] L_adv_D=-E[log(D(X_real))]-E[log(1-D(G(X_incomplete, M)))]
[0087] Where L_adv_G is the second adversarial loss value. The discriminator makes the score D(X_real) for the real sample X_real close to 1, and the score for the generated sample G(...) close to 0.
[0088] For S523, the first loss value can be obtained by summing the anomaly identification loss value and the second adversarial loss value, or by weighted summation. For example, the first loss value L_D = L_D = L_adv_D +λ_anom * L_anom, where λ_anom can be a hyperparameter that controls the weight of the loss.
[0089] In this embodiment, the first loss value is obtained by fusing the adversarial loss and anomaly recognition loss of the discriminator, thereby improving the accuracy of the discriminator's performance quantification.
[0090] In an exemplary embodiment, the discriminator includes a feature extraction layer and an output layer. The discriminator in the data processing model extracts local morphological features from the power time-series data and identifies outliers in the power time-series data based on these local morphological features, resulting in outlier identification results including steps S201 to S203. Wherein:
[0091] S201, extract local morphological features of power time series data based on the feature extraction layer.
[0092] S202, based on the flattening layer and fully connected layer connected in sequence in the output layer, feature flattening and outlier probability prediction are performed on the local morphological features to obtain the outlier prediction probability sequence of the power time series data.
[0093] S203, Generate outlier identification results based on the statistical characteristics of the outlier prediction probability sequence.
[0094] Local morphological features can include information such as the waveform shape, trend of change, and distribution of local extrema of the input data within a time window. The feature extraction layer consists of multiple stacked one-dimensional convolutional layers, followed by a LeakyReLU activation function and a Dropout layer.
[0095] In practice, for each sequence W_i of the power time-series data, the convolutional layer in the feature extraction layer slides across the input sequence data along the time dimension to extract local morphological features. These local morphological features can include information such as the waveform shape, trend, and distribution of local extrema within the time window. Subsequently, the flattening layer of the output layer flattens the local morphological features extracted by the convolutional layer, and then outputs the probability that each data point in the sequence is an outlier through a fully connected layer, i.e., the outlier prediction probability, thus obtaining the outlier prediction probability sequence.
[0096] Subsequently, for each time point t, the outlier prediction probability of the data point at time point t within different windows is obtained, and the target outlier prediction probability of the data point is determined by statistical methods. Specifically, the target outlier prediction probability Score(t) is obtained by averaging, medianing, or maximizing the obtained multiple outlier prediction probabilities. Then, for each time point t, the target outlier prediction probability Score(t) at time point t is compared with a preset outlier prediction probability threshold τ. If Score(t) > τ, the data point at time point t is determined to be an outlier; otherwise, the data point is determined to be a normal value. Based on the above judgment results, an outlier identification result is generated. Specifically, the outlier identification result is a binary label sequence L = [l_1, l_2, ...], where l_t = 1 indicates that the data point is an outlier, and l_t = 0 indicates that the data point is a normal value.
[0097] The outlier prediction probability threshold τ can be set by using a trained discriminator to identify outliers on a known validation set, obtaining the outlier prediction probability for each data point, and then setting the outlier prediction probability threshold τ based on the outlier prediction probability of each data point. For example, the outlier prediction probability threshold τ can be higher than the maximum outlier prediction probability. Alternatively, the threshold τ can be recalculated periodically based on the outlier prediction probability distribution over a period of time to adapt to slow changes in data distribution.
[0098] In this embodiment, a discriminator is used to extract local morphological features and identify outliers, thereby improving the accuracy of outlier identification.
[0099] In an exemplary embodiment, the generator includes an encoder and a decoder. Using the generator in the data processing model, missing values are imputed in the power time-series data based on contextual information, resulting in imputed power time-series data, including S301 to S302, wherein:
[0100] S301, the power time-series data is encoded into a hidden state sequence according to the decoder.
[0101] S302, the decoder reconstructs the hidden state sequence based on the context information of the hidden state sequence to obtain the filled power time series data.
[0102] Following the steps above, in this embodiment, the generator consists of an encoder and a decoder. The encoder is a multi-layer bidirectional gated loop unit that reads the masked sequence X_incomplete and encodes it into a hidden state sequence H_enc. It then transforms the final hidden state of the encoder to serve as the potential representation z of the sequence, introducing a random noise vector into z.
[0103] The decoder is a multi-layer bidirectional gated recurrent unit. Starting with a latent representation z and noise as initial states, the decoder progressively reconstructs the entire sequence. During reconstruction, at each decoding step t, the decoder passes through an attention layer, focusing on all hidden states H_enc of the encoder. It dynamically acquires the most relevant contextual information from the entire input sequence, calculates attention weights based on this contextual information, and obtains a context vector. This context vector is then fused with the current decoder hidden state to generate candidate values. A gated fusion layer is used to determine, based on the mask M, whether the output at the current time step depends on the candidate values generated by the decoder or on the original true input values, thus obtaining the reconstructed power time-series data. Specifically, if the component representation data points in the mask vector at the current time step exist, the original input data points are retained; if the component representation data points are missing, the generated candidate values are used to fill the gaps.
[0104] In this embodiment, the context information of the input sequence is dynamically obtained through the multi-layer bidirectional gated loop unit of the generator, and the value to be filled is generated by combining the mask, which improves the reliability of missing value filling.
[0105] To provide a clearer explanation of the data processing method provided in this application, a specific embodiment is described below, which includes the following steps:
[0106] S1, acquire power time-series data of the power system.
[0107] S2 takes power time series data as input, calls the trained data processing model, extracts the local morphological features of the power time series data based on the feature extraction layer of the discriminator of the data processing model, and performs feature flattening and outlier probability prediction on the local morphological features based on the flattening layer and fully connected layer connected in sequence in the output layer, to obtain the outlier prediction probability sequence of the power time series data.
[0108] S3 generates outlier identification results based on the statistical characteristics of the outlier prediction probability sequence.
[0109] S4, the decoder in the generator of the data processing model encodes the power time series data into a hidden state sequence. The decoder reconstructs the hidden state sequence based on the context information of the hidden state sequence to obtain the filled power time series data.
[0110] S5. Based on the outlier identification results, the outliers in the filled power time series data are corrected to obtain the target power time series data.
[0111] The training process of the data processing model includes S101 to S104:
[0112] S101: Obtain historical power time-series data of the power system and generate training samples based on the historical power time-series data.
[0113] S102, the discriminator extracts local morphological features of training samples and generated samples, generates real data prediction probability and outlier prediction probability based on local morphological features, determines an anomaly identification loss value based on the outlier prediction probability and a preset anomaly identification loss function, determines a second adversarial loss value based on the real data prediction probability and a preset second adversarial loss function, determines a first loss value based on the anomaly identification loss value and the second adversarial loss value, and updates the hyperparameters of the discriminator based on the first loss value.
[0114] S103: The generator encodes the training samples into a hidden state sequence, reconstructs the hidden state sequence based on the context information of the hidden state sequence to obtain the generated samples, determines the first adversarial loss value based on the generated samples and the preset first adversarial loss function, determines the reconstruction loss value based on the training samples, the generated samples and the preset reconstruction loss function, determines the mask region loss value based on the training samples and the preset mask region loss function, determines the second loss value based on the first adversarial loss value, the reconstruction loss value and the mask region loss value, and updates the generator's hyperparameters based on the second loss value.
[0115] S104, alternately execute S102 and S103 until the preset training termination condition is met, and obtain the trained data processing model.
[0116] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0117] In one exemplary embodiment, such as Figure 5 As shown, a data processing device 600 is provided, including: a data acquisition module 610, a first data processing module 620, and a second data processing module 630, wherein:
[0118] Data acquisition module 610 is used to acquire power time-series data of the power system;
[0119] The first data processing module 620 is used to take power time series data as input, call a trained data processing model, extract local morphological features of the power time series data through the discriminator in the data processing model, and identify outliers in the power time series data based on the local morphological features to obtain outlier identification results; wherein, the local morphological features at least characterize the changing trend and waveform shape of the power time series data; through the generator in the data processing model, missing values are filled in the power time series data based on the context information of the power time series data to obtain filled power time series data;
[0120] The second data processing module 630 is used to correct the outliers in the filled power time series data based on the outlier identification results, so as to obtain the target power time series data.
[0121] In an exemplary embodiment, the first data processing module 620 is further configured to extract local morphological features of power time series data based on the feature extraction layer; perform feature flattening and outlier probability prediction on the local morphological features based on the flattening layer and fully connected layer connected in sequence in the output layer to obtain an outlier prediction probability sequence of power time series data; and generate an outlier identification result based on the statistical characteristics of the outlier prediction probability sequence.
[0122] In an exemplary embodiment, the first data processing module 620 is further configured to encode the power time series data into a hidden state sequence according to the decoder; and to reconstruct the hidden state sequence according to the context information of the hidden state sequence to obtain the filled power time series data.
[0123] In one exemplary embodiment, the data processing device 600 further includes a model training module 640:
[0124] The data acquisition module 610 is also used to acquire historical power time series data of the power system and generate training samples based on the historical power time series data;
[0125] The model training module 640 is used to alternately and iteratively execute the generator training step and the discriminator training step until the preset training termination condition is met, resulting in a trained data processing model. The discriminator training step involves: extracting local morphological features from training samples and generated samples using the discriminator; generating true data prediction probabilities and outlier prediction probabilities based on these local morphological features; determining a first loss value based on these probabilities; and updating the discriminator's hyperparameters based on the first loss value. The generator training step involves: encoding training samples into hidden state sequences using the generator; reconstructing the hidden state sequences based on the context information of the hidden state sequences to obtain generated samples; determining a second loss value based on the generated samples and training samples; and updating the generator's hyperparameters based on the second loss value.
[0126] In an exemplary embodiment, the model training module 640 is further configured to determine a first adversarial loss value based on the generated samples and a preset first adversarial loss function; determine a reconstruction loss value based on the training samples, the generated samples, and a preset reconstruction loss function; determine a mask region loss value based on the training samples and a preset mask region loss function; and determine a second loss value based on the first adversarial loss value, the reconstruction loss value, and the mask region loss value.
[0127] In an exemplary embodiment, the model training module 640 is further configured to determine an anomaly identification loss value based on the outlier prediction probability and a preset anomaly identification loss function; determine a second adversarial loss value based on the real data prediction probability and a preset second adversarial loss function; and determine a first loss value based on the anomaly identification loss value and the second adversarial loss value.
[0128] Each module in the aforementioned data processing device 600 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0129] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.
[0130] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0131] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above-described data processing method embodiments.
[0132] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above data processing method embodiments.
[0133] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.
[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0135] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A data processing method, characterized in that, The method includes: Acquire power time-series data from the power system; Using the power time-series data as input, a trained data processing model is invoked. Through the discriminator in the data processing model, local morphological features of the power time-series data are extracted, and outliers in the power time-series data are identified based on the local morphological features to obtain outlier identification results. The local morphological features at least characterize the changing trend and waveform shape of the power time-series data. The generator in the data processing model fills in missing values in the power time series data according to the context information of the power time series data to obtain the filled power time series data. Based on the outlier identification results, the outliers in the filled power time series data are corrected to obtain the target power time series data.
2. The method according to claim 1, characterized in that, The discriminator includes a feature extraction layer and an output layer; the process of extracting local morphological features of the power time-series data through the discriminator in the data processing model, and identifying outliers in the power time-series data based on the local morphological features, to obtain outlier identification results, includes: The local morphological features of the power time-series data are extracted based on the feature extraction layer; Based on the flattening layer and fully connected layer connected in sequence in the output layer, feature flattening and outlier probability prediction are performed on the local morphological features to obtain the outlier prediction probability sequence of the power time series data. Anomaly identification results are generated based on the statistical characteristics of the outlier prediction probability sequence.
3. The method according to claim 1, characterized in that, The generator includes an encoder and a decoder; the process of filling missing values in the power time series data using the generator in the data processing model, based on the context information of the power time series data, to obtain the filled power time series data includes: The power time-series data is encoded into a hidden state sequence according to the decoder; The decoder reconstructs the hidden state sequence based on the context information of the hidden state sequence to obtain the filled power time series data.
4. The method according to claim 1, characterized in that, The training steps of the data processing model include: Acquire historical power time-series data of the power system, and generate training samples based on the historical power time-series data; Discriminator training steps: Extract local morphological features of the training samples and generated samples through the discriminator, generate true data prediction probability and outlier prediction probability based on the local morphological features, determine a first loss value based on the true data prediction probability and the outlier prediction probability, and update the hyperparameters of the discriminator based on the first loss value; Generator training steps: The training samples are encoded into a hidden state sequence by the generator, the hidden state sequence is reconstructed according to the context information of the hidden state sequence to obtain generated samples, a second loss value is determined according to the generated samples and the training samples, and the hyperparameters of the generator are updated according to the second loss value; The generator training step and the discriminator training step are executed alternately and iteratively until the preset training termination condition is met, and the trained data processing model is obtained.
5. The method according to claim 4, characterized in that, Determining the second loss value based on the generated samples and the training samples includes: The first adversarial loss value is determined based on the generated sample and the preset first adversarial loss function; The reconstruction loss value is determined based on the training samples, the generated samples, and the preset reconstruction loss function; The mask region loss value is determined based on the training samples and the preset mask region loss function; The second loss value is determined based on the first adversarial loss value, the reconstruction loss value, and the mask region loss value.
6. The method according to claim 4, characterized in that, The step of determining the first loss value based on the predicted probability of the real data and the predicted probability of the outlier includes: The anomaly identification loss value is determined based on the outlier prediction probability and the preset anomaly identification loss function; The second adversarial loss value is determined based on the predicted probability from the real data and the preset second adversarial loss function; The first loss value is determined based on the anomaly identification loss value and the second adversarial loss value.
7. A data processing apparatus, characterized in that, The device includes: The data acquisition module is used to acquire power time-series data of the power system; The first data processing module is used to take the power time-series data as input, call a trained data processing model, extract local morphological features of the power time-series data through the discriminator in the data processing model, and identify outliers in the power time-series data based on the local morphological features to obtain outlier identification results; wherein, the local morphological features at least characterize the changing trend and waveform shape of the power time-series data; and use the generator in the data processing model to fill missing values in the power time-series data based on the context information of the power time-series data to obtain filled power time-series data. The second data processing module is used to correct the outliers in the filled power time series data according to the outlier identification results, so as to obtain the target power time series data.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.