Electric power big data model-based electric power data cleaning method and system

Abstract

The invention relates to the technical field of data cleaning, and particularly discloses an electric power big data model-based electric power data cleaning method and system, and the method comprises the steps: firstly obtaining an electric power abnormal data set, introducing a deep learning-based data processing technology to carry out the data feature aggregation analysis of the electric power abnormal data set based on self-learning reinforcement, and carrying out the data feature aggregation analysis of the electric power abnormal data set; the method comprises the following steps of: intelligently decoding the optimal sampling parameter setting of load data to mine key characteristics such as noise distribution, periodicity and abnormal trend of power abnormal data so as to intelligently decode the optimal sampling parameter setting of the load data, and on the basis, collecting, classifying and storing the load data and performing data cleaning based on time sequence data similarity so as to complete the power data cleaning operation. In this way, the dynamic nonlinear characteristics of the power abnormal data can be automatically adapted, the reliability of optimal parameter setting is improved, and then the intelligent level and efficiency of data cleaning are improved.

Description

Technical Field

[0001] The present application relates to the field of data cleaning technology, and more specifically, to a power data cleaning method and system based on a power big data model. Background Art

[0002] With the deepening of the intelligent and digital transformation of power systems, the scale of power data has experienced explosive growth. Multi-source, heterogeneous data covering all aspects of power generation, transmission, and consumption has become the core basis for grid optimization, scheduling, and fault warning. However, raw power data is commonly subject to noise interference, abnormal jumps, and missing values, which directly affect the accuracy of data analysis and the reliability of decision-making, and may even cause false alarms or equipment malfunctions. Therefore, data cleaning is a fundamental step in the application of power big data. Its goal is to improve the integrity, consistency, and credibility of data by filtering out noise, repairing missing values, and correcting anomalies, thereby providing high-quality input for subsequent analysis and modeling.

[0003] In response to this, patent application CN117763284A proposes a power data cleaning method based on a power big data model. This method first marks abnormal data and dynamically adjusts optimal sampling parameters for different periodic data (long and short periods). It then uses the Pearson correlation coefficient to filter weakly correlated historical data sets, constructs a univariate regression model to fill missing values, and achieves high-precision data repair. Data is then classified and stored based on the power data model, and cleaned using a time series data similarity metric. This method can clean power data in a timely manner, improving data credibility and availability, and ultimately enhancing data quality and value.

[0004] However, in the existing technology, it relies on linear formulas to dynamically set smoothing parameters, requires manual pre-setting of free variables and iterative detection of abnormal intervals step by step to determine the optimal sampling parameters. Since power anomaly patterns are highly nonlinear (such as sudden load spikes and intermittent sensor failures), parameter rules driven by manual experience are difficult to accurately adapt to the dynamic changes of noise distribution and periodic characteristics, resulting in frequent over-smoothing or under-smoothing phenomena. On the other hand, when iterative detection of abnormal intervals step by step to determine the optimal sampling parameters, a simple repeated detection strategy is inefficient. When the amount of data is large and the abnormal situation is complex, checking each interval will consume a lot of time and computing resources, seriously affecting the timeliness of data cleaning, and cannot meet the power data processing scenarios with high real-time requirements.

[0005] Therefore, an optimized power data cleaning method and system based on the power big data model is expected. Summary of the Invention

[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a power data cleaning method and system based on the power big data model, which first obtains a collection of power anomaly data, and introduces a data processing technology based on deep learning to perform data feature aggregation analysis based on self-learning reinforcement on the power anomaly data set, so as to mine the key features such as noise distribution, periodicity, and abnormal trend of the power anomaly data, thereby intelligently decoding the optimal sampling parameter setting of the load data, and then on this basis, by collecting, classifying, storing and cleaning the load data based on the similarity of time series data, the power data cleaning operation is completed. In this way, it can automatically adapt to the dynamic nonlinear characteristics of power anomaly data, improve the reliability of the optimal parameter setting, and thus improve the intelligence level and efficiency of data cleaning.

[0007] According to one aspect of the present application, a power data cleaning method based on a power big data model is provided, which includes: Acquire a collection of power anomaly data, and intelligently configure optimal parameter settings based on the collection of power anomaly data; Load data were collected according to the optimal parameter settings and based on the Pearson correlation coefficient; Based on the power data model, the collected load data is classified and stored using attributes; The classified and stored load data is cleaned using time series data similarity measurement to complete the power data cleaning operation; Among them, intelligently configuring optimal parameter settings based on the set of power abnormality data includes: performing data feature aggregation analysis based on self-learning reinforcement on the set of power abnormality data to determine the optimal parameter settings.

[0008] According to another aspect of the present application, a power data cleaning system based on a power big data model is provided, comprising: An abnormal data acquisition and parameter configuration module, used to acquire a collection of power abnormal data and intelligently configure optimal parameter settings based on the collection of power abnormal data; A load data acquisition module is used to collect load data according to the optimal parameter settings and based on the Pearson correlation coefficient; The load data classification and storage module is used to classify and store the collected load data using attributes based on the power data model; The load data cleaning module is used to clean the classified and stored load data using the time series data similarity measurement to complete the power data cleaning operation; The abnormal data acquisition and parameter configuration module is used to perform data feature aggregation analysis based on self-learning reinforcement on the set of abnormal power data to determine the optimal parameter settings.

[0009] Compared with the existing technology, the power data cleaning method and system based on the power big data model provided by this application first obtains a collection of power anomaly data, and introduces data processing technology based on deep learning to perform data feature aggregation analysis based on self-learning reinforcement on the power anomaly data set to mine the noise distribution, periodicity, abnormal trend and other key features of the power anomaly data, thereby intelligently decoding the optimal sampling parameter setting of the load data, and then on this basis, by collecting, classifying, storing and cleaning the load data based on the similarity of time series data, the power data cleaning operation is completed. In this way, it can automatically adapt to the dynamic nonlinear characteristics of power anomaly data, improve the reliability of the optimal parameter setting, and thus improve the intelligence level and efficiency of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 Flowchart of a power data cleaning method based on a power big data model according to an embodiment of the present application.

[0012] Figure 2 This is a flowchart of sub-step S1 of the power data cleaning method based on the power big data model according to an embodiment of the present application.

[0013] Figure 3 Schematic diagram of data flow of sub-step S1 of the power data cleaning method based on the power big data model according to an embodiment of the present application.

[0014] Figure 4 This is a flowchart of sub-step S13 of the power data cleaning method based on the power big data model according to an embodiment of the present application.

[0015] Figure 5 This is a flowchart of sub-step S132 of the power data cleaning method based on the power big data model according to an embodiment of the present application.

[0016] Figure 6 4 is a block diagram of an electric power data cleaning system based on an electric power big data model according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0018] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are illustrative only, and different aspects of the system and method can use different modules.

[0019] Flowcharts are used in this application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the various steps may be processed in reverse order or simultaneously, as needed. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0020] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0021] It is worth noting that in this application, all actions to obtain data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0022] As mentioned in the background technology section above, patent CN117763284A proposes a power data cleaning method based on a power big data model. This method first marks abnormal data and dynamically adjusts optimal sampling parameters for different periodic data (long and short periods). It then uses the Pearson correlation coefficient to filter weakly correlated historical data sets, constructs a univariate regression model to fill missing values, and achieves high-precision data repair. Data is then classified and stored based on the power data model, and data is cleaned using a time series data similarity metric. This method can clean power data in a timely manner, improving data credibility and availability, and ultimately enhancing data quality and value.

[0023] However, in the prior art, the dynamic setting of smoothing parameters relies on linear formulas, which requires manual pre-setting of free variables and step-by-step iteration to detect abnormal intervals to determine the optimal sampling parameters. However, power anomaly patterns usually have highly nonlinear characteristics, such as sudden load spikes and intermittent sensor failures. These complex anomaly patterns make it difficult for parameter rules based on manual experience to accurately match the dynamic changes of noise distribution and periodic characteristics, resulting in frequent problems of over-smoothing or under-smoothing. In addition, the step-by-step iterative detection method used to determine the optimal sampling parameters is inefficient. When faced with large amounts of data and complex anomalies, interval-by-interval inspection consumes a lot of time and computing resources, seriously affecting the efficiency of data cleaning, and it is difficult to meet the needs of power data processing with high real-time requirements. In response to the above technical problems, this application proposes a power data cleaning method based on a power big data model, which first obtains a collection of power anomaly data and introduces data processing technology based on deep learning to perform data feature aggregation analysis based on self-learning reinforcement on the power anomaly data set to mine the noise distribution, periodicity, abnormal trend and other key features of the power anomaly data, thereby intelligently decoding the optimal sampling parameter settings for the load data. On this basis, the power data cleaning operation is completed by collecting, classifying, storing and cleaning the load data based on the similarity of time series data. In this way, it can automatically adapt to the dynamic nonlinear characteristics of power anomaly data, improve the reliability of the optimal parameter setting, and thus improve the intelligence level and efficiency of data cleaning.

[0024] Figure 1 FIG is a flow chart of a method for cleaning power data based on a power big data model according to an embodiment of the present application. Figure 1 As shown, the power data cleaning method based on the power big data model includes the following steps: S1, obtaining a set of power abnormality data, and intelligently configuring optimal parameter settings based on the set of power abnormality data; S2, collecting load data according to the optimal parameter settings and based on the Pearson correlation coefficient; S3, based on the power data model, classifying and storing the collected load data using attributes; S4, cleaning the classified and stored load data using time series data similarity measurement to complete the power data cleaning operation.

[0025] In the above-mentioned power data cleaning method based on the power big data model, step S1 obtains a collection of power anomaly data and intelligently configures optimal parameter settings based on the collection of power anomaly data. In a specific example of the present application, step S1 includes performing a data feature aggregation analysis based on self-learning reinforcement on the collection of power anomaly data to determine the optimal parameter settings. It should be understood that existing technologies determine optimal parameters through preset linear formulas and iterative detection of anomaly intervals. However, due to the nonlinear and multimodal distribution characteristics of power anomaly data (such as load surges caused by lightning strikes and periodic sensor drift), linear formulas cannot capture the complex noise patterns and periodic coupling relationships. For example, short-period daily data may contain both high-frequency noise (such as industrial equipment startups and shutdowns) and low-frequency trends (such as diurnal load fluctuations). Fixed linear parameter rules can easily result in unsmoothing high-frequency noise or over-filtering low-frequency trends. Furthermore, iterative detection requires traversing all marked anomaly intervals. When the data volume reaches GB levels and the anomaly distribution is dense (such as during periods of frequent distribution network failures), the computational time increases exponentially, failing to meet the millisecond-level real-time response requirements of smart grids. To overcome this shortcoming, this application introduces self-supervised learning technology to automatically mine implicit features such as noise distribution, periodicity, and abnormal trends from a large amount of power anomaly data, and dynamically generate optimal parameters that adapt to complex scenarios to solve the problems of poor adaptability and lack of timeliness in the parameter configuration process. Figure 2 This is a flowchart of sub-step S1 of the power data cleaning method based on the power big data model according to an embodiment of the present application. Figure 3 Schematic diagram of data flow of sub-step S1 of the power data cleaning method based on the power big data model according to the embodiment of the present application. Figure 2 and Figure 3 As shown, the step S1 includes the steps of: S11, using the power abnormality data embedding coding matrix to embed code each power abnormality data in the set of power abnormality data to obtain a set of power abnormality data structured embedding coding vectors; S12, using a linear projection layer to map each power abnormality data structured embedding coding vector in the set of power abnormality data structured embedding coding vectors to the same latent semantic space to obtain a set of power abnormality data structured projection embedding coding vectors; S13, performing feature aggregation analysis based on self-learning reinforcement on the set of power abnormality data structured projection embedding coding vectors to obtain a power abnormality data aggregation analysis modeling vector; S14, performing feature decoding on the power abnormality data aggregation analysis modeling vector to obtain the optimal parameter setting.

[0026] Specifically, in step S11, each power anomaly data in the set of power anomaly data is embedded and encoded using the power anomaly data embedding coding matrix to obtain a set of power anomaly data structured embedded coding vectors. Specifically, since power anomaly data (such as jump values, missed sampling segments, and noise pulses) usually exist in an unstructured form, it is difficult to directly capture the implicit associations between the data. To this end, the present application uses a trainable weight matrix (i.e., a power anomaly data embedding coding matrix) to embed and encode each power anomaly data to map the original data into a low-dimensional dense vector, thereby obtaining a set of power anomaly data structured embedded coding vectors, thereby achieving structured expression of power anomaly data and providing a unified feature representation basis for subsequent optimal parameter configuration analysis.

[0027] Specifically, step S12 uses a linear projection layer to map each of the structured embedding code vectors of the power anomaly data in the set of structured embedding code vectors of the power anomaly data to the same latent semantic space to obtain a set of structured projection embedding code vectors of the power anomaly data. It should be understood that because different types of power anomaly data (such as leakage data from different devices) may have different feature distributions (such as the difference in embedding space between transformer tripping and cable leakage), direct aggregation analysis will make it difficult for the model to capture common patterns due to "feature space heterogeneity." To this end, the present application further uses a linear projection layer to align the structured embedding code vectors of power anomaly data from different sources and types to a unified semantic space to eliminate feature distribution offsets, allowing different types of power anomaly data to be effectively compared and aggregated in the same latent semantic space.

[0028] Specifically, the step S13 performs a feature aggregation analysis based on self-learning reinforcement on the set of structured projection embedded coding vectors of the power anomaly data to obtain a power anomaly data aggregation analysis modeling vector. It should be understood that in the field of power data monitoring and analysis, the characteristics of abnormal data (such as abnormal duration, frequency of occurrence) are directly related to the optimal sampling period. Since the characteristics of power anomaly data usually have long-range correlations, direct parameter configuration based on local features is prone to ignore global context information (such as seasonal fluctuations in historical load data and short-term load mutations may have similar patterns), resulting in unreasonable parameter settings. To this end, the present application further performs a feature aggregation analysis on the set of structured projection embedded coding vectors of the power anomaly data to utilize a self-learning reinforcement mechanism to capture high-order correlations across data samples, and to mine the deep-level characteristic laws of power anomaly data in the global context, such as noise distribution, periodic characteristics, and abnormal trends, thereby providing a more accurate feature basis for subsequent optimal parameter configuration. Among them, Figure 4 FIG is a flowchart of sub-step S13 of the power data cleaning method based on the power big data model according to an embodiment of the present application. Figure 4 As shown, the step S13 includes the steps of: S131, performing linear clustering analysis on the set of structured projection embedded coding vectors of the power anomaly data to obtain the initial linear cluster center coding vector of the power anomaly data; S132, based on the feature clustering compensation increment of the set of structured projection embedded coding vectors of the power anomaly data relative to the initial linear cluster center coding vector of the power anomaly data, compensating and correcting the initial linear cluster center coding vector of the power anomaly data to obtain the power anomaly data aggregation analysis modeling vector.

[0029] More specifically, in a specific example of the present application, step S131 includes: calculating the spatial reference average vector of the set of structured projection embedding code vectors of the power anomaly data to obtain the initial linear cluster center code vector of the power anomaly data, which is expressed as: in, Represents a set of structured projection embedding coding vectors of power anomaly data, 、 、 and They represent the first, second, and third vectors in the set of structured projection embedding coding vectors of power anomaly data. and The structured projection embedding coding vector of power anomaly data, represents the number of structured projection embedded coding vectors of power anomaly data in the set of structured projection embedded coding vectors of power anomaly data, Represents the initial linear cluster center encoding vector of power anomaly data.

[0030] Specifically, by calculating the spatial baseline average of a vector set, global statistical features are automatically extracted based on the data's inherent distribution characteristics, eliminating local interference from single data points and providing an objective benchmark for subsequent analysis. The resulting initial linear cluster center encoding vectors for power anomaly data serve as the global feature skeleton, revealing the primary linear patterns in the data and providing a global overview of the data distribution. This lays the foundation for subsequent compensation and correction operations, allowing subsequent steps to focus on processing nonlinear information and more accurately adapt to the dynamic characteristics of power anomaly data.

[0031] Figure 5 FIG is a flowchart of sub-step S132 of the power data cleaning method based on the power big data model according to an embodiment of the present application. Figure 5As shown, the step S132 includes the steps of: S1321, calculating the characteristic clustering compensation incremental operator of each power abnormality data structured projection embedded coding vector in the set of the power abnormality data structured projection embedded coding vector relative to the power abnormality data initial linear clustering center coding vector to obtain a set of power abnormality data clustering compensation incremental operators; S1322, based on the set of power abnormality data clustering compensation incremental operators, calculating the linear clustering compensation component coding vector of the set of the power abnormality data structured projection embedded coding vector to obtain the power abnormality data linear clustering compensation component coding vector; S1323, fusing the power abnormality data linear clustering compensation component coding vector and the power abnormality data initial linear clustering center coding vector to obtain the power abnormality data aggregation analysis modeling vector.

[0032] In a specific example of the present application, step S1321 includes: first, constructing a deep collaborative implicit coding vector between each power abnormality data structured projection embedded coding vector in the set of the power abnormality data structured projection embedded coding vector and the power abnormality data initial linear clustering center coding vector to obtain a set of power abnormality data deep collaborative implicit coding vectors, which is expressed as follows: in, represents the Sigmoid function, represents the activation encoding vector of the initial linear clustering center of power anomaly data, represents the bias term, represents the weight matrix, Represents a cascade function, The first vector in the set of deep collaborative implicit coding vectors representing power anomaly data Deep collaborative implicit coding vector of power anomaly data.

[0033] That is, a dynamic relationship mapping between data points and cluster centers is established, and a nonlinear compensation mechanism between the structured projection embedding coding vector of power anomaly data and the initial linear clustering center coding vector of power anomaly data is learned through a deep model. Specifically, the structured projection embedding coding vector of power anomaly data and the initial linear clustering center coding vector of power anomaly data are input into a deep neural network. The model generates a deep implicit feature coding for characterizing the implicit relationship between the two by mining the implicit associations between the two in dimensions such as temporal dependency and anomaly propagation path, that is, a set of deep collaborative implicit coding vectors of power anomaly data. In this way, the nonlinear offset of each structured projection embedding coding vector of power anomaly data relative to the initial linear clustering center coding vector of power anomaly data in high-dimensional space can be effectively captured, providing a nonlinear feature representation that is closer to the actual distribution of the data for subsequent feature compensation, thereby improving the accuracy of feature aggregation analysis.

[0034] Then, each power anomaly data deep collaborative implicit coding vector in the set of the power anomaly data deep collaborative implicit coding vector is evaluated with the power anomaly data initial linear cluster center coding vector to obtain the set of the power anomaly data clustering compensation increment operator, which is expressed as follows: in, represents the logarithmic function with base 2, express Middle The eigenvalues ​​at the positions, express Middle The eigenvalues ​​at the positions, The characteristic scale of the deep collaborative implicit coding vector of power anomaly data, express The corresponding power anomaly data clustering compensation increment operator.

[0035] Specifically, by calculating the differentiated features of the deep collaborative implicit coding vector of power anomaly data and the coding vector of the initial linear clustering center of power anomaly data, compensation weights positively correlated with the anomaly intensity are generated for non-steady-state anomalies such as sudden load spikes, thereby achieving a balance between anomaly sensitivity and noise resistance through a dynamic compensation mechanism. The resulting set of incremental compensation operators for power anomaly data clustering is used to characterize the unique nonlinear characteristics of each deep collaborative implicit coding vector of power anomaly data outside the linear clustering framework. This helps to form personalized compensation strategies for different data points, capture the complex nonlinear structure of local regions in the dataset, and thus improve the adaptability of feature representation to the dynamic nonlinear patterns of power anomaly data.

[0036] Here, when calculating the power anomaly data clustering compensation increment operator, due to the structured projection embedding coding vector in the power anomaly data In addition to the representation of , the initial linear clustering center encoding vector based on power anomaly data is added The linear clustering results of the incremental nonlinear information result in the superposition of elements within the clustering system, which leads to systematic imbalance in spatial distribution. In a preferred example of the present application, step S1322 includes: first, performing resonant interaction reinforcement based on cluster balance correction on each power anomaly data clustering compensation incremental operator in the set of power anomaly data clustering compensation incremental operators to obtain a set of optimized power anomaly data clustering compensation incremental operators.

[0037] Specifically, we first use the power anomaly data structured projection embedding encoding vector and the initial linear clustering center encoding vector of power anomaly data The inner product of Constructing joint feature clustering energy factor , and then calculate the cooperative dynamic convergence coefficient : in, Indicates the calculation of the inner product, Represents a vector The variance of express and The joint feature clustering energy factor between Representation and joint feature clustering energy factor The related intrinsic diffusion energy factor, for The collaborative dynamic convergence coefficient of For the degree of convergence of linear cluster centers, is the length of the eigenvector.

[0038] Finally, consider the joint feature clustering energy factor Related cooperative dynamic convergence coefficient The energy relaxation response under the scattering relation is given by To perform resonant coupling based on energy relaxation driving on the fractal structure in cluster space, represents an exponential function with a natural constant as the base, It represents the optimization of the power abnormal data clustering compensation incremental operator to achieve the resonance coupling enhancement of the feature clustering compensation incremental operator under the cluster space fractal structure representation, thereby obtaining the cluster equilibrium state correction effect of nonlinear incremental coupling.

[0039] Then, the set of optimized power abnormality data clustering compensation increment operators is normalized based on the Softmax function to obtain a set of normalized power abnormality data clustering compensation increment operators, which is expressed as follows: in, represents the softmax function, express The corresponding normalized power anomaly data clustering compensation increment operator.

[0040] That is, the Softmax function is used to convert each optimized power anomaly data clustering compensation incremental operator into a weight coefficient in the form of a probability distribution, so that the weight of each operator is in the range of 0-1 and the sum is 1, thereby eliminating the interference of numerical deviation on the aggregation process and highlighting the weight of compensation information that is more relevant to the true distribution of the data. The set of normalized power anomaly data clustering compensation incremental operators obtained in this way can quantify the importance of each optimized power anomaly data clustering compensation incremental operator in the global feature aggregation in the form of probability weight, suppress the influence of noise and redundant information, and enable subsequent weighted aggregation operations to more accurately extract common and global nonlinear compensation patterns, forming an effective supplement to the linear clustering results.

[0041] Finally, based on the set of normalized power anomaly data clustering compensation increment operators, the set of power anomaly data structured projection embedding code vectors is weightedly aggregated to obtain the power anomaly data linear clustering compensation component code vector, which is expressed as follows: in, Represents the linear clustering compensation component encoding vector of power anomaly data.

[0042] Specifically, the normalized power anomaly data clustering compensation increment operator is used to perform weighted aggregation on the structured projection embedding code vector of the power anomaly data. The scattered individual nonlinear compensation features are screened and integrated according to their importance, extracting common compensation patterns that reflect the overall deviation of the entire dataset from the linear clustering structure, and obtaining the linear clustering compensation component code vector for the power anomaly data. In this way, by weight allocation, redundant noise is suppressed and core nonlinear features are highlighted, achieving an abstract generalization from personalized compensation of individual data points to the global compensation pattern of the dataset. This provides a structurally balanced and feature-rich nonlinear compensation component for subsequent fusion with the initial linear clustering center code vector, ensuring that the final aggregate analysis modeling vector combines the global framework of the linear structure with the local correction capability of nonlinear details.

[0043] In a specific example of the present application, step S1323 includes: calculating the position-weighted sum between the linear clustering compensation component encoding vector of the power anomaly data and the initial linear clustering center encoding vector of the power anomaly data to obtain the power anomaly data aggregation analysis modeling vector, which is expressed as: in, and Represents different weight parameters, Represents the power anomaly data aggregation analysis modeling vector.

[0044] That is, by weighting the sum by position, the global linear structure information represented by the initial linear clustering center encoding vector of the power anomaly data and the nonlinear detail correction information contained in the linear clustering compensation component encoding vector of the power anomaly data are organically combined. On the basis of retaining the macroscopic linear distribution framework of the data, local nonlinear feature deviations are incorporated to form a power anomaly data aggregation analysis modeling vector with both structural stability and detail adaptability. It not only retains the global structural characteristics of the initial clustering center, but also integrates the detail optimization brought by nonlinear compensation, so that the feature representation covers the multi-dimensional characteristics of the data more comprehensively, and provides comprehensive information support including data distribution laws and dynamic anomaly characteristics for the subsequent decoding optimal parameter setting.

[0045] Specifically, in one specific example of the present application, step S14 includes inputting the power anomaly data aggregation and analysis modeling vector into an optimal parameter decoder based on a multi-layer perceptron structure to obtain the optimal parameter settings. That is, a nonlinear mapping between high-dimensional features and target output parameters is established through the decoding network, achieving end-to-end parameter self-optimization. Specifically, the optimal parameter decoder is based on a multi-layer perceptron structure, which is composed of multiple fully connected layers, each of which uses a nonlinear activation function to enhance the model's expressiveness. The input layer receives the power anomaly data aggregation and analysis modeling vector, then learns the nonlinear anomaly patterns of the power data through layer-by-layer transmission and transformation. Ultimately, the output layer generates optimal parameter settings adapted to the current power anomaly data scenario, such as the smoothing parameter h and the sampling period Δt. This achieves intelligent configuration of optimal parameters and optimizes the temporal granularity of power data acquisition. In this way, the present application can automatically adapt to different types of power anomaly data without the need for manually pre-setting parameter rules, significantly improving the flexibility and accuracy of parameter configuration and avoiding feature aliasing caused by fixed-period sampling (e.g., loss of peak information when sampling short-period data with long-period sampling).

[0046] In the above-mentioned power data cleaning method based on the power big data model, step S2 collects load data according to the optimal parameter settings and based on the Pearson correlation coefficient. It should be understood that during power load data collection, historical load data may exhibit strong correlations (e.g., load curves for consecutive working days). Direct collection can easily lead to data redundancy. Furthermore, directly using strongly correlated data for missing value repair essentially fills gaps with similar data, which can easily lead to repair bias. For example, strongly correlated data may carry the same noise or abnormal patterns as the target data. For example, if a substation experiences load peaks for two consecutive days due to rising temperatures, using data from adjacent days to repair today's missing values ​​will synchronously copy the noise from adjacent days (e.g., accidental equipment fluctuations) into the repair results, resulting in cumulative errors. Therefore, by using the Pearson correlation coefficient to quantify the similarity of load data and screening historical data sets that are weakly correlated with the target data, redundant data can be avoided while providing more differentiated reference samples for missing value repair. Specifically, first, the data sampling density is determined using the optimal parameters (such as Δt = 15min) to balance data integrity and storage costs. Then, for all historical daily load data sets (set as N), the Pearson correlation coefficients between each two are calculated, and an N×N matrix is ​​constructed based on the Pearson correlation coefficient. Next, based on the correlation coefficient threshold of the target data, the historical data set with the weakest correlation with the target data is selected to fill in the missing values ​​and avoid the bias caused by "inbreeding repair". Then, a univariate regression model is constructed (such as ), using the characteristic variables of weakly correlated data as independent variables and the missing points of the target data as dependent variables to calculate the values ​​to be filled. If the repair is incomplete at one time, the calculation is repeated using the less weakly correlated data until the data is complete.

[0047] In the above-mentioned power data cleaning method based on the power big data model, the step S3, based on the power data model, classifies and stores the collected load data using attributes. It should be understood that power data has multiple sources (such as voltage, current, power) and multi-level processing requirements (such as real-time monitoring, historical analysis). By constructing a layered power data model, data can be classified and structured by attributes, so as to facilitate efficient data management and provide standardized input for subsequent data cleaning. It is worth mentioning that the classification and storage of the collected load data using attributes based on the power data model involved in this application can use the method disclosed in Chinese patent CN117763284A to classify and store load data. Of course, other equivalent classification and storage methods can also be used, and this is not limited to this application.

[0048] Specifically, to effectively manage and utilize this load data, it is particularly important to classify it according to its attributes. Attributes include, but are not limited to, timestamps, geographic location information, device type, and specific data types (such as the aforementioned voltage, current, and power). By identifying and defining these attributes, the raw data can be grouped according to specific criteria. For example, data can be divided into real-time and historical data based on the time of collection, or data from different regions can be stored separately based on geographic location. This attribute-based classification approach helps improve data retrieval efficiency and simplify the data analysis process.

[0049] When it comes to storing load data, choosing the right storage solution is crucial, given the large and rapidly growing volume of data. One viable approach is to use a distributed database system, which offers excellent scalability and fault tolerance, making it suitable for processing large datasets. In a distributed database, data is stored across multiple nodes, with each node responsible for processing a portion of the data. This approach not only speeds up data processing but also ensures that the entire system remains operational even if a node fails.

[0050] During implementation, the storage of load data also requires consideration of data format consistency and standardization. A unified data format helps reduce errors during data conversion and facilitates data exchange between different systems. To this end, a detailed set of data specifications can be developed to clarify encoding rules and unit representation for various data types. Furthermore, during storage, necessary metadata information, such as the time and source of data generation, must be added to each record to enhance data traceability and usability.

[0051] Furthermore, ensuring data security and privacy protection is equally crucial. As the digitalization of power systems continues to increase, the risk of data leakage also increases. Therefore, when designing load data classification and storage solutions, effective security measures, such as encryption technology and access control mechanisms, must be implemented to ensure that sensitive information is properly protected. Furthermore, compliance with relevant laws and regulations and respect for user privacy are prerequisites for any data processing. Load data classification and storage strategies must be adjusted to meet the specific needs of different application scenarios. For example, in power grid dispatching scenarios, greater emphasis may be placed on the rapid acquisition and processing of real-time data; whereas, in long-term planning studies, the focus is on the accumulation and in-depth analysis of historical data.

[0052] In the aforementioned power data cleaning method based on the power big data model, step S4 cleans the classified and stored load data using a time series data similarity metric, completing the power data cleaning process. Specifically, since the classified and stored load data may still contain residual anomalies (such as unlabeled transition points and periodic noise), further cleaning is required using a similarity metric. To this end, the initial load data time series is first normalized to eliminate dimensionality effects. Next, a symbol set approximation algorithm is used to convert the normalized sequence into a symbol sequence (for example, using "A" to represent an upward trend and "B" to represent a stationary state) to reduce computational complexity. A similarity matrix is ​​then generated by calculating the Euclidean distance between data from the same device on different dates. A similarity curve is then plotted based on the matrix, with the horizontal axis representing time points and the vertical axis representing similarity values ​​(ranging from 0 to 1). Furthermore, a similarity threshold (such as 0.8) is set, and points below the threshold are identified as anomalies (such as transition data). Abnormal points are corrected by interpolating neighboring normal data, predicting regression models, or directly deleting redundant points, so as to accurately identify and eliminate / correct abnormal data, improve data integrity and reliability, and maximize data value.

[0053] During implementation, standardization is achieved by subtracting the mean from each data set and then dividing it by the standard deviation, ensuring that each data set has the same mean of 0 and variance of 1. Next, the standardized sequence is converted into a symbolic sequence using the Symbolic Aggregate approXimation (SAX) algorithm. The SAX algorithm is an effective dimensionality reduction technique that converts continuous time series into discrete symbolic sequences. In this process, the numerical trend within each time period is simplified to a symbol. For example, an upward trend can be represented by "A," a stationary trend by "B," and a downward trend by "C." This conversion not only reduces computational complexity but also preserves the key characteristics of the original time series, facilitating subsequent analysis and processing.

[0054] After symbolization, the similarity calculation phase begins. For data collected on different days by the same device, the Euclidean distance is calculated to measure similarity. Based on these calculation results, a similarity matrix is ​​generated, where each element represents the similarity score between the data at two specific time points. A similarity curve is then plotted based on the similarity matrix, with the horizontal axis representing the time point and the vertical axis representing the similarity value, ranging from 0 to 1. This graphical display helps intuitively identify data points that significantly deviate from the normal pattern, namely potential outliers.

[0055] Typically, a threshold of 0.8 is chosen, meaning that when the similarity value falls below this level, the corresponding time point is considered an anomaly. Anomalies here may include subtle issues such as jumpy data and periodic noise. Once an anomaly is identified, appropriate measures must be taken to correct or remove it. Minor anomalies can be corrected by interpolating adjacent normal data; more severe anomalies can be filled in through regression model prediction. Furthermore, in some special cases, simply deleting redundant points is also a viable option.

[0056] Accurately identifying and correcting abnormal data is crucial throughout the data cleansing process. This series of operations not only improves data integrity and reliability, but also maximizes its value. It's important to note that during implementation, careful attention to detail is crucial to ensure that each step accurately reflects the true state of the data. For example, when selecting the appropriate symbol set, data characteristics and business requirements should be fully considered. When setting the similarity threshold, flexible adjustments should be made based on actual circumstances to avoid a one-size-fits-all approach.

[0057] In summary, the power data cleaning method based on the power big data model of the embodiment of the present application is explained. It first obtains a collection of power anomaly data and introduces data processing technology based on deep learning to perform data feature aggregation analysis based on self-learning reinforcement on the power anomaly data set to mine the noise distribution, periodicity, abnormal trend and other key features of the power anomaly data, thereby intelligently decoding the optimal sampling parameter setting of the load data, and then on this basis, by collecting, classifying, storing and cleaning the load data based on the similarity of time series data, the power data cleaning operation is completed. In this way, it can automatically adapt to the dynamic nonlinear characteristics of power anomaly data, improve the reliability of the optimal parameter setting, and thus improve the intelligence level and efficiency of data cleaning.

[0058] Furthermore, a power data cleaning system based on a power big data model is also provided.

[0059] Figure 6 FIG is a block diagram of a power data cleaning system based on a power big data model according to an embodiment of the present application. Figure 6As shown, according to the embodiment of the present application, the power data cleaning system 100 based on the power big data model includes: an abnormal data acquisition and parameter configuration module 110, which is used to acquire a collection of power abnormal data and intelligently configure the optimal parameter settings based on the collection of power abnormal data; a load data acquisition module 120, which is used to collect load data according to the optimal parameter settings and based on the Pearson correlation coefficient; a load data classification and storage module 130, which is used to classify and store the collected load data using attributes based on the power data model; a load data cleaning module 140, which is used to clean the classified and stored load data using a time series data similarity metric to complete the power data cleaning operation. Among them, the abnormal data acquisition and parameter configuration module 110 is used to: perform data feature aggregation analysis based on self-learning reinforcement on the collection of power abnormal data to determine the optimal parameter settings.

[0060] Here, those skilled in the art will understand that the specific operations of each module in the power data cleaning system based on the power big data model have been described above. Figures 1 to 5 The description of the power data cleaning method based on the power big data model has been introduced in detail, and therefore, its repeated description will be omitted.

[0061] The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in the present invention are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present invention. Furthermore, the specific details of the above embodiments are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. These details do not necessarily limit the present invention to being implemented using these specific details.

[0062] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiment described above is only schematic. For example, the unit division is only a logical function division, and there may be other division methods in actual implementation. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0063] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be encompassed therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0064] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units stated in the system claims can also be implemented by one unit through software or hardware.

[0065] Finally, it should be noted that the above description has been provided for purposes of illustration and description. Furthermore, the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to be limiting. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art will appreciate that the technical solutions of the present invention may be modified or replaced with equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A power data cleaning method based on a power big data model, characterized in that: include: Acquire a collection of power anomaly data, and intelligently configure optimal parameter settings based on the collection of power anomaly data; Load data were collected according to the optimal parameter settings and based on the Pearson correlation coefficient; Based on the power data model, the collected load data is classified and stored using attributes; The classified and stored load data is cleaned using time series data similarity measurement to complete the power data cleaning operation; Among them, intelligently configuring optimal parameter settings based on the set of power abnormality data includes: performing data feature aggregation analysis based on self-learning reinforcement on the set of power abnormality data to determine the optimal parameter settings.

2. The power data cleaning method based on the power big data model according to claim 1 is characterized in that: Performing a data feature aggregation analysis based on self-learning reinforcement on the set of power anomaly data to determine the optimal parameter setting includes: Using the power abnormality data embedding coding matrix, embedding coding is performed on each power abnormality data in the power abnormality data set to obtain a set of power abnormality data structured embedding coding vectors; Using a linear projection layer, each power anomaly data structured embedding coding vector in the set of power anomaly data structured embedding coding vectors is mapped to the same latent semantic space to obtain a set of power anomaly data structured projection embedding coding vectors; Performing a feature aggregation analysis based on self-learning reinforcement on the set of structured projection embedded coding vectors of the power anomaly data to obtain a power anomaly data aggregation analysis modeling vector; Feature decoding is performed on the power anomaly data aggregation analysis modeling vector to obtain the optimal parameter setting.

3. The power data cleaning method based on the power big data model according to claim 2 is characterized in that: Performing a feature aggregation analysis based on self-learning reinforcement on the set of structured projection embedded coding vectors of the power anomaly data to obtain a power anomaly data aggregation analysis modeling vector, including: Performing linear cluster analysis on the set of structured projection embedding coding vectors of the power anomaly data to obtain an initial linear cluster center coding vector of the power anomaly data; Based on the feature clustering compensation increment of the set of structured projection embedded coding vectors of the power anomaly data relative to the initial linear clustering center coding vector of the power anomaly data, the initial linear clustering center coding vector of the power anomaly data is compensated and corrected to obtain the power anomaly data aggregation analysis modeling vector.

4. The power data cleaning method based on the power big data model according to claim 3 is characterized in that: Performing linear cluster analysis on the set of structured projection embedding coding vectors of the power anomaly data to obtain an initial linear cluster center coding vector of the power anomaly data, including: The spatial reference average vector of the set of structured projection embedding coding vectors of the power anomaly data is calculated to obtain the initial linear clustering center coding vector of the power anomaly data.

5. The power data cleaning method based on the power big data model according to claim 4 is characterized in that: Based on the feature clustering compensation increment of the set of structured projection embedded coding vectors of the power anomaly data relative to the initial linear clustering center coding vector of the power anomaly data, the initial linear clustering center coding vector of the power anomaly data is compensated and corrected to obtain the power anomaly data aggregation analysis modeling vector, including: Calculating a feature clustering compensation increment operator of each power abnormality data structured projection embedded code vector in the set of power abnormality data structured projection embedded code vectors relative to the power abnormality data initial linear cluster center code vector to obtain a set of power abnormality data clustering compensation increment operators; Based on the set of power abnormality data clustering compensation increment operators, calculating the linear clustering compensation component code vector of the set of power abnormality data structured projection embedding code vectors to obtain the power abnormality data linear clustering compensation component code vector; The power anomaly data linear clustering compensation component encoding vector and the power anomaly data initial linear clustering center encoding vector are fused to obtain the power anomaly data aggregation analysis modeling vector.

6. The power data cleaning method based on the power big data model according to claim 5 is characterized in that: Calculating a feature clustering compensation increment operator of each power anomaly data structured projection embedded code vector in the set of the power anomaly data structured projection embedded code vector relative to the power anomaly data initial linear cluster center code vector to obtain a set of power anomaly data clustering compensation increment operators, including: Constructing a deep collaborative implicit coding vector between each power anomaly data structured projection embedded coding vector in the set of power anomaly data structured projection embedded coding vectors and the power anomaly data initial linear clustering center coding vector to obtain a set of power anomaly data deep collaborative implicit coding vectors; Each power anomaly data deep collaborative implicit coding vector in the set of the power anomaly data deep collaborative implicit coding vector is evaluated with the power anomaly data initial linear clustering center coding vector based on characteristic grouping differences to obtain a set of the power anomaly data clustering compensation increment operators.

7. The power data cleaning method based on the power big data model according to claim 6 is characterized in that: Based on the set of power abnormality data clustering compensation increment operators, calculating the linear clustering compensation component code vector of the set of power abnormality data structured projection embedding code vectors to obtain the power abnormality data linear clustering compensation component code vector, including: Performing a resonant interaction reinforcement based on cluster balance correction on each power abnormality data cluster compensation incremental operator in the set of power abnormality data cluster compensation incremental operators to obtain an optimized set of power abnormality data cluster compensation incremental operators; performing a normalization process based on a Softmax function on the set of optimized power abnormality data clustering compensation increment operators to obtain a set of normalized power abnormality data clustering compensation increment operators; Based on the set of normalized power anomaly data clustering compensation increment operators, weighted aggregation is performed on the set of power anomaly data structured projection embedding code vectors to obtain the power anomaly data linear clustering compensation component code vector.

8. The power data cleaning method based on the power big data model according to claim 7 is characterized in that: The power anomaly data linear clustering compensation component encoding vector and the power anomaly data initial linear clustering center encoding vector are integrated to obtain the power anomaly data aggregation analysis modeling vector, including: A position-weighted sum between the power anomaly data linear cluster compensation component encoding vector and the power anomaly data initial linear cluster center encoding vector is calculated to obtain the power anomaly data aggregation analysis modeling vector.

9. The power data cleaning method based on the power big data model according to claim 8 is characterized in that: Performing feature decoding on the power anomaly data aggregation analysis modeling vector to obtain the optimal parameter setting includes: The power anomaly data aggregation analysis modeling vector is input into an optimal parameter decoder based on a multi-layer perceptron structure to obtain the optimal parameter setting.

10. A power data cleaning system based on a power big data model, characterized in that: include: An abnormal data acquisition and parameter configuration module, used to acquire a collection of power abnormal data and intelligently configure optimal parameter settings based on the collection of power abnormal data; A load data acquisition module is used to collect load data according to the optimal parameter settings and based on the Pearson correlation coefficient; The load data classification and storage module is used to classify and store the collected load data using attributes based on the power data model; The load data cleaning module is used to clean the classified and stored load data using the time series data similarity measurement to complete the power data cleaning operation; The abnormal data acquisition and parameter configuration module is used to perform data feature aggregation analysis based on self-learning reinforcement on the set of abnormal power data to determine the optimal parameter settings.

Images