Data management method and system for distributed new energy system

Abstract

The embodiment of the invention discloses a data management method and system for a distributed new energy system, which can solve the problems of data missing and short-term data missing and effectively improve the integrity and accuracy of data. The method comprises the steps of S1, data acquisition, S2, data missing identification, S3, data anomaly identification, S4, intelligent filling and S5, reasonable verification. The method comprises the following steps: firstly, for all connection lines between nodes in the distributed new energy system, periodically carrying out trend feature analysis on a time sequence corresponding to electric quantity data on each line, and carrying out data missing identification; secondly, extracting abnormal data features by using a DBSCAN algorithm, and then analyzing the extracted abnormal data features by using an abnormal data joint detection method of adaptive binning and isolated forest to determine abnormal data; and finally, performing multiple interpolation on missing and abnormal data by using a GAN (Generative Adversarial Network) technology and a joint distribution method to obtain prediction data so as to complete data governance.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of new energy and data processing technology, and in particular, to a data governance method and system for a distributed new energy system. Background Art

[0002] The new energy system covers a variety of energy types such as wind energy and solar energy. Since the generation and operation of these energy sources involve a variety of equipment and complex environmental conditions, new energy data is highly diverse and complex. The frequency and accuracy of data from different sources vary greatly. Artificial intelligence-based intelligent governance technology for new energy data has become a key way to solve these problems.

[0003] In the new energy system, missing and abnormal data may cause the system to lack reliable information support at critical moments. Data mutations will mislead the prediction model, leading to scheduling decision errors, and thus affecting the stable operation of the power grid. New energy power generation is an important part of the new power system, mainly including photovoltaic power generation, wind power generation and biomass power generation. The scale of new energy power generation data is rapidly expanding, and the types of abnormal data are also increasing. The penetration of distributed new energy poses a major challenge to the security and stability of smart grids.

[0004] The problem of missing operating data on the renewable energy power generation side is caused by a variety of reasons, including measurement equipment failure, communication network problems, data storage failure, bad weather and human factors. Measurement equipment failure and sensor aging lead to interrupted data collection; imperfect communication network coverage and data packet loss lead to incomplete data transmission; server failure and storage device damage lead to data failure; extreme weather damages equipment or interrupts power supply; human error and improper maintenance can also cause data loss. Solving these problems requires comprehensive technical means and management measures to ensure the integrity and reliability of data, thereby improving the operating efficiency and stability of renewable energy power generation systems. Summary of the invention

[0005] In order to overcome the above-mentioned technical defects, the embodiments of the present application provide a data governance method and system for a distributed new energy system, which can effectively reduce accidental errors caused by random factors and improve data accuracy.

[0006] In a first aspect, an embodiment of the present application provides a data governance method for a distributed new energy system, comprising the following steps:

[0007] S1. Data collection: Use intelligent sensors to collect power data related to the distributed new energy system, including: voltage, current, frequency, power factor data in the distributed new energy system, load data of the distributed new energy distribution network, and operation data of the power grid to which the distributed new energy system is connected;

[0008] S2. Data missing identification: For all connection lines between nodes in the distributed new energy system, periodically analyze the trend characteristics of the time series corresponding to the power data on each line to obtain a difference sequence representing the trend characteristics. Data with sudden changes or jumps in the difference sequence are identified as missing data and the missing positions are marked;

[0009] S3. Data anomaly identification: First, use the DBSCAN algorithm to extract abnormal data features in the power data. The abnormal data features include numerical features, categorical features, and time series features. Then, use the abnormal data joint detection method of adaptive binning and isolation forest to analyze the extracted abnormal data features to obtain the abnormal score value, and determine the data with an abnormal score value less than the threshold as abnormal data, and mark the abnormal position.

[0010] S4. Intelligent filling of missing data and abnormal data: For the identified missing data and abnormal data, the generative adversarial network (GAN) technology is used to approximate the real data distribution through adversarial training of the generator and the discriminator, and generate filling values ​​for multiple interpolation data; the joint distribution method is used to perform multiple interpolation of the filling values ​​to the corresponding marked abnormal positions and missing positions to obtain the predicted data;

[0011] S5. Reasonableness check: After completing intelligent interpolation, use the rule base to perform data reasoning to check the rationality of the predicted data. The data verification of the rule base is completed by analyzing the statistical characteristics of historical data, data distribution and physical constraints.

[0012] Optionally, in a possible implementation of the first aspect,

[0013] The time series corresponding to the power data on each line is recorded as x n The trend characteristics include an upward trend characteristic and a downward trend characteristic. The specific steps of the trend characteristic analysis include:

[0014] S2.1. Calculate the time series x n The corresponding moving average sequence A, n represents each moment in the time period, and the i-th element Ai in the moving average sequence A can be expressed as:

[0015]

[0016] Among them, X j is the jth data point in the original time series, m is the window size of the moving average, and i is the position of the current moving average;

[0017] S2.2. Calculate the time series x n The difference sequence diff of the corresponding moving average sequence, where:

[0018] S2.3. Calculate the time series x based on the difference sequence diff n Corresponding to the upward trend characteristics and downward trend characteristics of the running data curve.

[0019] Optionally, in a possible implementation manner of the first aspect, the specific step of extracting the abnormal data features includes:

[0020] S3.1. First, use an autoencoder to encode the original data from high-dimensional data into a low-dimensional representation, and then decode the low-dimensional representation back to the high-dimensional space to obtain reconstructed data. In the above encoding and decoding process, extract abnormal data features;

[0021] The training of the autoencoder uses the mean square error MSE as the loss function to measure the difference between the reconstructed data and the original data. The loss function is as follows:

[0022]

[0023] Where n is the number of samples, input i and decoded i are the i-th original data and reconstructed data respectively.

[0024] Optionally, in a possible implementation manner of the first aspect, when the abnormal data feature is a numerical feature, the specific steps of using the abnormal data joint detection method for analysis include:

[0025] S3.2. Use adaptive binning technology to determine the number and range of bins according to data distribution, and calculate the statistical characteristics of the data in each bin. The statistical characteristics include: mean, median and standard deviation; at the same time, use isolation forest to process numerical feature data, divide the data into different subspaces by randomly selecting features and split points, and the abnormal data is isolated in the subspace with fewer steps. Whether the data point is abnormal is determined based on the anomaly score.

[0026] Optionally, in a possible implementation manner of the first aspect, when the abnormal data feature is a categorical feature, the specific steps of using the abnormal data joint detection method for analysis include:

[0027] S3.3. Use adaptive binning technology to analyze and count the frequency of each category, and mark the categories with abnormally low frequency or large differences from the expected distribution as abnormal; at the same time, use isolation forest to analyze the abnormal situation of categorical features under specific values.

[0028] Optionally, in a possible implementation manner of the first aspect, when the abnormal data feature is a time series feature, the specific steps of using the abnormal data joint detection method for analysis include:

[0029] S3.4. Use adaptive binning technology to analyze the characteristics of time period data, and use isolation forest to process time point data and detect mutation points or abnormal fluctuations.

[0030] Optionally, in a possible implementation manner of the first aspect, the specific step of using the abnormal data joint detection method for analysis further includes:

[0031] S3.5. For multiple anomaly detection results of different types of features in the joint analysis of adaptive binning technology and isolation forest, a weighted fusion method is used and weights are assigned according to the importance of the features to comprehensively judge whether the data points are abnormal.

[0032] Optionally, in a possible implementation manner of the first aspect, the specific step of performing multiple interpolation using the joint distribution method includes: an interpolation step and a posterior distribution estimation step, wherein the two steps are iterated to interpolate missing data and estimate parameters to complete multiple interpolation;

[0033] S4.1. Interpolation step: Generate interpolation values ​​of missing variables by random sampling, and establish joint conditional distribution based on non-missing variables according to the missing variables, which is expressed as:

[0034]

[0035] in, represents the missing data at time point t, Y obs represents the observed data, θ is the parameter of the model, and θ t-1 represents the model parameters at the t-1th iteration, P(Y mis |Y obs ,θ t-1 ) means that given the observed data Y obs and parameter θ t-1 Under the condition that the missing data Y mis The conditional probability distribution of ;

[0036] S4.2, posterior distribution estimation step: re-estimate the conditional distribution parameters based on the interpolation values ​​obtained in the interpolation step, and its expression is:

[0037]

[0038] Among them, θ is the parameter of the new energy prediction data model, t usually represents the number of iterations or a specific time step, Yobs represents the observed data, represents the missing data at time point t, Represents the missing data estimate given the observed data Yobs and the t-th iteration The conditional probability distribution of parameter θ under the condition of .

[0039] Optionally, in a possible implementation manner of the first aspect, the specific steps of using the rule base to perform data reasoning include:

[0040] S5.1. Place the input data into a dynamic database;

[0041] S5.2. Use the rules in the rule base to match the data in the dynamic database. If the rule is successfully matched, the flag corresponding to it will be marked as 0, otherwise it will be marked as 1;

[0042] S5.3. If the rule base is empty, the data in the dynamic database is stored in the corresponding data table, and the corresponding table entry is added to the fact base table, and the reasoning ends;

[0043] S5.4. Determine whether all rule marks are 0. If so, store the data in the dynamic database into the corresponding data table, and add the corresponding table entry in the fact database table, and the reasoning ends; otherwise, generate a corresponding error prompt according to the item marked as 1, and wait for re-entry into the dynamic database.

[0044] In a second aspect, an embodiment of the present application provides a data management system for a distributed new energy system, including: a smart sensor, a data collection layer, a data analysis layer, and a data intelligent management layer;

[0045] The data collection layer performs the following steps:

[0046] S1. Data collection: Use intelligent sensors to collect electricity data related to distributed new energy systems, including voltage, current, power, output data in distributed new energy systems, load data of distributed new energy distribution networks, and operation data of the power grid to which distributed new energy systems are connected;

[0047] The data analysis layer performs the following steps:

[0048] S2. Data missing identification: For all connection lines between nodes in the distributed new energy system, periodically analyze the trend characteristics of the time series corresponding to the power data on each line to obtain a difference sequence representing the trend characteristics. Determine the data with sudden changes or jumps in the difference sequence as missing data and mark the missing position;

[0049] S3. Data anomaly identification: First, use the DBSCAN algorithm to extract abnormal data features in the power data. The abnormal data features include numerical features, categorical features, and time series features. Then, use the abnormal data joint detection method of adaptive binning and isolation forest to analyze the extracted abnormal data features to obtain the abnormal score value, and determine the data with an abnormal score value less than the threshold as abnormal data, and mark the abnormal position.

[0050] The data intelligence governance layer performs the following steps:

[0051] S4. Intelligent filling of missing data and abnormal data: For the identified missing data and abnormal data, the generative adversarial network (GAN) technology is used to approximate the real data distribution through adversarial training of the generator and the discriminator, and generate filling values ​​for multiple interpolation data; the joint distribution method is used to perform multiple interpolation of the filling values ​​to the corresponding marked abnormal positions and missing positions to obtain the predicted data;

[0052] S5. After completing intelligent interpolation, use the rule base to perform data reasoning to verify the rationality of the predicted data. The data verification of the rule base is completed by analyzing the statistical characteristics of historical data, data distribution and physical constraints.

[0053] In a third aspect, an embodiment of the present application provides a computer device, including:

[0054] at least one processor; and

[0055] a memory communicatively coupled to the at least one processor;

[0056] Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect and any one of its implementations.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, the method described in the first aspect and any one of its implementation methods is implemented.

[0058] In a fifth aspect, a computer program product is provided in an embodiment of the present application, including a computer program, which, when executed by a processor, implements the method described in the first aspect and any one of its implementation methods.

[0059] The technical scheme adopted in the embodiment of the present application can achieve the following technical effects: first, for all connection lines between nodes in the distributed new energy system, the time series corresponding to the power data on each line is periodically analyzed for trend characteristics; secondly, the DBSCAN algorithm is first used to extract and classify the abnormal data features in the power data, and then the abnormal data joint detection method of adaptive binning and isolation forest is used to analyze the extracted abnormal data features to obtain the abnormal score value, and the data with an abnormal score value less than the threshold is determined as abnormal data; through the above steps, the missing pattern of the data can be effectively identified and classified, and the generative adversarial network GAN technology combined with the joint distribution method is used again to intelligently fill in the abnormal data; finally, the rule library is used to perform data reasoning and the rationality of the predicted data is checked to ensure that the interpolated data is reasonable, thereby solving the problems of data missing and short-term data missing, and effectively improving the integrity and accuracy of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0061] Figure 1 This is a schematic diagram of an overall process of the data management method for a distributed new energy system in Example 1;

[0062] Figure 2 A schematic diagram of a recognition process for data missing recognition in Example 2;

[0063] Figure 3 This is a schematic diagram of the unsupervised clustering principle in Example 3;

[0064] Figure 4 A schematic diagram of a recognition process of data anomaly recognition in Example 3;

[0065] Figure 5 Schematic diagram of the network structure of the autoencoder in Example 3;

[0066] Figure 6 This is a schematic diagram of the abnormal data joint detection process in Example 3;

[0067] Figure 7 This is a schematic diagram of the isolation forest structure in Example 3;

[0068] Figure 8 A schematic diagram of a filling process for intelligently filling missing data and abnormal data in Embodiment 4;

[0069] Fig. 9A schematic diagram of a model structure of a generative adversarial network in Example 4;

[0070] Fig.10 A schematic diagram of an interpolation process of the data multiple interpolation method in Embodiment 4;

[0071] Fig.11 A schematic diagram of a verification process for rationality verification in Embodiment 5;

[0072] Fig.12 A schematic diagram of the principle of the rule base in the rationality check in the fifth embodiment;

[0073] Fig.13 This is a schematic diagram of the architecture of a data management system for a distributed new energy system in Example 6;

[0074] Fig.14 A hardware structure diagram of a computer device suitable for implementing the data governance method in Example 7. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0076] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0077] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0078] To facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below:

[0079] Embodiment 1

[0080] See also Figure 1 The data governance method for a distributed new energy system in an embodiment of the present application is described, including the following steps:

[0081] S1. Data collection: Use smart sensors to collect power data related to distributed new energy systems.

[0082] Specifically, the electricity data includes: voltage, current, frequency, power factor data in the distributed new energy system, load data of the distributed new energy distribution network, and operation data of the power grid to which the distributed new energy system is connected.

[0083] Specifically, it can be achieved by installing sensors on new energy power equipment and its power grid system to regularly capture various electrical quantity parameters that reflect the dynamic changes of distributed new energy power systems.

[0084] For example, a sensor array is configured in a group of wind turbines. It not only collects the electrical parameter information of each independent power generation unit (such as output power, etc.), but also records the real-time load fed by the cluster to the local power grid. At the same time, it feeds back the operation status of the power system to monitor the status of the power distribution network, thereby ensuring the stable operation of the power system, improving the efficiency of the new energy system and ensuring that users enjoy high-quality energy supply services.

[0085] S2. Data missing identification: For all connection lines between nodes in the distributed new energy system, the trend feature analysis is performed periodically on the time series corresponding to the power data on each line to obtain a difference sequence that represents the trend feature. Data with mutations or jumps in the difference sequence is identified as missing data and the missing position is marked.

[0086] In the S2 data, the electrical data of consecutive time series points are compared to determine whether there is a discontinuity (such as the sudden loss of information within a certain time and the inability to read the data), and then the location of the missing information is marked for subsequent processing. If the operation record of a wind power generation equipment is found to be interrupted within a specific period of time, this method can quickly locate the specific time period when the problem occurred, and make annotations for subsequent remediation.

[0087] S3. Data anomaly identification: First use the DBSCAN algorithm to extract abnormal data features in the power data; then use the abnormal data joint detection method of adaptive binning and isolation forest to analyze the extracted abnormal data features to obtain the abnormal score value, and determine the data with anomaly score value less than the threshold as abnormal data, and mark the abnormal location.

[0088] Specifically, the abnormal data features include numerical features, categorical features, and time series features. The DBSCAN cluster analysis method is used to preliminarily extract the outlier data features. The adaptive binning and isolation forest joint detection method of abnormal data is further used to comprehensively determine and identify the data that exceeds the conventional values ​​or behaves abnormally, and finally obtain the locations of all abnormal points. For example, when a household measured by a smart meter consumes much more electricity than the normal value during the night period (which may be caused by problems such as inflated consumption caused by equipment failure), the system will define this data that does not conform to common sense as a problem record.

[0089] S4. Intelligently fill in missing data and abnormal data: For the identified missing data and abnormal data, use the generative adversarial network (GAN) technology to approximate the real data distribution through adversarial training of the generator and the discriminator, and generate filling values ​​for multiple interpolation data; use the joint distribution method to multiple interpolate the filling values ​​to the corresponding marked abnormal positions and missing positions to obtain the predicted data.

[0090] After completing the above preparations, we enter the S4 stage, which is defect repair. In this stage, the adversarial neural network system GANS is used to obtain new supplementary data close to the original sample by imitating the normal probability function training to fill the gap area, and the joint distribution method is used to fill the gap, and finally the complete information flow is repaired to provide accurate and reliable data support for the next analysis process.

[0091] S5. Reasonableness check: After completing intelligent interpolation, use the rule base to perform data reasoning to check the rationality of the predicted data. The data verification of the rule base is completed by analyzing the statistical characteristics of historical data, data distribution and physical constraints.

[0092] The last stage, S5, is to use a series of preset judgment criteria to conduct a logical review of the processed numbers to confirm their correctness and authenticity (such as ensuring that the numerical range meets the actual engineering requirements and other details). For example, after completing a set of new power forecast value correction processing procedures, it is necessary to ensure that these data meet physical principles such as the law of conservation of energy and are consistent with records in similar situations over a period of time in the past. In this way, we can effectively make up for the imperfections in the data analysis of traditional distributed new energy systems, ensure the quality and accuracy of the entire process, and help optimize power grid management and effective resource utilization.

[0093] In an embodiment of the present invention, firstly, for all connection lines between nodes in a distributed new energy system, a trend feature analysis is periodically performed on the time series corresponding to the electricity data on each line; secondly, the DBSCAN algorithm is first used to extract and classify abnormal data features in the electricity data, and then the abnormal data joint detection method of adaptive binning and isolation forest is used to analyze the extracted abnormal data features to obtain abnormal score values, and the data with abnormal score values ​​less than the threshold is determined as abnormal data; through the above steps, the missing pattern of the data can be effectively identified and classified, and the generative adversarial network GAN technology combined with the joint distribution method is used again to intelligently fill in the abnormal data; finally, the rule library is used to perform data reasoning and the rationality of the predicted data is checked to ensure that the interpolated data is reasonable, thereby solving the problems of data missing and short-term data missing, and effectively improving the integrity and accuracy of the data.

[0094] Embodiment 2

[0095] See also Figure 2 The specific method flow of data missing identification in the embodiment of the present invention is described.

[0096] The time series is used to analyze the periodicity and trend characteristics of missing voltage, current and power data. Assuming that the data set X contains the operation data of D lines for D days, and the data contains M points per day, the operation data curve of each line can be expressed as an M-dimensional vector The data set can be represented as X = {x n ,n=1,2,……,N}. Simple moving average is used for analysis.

[0097] The time series corresponding to the power data on each line recorded in the first embodiment is denoted as x n , the trend characteristics include upward trend characteristics and downward trend characteristics, then the specific steps of the above trend characteristic analysis include:

[0098] S2.1. Calculate the time series x n The corresponding moving average sequence A.

[0099] Specifically, n represents each moment in the time period, and the i-th element Ai in the moving average sequence A can be expressed as:

[0100]

[0101] Among them, X j is the jth data point in the original time series, m is the window size of the moving average, and i is the position of the current moving average.

[0102] S2.2. Calculate the time series x n The difference sequence diff of the corresponding moving average sequence.

[0103] Specifically,

[0104] S2.3. Calculate the time series x based on the difference sequence diff n Corresponding to the upward trend characteristics and downward trend characteristics of the running data curve.

[0105] The difference sequence (diff) function can calculate the differences between adjacent elements in a time series or data list, so that a new sequence can be obtained in which each element represents the difference between two original data points. When unexplained mutations or jumps appear in the difference sequence, it may indicate that there are missing values ​​in the data.

[0106] Through these steps, the differences between adjacent elements in a time series or data list can be calculated to obtain a new sequence in which each element represents the difference between two original data points. A positive difference usually indicates an upward trend, while a negative difference indicates a downward trend. When unexplained mutations or jumps occur in the difference sequence, it may indicate that there are missing values ​​in the data. Finally, the missing patterns in the new energy operation data are effectively identified and classified. It provides a solid theoretical basis for subsequent data interpolation, thereby improving data integrity and analysis accuracy. Specifically, by calculating the differences between adjacent data points, the difference sequence directly reflects the upward or downward trend of the data, making the amount and trend of data changes more obvious; the difference sequence helps to discover anomalies and missing data, which is crucial to ensuring data quality and reliability of analysis; differential processing can effectively reduce fixed deviations caused by systematic errors, and is suitable for scenarios requiring high precision; by calculating relative change values, the difference sequence provides a more comparable measurement method, which helps to fairly evaluate performance in different situations; by measuring and calculating the difference sequence multiple times, the accidental errors caused by random factors can be effectively reduced, and the data accuracy can be improved.

[0107] Embodiment 3

[0108] See also Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 The specific method flow of data anomaly identification in an embodiment of the present invention is described.

[0109] In the embodiment of the present invention, the data anomaly identification adopts the DBSCAN algorithm (Density-Based Spatial Clustering of Applications with Noise), which is a density-based clustering algorithm. The algorithm adopts the unsupervised clustering principle, and its principle diagram is as follows: Figure 3As shown in the figure, clusters are defined by presetting two key parameters: ε and MinPts. ε represents the distance threshold near a point, while MinPts refers to the minimum number of neighbors required for a point within the distance. The number of neighbors in the ε neighborhood is calculated for each point in the prediction data set. If the number of neighbors is ≥ MinPts, it is marked as a core point; neighbors less than MinPts but belonging to other core points are marked as boundary points; the rest are noise points. After the classification is completed, the clustering process starts from any unvisited core point and merges the points that can be directly reached by density into a cluster. Repeat this operation until all core points are visited. Points that are not included in the cluster and are not core or boundary are considered noise points, which usually represent abnormal situations in abnormal data detection and reflect the unreasonable characteristics of new energy prediction data.

[0110] like Figure 4 As shown, the specific steps of extracting abnormal data features in the first embodiment include:

[0111] S3.1. First, use the autoencoder to encode the original data from high-dimensional data into a low-dimensional representation, and then decode the low-dimensional representation back to the high-dimensional space to obtain the reconstructed data. In the above encoding and decoding process, the abnormal data features are extracted.

[0112] Specifically, in S3.1, the dimensionality reduction of abnormal data features uses an autoencoder to compress high-dimensional data into a low-dimensional representation, i.e., encoding, and then decompress the low-dimensional representation, i.e., decode it back to the high-dimensional space, trying to restore the original data as much as possible, where the low-dimensional representation is regarded as a valuable feature of the data.

[0113] The network structure of the autoencoder is as follows Figure 5 As shown, it specifically includes three processes: encoding, decoding and training, which are as follows:

[0114] 1. Coding:

[0115] The goal of the encoding process is to compress the input data into a smaller hidden representation:

[0116] h=f(W1x+b1), where f is the activation function, W1 is the weight matrix from the input layer to the hidden layer, and b1 is the hidden layer bias. This is achieved through a multi-layer neural network, which reduces the number of neurons layer by layer to capture the main features of the data.

[0117] The encoding process can be expressed as:

[0118] encoded=f encoder (input), where f cncoder It is an encoding function, usually composed of a series of neural network layers, that maps raw data into a low-dimensional space.

[0119] 2. Decoding

[0120] The goal of the decoding process is to decode the hidden representation into the original data. The decoder structure is usually similar to the encoder, but the number of neurons gradually increases, and finally generates an output that matches the input data:

[0121] decoded=f decoder (encoded),

[0122] Among them, f decoder It is the decoding function, which is also composed of neural network layers and aims to restore the compressed data to its original dimension.

[0123] 3. Model training

[0124] The training of the autoencoder uses the mean square error MSE as the loss function to measure the difference between the reconstructed data and the original data. The loss function is as follows:

[0125]

[0126] Where n is the number of samples, input i and decoded i are the i-th original data and reconstructed data respectively.

[0127] In general, the autoencoder achieves effective dimensionality reduction and feature extraction of data through the above encoding and decoding processes. It can not only reduce the complexity of data processing, but also retain the main features of the original data to a certain extent.

[0128] In order to ensure the effectiveness of training and the effective application of results, the goal of MSE is to minimize the loss of all reconstructions. The reason for choosing mean square error in the design of this formula is that it can better measure the deviation between prediction and reality, and when the difference is large, it will give more penalties to larger deviations, so that the model can focus more on those more important or abnormal situations that deviate from the normal state.

[0129] S3.2. When the abnormal data features are numerical features, the adaptive binning technology is used to determine the number and range of bins according to the data distribution, and the statistical features of the data in each bin are calculated. The statistical features include: mean, median and standard deviation; at the same time, the isolation forest is used to process the numerical feature data. The data is divided into different subspaces by randomly selecting features and split points. The abnormal data is isolated in the subspace with fewer steps, and the data point is judged whether it is abnormal based on the anomaly score.

[0130] These steps first involve binning technology and statistical feature analysis, and in subsequent steps, the isolation forest algorithm is used to divide the data subspace to identify and determine abnormal values. First, the number of boxes and their ranges are defined in the adaptive binning process, and the mean, median and standard deviation are estimated based on the values ​​in each box, so that a deeper understanding of data features can be achieved based on the understanding of the information in each box.

[0131] Secondly, the isolation forest algorithm is used, which usually includes randomly selecting several attributes in a series of different sub-dimensions and determining the split positions, so as to achieve effective classification of the information points involved. Abnormal information is generally located at a shallower level, which makes it particularly isolated in a smaller number of sub-divisions; the abnormal scoring performance of the data point under this model is used to determine whether it is an information point with abnormal properties.

[0132] For example, in a method for processing data of distributed new energy systems, if we focus on the quantitative characteristics of the output power of a certain power generation system, this characteristic is obviously a data feature in the form of digital quantity. Therefore, in this case, the discretization technique that is adaptive to the distribution trend of this characteristic can be used to construct a reasonable interval distribution system, and then the central value, central number and deviation from the mean are calculated for each element in each box, and at the same time, the quantitative characteristics are studied with the help of the Lonely Model. In this process, several random features are selected and the splitting base points are set to isolate the data points. For example, for a historical power data set of a new energy site, the box area in the first step may cover multiple power ranges (such as 0-100kW, 101-200kW, etc.), and the average value in each partition can reflect a representative level of this type of value. Then, the divorce tree strategy can be used to find abnormal data that can be separated with only a few splits.

[0133] The abnormal data joint detection method of adaptive binning and isolation forest is adopted. By analyzing different feature types, it is mainly composed of numerical features, categorical features, and time series features. The specific analysis is as shown in S3.3-S3.5.

[0134] S3.3. When the abnormal data feature is a categorical feature, use adaptive binning technology to analyze and count the frequency of each category, and mark the category with abnormally low frequency or large difference from the expected distribution as abnormal; at the same time, use isolation forest to analyze the abnormal situation of categorical features under specific values.

[0135] This step includes using adaptive binning technology to count the frequency of occurrence of different categories of each feature, and further marking those categories with abnormally low significant frequencies or large deviations from the expected data distribution as suspicious abnormal data; secondly, within the same framework, the isolation forest algorithm is also used to detect abnormalities in the category data of a certain feature under specified conditions, thereby improving detection accuracy.

[0136] When executing step S3.3, the adaptive binning technology can dynamically adjust the size or number of boxes according to the characteristics of the categorical features in the actual data (such as frequency, trend changes, etc.), rather than simply using fixed width or the same number of category boxes. Such an adaptive mechanism helps to more accurately discover low-frequency anomalies or trend patterns that deviate from the normal. In this process, categories with extremely low frequencies (such as those below a certain defined standard value or several standard deviations lower than the overall average level) can be identified for further review and confirmation to determine whether they are abnormal conditions such as noise in the real data or input errors.

[0137] In addition, under certain value settings, categorical variables may exhibit abnormal behavior due to the particularity of the data they are associated with. Here, we can use the isolation forest to assess its potential risks. This method is based on building a tree structure to simulate the isolation of normal points and abnormal points for identification. For example, in a data set about distributed energy systems, there may be a certain type of equipment model that appears unusually rarely or all energy consumption levels in a certain area at a certain point in time suddenly mutate. These can be identified and processed in data quality control. The effect of improving detection accuracy is achieved by combining comprehensive detection from two perspectives: frequency and model.

[0138] 3.4 When the abnormal data features are time series features, adaptive binning technology is used to analyze the time period data features, and isolation forest is used to process time point data and detect mutation points or abnormal fluctuations.

[0139] Step S3.4 mentions the use of two core technologies: one is the adaptive binning technology to process the analysis of time period data features; the other is the isolation forest algorithm applied to the analysis of individual time point data and the detection of possible mutations or abnormal fluctuations in the time series. These operations combine statistical modeling methods and machine learning models.

[0140] Specifically, step S3.4 is divided into two parts: the application of adaptive binning technology and the isolation forest algorithm, which work together on data with time features to identify different forms of data anomalies. The former is used to dynamically allocate segments according to specific rules and perform statistical tests within different segments to reveal potential trends and laws, while the latter can identify whether a single data point conforms to the overall distribution and highlight behaviors that do not conform to the trend. Since the isolation forest algorithm is unsupervised, the selection of these values ​​mainly depends on experience adjustment, understanding of the given scenario, and evaluation of experimental results; the ideal state is to maintain accuracy while ensuring detection efficiency.

[0141] 3.5 For multiple anomaly detection results of different types of features in the joint analysis of adaptive binning technology and isolation forest, a weighted fusion method is used and weights are assigned according to the importance of the features to comprehensively judge whether the data points are abnormal.

[0142] Through the above analysis process of the abnormal data joint detection method of adaptive binning and isolation forest, abnormal data in the new energy system can be effectively detected to improve the stability and reliability of the system.

[0143] Different types of data are binned, and the isolation forest model is trained for different bins. Finally, different abnormal data are detected. The process of joint detection of outliers is as follows: Figure 6 As shown, including:

[0144] 1) In outlier detection, the new energy operation data is cleaned and preprocessed, including missing value processing, outlier processing, etc.;

[0145] 2) Divide the key parameters into boxes, analyze the distribution of data in each box, and identify possible abnormal boxes;

[0146] 3) Isolation forest outlier identification and judgment: Integrate the results of binning and isolation forest, and use the isolation forest technology to isolate normal and abnormal data points.

[0147] First, the path length is calculated. For a data set containing n samples, the path length h(x) of a given sample point x in an isolated tree is given. The path length estimation formula is: h(x) = c(m) + e(x), where c(m) = 2H(m-1)-2(m-1) / n, H(i) is the harmonic number, which is approximately ln(i) + 0.5772156649 (Euler constant), m is the number of nodes in the tree (counting from the root node), and e(x) is the number of edges that the sample point x passes through from the root node to the terminal node; secondly, the anomaly score is calculated. For a sample point x in an isolated forest consisting of t isolated trees, the anomaly score calculation formula is: Where E(h(x)) is the average path length of sample point x in t trees, c(n) is a correction term related to the number of samples n, and the abnormality of the sample point is evaluated by building an isolated tree and calculating the path length. The closer the abnormality score is to 1, the more likely it is an abnormal point, and the closer it is to 0, the more likely it is a normal point. The final abnormal point is identified, verified and analyzed to ensure its authenticity. Figure 7 Shown is the tree structure of an isolation forest.

[0148] 4) Comprehensively evaluate data, mark and process outliers, and improve detection accuracy and comprehensiveness.

[0149] In the embodiment of the present invention, after the abnormality is identified, data abnormality monitoring and early warning can also be performed. Specifically, the data abnormality monitoring and early warning adopts the FreeMatch algorithm based on semi-supervised learning to collect the benchmark values ​​during normal operation. According to the collected benchmark values, a reasonable threshold is set. Q1 is the first quartile, that is, the upper limit of the smallest 25% of all values ​​in the data set. Q2 is the second quartile, that is, the point where exactly half of the data points in the data set are less than or equal to its value. Q3 is the third quartile, that is, the value of the data points in the data set that are greater than or equal to 75% of its value. IQR is the interquartile range, that is, IQR = Q3-Q1, and the upper quartile Q1-1.5×IQR is used as the upper abnormal threshold, and the lower quartile Q3+1.5×IQR is used as the lower abnormal threshold. Any data point that falls outside the lower and upper limits is considered an abnormal value, and the cause of the abnormality is analyzed, and the benchmark value and threshold are adjusted to more accurately monitor the abnormality. When an abnormality is confirmed, the system will automatically generate an event and notify the monitoring personnel.

[0150] Embodiment 4

[0151] See also Figure 8 , Fig. 9 and Fig.10 The specific technical solution for intelligently filling missing data and abnormal data in an embodiment of the present invention is described.

[0152] Data interpolation technology handles missing or incomplete data sets. It is inevitable that there are missing values ​​in multiple types of operating data of new energy. In order to maintain the original data distribution and data volume of new energy data, the joint step-by-step method based on multiple interpolation technology is adopted. The joint step-by-step method is a kind of multiple interpolation method. It combines the joint model method and the complete condition method. It is suitable for data sets with hierarchical nested structure characteristics and can effectively handle the interpolation problem of multiple variables with missing data. The joint step-by-step method has some important improvements in new energy systems: First, in new energy systems, missing values ​​may be caused by sensor failures, communication interference and other reasons. By reasonably estimating and filling missing data, the completeness and accuracy of the data are improved, and the reliability of monitoring, control and optimization of the dispatching system is enhanced; secondly, the reliability assessment of new energy systems requires a large amount of operating data. However, missing data may lead to inaccurate reliability assessment results. The joint step-by-step method can provide a more complete data set for reliability assessment by filling missing data, thereby improving the accuracy of the assessment results; thirdly, the control strategy of the new energy system is usually adjusted based on real-time data. Missing data may affect the accuracy and effectiveness of the control strategy. The joint step-by-step multiple imputation technique can provide more accurate input for the control strategy by filling in missing data, thereby optimizing the control performance of the system; finally, data analysis and modeling of new energy systems require a large amount of high-quality data. The joint step-by-step multiple imputation technique can provide a more complete data set for data analysis and modeling, thereby improving the accuracy and reliability of the model.

[0153] The specific steps of using the joint distribution method to perform multiple interpolation in the above-mentioned embodiment 1 include: an interpolation step and a posterior distribution estimation step, and the two steps are iterated to interpolate the missing data and estimate the parameters to complete the multiple interpolation;

[0154] S4.1. Interpolation step: Generate interpolation values ​​for missing variables through random sampling, and establish a joint conditional distribution based on the non-missing variables based on the missing variables.

[0155] Specifically, the expression of joint conditional distribution is:

[0156]

[0157] in, represents the missing data at time point t, Y obs represents the observed data, θ is the parameter of the model, and θ t-1 represents the model parameters at the t-1th iteration, P(Y mis |Y obs ,θ t-1 ) means that given the observed data Y obs and parameter θ t-1Under the condition that the missing data Y mis The conditional probability distribution of .

[0158] S4.2, posterior distribution estimation step: re-estimate the conditional distribution parameters based on the interpolation values ​​obtained in the interpolation step.

[0159] Specifically, the expression is:

[0160]

[0161] Among them, θ is the parameter of the new energy prediction data model, t usually represents the number of iterations or a specific time step, and Y obs represents the observed data, represents the missing data at time point t, Represents the missing data estimate given the observed data, obs, and the t-th iteration The conditional probability distribution of parameter θ under the condition of .

[0162] Through repeated iterations of the interpolation step and the posterior distribution estimation step, the interpolation value and the parameter finally converge. ij ; i = 1, 2, ... n; j = 1, 2, ... p}, where n usually represents the number of samples, p represents the number of features of each sample, and y ij Represents the jth eigenvalue of the i-th sample. The initial value of the model parameter θ=(μ,∑) is θ0=(μ0,∑0), where μ is usually the mean vector and ∑ is the covariance matrix, which can be composed of the mean vector and covariance matrix after column deletion or pairwise deletion. Repeat the previous process m times to obtain m complete data, realize multiple interpolation of missing data in the original data set, and solve the multivariate model interpolation problem.

[0163] It should be noted that the generation of multiple interpolation data in the embodiment of the present invention uses the generative adversarial network (GAN) technology to approximate the real data distribution through adversarial training of the generator and the discriminator, effectively filling in the missing values. This method is particularly suitable for data scenarios with time dependence and complex distribution. Fig. 9 As shown in the figure, it consists of a generator (G) and a discriminator (D). The task of the generator (G) is to generate data (Xf) as close to the real data distribution as possible based on the input random noise (Z), and input the generated data Xf and the real data x into the discriminator (D) for true or false judgment. The discriminator outputs a judgment result. At the same time, it is input into the generator again. After repeated iterations, until the discriminator can hardly distinguish between the real data and the generated data, the generator is considered to have learned to simulate the distribution of the real data. Finally, for given missing data, the generator will accept the position mark and observation data of the missing value, and then generate the corresponding filling value. Finally, it will output a complete data set.

[0164] Furthermore, the missing values ​​are imputed m times to generate m complete data sets, which are then combined into a total estimate after parameter estimation. When calculating the total estimated standard deviation, the estimation differences between different data sets are considered. Multiple imputation is performed using the joint distribution method to randomly generate imputed values ​​for missing variables by considering the posterior joint distribution of non-missing variables. The flowchart of multiple imputation of new energy missing values ​​is as follows: Fig.10 shown.

[0165] Embodiment 5

[0166] See also Fig.11 and Fig.12 The specific technical solution of the rationality check in the embodiment of the present invention is described as follows:

[0167] The data verification of the rule base in the above embodiment 1 is constructed by analyzing the statistical characteristics of historical data, data distribution and physical constraints to construct a new energy prediction data verification rule base, and its specific structure is as follows: Fig.11 As shown. The data is inferred based on the constructed rule base. Specifically, Fig.12 The process shown consists of four main steps:

[0168] S5.1. Place the input data into a dynamic database;

[0169] S5.2, matching the data in the dynamic database by using the rules in the rule base;

[0170] S5.3. After all the rule base is processed, the valid data is stored and recorded in the database according to the situation;

[0171] S5.4. Determine whether there are invalid rules and give corresponding responses.

[0172] In the specific implementation steps, S5.1 indicates that after the system receives the input data, it will first be placed in a dynamic database. This arrangement is for dynamic management through rules in the future. For example, in a distributed energy system, there may be a new distributed power generation facility that submits its operating data to the system for processing. In this step, the data will be entered into the system's dynamic database for standby for the next step. The data at this time may be real-time power generation, voltage, and frequency measurement information. The main purpose of this step is to organize the original information provided by the user.

[0173] At S5.2, after the dynamic database is ready, the system will automatically use a set of predetermined logical judgment criteria or rules to compare the stored data. If the data meets the rules, the mark of the rule will be set to 0 to indicate that the match is correct, otherwise it will be set to 1, indicating that the data does not meet the set rules. As a specific implementation in the above-mentioned embodiment, the power factor range that the distributed power generation unit needs to comply with is a fixed value. If the detected power factor falls outside the specified range, it means that the data does not meet the established requirements and will be marked as inappropriate.

[0174] Entering S5.3, when there are no longer any applicable standards in the rule base to detect the input data, it is necessary to save all valid data that have been judged to be suitable into the corresponding storage table and add relevant items in the actual database to update the system content and complete the current task flow. At the same time, the completion of the reasoning process is announced, which means that the input information has been fully verified.

[0175] In the final stage, step S5.4, the entire system will check whether the marker variables used to identify whether each checkpoint is qualified are all zero. If these markers are all valid zeros, the update process described above will be executed again: the qualified data will be recorded in the corresponding table, and the database information will be updated and the entire reasoning process will be terminated. However, if there is any 1 left, it means that some input information does not meet the rules, so it will trigger the generation of information feedback about the part that does not meet the specifications, and notify the user to supplement or correct the dynamic database accordingly.

[0176] The above four specific operation stages constitute the core link of the overall data judgment process through logical rules. They are of key significance in the management and optimization process of distributed new energy systems, and can help ensure the normal operation of the system and continue to provide efficient and safe power supply. Please note that the formula here refers to the pre-established standard rules rather than the real mathematical calculation formula. Each operation of mark = 0 or 1 is actually expressing a binary conclusion under a series of conditional settings for whether a certain data meets the standard; as for why the system is configured in this way, it is because it can ensure that all information passed to subsequent processes can pre-exclude inaccurate or non-compliant factors. This setting is intended to ensure the quality and efficiency of subsequent data analysis, ensure that the data of distributed power sources can be properly processed and analyzed in accordance with established business requirements, and thereby promote the stability and coordinated operation of the entire distributed energy network.

[0177] Improve the rationality and availability of predicted data through data reasoning. After the original data is jointly identified by adaptive binning and isolation forest outliers, corresponding verification rules need to be generated for different anomaly types. The types of anomaly data include outlier anomalies, distribution inconsistency anomalies, and isolated point anomalies. a) Outlier anomalies are characterized by a data point that is far away from most of the data and obviously deviates from the distribution range of the overall data. The verification rules used for outlier anomalies are: first, set a reasonable data range. According to the historical distribution of the data and business needs, determine the reasonable value range of the data. Secondly, compare the difference with adjacent data points. Calculate the difference between the outlier and the adjacent data point. If the difference exceeds a certain threshold, the data is considered abnormal. The size of the threshold can be determined according to the characteristics of the data and business needs. b) Distribution inconsistency anomalies are characterized by a large difference between the data distribution in the box where the data point is located and the overall data distribution. The verification rules used are: First, check the statistical characteristics of the data in the box. Calculate the statistical characteristics of the mean, median, standard deviation, etc. of the data in the box where the abnormal data point is located, and compare them with the statistical characteristics of the overall data. If the difference is large, you can further analyze whether the data point is reasonable. Secondly, perform data distribution fitting. Use a suitable probability distribution model to fit the overall data, and then determine whether the abnormal data point conforms to the distribution model. If not, it can be marked as an anomaly. c) The anomaly of the isolated point is manifested as a point that is judged to be isolated in the isolation forest algorithm and lacks correlation with the surrounding data points. The verification rules are as follows: first, analyze the correlation of the data. Check the correlation between the isolated point and other data points, such as by calculating the correlation coefficient or performing cluster analysis. If there is no obvious correlation between the isolated point and other data points, its rationality can be further verified. Secondly, combine the business logic judgment. Consider the rationality of the isolated point in the business scenario, and judge that it may be caused by data entry errors, system failures or special circumstances. If the isolated point cannot be explained from the business logic, it can be marked as an anomaly. The above analysis is conducive to identifying and processing abnormal data in the process of data analysis and prediction model construction, ensuring the quality of data and the accuracy of the model.

[0178] Embodiment 6

[0179] See also Fig.13 The data management system for a distributed new energy system in an embodiment of the present invention is described as follows:

[0180] like Fig.13 As shown, the data management system 13 for the distributed new energy system includes: a smart sensor 1301, a data acquisition layer 1302, a data analysis layer 1303 and a data intelligent management layer 1304;

[0181] The data collection layer 1302 performs the following steps:

[0182] S1. Data collection: Use intelligent sensors to collect power data related to the distributed new energy system, including: voltage, current, frequency, power factor data in the distributed new energy system, load data of the distributed new energy distribution network, and operation data of the power grid to which the distributed new energy system is connected;

[0183] The data analysis layer 1303 performs the following steps:

[0184] S2. Data missing identification: For all connection lines between nodes in the distributed new energy system, periodically analyze the trend characteristics of the time series corresponding to the power data on each line to obtain a difference sequence representing the trend characteristics. Determine the data with sudden changes or jumps in the difference sequence as missing data and mark the missing position;

[0185] S3. Data anomaly identification: First, use the DBSCAN algorithm to extract abnormal data features in the power data. The abnormal data features include numerical features, categorical features, and time series features. Then, use the abnormal data joint detection method of adaptive binning and isolation forest to analyze the extracted abnormal data features to obtain the abnormal score value, and determine the data with an abnormal score value less than the threshold as abnormal data, and mark the abnormal position.

[0186] The data intelligence governance layer 1304 performs the following steps:

[0187] S4. Intelligent filling of missing data and abnormal data: For the identified missing data and abnormal data, the generative adversarial network (GAN) technology is used to approximate the real data distribution through adversarial training of the generator and the discriminator, and generate filling values ​​for multiple interpolation data; the joint distribution method is used to perform multiple interpolation of the filling values ​​to the corresponding marked abnormal positions and missing positions to obtain the predicted data;

[0188] S5. After completing intelligent interpolation, use the rule base to perform data reasoning to verify the rationality of the predicted data. The data verification of the rule base is completed by analyzing the statistical characteristics of historical data, data distribution and physical constraints.

[0189] It should be noted that the above-mentioned data collection layer 1302, data analysis layer 1303 and data intelligent governance layer 1304 are also used to execute the steps or operations of the relevant parts of the method embodiments of the above-mentioned embodiments one to five.

[0190] Embodiment 7

[0191] Fig.14The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing a data governance method according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.14 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them:

[0192] The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the data management method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0193] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0194] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0195] It should be pointed out that Fig.14 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0196] In this embodiment, the data governance method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0197] Embodiment 8

[0198] A computer-readable storage medium is also provided in an embodiment of the present application, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the data governance method in the embodiment.

[0199] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store the operating system and various application software installed on the computer device, such as the program code of the data governance method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0200] Embodiment 9

[0201] A computer program product is also provided in an embodiment of the present application, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0202] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0203] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data management method for a distributed new energy system, characterized in that: The following steps are involved: S1. Data collection: Use intelligent sensors to collect power data related to the distributed new energy system, including: voltage, current, frequency, power factor data in the distributed new energy system, load data of the distributed new energy distribution network, and operation data of the power grid to which the distributed new energy system is connected; S2. Data missing identification: For all connection lines between nodes in the distributed new energy system, periodically analyze the trend characteristics of the time series corresponding to the power data on each line to obtain a difference sequence representing the trend characteristics. Determine the data with sudden changes or jumps in the difference sequence as missing data and mark the missing position; S3. Data anomaly identification: First, use the DBSCAN algorithm to extract abnormal data features in the power data. The abnormal data features include numerical features, categorical features, and time series features. Then, use the abnormal data joint detection method of adaptive binning and isolation forest to analyze the extracted abnormal data features to obtain the abnormal score value, and determine the data with an abnormal score value less than the threshold as abnormal data, and mark the abnormal position. S4. Intelligent filling of missing data and abnormal data: For the identified missing data and abnormal data, the generative adversarial network (GAN) technology is used to approximate the real data distribution through adversarial training of the generator and the discriminator, and generate filling values ​​for multiple interpolation data; the joint distribution method is used to perform multiple interpolation of the filling values ​​to the corresponding marked abnormal positions and missing positions to obtain the predicted data; S5. Reasonableness check: After completing intelligent interpolation, use the rule base to perform data reasoning to check the rationality of the predicted data. The data verification of the rule base is completed by analyzing the statistical characteristics of historical data, data distribution and physical constraints.

2. The method according to claim 1, characterized in that The time series corresponding to the power data on each line is recorded as x n The trend characteristics include an upward trend characteristic and a downward trend characteristic. The specific steps of the trend characteristic analysis include: S2.

1. Calculate the time series x n The corresponding moving average sequence A, n represents each moment in the time period, and the i-th element A in the moving average sequence A i It can be expressed as: Among them, X j is the jth data point in the original time series, m is the window size of the moving average, and i is the position of the current moving average; S2.

2. Calculate the time series x n The difference sequence diff of the corresponding moving average sequence, where: S2.

3. Calculate the time series x based on the difference sequence diff n Corresponding to the upward trend characteristics and downward trend characteristics of the running data curve.

3. The method according to claim 1, characterized in that The specific steps of extracting the abnormal data features include: S3.

1. First, use an autoencoder to encode the original data from high-dimensional data into a low-dimensional representation, and then decode the low-dimensional representation back to the high-dimensional space to obtain reconstructed data. In the above encoding and decoding process, extract abnormal data features; The training of the autoencoder uses the mean square error MSE as the loss function to measure the difference between the reconstructed data and the original data. The loss function is as follows: Where n is the number of samples, input i and decoded i are the i-th original data and reconstructed data respectively.

4. The method according to claim 1, characterized in that When the abnormal data feature is a numerical feature, the specific steps of using the abnormal data joint detection method for analysis include: S3.

2. Use adaptive binning technology to determine the number and range of bins according to data distribution, and calculate the statistical characteristics of the data in each bin. The statistical characteristics include: mean, median and standard deviation; at the same time, use isolation forest to process numerical feature data, divide the data into different subspaces by randomly selecting features and split points, and the abnormal data is isolated in the subspace with fewer steps. Whether the data point is abnormal is determined based on the anomaly score.

5. The method according to claim 1, characterized in that When the abnormal data feature is a categorical feature, the specific steps of using the abnormal data joint detection method for analysis include: S3.

3. Use adaptive binning technology to analyze and count the frequency of each category, and mark the categories with abnormally low frequency or large differences from the expected distribution as abnormal; at the same time, use isolation forest to analyze the abnormal situation of categorical features under specific values.

6. The method according to claim 1, characterized in that When the abnormal data feature is a time series feature, the specific steps of using the abnormal data joint detection method for analysis include: S3.

4. Use adaptive binning technology to analyze the characteristics of time period data, and use isolation forest to process time point data and detect mutation points or abnormal fluctuations.

7. The method according to claim 4, 5 or 6, characterized in that: The specific steps of using the abnormal data joint detection method for analysis also include: S3.

5. For multiple anomaly detection results of different types of features in the joint analysis of adaptive binning technology and isolation forest, a weighted fusion method is used and weights are assigned according to the importance of the features to comprehensively judge whether the data points are abnormal.

8. The method according to claim 1, characterized in that The specific steps of using the joint distribution method for multiple interpolation include: an interpolation step and a posterior distribution estimation step, and the two steps are iterated to interpolate the missing data and estimate the parameters to complete the multiple interpolation; S4.

1. Interpolation step: Generate interpolation values ​​of missing variables by random sampling, and establish joint conditional distribution based on non-missing variables according to missing variables, which is expressed as: in, represents the missing data at time point t, Yobs represents the observed data, θ is the parameter of the model, and θ t-1 represents the model parameters at the t-1th iteration, P(Y mis |Y obs ,θ t-1 ) means that given the observed data Y obs and parameter θ t-1 Under the condition that the missing data Y mis The conditional probability distribution of ; S4.2, posterior distribution estimation step: re-estimate the conditional distribution parameters based on the interpolation values ​​obtained in the interpolation step, and its expression is: Among them, θ is the parameter of the new energy prediction data model, t usually represents the number of iterations or a specific time step, and Y obs represents the observed data, represents the missing data at time point t, Indicates that given the observed data Y obs and the missing data estimate for the tth iteration The conditional probability distribution of parameter θ under the condition of .

9. The method according to claim 1, characterized in that: The specific steps of using the rule base to perform data reasoning include: S5.

1. Place the input data into a dynamic database; S5.

2. Use the rules in the rule base to match the data in the dynamic database. If the rule is successfully matched, the flag corresponding to it will be marked as 0, otherwise it will be marked as 1; S5.

3. If the rule base is empty, the data in the dynamic database is stored in the corresponding data table, and the corresponding table entry is added to the fact base table, and the reasoning ends; S5.

4. Determine whether all rule marks are 0. If so, store the data in the dynamic database into the corresponding data table, and add the corresponding table entry in the fact database table, and the reasoning ends; otherwise, generate a corresponding error prompt according to the item marked as 1, and wait for re-entry into the dynamic database.

10. A data management system for a distributed new energy system, characterized in that: include: Smart sensors, data collection layer, data analysis layer and data intelligent governance layer; The data collection layer performs the following steps: S1. Data collection: Use intelligent sensors to collect power data related to the distributed new energy system, including: voltage, current, frequency, power factor data in the distributed new energy system, load data of the distributed new energy distribution network, and operation data of the power grid to which the distributed new energy system is connected; The data analysis layer performs the following steps: S2. Data missing identification: For all connection lines between nodes in the distributed new energy system, periodically analyze the trend characteristics of the time series corresponding to the power data on each line to obtain a difference sequence representing the trend characteristics. Determine the data with sudden changes or jumps in the difference sequence as missing data and mark the missing position; S3. Data anomaly identification: First, use the DBSCAN algorithm to extract abnormal data features in the power data. The abnormal data features include numerical features, categorical features, and time series features. Then, use the abnormal data joint detection method of adaptive binning and isolation forest to analyze the extracted abnormal data features to obtain the abnormal score value, and determine the data with an abnormal score value less than the threshold as abnormal data, and mark the abnormal position. The data intelligence governance layer performs the following steps: S4. Intelligent filling of missing data and abnormal data: For the identified missing data and abnormal data, the generative adversarial network (GAN) technology is used to approximate the real data distribution through adversarial training of the generator and the discriminator, and generate filling values ​​for multiple interpolation data; the joint distribution method is used to perform multiple interpolation of the filling values ​​to the corresponding marked abnormal positions and missing positions to obtain the predicted data; S5. After completing intelligent interpolation, use the rule base to perform data reasoning to verify the rationality of the predicted data. The data verification of the rule base is completed by analyzing the statistical characteristics of historical data, data distribution and physical constraints.

Images