New energy station monitoring data quality evaluation method and system based on multi-source data
Through multi-source data processing and integrated learning network, identifying data deviations of new energy stations is solved, and the problem that traditional methods are difficult to reflect complex environmental factors is achieved, accurate assessment and early warning of data quality is achieved, and stable operation of stations is ensured.
Patent Information
- Application Number
- CN202410958735.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-12-03
AI Technical Summary
The traditional single data source based new energy station data quality evaluation method is difficult to fully reflect complex and changeable environmental factors, and is susceptible to problems such as sensor failure and communication interruption, resulting in incomplete or distorted data, which in turn affects new energy stations. precise management and optimized operations.
The quality evaluation method of monitoring data of new energy stations based on multi-source data is adopted, and the data deviation recognition model is trained by obtaining multi-source monitoring data. The data deviation recognition model is trained by using the Baum-Welch algorithm, combined with the integrated learning network to identify data deviations, calculate the abnormal score, and Establish a real-time monitoring platform for early warning.
It realizes accurate identification and dynamic quality evaluation of data deviations of new energy stations, improves data quality evaluation and early warning capabilities, and ensures stable operation and optimized management of stations.
Smart Images

Figure CN119066541A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of enterprise production operation and maintenance quality evaluation, and in particular to a method and system for evaluating the quality of new energy station monitoring data based on multi-source data. Background Art
[0002] New energy stations rely on a large amount of real-time data for operation and management, including power generation, equipment status, environmental conditions, etc. The accuracy of the data directly affects the operation efficiency and decision-making quality of the station. High-quality monitoring data helps to optimize the operation strategy of new energy stations and improve energy conversion efficiency and equipment utilization. The power generation of new energy stations is greatly affected by environmental factors, such as wind speed and light intensity. Accurate data helps to improve the accuracy of the power generation prediction model. The quality of monitoring data is directly related to the safe operation of the station. Abnormal data may indicate equipment failure or other safety hazards. As the global energy structure transforms to clean and low-carbon, the development and utilization of new energy such as wind power and solar energy are becoming more and more extensive. The efficient operation of new energy stations depends on accurate and real-time monitoring data, which is used for equipment status monitoring, fault warning, performance evaluation and operation and maintenance decision-making. However, traditional data quality evaluation methods are mostly based on a single data source, such as the SCADA (Supervisory Control And Data Acquisition) system. This method is difficult to fully reflect the complex and changeable new energy environmental factors, such as weather changes and equipment aging, on data quality. In addition, a single data source is susceptible to problems such as sensor failure and communication interruption, resulting in incomplete or distorted data, which in turn affects the precise management and optimized operation of new energy sites.
[0003] In recent years, with the rapid development of technologies such as the Internet of Things, big data, and artificial intelligence, multi-source data fusion has become an effective way to improve data quality. Multi-source data includes but is not limited to SCADA data, meteorological data, historical equipment maintenance records, geographic information system (GIS) data, etc. By comprehensively analyzing these data, we can more comprehensively evaluate data quality, identify abnormal situations, predict equipment status, and achieve refined management and intelligent operation and maintenance. However, the application of multi-source data fusion in new energy stations is still in its early stages, and there are challenges such as difficulty in data integration, complex model construction, and limited real-time processing capabilities. Especially in terms of data quality evaluation, how to effectively integrate multi-source information and establish a scientific and reasonable evaluation system is a key issue that needs to be solved urgently. Summary of the invention
[0004] In view of the above existing problems, the present invention is proposed.
[0005] Therefore, the present invention provides a new energy station monitoring data quality evaluation method and system based on multi-source data to solve the problems of accurate identification and dynamic quality evaluation of new energy station monitoring data deviations in a multi-source data environment, as well as to achieve efficient evaluation and intelligent early warning of the impact of data deviations on operating performance.
[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0007] In a first aspect, an embodiment of the present invention provides a method for evaluating the quality of monitoring data of a new energy station based on multi-source data, which comprises:
[0008] Obtain multi-source monitoring data collected by various monitoring devices in new energy stations and pre-process the multi-source monitoring data;
[0009] Based on the preprocessed multi-source monitoring data, the Baum-Welch algorithm is used to train the data deviation recognition model for the preprocessed multi-source monitoring data;
[0010] Integrate learning networks and data deviation identification models to construct observation sequences, convert predicted latent state sequences into state label data, and identify data deviations between various monitoring data sources;
[0011] Calculate data deviation anomaly scores, set data quality levels for each data deviation point, and evaluate the impact of data deviation on the operating performance of new energy stations;
[0012] Based on the data quality level, a real-time monitoring platform is established to track the changing trend of data deviations, and a change rate threshold is set to issue an early warning for data deviations that exceed the change rate threshold.
[0013] As a preferred solution of the method for evaluating the quality of monitoring data of new energy stations based on multi-source data of the present invention, the main steps of obtaining the multi-source monitoring data collected by each monitoring device in the new energy station are as follows:
[0014] List all monitoring equipment in the new energy station and specify the types of data and indicators that need to be collected;
[0015] Install monitoring equipment, ensure that the equipment location and installation method meet the standards, and debug the equipment to ensure that the equipment operates normally and can accurately collect data;
[0016] The collected data is transmitted to the cloud server through the network to pre-process the original data.
[0017] As a preferred solution of the method for evaluating the quality of monitoring data of new energy stations based on multi-source data described in the present invention, the Baum-Welch algorithm is used to train the data deviation recognition model of the preprocessed multi-source monitoring data, and the main steps are:
[0018] Normalize the multi-source monitoring data and set the parameters of the data deviation identification model based on the normalization results;
[0019] Initialize the initial state probability vector, state transition probability matrix and observation probability matrix to construct a hidden Markov model;
[0020] The Baum-Welch algorithm is used to calculate the forward probability at the initial moment, and the forward probability at each moment is calculated through forward recursion to calculate the total probability of the observation sequence;
[0021] Set the backward probability at the final moment, and calculate the backward probability at each moment through backward recursion;
[0022] Calculate state occupancy probability based on forward probability and backward probability;
[0023] Calculate the state transition probability based on the forward probability and the backward probability;
[0024] The formula for calculating the state occupancy probability based on the forward probability and the backward probability is:
[0025]
[0026] The formula for calculating the state transition probability based on the forward probability and the backward probability is:
[0027]
[0028] Among them, γ t (i) represents the state occupancy probability of being in state i at time t, Ω t (i) represents the forward probability of being in state i at time t, Β t (i) represents the backward probability of being in state i at time t, W(O|λ) represents the total probability of the observation sequence O appearing when the model parameter λ is given, and E t (i,j) represents the state transition probability from state i to state j at time t, a ij represents the probability of state i transitioning to state j, b j (O t+1 ) represents the observation data O generated under state j t+1 The probability of O t+1 represents the data at time t+1, β t+1 (i) represents the backward probability of being in state j at time t+1;
[0029] The initial state probability vector, state transition probability matrix and observation probability matrix are updated based on the state occupancy probability and state transition probability.
[0030] As a preferred solution of the method for evaluating the quality of monitoring data of new energy stations based on multi-source data described in the present invention, the integrated learning network and the data deviation identification model construct an observation sequence, and convert the predicted hidden state sequence into state label data, and identify the data deviation between the monitoring data sources. The main steps are:
[0031] Integrated learning network and data bias identification model;
[0032] Based on the initial state probability vector, the state transition probability matrix and the observation probability matrix;
[0033] Align the observation data from each data source in time and fuse them into a multidimensional time series;
[0034] The fused multi-dimensional time series is input into the trained data deviation recognition model as the observation sequence to construct the observation sequence;
[0035] Initialize the path probability and path record matrix, and generate the initialized path probability matrix and path record matrix;
[0036] Repeat the process for each moment, respectively calculate the path probability at each moment and record the optimal predecessor state of each state while calculating the path probability, update the path probability matrix and the path record matrix, until the optimal predecessor state of each state is reached;
[0037] The calculation formula of the path probability at each moment is:
[0038] δ t+1 (i) = max[δ t (i) a ij ] j O t+1 ;
[0039] Among them, δ t+1 (i) represents the optimal path probability at the i-th data point at time t+1, δ t (i) represents the optimal path probability at the i-th data point at time t;
[0040] Based on the optimal predecessor state, calculate the optimal path probability and optimal state at the final moment;
[0041] Extract the optimal path probability from the final moment, determine the optimal state at the final moment, and initialize the optimal hidden state sequence;
[0042] Using the path record matrix, trace back from time to time to determine the optimal state at each time;
[0043] According to the optimal state at each moment obtained by backtracking, the optimal hidden state sequence is constructed, and the predicted hidden state sequence is converted into state label data;
[0044] By comparing the differences between the observed data and the state label data of each monitoring data, the data deviations between the various monitoring data sources can be accurately identified.
[0045] As a preferred solution of the method for evaluating the quality of monitoring data of new energy stations based on multi-source data of the present invention, the main steps of the integrated learning network and data deviation identification model are as follows:
[0046] Through the logging module, the prediction results output by the model are collected once a minute for comparison with the prediction deviation, the actual deviation and the corresponding timestamp data;
[0047] Use automated scripts to classify and annotate the collected data according to preset deviation types and severity standards, distinguish true positives, false positives, true negatives, and false negatives, and form annotated feedback data sets;
[0048] Set the retraining day to trigger model retraining on that day of each month;
[0049] For the hidden Markov model, the Baum-Welch algorithm is used to re-estimate the state transition probability matrix and the observation probability matrix;
[0050] For the Isolation Forest model, adjust the number of trees and subsample size to ensure that the model adapts to the new data characteristics;
[0051] Stochastic gradient descent is used as the online learning algorithm. A small batch of data is randomly extracted from the latest feedback data set each time to update the model parameters. The formula is:
[0052]
[0053] The formula for the loss function is;
[0054]
[0055] Among them, P(y|x; θ) represents the probability that the model predicts the category y under parameter θ, N represents the total number of samples, θ represents the model parameter, η represents the learning rate, and L(θ t ;x t ,y t ) represents the loss function, x t and t Represents the features and labels of the current sample, x i andi Represent the i-th input feature and output label respectively;
[0056] The model parameters θ0 are randomly initialized. The samples in the data set are input in a stream manner. A small batch of data is obtained each time. For each small batch of data, the gradient of the model parameters with respect to the loss function is calculated. According to the calculated gradient and the preset learning rate, the model parameters are updated and iterated continuously until the data stream reaches the predetermined number of iterations.
[0057] Using the deep Q network algorithm, the reward function is set to give a positive reward r for each correct warning. + , for each false alarm, r - , the reward design follows the following principles:
[0058] For each correct warning: + =log e (1+V);
[0059] For each error warning: r - =-log e (1+RL);
[0060] Among them, V and RL represent the response speed and response delay of the system in the case of correct and false warning, respectively;
[0061] Set the change rate threshold δ. When the change rate of the monitored data δ new When >δ, the system will determine that there may be an abnormal situation and trigger an early warning. The formula is:
[0062]
[0063] Set the abnormal score threshold s. When the abnormal score of a monitoring point is s new >s, trigger an early warning;
[0064]
[0065] δ new and new represent the new change rate threshold and anomaly score threshold, α and β are the false positive rate and false negative rate, respectively, and T is the average response time;
[0066] One working day of each month is set as the evaluation day. Based on the warning effect and operation and maintenance efficiency improvement of the past month, the strategy evaluation is performed. The data deviation identification accuracy ACC formula is:
[0067]
[0068] Among them, TP represents the number of anomalies correctly identified by the system, TN represents the amount of data correctly judged as normal by the system, FP represents the number of normal data incorrectly judged as anomalies by the system, and FN represents the actual number of abnormal data that the system failed to identify;
[0069] The formula for early warning response time RT is:
[0070]
[0071] Among them, T r,i represents the time point when the i-th abnormal event is responded by the system, T r Represents the response time, T t,i represents the time point when the ith abnormal event actually occurred, T t represents the actual occurrence time of the abnormal event, and N represents the total number of abnormal events during the monitoring period;
[0072] The formula for improving the operation and maintenance efficiency percentage E is:
[0073]
[0074] Among them, OT represents the manual operation and maintenance time required to solve the same scale of problems before the introduction of the automated early warning system, and NT represents the actual time spent on solving the same problem after the system intervenes;
[0075] The CPI formula that reflects overall performance is:
[0076]
[0077] Among them, RT max represents the maximum acceptable warning response time, α represents the adjustment factor, and w1, w2, and w3 represent the weight coefficient ratios of accuracy, response time, and operation and maintenance efficiency improvement percentage, respectively.
[0078] As a preferred solution of the data quality evaluation method for monitoring new energy stations based on multi-source data of the present invention, wherein: the data deviation anomaly score is calculated, the data quality level is set for each data deviation point, and the impact of data deviation on the operating performance of the new energy station is evaluated. The main steps are:
[0079] Standardize each data deviation, and form a test data set with the standardized data deviations;
[0080] Construct an anomaly monitoring model, set the parameters of the anomaly monitoring model, and use the test data set as training data to train the anomaly monitoring model. Use the trained anomaly monitoring model to predict the anomaly score of each data deviation in the test data set, traverse each data deviation in the test data set, input each data deviation into the isolated tree of the anomaly monitoring model, start from the root node, traverse downward according to the splitting condition of the isolated tree until reaching the leaf node, record the traversal path length of each data point in each tree, and for each data deviation, calculate the average of the path lengths of the data deviation in all isolated trees, and use the average as the final anomaly score of the current data deviation, and normalize the anomaly score to a preset range;
[0081] The calculation formula of the abnormal score is:
[0082]
[0083] Among them, S(x) represents the anomaly score of data deviation x, represents the average path length of the data deviation x in all isolated trees, and C(n) represents the adjustment coefficient;
[0084] Set data quality levels based on anomaly scores and score anomaly data points;
[0085] Set the data quality level based on the anomaly score. The level can be set based on the score range:
[0086] High-quality data: anomaly scores are in the range of 0-a;
[0087] Good data: the anomaly score is within the range of ab;
[0088] Medium data: anomaly score is within the bc range;
[0089] Low-quality data: anomaly scores are within the cd range;
[0090] Abnormal data: the abnormal score is within the range of de;
[0091] Assign a corresponding quality level to each data point according to its anomaly score
[0092] Each abnormal data point is assigned a score, which can be set based on the anomaly score:
[0093] Data with abnormal scores within the range of de are scored as E' (the highest score);
[0094] Data with abnormal scores within the range of cd are scored as D';
[0095] Data with abnormal scores within the range of bc are scored as C';
[0096] Data with abnormal scores within the range of ab are classified as B';
[0097] Data with anomaly scores in the range of 0-a are scored as A' (the lowest score);
[0098] The impact of data deviation on the operating performance of new energy stations is evaluated based on the scoring of abnormal data points.
[0099] As a preferred solution of the data quality evaluation method for monitoring new energy stations based on multi-source data of the present invention, wherein: based on the data quality level, a real-time monitoring platform is established to track the change trend of data deviation, and a change rate threshold is set to warn of data deviation exceeding the change rate threshold. The main steps are:
[0100] Establish a real-time monitoring platform based on data quality levels;
[0101] The data deviation index is subjected to a differential operation to obtain a differential value, and a stationarity test is performed on the differential value. The deviation value after the stationarity test is used as the rate of change of the data deviation;
[0102] The threshold value of the change rate of data deviation is set to μ0, and the formula of the deviation change rate μ is:
[0103] μ=k〃σ;
[0104] Where σ represents the standard deviation of the data, and k represents the safety factor;
[0105] The objective function G(k) is introduced to evaluate the early warning effect under different k values, so as to obtain the optimal safety factor k, the formula is:
[0106]
[0107] Among them, FPR(k) represents the false alarm rate, FNR(k) represents the false alarm rate, w4 and w5 represent the weight coefficients of the false alarm rate and false alarm rate respectively;
[0108] The formula for the optimal safety factor k is:
[0109]
[0110] When μ>μ0, an early warning is triggered and the early warning information is conveyed to the operation and maintenance personnel, who then analyze the early warning information:
[0111] Collect all data deviations whose change rate exceeds a preset threshold;
[0112] Classify the data deviations exceeding the threshold by type to obtain abnormal data points of each category;
[0113] Conduct root cause analysis for each category of abnormal data points to obtain the causes of data deviation;
[0114] Based on the analysis results, the early warning rules are regularly evaluated and adjusted to optimize the real-time monitoring platform.
[0115] In a second aspect, the present invention provides a new energy station monitoring data quality evaluation system based on multi-source data, including a data acquisition and preprocessing module, a data deviation identification model training module, an integrated learning network and deviation identification module, a data deviation anomaly score and quality assessment module, and a real-time monitoring and early warning response module;
[0116] The data acquisition and preprocessing module is used to obtain the multi-source monitoring data collected by various monitoring devices in the new energy station and pre-process the multi-source monitoring data;
[0117] A data deviation identification model training module is used to train a data deviation identification model on the preprocessed multi-source monitoring data using the Baum-Welch algorithm;
[0118] The integrated learning network and deviation identification module is used to integrate the learning network and the data deviation identification model, construct the observation sequence, convert the predicted hidden state sequence into state label data, and identify the data deviation between various monitoring data sources;
[0119] The data deviation anomaly score and quality assessment module is used to calculate the data deviation anomaly score, set the data quality level for each data deviation point, and evaluate the impact of data deviation on the operating performance of new energy stations;
[0120] The real-time monitoring and early warning response module is used to establish a real-time monitoring platform, track the changing trend of data deviations, set change rate thresholds, and issue early warnings for data deviations that exceed the change rate thresholds.
[0121] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the new energy station monitoring data quality evaluation method based on multi-source data as described in the first aspect of the present invention.
[0122] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the new energy station monitoring data quality evaluation method based on multi-source data as described in the first aspect of the present invention.
[0123] The beneficial effects of the present invention are: improving the accuracy and reliability of monitoring data of new energy stations, and through real-time monitoring and early warning mechanisms, greatly enhancing the foresight and response capabilities to potential risks, ultimately ensuring the stable operation and optimized management of new energy stations. BRIEF DESCRIPTION OF THE DRAWINGS
[0124] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0125] Figure 1 This is a flow chart of the method for evaluating the quality of new energy station monitoring data based on multi-source data in Example 1.
[0126] Figure 2 This is a flow chart of the data deviation identification model training in Example 1. DETAILED DESCRIPTION
[0127] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.
[0128] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0129] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.
[0130] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a method for evaluating the quality of monitoring data of a new energy station based on multi-source data, comprising the following steps:
[0131] S1. Obtain multi-source monitoring data collected by various monitoring devices in the new energy station, and pre-process the multi-source monitoring data.
[0132] List all monitoring equipment in the new energy station and specify the types of data and indicators that need to be collected;
[0133] Install monitoring equipment, ensure that the equipment location and installation method meet the standards, and debug the equipment to ensure that the equipment operates normally and can accurately collect data;
[0134] The collected data is transmitted to the cloud server through the network, and the raw data is cleaned, integrated, feature engineered and normalized.
[0135] It needs to be explained that all monitoring equipment in the new energy station should be listed, including sensors, instruments, etc., and the data types and indicators that need to be collected should be clearly defined, such as temperature, humidity, wind speed, power generation, equipment status, etc.; monitoring equipment should be installed to ensure that the equipment location and installation method meet the standards, and equipment should be debugged to ensure that the equipment operates normally and can accurately collect data.
[0136] S2. Based on multi-source monitoring data, the Baum-Welch algorithm is used to train the data deviation identification model on the pre-processed multi-source monitoring data.
[0137] Normalize the multi-source monitoring data and set the parameters of the data deviation identification model based on the normalization results;
[0138] It is necessary to explain that all multi-source monitoring data should be collected to ensure that the data coverage is comprehensive and includes all parameters that need to be monitored. Data from different sources should be sorted to ensure that the timestamps are aligned so that the data has a consistent time dimension. A suitable normalization method should be selected, such as Min-Max Normalization or Z-score normalization. Each monitoring parameter should be normalized so that the data values fall within the same range, such as [0, 1] or standard normal distribution. The normalized data should be checked to ensure that it is correct and meets expectations.
[0139] The data deviation identification model in this application is a hidden Markov model. The hidden Markov model (HMM) is a statistical model that describes an observation sequence generated by a hidden state sequence. The hidden Markov model is often used for modeling and analysis of time series data.
[0140] Initialize the initial state probability vector, state transition probability matrix and observation probability matrix, and construct a hidden Markov model;
[0141] The Baum-Welch algorithm is used to calculate the forward probability at the initial moment, and the forward probability at each moment is calculated through forward recursion to calculate the total probability of the observation sequence;
[0142] Set the backward probability at the final moment, and calculate the backward probability at each moment through backward recursion;
[0143] Calculate state occupancy probability based on forward probability and backward probability;
[0144] Calculate the state transition probability based on the forward probability and the backward probability;
[0145] Among them, the formula for calculating the state occupancy probability based on the forward probability and the backward probability is:
[0146]
[0147] The formula for calculating the state transition probability based on the forward probability and the backward probability is:
[0148]
[0149] Among them, γ t (i) represents the state occupancy probability of being in state i at time t, Ω t (i) represents the forward probability of being in state i at time t, Β t (i) represents the backward probability of being in state i at time t, W(O|λ) represents the total probability of the observation sequence O appearing when the model parameter λ is given, and E t (i,j) represents the state transition probability from state i to state j at time t, a ij represents the probability of state i transitioning to state j, b j (O t+1 ) represents the observation data O generated under state j t+1 The probability of O t+1 represents the data at time t+1, β t+1 (i) represents the backward probability of being in state j at time t+1;
[0150] The initial state probability vector, state transition probability matrix and observation probability matrix are updated based on the state occupancy probability and state transition probability.
[0151] It needs to be explained that the initial state probability vector, state transition probability matrix and observation probability matrix are all parameters of the data deviation identification model;
[0152] Among them, the initial state probability vector represents the probability that the data deviation recognition model is initially in each hidden state;
[0153] The state transition probability matrix represents the probability of transitioning from one hidden state to another hidden state;
[0154] The observation probability matrix represents the probability of generating a certain observation value under a certain hidden state.
[0155] The forward algorithm is a dynamic programming algorithm used to calculate the probability of an observation sequence under given hidden Markov model (HMM) parameters. The algorithm uses recursive calculations to propagate the state probability of the previous moment to the next moment at each step, and updates the probability based on the observation value at the current moment. The forward probability is defined as the probability of being in state i at time t and observing the first t observation values.
[0156] The backward algorithm is also a dynamic programming algorithm that is used to calculate the probability of being in a certain state starting from a certain moment and generating subsequent observations under the condition of a given observation sequence. It is defined as the probability of being in state i at time t and observing the observations from time t to the final moment T.
[0157] S3, integrates learning networks and data deviation identification models, constructs observation sequences, converts predicted latent state sequences into state label data, and identifies data deviations between various monitoring data sources.
[0158] Based on the initial state probability vector, the state transition probability matrix and the observation probability matrix;
[0159] Align the observation data from each data source in time and fuse them into a multidimensional time series;
[0160] The fused multi-dimensional time series is input into the trained data deviation recognition model as the observation sequence to construct the observation sequence;
[0161] Initialize the path probability and path record matrix, and generate the initialized path probability matrix and path record matrix;
[0162] Repeat the process for each moment, respectively calculate the path probability at each moment and record the optimal predecessor state of each state while calculating the path probability, update the path probability matrix and the path record matrix, until the optimal predecessor state of each state is reached;
[0163] The calculation formula of the path probability at each moment is:
[0164] δ t+1 (i) = max[δ t (i) a ij ] j O t+1 ;
[0165] Among them, δ t+1 (i) represents the optimal path probability at the i-th data point at time t+1, δ t (i) represents the optimal path probability at the i-th data point at time t;
[0166] What needs to be explained is that the path probability of each possible state under the initial state is determined. The path probability is based on the initial state probability and the probability of the first observation value. A matrix is created to store these initial probabilities. Each row of the matrix represents a state and each column represents a time point. At the same time, a matrix is created to record the predecessor state to each state. Because this is the initial moment, each item of the predecessor state matrix can be set to 0, indicating that there is no predecessor state. Starting from time point t=2, the path probability of each time point is calculated step by step. For each time point, the path probability of transferring from all possible predecessor states to the current state is considered. In order to calculate the optimal path probability to reach each state at the current moment, it is necessary to compare all paths from each predecessor state to the current state, compare all possible paths, and select the path with the largest path probability as the optimal path. This optimal path probability is stored in the path probability matrix, corresponding to the current time point and state position.
[0167] When calculating the path probability at each time point, the optimal predecessor state to reach the current state is recorded at the same time; the specific operation is that in the process of determining the optimal path, the index of the predecessor state that makes the path probability the largest is saved, which means at the current moment, from which predecessor state is transferred to the current state, the index of this predecessor state is stored in the path record matrix, corresponding to the position of the current time point and state, the path record matrix is continuously updated, and the optimal predecessor state is recorded for each state at each time point. In this way, the path record matrix will save the predecessor state information of each step on the entire path, and the previous steps are continuously executed to recursively calculate and record each time point and state. At each time point, the optimal path probability of the current time point is calculated by the previous state probability and transfer probability, and the optimal predecessor state of each state is recorded at the same time. The path probability matrix is continuously updated to reflect the optimal path probability of the current state at each time point, and the path record matrix is updated to record the optimal predecessor state of each state at each time point, ensuring that the complete optimal path can be traced back in the end. Based on the optimal predecessor state, the optimal path probability and optimal state at the final moment are calculated, the optimal path probability is extracted from the final moment, the optimal state at the final moment is determined, and the optimal hidden state sequence is initialized;
[0168] It needs to be explained that the observation data of all monitoring devices are collected to ensure that each data source has an accurate timestamp, and the interpolation method is used to time-align the data at different times to ensure that each time point has complete multi-dimensional data.
[0169] The data format after alignment is: X = {(t1, x 11 , x 12 , …, x 1n ),(t2,x 21 , x 22 , …, x2n ),…,(t T , x T1 , x T2 , …, x Tn )}, where t i Indicates the timestamp, x ij represents the j-th dimension observation value at the i-th time point;
[0170] The aligned multi-source data are fused into a multi-dimensional time series, O = {o1, o2, ..., o T}, where o i =(x i1 , x i2 , …, x in ) represents the multidimensional observation vector at the i-th time point;
[0171] The fused multidimensional time series O is input into the trained data deviation recognition model to ensure that the model can process multidimensional observation vectors, and the probability of each observation vector is calculated according to the observation probability matrix B of the model, and the path probability matrix δ is initialized to represent the probability of the optimal path to each state at each time;
[0172] δ1(i)=π i b i (o1),π i represents the initial state probability, b i (o1) is the probability of observing o1 in state i at time point.
[0173] Initialize the path record matrix ψ to record the optimal predecessor state of each state at each moment.
[0174] Based on the initialized path probability and path record matrix, the Viterbi algorithm is used based on the observation sequence to recursively calculate the path probability at each moment and record the optimal predecessor state to reach each state. Based on the optimal predecessor state, the optimal path probability and optimal state at the final moment are calculated. Starting from the optimal state at the final moment, the optimal hidden state sequence is obtained by backtracking through the recorded optimal predecessor state.
[0175] The Viterbi algorithm is a dynamic programming algorithm used to find the most likely hidden state sequence for a given observation sequence in a hidden Markov model (HMM). It is widely used in speech recognition, bioinformatics, communication signal processing and other fields.
[0176] Using the path record matrix, trace back from time to time to determine the optimal state at each time;
[0177] According to the optimal state at each moment obtained by backtracking, the optimal hidden state sequence is constructed, and the predicted hidden state sequence is converted into state label data;
[0178] By comparing the differences between the observed data and the state label data of each monitoring data, the data deviations between the various monitoring data sources can be accurately identified.
[0179] It should be explained that at the final moment, all values in the path probability matrix are checked and the largest value is found, which represents the optimal path probability of the entire observation sequence from the beginning to the final moment. After determining the optimal path probability, record it for subsequent analysis and find the state corresponding to the above maximum path probability. This state is the optimal state at the final moment. Record this optimal state to initialize the optimal hidden state sequence, create an array or list to store the optimal hidden state sequence, and use the optimal state at the final moment as the last element of the sequence. Starting from the known optimal state at the final moment, trace back step by step through the path record matrix to find the optimal predecessor state at each time point. For each time point, start from the current The optimal state of the path records the optimal predecessor state of each state at each moment. The optimal state of the current moment is read from the path record matrix and recorded in the optimal hidden state sequence. At each time point, the optimal state found is inserted into the corresponding position of the optimal hidden state sequence. The optimal hidden state sequence is completed by backtracking step by step until it returns to the first moment. The optimal hidden state sequence is completed. The optimal state of each time point has been determined, and finally a complete optimal hidden state sequence is constructed. The optimal hidden state sequence contains the optimal hidden state of each step from the initial moment to the final moment. The optimal hidden state sequence is checked and verified to ensure that no time point is missed during the backtracking process and the optimal hidden state sequence is complete.
[0180] The optimal hidden state sequence obtained by backtracking is converted into state label data that is easier to understand and process; each hidden state can correspond to a specific label, such as "normal" or "abnormal", and a corresponding label is assigned to the hidden state at each time point to form a complete state label data sequence. For each time point, the observed data and the state label data are compared to check the differences between them. Through this comparison, it can be found whether there are inconsistencies or deviations between the various monitoring data sources, the differences between the observed data and the state label data are analyzed, and possible data deviations are identified.
[0181] S4 integrated learning network and data deviation identification model.
[0182] Through the logging module, the prediction results output by the model are collected once a minute for comparison with the prediction deviation, the actual deviation and the corresponding timestamp data;
[0183] Use automated scripts to classify and annotate the collected data according to preset deviation types and severity standards, distinguish true positives, false positives, true negatives, and false negatives, and form annotated feedback data sets;
[0184] Set the retraining day to trigger model retraining on that day of each month;
[0185] For the hidden Markov model, the Baum-Welch algorithm is used to re-estimate the state transition probability matrix and the observation probability matrix;
[0186] For the Isolation Forest model, adjust the number of trees and subsample size to ensure that the model adapts to the new data characteristics;
[0187] Stochastic gradient descent is used as the online learning algorithm. A small batch of data is randomly extracted from the latest feedback data set each time to update the model parameters. The formula is:
[0188]
[0189] The formula for the loss function is;
[0190]
[0191] Among them, P(y|x; θ) represents the probability that the model predicts the category y under parameter θ, N represents the total number of samples, θ represents the model parameter, η represents the learning rate, and L(θ t ;x t ,y t ) represents the loss function, x t and t Represents the features and labels of the current sample, x i and i Represent the i-th input feature and output label respectively;
[0192] The model parameters θ0 are randomly initialized. The samples in the data set are input in a stream manner. A small batch of data is obtained each time. For each small batch of data, the gradient of the model parameters with respect to the loss function is calculated. According to the calculated gradient and the preset learning rate, the model parameters are updated and iterated continuously until the data stream reaches the predetermined number of iterations.
[0193] Using the deep Q network algorithm, the reward function is set to give a positive reward r for each correct warning. + , for each false alarm, r - , the reward design follows the following principles:
[0194] For each correct warning: + =log e (1+V);
[0195] For each error warning: r - =-log e (1+RL);
[0196] Among them, V and RL represent the response speed and response delay of the system in the case of correct and false warning, respectively;
[0197] Set the change rate threshold δ. When the change rate of the monitored data δ new When >δ, the system will determine that there may be an abnormal situation and trigger an early warning. The formula is:
[0198]
[0199] Set the abnormal score threshold s. When the abnormal score of a monitoring point is s new >s, trigger an early warning;
[0200]
[0201] δ new and new represent the new change rate threshold and anomaly score threshold, α and β are the false positive rate and false negative rate, respectively, and T is the average response time;
[0202] It should be explained that the anomaly monitoring model is an isolation forest model. The main parameters of the isolation forest model are determined, including the number of trees (i.e., the number of isolated trees), the size of subsamples (i.e., the number of data samples used for training each tree), the maximum number of features, etc. The parameters affect the performance and detection accuracy of the model, and the parameters can be adjusted according to actual needs and data characteristics. For example, the more trees there are, the higher the stability of the model. The size of the subsample affects the training speed and accuracy of the model. The test data set is used as the training data to ensure that the data set has been preprocessed, such as removing missing values and normalizing, to ensure the effectiveness of the model training. The data set should contain a variety of data deviations so that the model can learn the normal and abnormal patterns of the data. The set parameters and accuracy are used as the training data. The prepared training data is input into the isolation forest model for training; the model learns the distribution of data by constructing multiple isolated trees; each isolated tree recursively and randomly divides the data sample to form a series of partitioning rules. The purpose of the isolation tree is to isolate the outliers with a small number of partitions. The trained isolation forest model is used to predict the test data set. The model calculates the anomaly score of each data point based on the partition of the isolated tree. The anomaly score reflects the degree of anomaly of the data point. The higher the score, the more likely the data point is an anomaly. The predicted anomaly score is normalized and adjusted to a preset range (such as between 0 and 1). Normalization helps to unify the scoring standard and facilitate subsequent threshold setting and outlier identification.
[0203] According to the normalized anomaly score, a threshold is set. This threshold can be set according to actual needs and data distribution. Common methods include selecting a fixed score value as the threshold, or setting a percentile according to the distribution of the data (such as marking points with scores in the top 5% as anomalies), and marking all data points in the test data set with scores higher than the set threshold as abnormal data points.
[0204] One working day of each month is set as the evaluation day. Based on the warning effect and operation and maintenance efficiency improvement of the past month, the strategy evaluation is performed. The data deviation identification accuracy ACC formula is:
[0205]
[0206] Among them, TP is the number of true positive cases, which represents the number of anomalies correctly identified by the system; TN is the number of true negative cases, which represents the number of data correctly judged as normal by the system; FP is the number of false positive cases, which represents the number of normal data mistakenly judged as anomalies by the system; FN is the number of false negative cases, which represents the number of actual abnormal data that the system failed to identify;
[0207] The formula for early warning response time RT is:
[0208]
[0209] Among them, T r,i represents the time point when the i-th abnormal event is responded by the system, T r Represents the response time, T t,i represents the time point when the ith abnormal event actually occurred, T t represents the actual occurrence time of the abnormal event, and N represents the total number of abnormal events during the monitoring period;
[0210] The formula for improving the operation and maintenance efficiency percentage E is as follows:
[0211]
[0212] Among them, OT is the old operation and maintenance time, which represents the manual operation and maintenance time required to solve the same scale problem before the introduction of the automated early warning system; NT is the new operation and maintenance time, which represents the actual time spent on handling the same problem after the system is involved;
[0213] The CPI formula that reflects overall performance is:
[0214]
[0215] Among them, RT max represents the maximum acceptable warning response time, α represents the adjustment factor, and w1, w2, and w3 represent the weight coefficient ratios of accuracy, response time, and operation and maintenance efficiency improvement percentage, respectively.
[0216] S5. Calculate the data deviation anomaly score, set the data quality level for each data deviation point, and evaluate the impact of data deviation on the operating performance of new energy stations.
[0217] Standardize each data deviation, and form a test data set with the standardized data deviations;
[0218] Construct an anomaly monitoring model, set the parameters of the anomaly monitoring model, and use the test data set as training data to train the anomaly monitoring model. Use the trained anomaly monitoring model to predict the anomaly score of each data deviation in the test data set, traverse each data deviation in the test data set, input each data deviation into the isolated tree of the anomaly monitoring model, start from the root node, traverse downward according to the splitting condition of the isolated tree until reaching the leaf node, record the traversal path length of each data point in each tree, and for each data deviation, calculate the average of the path lengths of the data deviation in all isolated trees, and use the average as the final anomaly score of the current data deviation, and normalize the anomaly score to a preset range;
[0219] The calculation formula of anomaly score is;
[0220]
[0221] Among them, S(x) represents the anomaly score of data deviation x, represents the average path length of the data deviation x in all isolated trees, C(n) represents the adjustment coefficient, T represents the total number of trees in the isolation forest, and n represents any positive integer;
[0222] It should be explained that each data point in the test data set is traversed to ensure that each data point is evaluated by the isolation forest model; each data point in the data set is checked, and the anomaly detection of each point is processed in turn, and the current data point is input into each isolated tree of the trained isolation forest model. Each isolated tree is constructed by randomly selecting features and randomly selecting split points to isolate anomalies to the maximum extent. Starting from the root node, according to the feature value of the current data point, the tree is traversed downward according to the splitting condition of the isolation tree. At each node, the comparison result between the feature value and the split point determines whether to enter the left child node or the right child node, and the traversal downward continues until the leaf node is reached.
[0223] In each isolated tree, the traversal path length of the current data point from the root node to the leaf node is recorded. The path length indicates the number of splits required by the isolated tree to isolate the data point. The shorter the path, the easier it is to isolate the data point and the more likely it is an anomaly. The traversal path lengths of the current data point in all isolated trees are collected for subsequent calculation of the average path length. For each data point, the path lengths in all isolated trees are averaged. The average path length reflects the difficulty of isolating the data point in the isolation forest model. The shorter the path length, the easier it is to isolate the data point and the higher the anomaly score. The average path length is converted into an anomaly score. The isolation forest model usually corresponds a shorter average path length to a higher anomaly score. The anomaly score is normalized to ensure that all scores are within a preset range, such as between 0 and 1. Normalization helps to unify the standard of anomaly scores and facilitates subsequent threshold setting and anomaly point identification.
[0224] Set data quality levels based on anomaly scores and score anomaly data points;
[0225] Quality levels can be set based on a range of scores. For example, the following levels can be set:
[0226] High-quality data: anomaly scores are in the range of 0-a;
[0227] Good data: the anomaly score is within the range of ab;
[0228] Medium data: anomaly score is within the bc range;
[0229] Low-quality data: anomaly scores are within the cd range;
[0230] Abnormal data: the abnormal score is within the range of de;
[0231] Where a = 0.2, b = 0.4, c = 0.6, d = 0.8, e = 1.0;
[0232] Each data point is assigned a corresponding quality level according to its anomaly score.
[0233] Each abnormal data point is assigned a score, which can be set based on the anomaly score.
[0234] For example:
[0235] Data with abnormal scores within the range of de are scored as E' (the highest score);
[0236] Data with abnormal scores within the range of cd are scored as D';
[0237] Data with abnormal scores within the range of bc are scored as C';
[0238] Data with abnormal scores within the range of ab are classified as B';
[0239] Data with anomaly scores in the range of 0-a are scored as A' (the lowest score);
[0240] Among them, A'=1 point, B'=2 points, C'=3 points, D'=4 points, E'=5 points;
[0241] The impact of data deviation on the operating performance of new energy stations is evaluated based on the scoring of abnormal data points.
[0242] Summarize the scores of all abnormal data points, calculate the proportion and distribution of data points of each quality level, count the number of data points of each rating level and their proportion in the total data, analyze the overall status of data quality, and evaluate the impact of high-scoring (i.e. high abnormality score) data points on the station's operating performance. High-scoring data points often mean large data deviations, which may have an adverse impact on the station's operational decisions, predictions, and controls.
[0243] Specific analysis may include:
[0244] (1) Deviation of performance indicators: Compare the key performance indicators (such as power generation, equipment utilization, etc.) in the time period containing high-scoring data points with the normal time period to identify the specific impact of abnormal data on performance.
[0245] (2) Adjustment of operation strategy: Based on the distribution and scoring of abnormal data points, adjust the operation strategy and control method of the station, for example, eliminate high-scoring data points or increase data correction efforts.
[0246] (3) Evaluation of forecast accuracy: Check the impact of historical data containing abnormal data on the future operation of the forecast model and evaluate the accuracy and reliability of the forecast model.
[0247] Based on the evaluation results, formulate corresponding data management and improvement measures to reduce the impact of data deviation on the operating performance of new energy stations.
[0248] Improvement measures may include:
[0249] (1) Data cleaning and preprocessing: Strengthen data cleaning and preprocessing, and correct or eliminate abnormal data points in a timely manner.
[0250] (2) Monitoring equipment optimization: Optimize the monitoring equipment and sensor layout to improve the accuracy and reliability of data collection.
[0251] (3) Data fusion and correction: Use multi-source data fusion and correction technology to reduce the impact of the deviation of a single data source on the overall data quality.
[0252] S6. Based on the data quality level, a real-time monitoring platform is established to track the changing trend of data deviations, and a change rate threshold is set to issue an early warning for data deviations that exceed the change rate threshold.
[0253] Establish a real-time monitoring platform based on data quality levels;
[0254] The data deviation index is subjected to a differential operation to obtain a differential value, and a stationarity test is performed on the differential value. The deviation value after the stationarity test is used as the rate of change of the data deviation;
[0255] The threshold value of the change rate of data deviation is set to μ0, and the formula of the deviation change rate μ is:
[0256] μ=k〃σ;
[0257] Where σ is the standard deviation of the data, and k is the safety factor;
[0258] The objective function G(k) is introduced to evaluate the early warning effect under different k values, so as to obtain the optimal safety factor k, the formula is:
[0259]
[0260] Among them, FPR(k) represents the false alarm rate, FNR(k) represents the false alarm rate, w4 and w5 represent the weight coefficients of the false alarm rate and false alarm rate respectively;
[0261] The formula for the optimal safety factor k is:
[0262]
[0263] When μ>μ0, an early warning is triggered and the early warning information is conveyed to the operation and maintenance personnel, who then analyze the early warning information:
[0264] Collect all data deviations whose change rate exceeds a preset threshold;
[0265] Classify the data deviations exceeding the threshold by type to obtain abnormal data points of each category;
[0266] Conduct root cause analysis for each category of abnormal data points to obtain the causes of data deviation;
[0267] Based on the analysis results, the early warning rules are regularly evaluated and adjusted to optimize the real-time monitoring platform.
[0268] It needs to be explained that we need to choose appropriate technical frameworks and tools to build a platform that can collect, process and display data in real time. Commonly used tools include Kafka, InfluxDB, Grafana, etc. The real-time monitoring platform needs to have the ability to collect, store, analyze and visualize data so as to monitor data changes and deviations at any time; collect various indicators in real-time data and calculate data deviation values. These deviation values represent the difference between actual observed data and expected data; perform differential operations on the collected data deviation values. Differentiation is a method of calculating the difference between data values at adjacent time points. Through differentiation, the trend and seasonal components in the time series data can be eliminated to make the data more stable.
[0269] Differential operation steps: For the deviation value at each time point, calculate the difference between it and the deviation value at the previous time point, that is: differential value = deviation value at the current time point - deviation value at the previous time point.
[0270] The purpose of the stationarity test is to ensure that the data after difference is stationary, that is, the statistical characteristics of the data (such as mean and variance) remain unchanged over time. Stationary data is more suitable for time series analysis and prediction. Common stationarity test methods include ADF (Augmented Dickey-Fuller) test and KPSS
[0271] (Kwiatkowski-Phillips-Schmidt-Shin) test.
[0272] ADF test: The null hypothesis is that the data has a unit root (that is, the data is not stable). If the test result is significant, the null hypothesis is rejected, indicating that the data is stable.
[0273] KPSS test: The null hypothesis is that the data is stable. If the test result is significant, the null hypothesis is rejected, indicating that the data is not stable.
[0274] The above-mentioned stationarity test is performed on the data deviation value after differentiation. If the data passes the stationarity test, the data deviation value is considered to be stable. The differential value that passes the stationarity test is regarded as the rate of change of the data deviation. The rate of change reflects the change of the data deviation over time. The calculated data deviation change rate is stored in the real-time monitoring platform for subsequent analysis and early warning.
[0275] According to historical data and business needs, set a reasonable data deviation change rate threshold. The change rate threshold can be a single fixed value or dynamically adjusted according to different time periods and conditions.
[0276] For example, a change rate threshold is set. When the change rate of the data deviation exceeds the change rate threshold, the data deviation is considered abnormal and needs to be paid attention to.
[0277] Define specific warning rules to trigger a warning when the rate of change of data deviation exceeds the set threshold. The warning rules can include SMS, email notifications and other forms to clarify the triggering conditions of the warning, such as the rate of change of data deviation at multiple consecutive time points exceeding the threshold, or the rate of change at a single time point is abnormally high; implement the warning function in the real-time monitoring platform to monitor the rate of change of data deviation in real time, and trigger the warning mechanism when the rate of change exceeds the preset threshold; through the warning mechanism, promptly notify relevant personnel to take corresponding measures to avoid data deviation from having a significant impact on the business.
[0278] Collect all data deviation points whose change rate exceeds the preset threshold from the real-time monitoring platform, record and save these data points for subsequent analysis, set classification standards according to the characteristics of data deviation (such as time period, equipment type, geographical location, etc.), classify the collected over-threshold data deviation according to the preset classification standards, obtain abnormal data points of each category, and organize and record them. Use various analysis tools and methods, such as fault tree analysis (FTA), fishbone diagram (Ishikawa diagram), cause-and-effect analysis, etc., and trace the abnormal data points of each category in detail according to the classification results to find the specific reasons for the data deviation, summarize and summarize the root causes of the data deviation in each category, which may include equipment failure, data transmission problems, environmental factors, etc., and analyze the root causes. The results are recorded to form a detailed analysis report. Based on the results of the root cause analysis, the effectiveness of the current warning rules is evaluated to determine whether the warning rules can identify important abnormal data points in a timely and accurate manner. Feedback from relevant personnel on the warning rules is collected to understand the effects and shortcomings of the warning rules in actual applications. Based on the analysis results of data deviations and change rates, the warning thresholds are adjusted to make the warnings more accurate and timely. Based on the newly discovered root causes, the warning rules are updated and improved. For example, more refined warning strategies are formulated for specific types of abnormal data points. The warning rules and monitoring platforms are regularly inspected and evaluated to ensure that they are always in the best condition. Based on the actual operation conditions and analysis results, the warning rules and monitoring platforms are continuously improved to enhance their performance and reliability.
[0279] The present embodiment also provides a new energy station monitoring data quality evaluation system based on multi-source data, including: a data acquisition and preprocessing module, a data deviation identification model training module, an integrated learning network and deviation identification module, a data deviation anomaly score and quality evaluation module and a real-time monitoring and early warning response module. The data acquisition and preprocessing module is used to obtain the multi-source monitoring data collected by each monitoring device in the new energy station and preprocess the multi-source monitoring data. The data deviation identification model training module is used to train the data deviation identification model on the preprocessed multi-source monitoring data using the Baum-Welch algorithm based on the preprocessed multi-source monitoring data. The integrated learning network and deviation identification module are used to integrate the learning network and the data deviation identification model, construct an observation sequence, and convert the predicted hidden state sequence into state label data, identify the data deviation between each monitoring data source, and the data deviation anomaly score and quality evaluation module is used to calculate the data deviation anomaly score, set the data quality level for each data deviation point, and evaluate the impact of the data deviation on the operating performance of the new energy station. The real-time monitoring and early warning response module is used to establish a real-time monitoring platform, track the change trend of the data deviation, set the change rate threshold, and issue an early warning for the data deviation exceeding the change rate threshold.
[0280] This embodiment also provides a computer device, which is suitable for the case of a new energy station monitoring data quality evaluation method based on multi-source data, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the new energy station monitoring data quality evaluation method based on multi-source data proposed in the above embodiment.
[0281] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.
[0282] The present embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for evaluating the quality of monitoring data of new energy stations based on multi-source data as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.
[0283] In summary, the present invention realizes the monitoring and understanding of user behavior through the analysis of user file operation behavior, can accurately capture the user's operation habits and behavior patterns, provide basic data for the subsequent generation of encryption strategies, and also help the system to detect and prevent abnormal behaviors; predict the user's future operations according to the user's behavior habits, so as to formulate encryption strategies more intelligently; more accurately protect user privacy, reduce the risk of data leakage, and improve the data security and confidentiality of the system; the present invention better adapts to the user's operation habits and needs, improves the intelligence level of the encryption algorithm and user experience, and enhances the flexibility and adaptability of the system.
[0284] Example 2, referring to Table 1, is the second example of the present invention. To further verify the advancement of the present invention, experimental simulation data of a new energy station monitoring data quality evaluation method based on multi-source data is provided.
[0285] A typical wind farm was selected as the research object. The wind farm deployed an advanced sensor network, including multi-source monitoring equipment such as wind speed, temperature, humidity, and equipment vibration, with a total of 200 monitoring points. Five monitoring points were randomly selected from these 200 monitoring points for the experiment. The experimental design compared the performance differences of the traditional monitoring method and the method proposed in this invention in terms of data preprocessing, deviation identification, anomaly detection, and real-time monitoring.
[0286] First, the raw data from all monitoring points were collected continuously, and data cleaning was performed to eliminate missing values and obvious outliers to ensure data integrity. The data were also standardized to make different types of monitoring data comparable.
[0287] Then, the Baum-Welch algorithm was used to train the preprocessed data and a data deviation identification model was constructed. This model can automatically identify potential deviations between various monitoring data sources.
[0288] Secondly, the trained model is combined with the integrated learning network to identify the data deviations between the monitoring data sources. By comparing the observed data with the model prediction results, the data deviations can be accurately identified.
[0289] Next, the anomaly score of data deviation was calculated, and the data quality was divided into four levels based on the score: excellent, good, fair, and poor. This helps to evaluate the impact of data deviation on the operating performance of wind farms.
[0290] Finally, a real-time monitoring platform was established to track the trend of data deviation and set a threshold for the rate of change. When the data deviation exceeds the threshold, the system automatically triggers an early warning and notifies the operation and maintenance personnel to respond in a timely manner.
[0291] Details are as shown in Table 1:
[0292] Table 1 Experimental record table
[0293]
[0294] By comparing the data before and after the embodiment, it is obvious that the method of the present invention is effective and superior. From the table above, it can be seen that the data deviation anomaly score of monitoring point 001 is 0.12, which is rated as "excellent" data quality level, which means that the deviation of this data source has a very small impact on the operating performance of the wind farm, which is only 0.03. In contrast, the score of monitoring point 004 is as high as 0.49, which is rated as "poor", and the impact index on the operating performance is 0.22, which is significantly higher than other monitoring points.
[0295] The system proposed by the present invention also shows obvious advantages in data deviation identification accuracy. For example, the deviation identification accuracy of monitoring point 003 is 88%, while the traditional method is only 78%. This shows that the present invention can not only identify data deviation more accurately, but also effectively evaluate the impact of deviation on operating performance, significantly improving the operation and maintenance efficiency and safety of wind farms.
[0296] In summary, this embodiment fully demonstrates the innovation and practicality of the new energy station monitoring data quality evaluation system based on multi-source data by comparing experimental data. The system has shown superior performance to traditional methods in data preprocessing, deviation identification, anomaly detection, real-time monitoring, etc., providing a more reliable and efficient monitoring solution for new energy stations.
[0297] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A method for evaluating the quality of monitoring data of new energy stations based on multi-source data. It is characterized in that include: Obtain multi-source monitoring data collected by various monitoring devices in new energy stations and pre-process the multi-source monitoring data; Based on the preprocessed multi-source monitoring data, the Baum-Welch algorithm is used to train the data deviation recognition model for the preprocessed multi-source monitoring data; Integrate learning networks and data deviation identification models to construct observation sequences, convert predicted latent state sequences into state label data, and identify data deviations between various monitoring data sources; Calculate data deviation anomaly scores, set data quality levels for each data deviation point, and evaluate the impact of data deviation on the operating performance of new energy stations; Based on the data quality level, a real-time monitoring platform is established to track the changing trend of data deviations, and a change rate threshold is set to issue an early warning for data deviations that exceed the change rate threshold.
2. The method for evaluating the quality of monitoring data of new energy stations based on multi-source data according to claim 1, It is characterized in that The main steps of obtaining multi-source monitoring data collected by each monitoring device in the new energy station and preprocessing the multi-source monitoring data are as follows: List all monitoring equipment in the new energy station and specify the types of data and indicators that need to be collected; Install monitoring equipment, ensure that the equipment location and installation method meet the standards, and debug the equipment to ensure that the equipment operates normally and can accurately collect data; The collected data is transmitted to the cloud server through the network to pre-process the original data.
3. The method for evaluating the quality of monitoring data of new energy stations based on multi-source data according to claim 2, It is characterized in that Based on the preprocessed multi-source monitoring data, the Baum-Welch algorithm is used to train the data deviation recognition model for the preprocessed multi-source monitoring data. The main steps are: Normalize the multi-source monitoring data and set the parameters of the data deviation identification model based on the normalization results; Initialize the initial state probability vector, state transition probability matrix and observation probability matrix to construct a hidden Markov model; The Baum-Welch algorithm is used to calculate the forward probability at the initial moment, and the forward probability at each moment is calculated through forward recursion to calculate the total probability of the observation sequence; Set the backward probability at the final moment, and calculate the backward probability at each moment through backward recursion; Calculate state occupancy probability based on forward probability and backward probability; Calculate the state transition probability based on the forward probability and the backward probability; The formula for calculating the state occupancy probability based on the forward probability and the backward probability is: The formula for calculating the state transition probability based on the forward probability and the backward probability is: Among them, γ t (i) represents the state occupancy probability of being in state i at time t, Ω t (i) represents the forward probability of being in state i at time t, Β t (i) represents the backward probability of being in state i at time t, W(O|λ) represents the total probability of the observation sequence O appearing when the model parameter λ is given, and E t (i,j) represents the state transition probability from state i to state j at time t, a ij represents the probability of state i transitioning to state j, b j (O t+1 ) represents the observation data O generated under state j t+1 The probability of O t+1 represents the data at time t+1, β t+1 (i) represents the backward probability of being in state j at time t+1; The initial state probability vector, state transition probability matrix and observation probability matrix are updated based on the state occupancy probability and state transition probability.
4. The method for evaluating the quality of monitoring data of new energy stations based on multi-source data according to claim 3, It is characterized in that The integrated learning network and data deviation identification model construct an observation sequence, convert the predicted hidden state sequence into state label data, and identify the data deviation between each monitoring data source. The main steps are: Integrated learning network and data bias identification model; Based on the initial state probability vector, the state transition probability matrix and the observation probability matrix; Align the observation data from each data source in time and fuse them into a multidimensional time series; The fused multi-dimensional time series is input into the trained data deviation recognition model as the observation sequence to construct the observation sequence; Initialize the path probability and path record matrix, and generate the initialized path probability matrix and path record matrix; Repeat the process for each moment, respectively calculate the path probability at each moment and record the optimal predecessor state of each state while calculating the path probability, update the path probability matrix and the path record matrix, until the optimal predecessor state of each state is reached; The calculation formula of the path probability at each moment is: d t+1 (i)=max[δ t (i)·a ij ]b j The t+1 ; Among them, δ t+1 (i) represents the optimal path probability at the i-th data point at time t+1, δ t (i) represents the optimal path probability at time t for the i-th group of data; Based on the optimal predecessor state, calculate the optimal path probability and optimal state at the final moment; Extract the optimal path probability from the final moment, determine the optimal state at the final moment, and initialize the optimal hidden state sequence; Using the path record matrix, trace back from time to time to determine the optimal state at each time; According to the optimal state at each moment obtained by backtracking, the optimal hidden state sequence is constructed, and the predicted hidden state sequence is converted into state label data; By comparing the differences between the observed data and the state label data of each monitoring data, the data deviations between the various monitoring data sources can be accurately identified.
5. The method for evaluating the quality of monitoring data of new energy stations based on multi-source data according to claim 4, It is characterized in that The integrated learning network and data deviation identification model mainly comprises the following steps: Through the logging module, the prediction results output by the model are collected once a minute for comparison with the prediction deviation, the actual deviation and the corresponding timestamp data; Use automated scripts to classify and annotate the collected data according to preset deviation types and severity standards, distinguish true positives, false positives, true negatives, and false negatives, and form annotated feedback data sets; Set a retraining day to trigger model retraining on that day of each month; For the hidden Markov model, the Baum-Welch algorithm is used to re-estimate the state transition probability matrix and the observation probability matrix; For the Isolation Forest model, adjust the number of trees and subsample size to ensure that the model adapts to the new data characteristics; Stochastic gradient descent is used as the online learning algorithm. A small batch of data is randomly extracted from the latest feedback data set each time to update the model parameters. The formula is: The formula for the loss function is; Among them, P(y|x; θ) represents the probability that the model predicts the category y under parameter θ, N represents the total number of samples, θ represents the model parameter, η represents the learning rate, and L(θ t ;x t ,y t ) represents the loss function, x t and t Represents the features and labels of the current sample, x i and i Represent the i-th input feature and output label respectively; Randomly initialize model parameters θ 0 , the samples in the data set are input in a stream manner, and a small batch of data is obtained each time. For each small batch of data, the gradient of the model parameters with respect to the loss function is calculated. According to the calculated gradient and the preset learning rate, the model parameters are updated, and it is continuously iterated until the data stream reaches the predetermined number of iterations; Using the deep Q network algorithm, the reward function is set to give a positive reward r for each correct warning. + , for each false alarm, r - , the reward design follows the following principles: For each correct warning: + =log e (1+V); For each error warning: r - =-log e (1+RL); Among them, V and RL represent the response speed and response delay of the system in the case of correct and false warning, respectively; Set the change rate threshold δ. When the change rate of the monitored data δ new When >δ, the system will determine that there may be an abnormal situation and trigger an early warning. The formula is: Set the abnormal score threshold s. When the abnormal score of a monitoring point is s new >s, trigger an early warning; δ new and new represent the new change rate threshold and anomaly score threshold, α and β are the false positive rate and false negative rate, respectively, and T is the average response time; One working day of each month is set as the evaluation day. Based on the warning effect and operation and maintenance efficiency improvement of the past month, the strategy evaluation is performed. The data deviation identification accuracy ACC formula is: Among them, TP represents the number of anomalies correctly identified by the system, TN represents the amount of data correctly judged as normal by the system, FP represents the number of normal data incorrectly judged as anomalies by the system, and FN represents the actual number of abnormal data that the system failed to identify; The formula for early warning response time RT is: Among them, T r,i represents the time point when the i-th abnormal event is responded by the system, T r Represents the response time, T t,i represents the time point when the ith abnormal event actually occurred, T t represents the actual occurrence time of the abnormal event, and N represents the total number of abnormal events during the monitoring period; The formula for improving the operation and maintenance efficiency percentage E is: Among them, OT represents the manual operation and maintenance time required to solve the same scale of problems before the introduction of the automated early warning system, and NT represents the actual time spent on solving the same problem after the system intervenes; The CPI formula that reflects overall performance is: Among them, RT max represents the maximum acceptable warning response time, α represents the adjustment factor, and w 1 、w 2 and w 3 They represent the weight coefficient ratios of accuracy, response time, and improvement percentage of operation and maintenance efficiency respectively.
6. The method for evaluating the quality of monitoring data of new energy stations based on multi-source data according to claim 5, It is characterized in that The calculation of data deviation anomaly scores, setting data quality levels for each data deviation point, and evaluating the impact of data deviation on the operating performance of new energy stations mainly include: Standardize each data deviation, and form a test data set with the standardized data deviations; Construct an anomaly monitoring model, set the parameters of the anomaly monitoring model, and use the test data set as training data to train the anomaly monitoring model. Use the trained anomaly monitoring model to predict the anomaly score of each data deviation in the test data set, traverse each data deviation in the test data set, input each data deviation into the isolated tree of the anomaly monitoring model, start from the root node, traverse downward according to the splitting condition of the isolated tree until reaching the leaf node, record the traversal path length of each data point in each tree, and for each data deviation, calculate the average of the path lengths of the data deviation in all isolated trees, and use the average as the final anomaly score of the current data deviation, and normalize the anomaly score to a preset range; The calculation formula of the abnormal score is: Among them, S(x) represents the anomaly score of data deviation x, represents the average path length of data deviation x in all isolated trees, C(n) represents the adjustment coefficient, and n represents any positive integer; Set the data quality level based on the anomaly score. The level can be set based on the score range: High-quality data: anomaly scores are in the range of 0-a; Good data: the anomaly score is within the range of ab; Medium data: anomaly score is within the bc range; Low-quality data: anomaly scores are within the cd range; Abnormal data: the abnormal score is within the range of de; Assign a corresponding quality level to each data point according to its anomaly score Each abnormal data point is assigned a score, which can be set based on the anomaly score: Data with abnormal scores within the range of de are scored as E' (the highest score); Data with abnormal scores within the range of cd are scored as D'; Data with abnormal scores within the range of bc are scored as C'; Data with abnormal scores within the range of ab are classified as B'; Data with anomaly scores in the range of 0-a are scored as A' (the lowest score); The impact of data deviation on the operating performance of new energy stations is evaluated based on the scoring of abnormal data points.
7. The method for evaluating the quality of monitoring data of new energy stations based on multi-source data according to claim 6, It is characterized in that Based on the data quality level, a real-time monitoring platform is established to track the changing trend of data deviations, set a change rate threshold, and issue an early warning for data deviations that exceed the change rate threshold. The main steps are: Establish a real-time monitoring platform based on data quality levels; Perform a difference operation on the data deviation index to obtain a difference value, perform a stability test on the difference value, and use the deviation value after the stability test as the rate of change of the data deviation; Set the data deviation change rate threshold to μ 0 , the formula for the deviation change rate μ is: μ=k〃σ; Where σ represents the standard deviation of the data, and k represents the safety factor; The objective function G(k) is introduced to evaluate the early warning effect under different k values, so as to obtain the optimal safety factor k, the formula is: Among them, FPR(k) represents the false alarm rate, FNR(k) represents the missed alarm rate, and w 4 and w 5 Represent the weight coefficients of false alarm rate and missed alarm rate respectively; The formula for the optimal safety factor k is: When μ>μ 0 When an alarm occurs, an early warning is triggered and the early warning information is conveyed to the operation and maintenance personnel, who then analyze the early warning information: Collect all data deviations whose change rate exceeds a preset threshold; Classify the data deviations exceeding the threshold by type and obtain abnormal data points of each category; Conduct root cause analysis for each category of abnormal data points to obtain the causes of data deviation; Based on the analysis results, the early warning rules are regularly evaluated and adjusted to optimize the real-time monitoring platform.
8. A new energy station monitoring data quality evaluation system based on multi-source data, used to implement the new energy station monitoring data quality evaluation method based on multi-source data according to any one of claims 1 to 7, It is characterized in that The new energy station monitoring data quality evaluation system includes: data acquisition and preprocessing module, data deviation identification model training module, integrated learning network and deviation identification module, data deviation anomaly score and quality assessment module and real-time monitoring and early warning response module; The data acquisition and preprocessing module is used to obtain the multi-source monitoring data collected by various monitoring devices in the new energy station and pre-process the multi-source monitoring data; A data deviation identification model training module is used to train a data deviation identification model based on the preprocessed multi-source monitoring data using the Baum-Welch algorithm; The integrated learning network and deviation identification module is used to integrate the learning network and the data deviation identification model, construct the observation sequence, convert the predicted hidden state sequence into state label data, and identify the data deviation between various monitoring data sources; The data deviation anomaly score and quality assessment module is used to calculate the data deviation anomaly score, set the data quality level for each data deviation point, and evaluate the impact of data deviation on the operating performance of new energy stations; The real-time monitoring and early warning response module is used to establish a real-time monitoring platform, track the changing trend of data deviations, set change rate thresholds, and issue early warnings for data deviations that exceed the change rate thresholds.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. Features: When the processor executes the computer program, the steps of the file encryption method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, Features: When the computer program is executed by a processor, the steps of the file encryption method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Intelligent data management system based on large model
CN119474770A
Multi-dimensional data quality detection method based on data medium table
CN119645984A
Multidimensional data quality detection method based on data center
CN119645984B
Power cable intelligent segmentation management method and system
CN119671205A
Quality management method based on standard data
CN119884102A
