Method, system and equipment for preprocessing and normalizing digital power grid protection efficiency evaluation data and medium

Abstract

The invention discloses a digital power grid protection efficiency evaluation data preprocessing and normalization method, system, equipment and medium, and belongs to the technical field of power system monitoring and information processing, and the method comprises the steps: collecting digital power grid monitoring original data, carrying out time alignment, generating a data record, carrying out the preprocessing, obtaining a historical data set, and storing the historical data set in a database; global statistical parameters are calculated and stored in a global statistical parameter storage library, real-time data of a power grid are received and processed, a real-time stream processing flow is constructed, the global statistical parameters are loaded, data conversion is carried out on the real-time data after data processing, standardized data are generated and analyzed, and the global statistical parameters are monitored in real time and periodically updated. And comparing the historical data set with the standardized data, calculating a deviation value, and generating an alarm notice and performing troubleshooting when the deviation value exceeds a set deviation value threshold, thereby solving the problems that the digital power grid operation data quality is different, the real-time analysis benchmark drifts and the system state evaluation is inconsistent.

Description

Technical Field

[0001] This invention relates to the field of power system monitoring and information processing technology, specifically to a method, system, equipment, and medium for preprocessing and normalizing digital power grid protection effectiveness assessment data. Background Technology

[0002] With the deepening of digital transformation, power systems are gradually developing towards intelligence and networking, and digital power grids have become an important form of modern power systems. Digital power grids, through the deployment of numerous intelligent sensors, monitoring equipment, and control terminals, achieve comprehensive perception and real-time monitoring of the power grid's operating status, generating massive amounts of multi-source heterogeneous operational data. To ensure the safe and stable operation of digital power grids, continuous evaluation and monitoring of their protective effectiveness are necessary, requiring the ability to process massive amounts of power grid data in real time and generate reliable evaluation indicators. Currently, streaming computing technologies such as Apache Flink and Apache Storm have been widely applied to real-time processing scenarios for power big data. Data normalization, as a key step in data preprocessing, can eliminate the influence of different dimensions and numerical ranges on subsequent analysis, providing a foundation for building a unified evaluation model.

[0003] Existing digital power grid protection effectiveness assessment technologies generally adopt an instantaneous normalization processing method based on real-time data streams, lacking a globally unified parameter benchmark based on historical data. This results in the use of local statistical parameters or fixed empirical values ​​for normalization processing at different time periods and data sources, leading to a lack of comparability and consistency in assessment results at different times and in different scenarios. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing technology generally adopts an instantaneous normalization processing method based on real-time data streams, which lacks a globally unified parameter benchmark based on historical data, resulting in the use of local statistical parameters or fixed empirical values ​​for normalization processing of different time periods and different data sources.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for preprocessing and normalizing data for digital power grid protection effectiveness assessment, comprising, Collect raw data from digital power grid monitoring, align the raw data to time, generate data records, preprocess the data records, and obtain historical datasets. Based on historical datasets, global statistical parameters are calculated and stored in a global statistical parameter storage library. Real-time power grid data is received, a real-time stream processing flow is constructed, the real-time power grid data is processed, global statistical parameters are loaded, and the processed real-time data is transformed to generate standardized data. The system performs data analysis on standardized data, monitors and updates global statistical parameters in real time and regularly, compares historical datasets with standardized data, calculates deviation values, and generates alarm notifications and conducts investigations when deviation values ​​exceed the set threshold.

[0007] As a preferred embodiment of the method for preprocessing and normalizing data for evaluating the protection effectiveness of a digital power grid according to the present invention, the following steps are included: collecting raw digital power grid monitoring data, performing time alignment on the raw digital power grid monitoring data to generate data records, preprocessing the data records to obtain a historical dataset. Collect raw data from digital power grid monitoring, perform time alignment on the raw data, and generate data records; Identify missing data in data records, fill in the missing data in data records using missing value processing, and remove data records that exceed the first threshold; The data records after removing data exceeding the first threshold are subjected to noise reduction processing to obtain the historical dataset.

[0008] As a preferred embodiment of the method for data preprocessing and normalization for digital power grid protection effectiveness assessment described in this invention, the steps include: calculating global statistical parameters based on historical datasets and storing them in a global statistical parameter storage library; receiving real-time power grid data; constructing a real-time stream processing flow; processing the real-time power grid data; loading global statistical parameters; and transforming the processed real-time data to generate standardized data. Based on historical datasets, calculate global statistical parameters and store them in a global statistical parameter repository; Based on global statistical parameters, real-time power grid data is received, a real-time stream processing flow is constructed, and the real-time power grid data is parsed and cleaned. Based on the parsed and cleaned real-time power grid data, global statistical parameters are loaded, and the parsed and cleaned real-time data is transformed to generate standardized data.

[0009] As a preferred embodiment of the method for data preprocessing and normalization for digital power grid protection effectiveness assessment described in this invention, the steps include: performing data analysis on the normalized data, monitoring and periodically updating global statistical parameters, comparing historical datasets and normalized data, calculating deviation values, and generating alarm notifications and conducting investigations when deviation values ​​exceed a set threshold. The standardized data is analyzed, a periodic scheduling strategy is set, and the offline batch processing process is triggered at regular intervals to generate updated global statistical parameters. Based on the updated global statistical parameters, the historical dataset and normalized data are compared, and the deviation value is calculated. Determine whether the deviation value exceeds the deviation value threshold. If the deviation value exceeds the set deviation value threshold, generate an alarm notification and conduct an investigation.

[0010] As a preferred embodiment of the method for preprocessing and normalizing data for evaluating the protection effectiveness of a digital power grid according to the present invention, wherein: the method of filling missing data in data records by handling missing values ​​includes, The completion is performed using a preset nearest-neighbor linear interpolation algorithm, expressed as: in, For the generated completion value, , These are the most recent valid data and timestamp before the missing point, respectively. , These are the most recent valid data and the timestamp, respectively. The timestamp for the missing data point.

[0011] As a preferred embodiment of the method for preprocessing and normalizing data for evaluating the protection effectiveness of a digital power grid according to the present invention, the step of performing data transformation on the parsed and cleaned real-time data to generate standardized data includes: The Z-Score normalization algorithm is used, and the expression is: in, For standardized scores, These are the real-time data values ​​obtained after noise filtering. and These are the global mean and global standard deviation corresponding to the standardized scores, obtained from the memory cache, respectively; The unbounded Z-score is mapped to a preset numerical range, and then the Sigmoid function is used for the mapping, with the expression: in, For the final generated normalized data, For standardized scores, It is the base of the natural logarithm. This is the real-time data value obtained after noise filtering.

[0012] This invention employs a two-step data transformation method combining Z-Score standardization and Sigmoid function mapping to preserve the distribution characteristics of data under a global statistical benchmark, ensuring the consistency of the output value range of all evaluation indicators. This provides numerically stable input characteristics for protection effectiveness evaluation and solves the problems of distribution adaptability and output range control in real-time data normalization processing.

[0013] As a preferred embodiment of the method for preprocessing and normalizing data for evaluating the protection effectiveness of a digital power grid as described in this invention, the calculated deviation value includes: The numerical deviation between offline and online verification data over the entire window is calculated using the mean absolute error algorithm, expressed as: in, The offline and online verification data show the numerical deviation within the entire window, where M is the total number of data points within the window. and These are the offline and online verification data at the j-th time point, respectively.

[0014] This invention introduces the mean absolute error algorithm to quantitatively compare offline and online verification data, which can capture the deviation of normalization results caused by global statistical parameter updates or real-time processing logic anomalies. It transforms the original process consistency check that relied on experience judgment into a quantifiable numerical indicator, thereby realizing real-time diagnosis of the health status of the data processing pipeline.

[0015] This invention provides a system for preprocessing and normalizing data for evaluating the protection effectiveness of digital power grids.

[0016] To address the aforementioned technical problems, this invention provides the following technical solution: a system for preprocessing and normalizing data for digital power grid protection effectiveness assessment, comprising a data acquisition and preprocessing module, a global parameter calculation and data processing module, and a verification and comparison module. The data acquisition and preprocessing module acquires raw data from digital power grid monitoring, performs time alignment on the raw data, generates data records, preprocesses the data records, and obtains historical datasets. The global parameter calculation and data processing module calculates global statistical parameters based on historical datasets and stores them in a global statistical parameter storage library. It receives real-time power grid data, constructs a real-time stream processing flow, processes the real-time power grid data, loads global statistical parameters, transforms the processed real-time data, and generates standardized data. The verification and comparison module performs data analysis on the standardized data, monitors and updates global statistical parameters in real time and periodically, compares the historical dataset with the standardized data, calculates the deviation value, and generates an alarm notification and conducts investigation when the deviation value exceeds the set threshold.

[0017] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the method for preprocessing and normalizing data for evaluating the effectiveness of digital power grid protection.

[0018] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the method for preprocessing and normalizing data for evaluating the effectiveness of digital power grid protection.

[0019] The beneficial effects of this invention are as follows: By constructing a global statistical parameter benchmark based on historical data and establishing a collaborative mechanism between real-time stream processing and offline batch processing, this invention solves the benchmark inconsistency problem caused by the reliance on local parameters or fixed thresholds in traditional real-time normalization methods. By utilizing historical full-set data to calculate statistical characteristics, the globality and stability of the normalization benchmark are ensured. Through dynamic parameter updates and consistency verification, the self-adaptation and self-monitoring of the data processing flow are realized, avoiding evaluation deviations caused by data distribution drift or abnormal processing logic, and improving the accuracy of digital power grid protection effectiveness evaluation results and system reliability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 The above is a flowchart of a method for preprocessing and normalizing data for evaluating the protection effectiveness of a digital power grid, provided as an embodiment of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0023] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a method for preprocessing and normalizing digital power grid protection effectiveness assessment data, including: To address the problem that existing technologies commonly employ real-time normalization based on real-time data streams, which lack a globally unified parameter benchmark based on historical data, leading to the use of local statistical parameters or fixed empirical values ​​for normalization processing at different time periods and data sources, this invention provides a method for data preprocessing and normalization of digital power grid protection effectiveness assessment.

[0024] S1: Collect raw data from digital power grid monitoring, align the raw data to time, generate data records, preprocess the data records, and obtain historical datasets.

[0025] S2: Based on historical datasets, calculate global statistical parameters and store them in a global statistical parameter storage library. Receive real-time power grid data, construct a real-time stream processing flow, process the real-time power grid data, load global statistical parameters, transform the processed real-time data, and generate standardized data.

[0026] S3: Perform data analysis on the normalized data, monitor and update global statistical parameters in real time and periodically, compare historical datasets and normalized data, calculate the deviation value, generate alarm notifications and conduct investigations when the deviation value exceeds the set threshold.

[0027] Therefore, this invention ensures the temporal consistency and quality of historical datasets through time alignment and preprocessing. It calculates global parameters offline based on historical data and applies them to real-time stream processing, ensuring the uniformity and stability of the real-time data normalization benchmark. It introduces a dynamic parameter update and data consistency verification process, periodically corrects global parameters to adapt to changes in system state, and uses cross-validation of offline and real-time processing results to solve the problem of evaluation distortion caused by data benchmark drift or abnormal processing logic, thereby improving the accuracy of state perception and the reliability of system operation and maintenance.

[0028] Example 2, an embodiment of the present invention, provides a method for preprocessing and normalizing digital power grid protection effectiveness assessment data based on the previous embodiment, including: In this embodiment of the application, step S1 involves collecting raw digital power grid monitoring data, aligning the raw digital power grid monitoring data by time, generating data records, preprocessing the data records, and obtaining a historical dataset, including the following steps A1-A3: A1: Collect raw data from digital power grid monitoring, align the raw data to the time, and generate data records.

[0029] By deploying a data acquisition server, which is equipped with various data access adapters, the system establishes data interfaces with different data sources in the digital power grid. Specifically, the system can communicate with monitoring equipment using the OPCUA interface protocol, with control terminals using the ModbusTCP protocol, with various sensors using the MQTT protocol, and is configured with a Syslog receiver to obtain the operation log data of the safety system. Through these preset data interfaces, the data acquisition server can continuously receive heterogeneous raw operation data from various source devices.

[0030] A2: Identify missing data in data records, fill in the missing data in data records using missing value processing, and remove data records that exceed the first threshold.

[0031] After the batch processing task starts, the system first performs field type and unit verification and conversion. The system loads a preset metadata mapping table, which defines in detail the standard data type (such as double-precision floating-point) and standard physical unit of each index field in the digital power grid. When the system detects that the index field of a record is "line voltage", the original data type is a string and the value is 220.3, the system converts it to double-precision floating-point according to the mapping table. If the original unit is kV and the standard unit is V, the system performs a conversion to convert 0.2203 to 220.3 to ensure that the data of the same index are completely consistent in units.

[0032] The system performs abnormal data cleaning. It obtains the preset validity threshold ranges for each indicator from the configuration library. For example, the threshold for the power grid frequency is 49.0Hz to 51.0Hz, and the threshold for the 220V nominal line is 200V to 240V. The batch processing checks each record one by one. If a frequency of 52.5Hz or a line voltage of 180V is found to exceed the threshold, it is regarded as severely distorted or faulty data and is removed.

[0033] A3: Perform noise reduction processing on the data records after removing data that exceeds the first threshold to obtain the historical dataset.

[0034] A preset noise reduction algorithm is applied to the cleaned data records to filter out high-frequency noise.

[0035] The N-point moving average algorithm is used, and its expression is: in, The output result is smoothed. The size of the sliding window (e.g., 5). These are the original data values ​​after abnormal cleaning.

[0036] After the stages of field unification, missing data completion, anomaly removal, and noise reduction filtering, the processed data set is persistently stored to form a preprocessed historical dataset.

[0037] In this embodiment of the application, the time alignment of the raw digital power grid monitoring data in step S1 specifically employs a time offset calibration algorithm: Alignment is achieved by applying a time calibration offset, the calculation formula of which is: in, The final timestamp generated after calibration and alignment. This is the device reporting timestamp extracted from the raw operational data; this input parameter This refers to data from a specific technical field, such as the raw voltage measurement time reported by a monitoring device in a substation within a digital power grid. The time calibration offset between the device clock and the global synchronization clock source is pre-determined or calculated in real time.

[0038] The system then fills the specific operating values ​​from the original operating data, such as voltage values, into the indicator field, and the system calculates... Enter the timestamp field and output the result. This is the unified reference time for the voltage measurement data throughout the entire evaluation system. The system fills the unique identifier of the data source device into the source field. Through mapping operations, the system generates data records in a unified format that meet the requirements from the original operating data.

[0039] In an optional implementation, the time alignment of the raw digital power grid monitoring data in step S1 can also be performed using the event sequence matching alignment method. By identifying the characteristic points of these related events across data sources, and using the precise event time recorded by the master station as a benchmark, the time offset of the data stream of each device is dynamically calculated and corrected to achieve event-triggered alignment.

[0040] In another optional implementation, the time alignment of the raw digital power grid monitoring data in step S1 can also be performed using the sliding window correlation alignment method. The time shift corresponding to the maximum correlation is found by calculating the cross-correlation function, and this amount is used as the dynamic time offset of the window for alignment.

[0041] In the embodiments of this application, the original data of digital power grid monitoring is time-aligned in step S1. The time offset calibration algorithm effectively solves the problem of data timing disorder caused by clock asynchrony of multi-source heterogeneous devices.

[0042] In this embodiment of the application, the missing data filling process in step A2 specifically employs a preset nearest neighbor linear interpolation algorithm: The completion is performed using a preset nearest-neighbor linear interpolation algorithm, expressed as: in, For the generated completion value, , These are the most recent valid data and timestamp before the missing point, respectively. , These are the most recent valid data and the timestamp, respectively. The timestamp for the missing data point.

[0043] In one alternative implementation, the missing data filling method in step A2 can also be the historical average imputation method. When data is missing, the average value of normal data from the same historical period is extracted and used to fill the missing data.

[0044] In another alternative implementation, the missing data filling method in step A2 can also be used, where the state-preserving filling method is used. When missing data is detected, the value of the previous valid data point is directly used as the filling value.

[0045] In this embodiment of the application, step A2 uses missing value processing to fill in the missing data in the data record, restores the continuous change trend of the data in the time series, and the completion result conforms to the actual change law of the physical quantity, thus ensuring the temporal integrity and consistency of the dataset.

[0046] In this embodiment of the application, step S2 involves calculating global statistical parameters based on historical datasets and storing them in a global statistical parameter storage library. It also includes receiving real-time power grid data, constructing a real-time stream processing flow, processing the real-time power grid data, loading global statistical parameters, transforming the processed real-time data, and generating standardized data. This includes the following steps B1-B3: B1: Based on historical datasets, calculate global statistical parameters and store them in the global statistical parameter storage library.

[0047] B2: Based on global statistical parameters, receive real-time power grid data, construct a real-time stream processing flow, and parse and clean the real-time power grid data.

[0048] By configuring a streaming data receiving channel to continuously capture real-time data streams from power grid equipment, specifically by subscribing to multiple preset message queue topics, these topics are dedicated to receiving operational data pushed in real time by power grid monitoring equipment, control terminals, and sensors. When the application receives a payload aho of a real-time data stream from the message queue, it parses the payload aho according to a preset communication protocol, such as JSON format, to extract the real-time data containing the indicator name and raw value.

[0049] To remove random interference components from the signal, the application immediately invokes a preset noise filtering algorithm to filter the extracted real-time data. This implementation uses the exponential moving average algorithm, which is suitable for real-time scenarios due to its low computational cost and higher weighting of new data. The expression is: ; in, It is the smoothed data output result after noise reduction generated at time t. It is the raw real-time data value parsed at time t. It is the previous time step The calculated smoothed data values ​​are stored as state in the stream processing application. It is a preset smoothing factor, with a value between 0 and 1, which determines the weight of new data and historical data. Specific input parameters... It could be the instantaneous current measurement reported by a feeder in a digital power grid, and the related output results This is a smoother representation of the current value with less jitter and a better reflection of the trend.

[0050] B3: Based on the parsed and cleaned real-time power grid data, load global statistical parameters, perform data transformation on the parsed and cleaned real-time data, and generate standardized data.

[0051] Before performing normalization on the real-time data, the real-time stream processing application establishes a memory cache upon startup, connects to the global parameter repository, reads all global statistical parameters at once, and loads them into the memory cache, forming a fast query mapping with the metric name as the key and the statistical parameter object as the value. For each piece of real-time data that has been filtered for noise, the application will quickly retrieve its corresponding global mean and global standard deviation from the memory cache based on the metric name and perform standardization calculations.

[0052] The Z-Score normalization algorithm is used, and the expression is: in, For standardized scores, These are the real-time data values ​​obtained after noise filtering. and These are the global mean and global standard deviation corresponding to the standardized scores, obtained from the memory cache, respectively; The unbounded Z-score is mapped to a preset numerical range, and then the Sigmoid function is used for the mapping, with the expression: in, For the final generated normalized data, For standardized scores, It is the base of the natural logarithm. This is the real-time data value obtained after noise filtering.

[0053] In this embodiment of the application, the calculation of global statistical parameters in step S2 is specifically manifested as follows: Offline statistical analysis was adopted, and a preset indicator configuration table was loaded to determine the normalization algorithm type and required statistical type of each protection effectiveness evaluation indicator in the historical dataset. For example, the configuration table specifies that the line load rate indicator will subsequently adopt the Z-Score normalization algorithm, so its required statistical type is global average and global standard deviation; at the same time, it specifies that the alarm event frequency indicator adopts the Min-Max normalization algorithm, and it is necessary to calculate the global maximum value and global minimum value.

[0054] Perform overall statistical calculations on all records in the preprocessed historical dataset.

[0055] For metrics requiring Z-score standardization, the module calculates the global mean and global standard deviation, expressed as follows: in, This represents the total number of records for this indicator. For the first Line load rate data values, This is the global average. This represents the global standard deviation.

[0056] For metrics that require Min-Max normalization, iterate through the metrics... Count records, get the global maximum value and global minimum After the calculation is completed, the system will display the statistical values ​​(such as...). , , and These are combined to form a global set of statistical parameters.

[0057] The global statistical parameter set is structured and written to a global parameter repository. This repository can be a cache database. The parameter data structure uses key-value pairs, where the key is a unique name of the evaluation metric (such as line load rate), and the value is a structured object containing all statistical parameters.

[0058] In an alternative implementation, the calculation of global statistical parameters in step S2 can also employ a sliding window statistical method, maintaining a historical dataset of a recent time window (such as the last 30 days), and periodically recalculating the global statistical parameters within this window.

[0059] In another optional implementation, the calculation of global statistical parameters in step S2 can also adopt an incremental statistical algorithm. The system does not store all historical data, but calculates new global statistical parameters based on the existing statistical results and the data of the newly added batch through an incremental update formula.

[0060] In the embodiments of this application, the calculation of global statistical parameters in step S2 solves the problem of inconsistency in benchmarks caused by the use of local statistics or empirical values ​​in traditional methods, thereby improving the accuracy and comparability of the protection effectiveness assessment results.

[0061] It should be noted that global statistical parameters are calculated offline based on historical data and cached to provide a stable normalized benchmark for real-time stream processing. In the real-time processing stage, a lightweight exponential moving average is used for noise filtering. By combining the Z-Score and the Sigmoid function, the real-time data is converted into a normallyized value with a stable distribution and limited range, reducing real-time calculation latency and enhancing the system's adaptability and reliability in the dynamic environment of the real power grid.

[0062] In this embodiment of the application, step S3 involves data analysis of the normalized data, real-time monitoring and periodic updating of global statistical parameters, comparison of historical datasets and normalized data, calculation of deviation values, and generation of alarm notifications and investigation when deviation values ​​exceed a set threshold. This includes the following steps C1-C3: C1: Perform data analysis on the normalized data, set a periodic scheduling strategy, trigger the offline batch processing process at regular intervals, and generate updated global statistical parameters.

[0063] It receives the generated normalized data stream. When a set of normalized data is received, it calculates and generates a comprehensive protection performance index value based on the preset protection performance evaluation model.

[0064] For each normalized index value Risk mapping is performed by using a piecewise linear function to map the risk contribution level. (Range 0~1), the expression is: like ,but ; like ,but ; like ,but .

[0065] in , These are the low and high thresholds configured by the system, respectively. For normalized index values, Contribution to risk.

[0066] Subsequently, the system uses a weighted summation algorithm to aggregate all risk contributions: in, The comprehensive protective effectiveness index is set at a value of 0 to 1, where 0 indicates good effectiveness and 1 indicates extremely poor effectiveness. The total number of normalized indicators to be evaluated. Preset weighting coefficients (satisfying) All are loaded from the configuration. Contribution to risk.

[0067] Calculated Then, the unit compares it with a preset alarm threshold, such as a minor alarm threshold. With critical alarm threshold .like This will trigger a minor alarm. This will trigger a critical alarm, and upon triggering, the system will generate alarm event data. Minor alarm threshold. With critical alarm threshold It can be set based on experience.

[0068] The structured message body includes a unique identifier, a trigger timestamp, and an alarm severity level. Values, the original normalized data components that caused the exceedance, and their corresponding risk contributions. This provides traceability information for operations and maintenance personnel.

[0069] After the offline batch processing flow is triggered, a comprehensive statistical analysis is performed on the historical dataset. A single-pass parallel algorithm is used to recalculate the global statistical parameters of each protection effectiveness evaluation index. Taking the transformer load rate, an index that requires ZScore standardization, as an example, the global mean and global standard deviation are recalculated. The calculation formula is as follows: in, , These are the recalculated global mean and global standard deviation, respectively, where N is the total number of records for this indicator in the current historical dataset. For each load rate data point, For the sum, The sum of squares is calculated, and an updated set of global statistical parameters is generated after the calculation.

[0070] The offline batch processing workflow writes the set to the global parameter repository and updates these parameters with a version identifier (incrementing sequence number or timestamp), such as 20251110001.

[0071] The real-time stream processing workflow continuously monitors the version identifier of the global parameter repository and uses timed queries (e.g., every 5 seconds) to obtain the latest version number. When a version number change is detected (e.g., from "20251103001" to "20251110001"), the real-time stream processing immediately reads the latest global statistical parameters from the repository and replaces the old parameters in the memory cache with them. The global average and standard deviation used for normalization of indicators such as "transformer load rate" are updated to the latest values ​​in real time.

[0072] C2: Based on the updated global statistical parameters, compare the historical dataset and the normalized data to calculate the deviation value.

[0073] Set a verification time window, extract source data samples within the window from the preprocessed historical dataset, connect to the global parameter repository, load the latest version of global statistical parameters, and then perform the same normalization calculation on the extracted source data samples as in the real-time process to generate offline verification data.

[0074] For the active power index of the line, its latest global average value is loaded. and global standard deviation Then, Z-Score normalization and the Sigmoid mapping function are applied to each source data sample V_sample: in, These are the raw measured values ​​of line active power extracted from historical datasets within the verification window. , These are the recalculated global mean and global standard deviation, respectively. This is for offline verification data.

[0075] After generating a complete offline verification sequence, query the data archive that stores the real-time process output results to obtain the normalized data (i.e., online verification data) generated in real time within the same verification window.

[0076] C3: Determine whether the deviation value exceeds the deviation value threshold. If the deviation value exceeds the set deviation value threshold, generate an alarm notification and conduct an investigation.

[0077] The module compares the calculated deviation Q with the preset deviation threshold. If Q exceeds the threshold, it determines that the normalization calculation logic and parameter benchmark are inconsistent, and generates a system-level alarm to notify the operation and maintenance personnel to investigate.

[0078] In this embodiment of the application, the calculation of the deviation value in step C2 is specifically manifested as follows: The numerical deviation between offline and online verification data over the entire window is calculated using the mean absolute error algorithm, expressed as: in, The offline and online verification data show the numerical deviation within the entire window, where M is the total number of data points within the window. and These are the offline and online verification data at the j-th time point, respectively.

[0079] In an alternative implementation, the root mean square error method can also be used to calculate the deviation value in step C2. By squaring the error, the contribution of larger errors can be amplified, and the calculation result can more severely punish outliers.

[0080] In another optional implementation, the deviation value in step C2 can also be calculated using the consistency correlation coefficient method to assess whether the data closely follows a linear distribution with a slope of 1, measure whether the data changes synchronously, verify whether the actual values ​​match, and evaluate the consistency between offline and online data in terms of trend and numerical level.

[0081] In the embodiments of this application, the calculation of deviation value in step C2 can effectively identify numerical deviations caused by global parameter update delays or real-time processing logic anomalies, transforming the consistency assessment of data processing from a qualitative judgment into a quantifiable monitoring indicator.

[0082] In summary, this invention employs time offset calibration and multi-protocol adaptation access to ensure the temporal consistency of heterogeneous data sources. By combining offline calculation of historical data with real-time stream processing, and utilizing methods such as moving average, exponential moving average, and Z-score standardization, it balances data quality and computational efficiency during denoising and normalization. Furthermore, it introduces a dynamic update and deviation verification mechanism for global statistical parameters, ensuring the adaptability of parameters to changes in power grid status through periodic recalculation and versioned release. Through consistency comparison of offline and real-time normalization results, it achieves closed-loop verification and anomaly alarm of the data processing flow, solving the core problems of inconsistent data quality, real-time analysis benchmark drift, and inconsistent system status assessment in digital power grid operation, thereby improving the accuracy of status assessment, the real-time performance of system response, and the reliability of operation and maintenance.

[0083] Example 3 is an embodiment of the present invention, which provides a system for preprocessing and normalizing digital power grid protection effectiveness assessment data, including a data acquisition and preprocessing module, a global parameter calculation and data processing module, and a verification and comparison module. The data acquisition and preprocessing module acquires raw data from digital power grid monitoring, performs time alignment on the raw data, generates data records, preprocesses the data records, and obtains historical datasets. The global parameter calculation and data processing module calculates global statistical parameters based on historical datasets and stores them in a global statistical parameter storage library. It receives real-time power grid data, constructs a real-time stream processing flow, processes the real-time power grid data, loads global statistical parameters, transforms the processed real-time data, and generates standardized data. The verification and comparison module performs data analysis on the standardized data, monitors and updates global statistical parameters in real time and periodically, compares the historical dataset with the standardized data, calculates the deviation value, and generates an alarm notification and conducts investigation when the deviation value exceeds the set threshold.

[0084] This embodiment also provides an electronic device applicable to a method for preprocessing and normalizing data for digital power grid protection effectiveness assessment, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method for preprocessing and normalizing data for digital power grid protection effectiveness assessment as proposed in the above embodiment.

[0085] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements a method for preprocessing and normalizing digital power grid protection effectiveness assessment data as proposed in the above embodiments.

[0086] The storage medium proposed in this embodiment and the method for preprocessing and normalizing digital power grid protection effectiveness assessment data proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0087] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0088] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for preprocessing and normalizing data for evaluating the protection effectiveness of a digital power grid, characterized in that: include, Collect raw data from digital power grid monitoring, align the raw data to time, generate data records, preprocess the data records, and obtain historical datasets. Based on historical datasets, global statistical parameters are calculated and stored in a global statistical parameter storage library. Real-time power grid data is received, a real-time stream processing flow is constructed, the real-time power grid data is processed, global statistical parameters are loaded, and the processed real-time data is transformed to generate standardized data. The system performs data analysis on standardized data, monitors and updates global statistical parameters in real time and regularly, compares historical datasets with standardized data, calculates deviation values, and generates alarm notifications and conducts investigations when deviation values ​​exceed the set threshold.

2. The method for preprocessing and normalizing digital power grid protection effectiveness assessment data as described in claim 1, characterized in that: The process involves collecting raw digital power grid monitoring data, performing time alignment on the raw data, generating data records, preprocessing the data records, and deriving a historical dataset. Collect raw data from digital power grid monitoring, perform time alignment on the raw data, and generate data records; Identify missing data in data records, fill in the missing data in data records using missing value processing, and remove data records that exceed the first threshold; The data records after removing data exceeding the first threshold are subjected to noise reduction processing to obtain the historical dataset.

3. The method for preprocessing and normalizing digital power grid protection effectiveness assessment data as described in claim 2, characterized in that: The process involves calculating global statistical parameters based on historical datasets and storing them in a global statistical parameter repository. It also includes receiving real-time power grid data, constructing a real-time stream processing workflow, processing the real-time power grid data, loading the global statistical parameters, transforming the processed real-time data, and generating standardized data. Based on historical datasets, calculate global statistical parameters and store them in a global statistical parameter repository; Based on global statistical parameters, real-time power grid data is received, a real-time stream processing flow is constructed, and the real-time power grid data is parsed and cleaned. Based on the parsed and cleaned real-time power grid data, global statistical parameters are loaded, and the parsed and cleaned real-time data is transformed to generate standardized data.

4. The method for preprocessing and normalizing digital power grid protection effectiveness assessment data as described in claim 3, characterized in that: The process of performing data analysis on standardized data, monitoring and periodically updating global statistical parameters, comparing historical datasets with standardized data, calculating deviation values, generating alarm notifications and conducting investigations when deviation values ​​exceed set thresholds includes... The standardized data is analyzed, a periodic scheduling strategy is set, and the offline batch processing process is triggered at regular intervals to generate updated global statistical parameters. Based on the updated global statistical parameters, the historical dataset and normalized data are compared, and the deviation value is calculated. Determine whether the deviation value exceeds the deviation value threshold. If the deviation value exceeds the set deviation value threshold, generate an alarm notification and conduct an investigation.

5. The method for preprocessing and normalizing digital power grid protection effectiveness assessment data as described in claim 4, characterized in that: The method of filling missing data in data records using missing value processing includes, The completion is performed using a preset nearest-neighbor linear interpolation algorithm, expressed as: in, For the generated completion value, , These are the most recent valid data and timestamp before the missing point, respectively. , These are the most recent valid data and the timestamp, respectively. The timestamp for the missing data point.

6. The method for preprocessing and normalizing digital power grid protection effectiveness assessment data as described in claim 5, characterized in that: The process of transforming the parsed and cleaned real-time data to generate standardized data includes: The Z-Score normalization algorithm is used, and the expression is: in, For standardized scores, These are the real-time data values ​​obtained after noise filtering. and These are the global mean and global standard deviation corresponding to the standardized scores, obtained from the memory cache, respectively; The unbounded Z-score is mapped to a preset numerical range, and then the Sigmoid function is used for the mapping, with the expression: in, For the final generated normalized data, For standardized scores, It is the base of the natural logarithm. This is the real-time data value obtained after noise filtering.

7. The method for preprocessing and normalizing digital power grid protection effectiveness assessment data as described in claim 6, characterized in that: The calculated deviation value includes, The numerical deviation between offline and online verification data over the entire window is calculated using the mean absolute error algorithm, expressed as: in, The offline and online verification data show the numerical deviation within the entire window, where M is the total number of data points within the window. and These are the offline and online verification data at the j-th time point, respectively.

8. A system for preprocessing and normalizing data for digital power grid protection effectiveness assessment, comprising applying the method for preprocessing and normalizing data for digital power grid protection effectiveness assessment as described in any one of claims 1 to 7, characterized in that, The system includes a data acquisition and preprocessing module, a global parameter calculation and data processing module, and a verification and comparison module. The data acquisition and preprocessing module acquires raw data from digital power grid monitoring, performs time alignment on the raw data, generates data records, preprocesses the data records, and obtains historical datasets. The global parameter calculation and data processing module calculates global statistical parameters based on historical datasets and stores them in a global statistical parameter storage library. It receives real-time power grid data, constructs a real-time stream processing flow, processes the real-time power grid data, loads global statistical parameters, transforms the processed real-time data, and generates standardized data. The verification and comparison module performs data analysis on the standardized data, monitors and updates global statistical parameters in real time and periodically, compares the historical dataset with the standardized data, calculates the deviation value, and generates an alarm notification and conducts investigation when the deviation value exceeds the set threshold.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for preprocessing and normalizing digital power grid protection effectiveness assessment data according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for preprocessing and normalizing digital power grid protection effectiveness assessment data according to any one of claims 1 to 7.

Images