Data cleaning method for grid regulation multi-source service flow instant processing and related device

Abstract

The application discloses a kind of data cleaning method and related device for power grid regulation multi-source service flow instant processing, the model data with power grid operation equipment as center is fused, the time sequence relationship and variation trend characteristics of power grid regulation multi-source data are fully analyzed, the instability of single method in data identification and correction effect is made up by the reasonable integration of multiple types of abnormal problem analysis methods, and the multi-source service flow instant processing method is not stored after the flow data generated by service system in a certain time window arrives, directly imported into memory for real-time calculation, valuable information is output from flowing, disordered data, effectively solve the problem of batch data processing timeliness, slow speed, data I / O multiple occupation, also can expand the entire service flow data processing capacity by the form of online increasing node, improve the accuracy, timeliness, high scalability of power grid regulation multi-source data cleaning.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and relates to a data cleaning method for instant processing of multi-source business flow of power grid regulation and related devices. BACKGROUND

[0002] With the construction of UHV AC / DC interconnected power grid, the development of new energy, and the deepening of power market reform, the operation characteristics of power grid have changed greatly, the service range of power grid regulation system has rapidly expanded, the obtained power dispatching data and data types have increased explosively, and great pressure is caused for data storage and processing of the power grid regulation system. At present, the power grid regulation system is constructed in stages, and the cloud service platform of power grid dispatching business adopts a hierarchical deployment design combining unification and distribution to meet the requirements of business continuity, real-time performance and collaboration of the power grid regulation system, form a two-level deployment of leading nodes and collaborative nodes, the leading nodes serve as the center of various models and data, are responsible for the management of metadata and dictionary data, and are responsible for the establishment of data models of various data and the collection of models and data within the jurisdiction of regulation, the collaborative nodes are responsible for the collection of provincial models and data and synchronously / forward the related data to the leading nodes. The data of leading nodes and collaborative nodes are shared and collected through synchronization / forwarding mode to realize the storage of full-network models and data of the power grid regulation system.

[0003] However, in the actual operation and data collection process of the power grid regulation system, various reasons such as collection error, ID mapping error, network congestion, and model maintenance error often lead to a large number of inconsistencies and abnormal phenomena in the collection and reporting data of each regional power dispatching center, which is difficult to support the comprehensive use of full-network information and centralized analysis and decision-making, and therefore, cleaning the multi-source abnormal data of the power grid regulation is an important means to improve the data use value. SUMMARY

[0004] The purpose of the present application is to solve the problems in the prior art and provide a data cleaning method for instant processing of multi-source business flow of power grid regulation and related devices.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a data cleaning method for instant processing of multi-source business flow of power grid regulation, comprising the following steps:

[0007] The data flow of multi-source operation data of the power grid regulation is analyzed, and the analyzed data is data-shunted according to single objects to form a plurality of non-continuous time slice sets;

[0008] The time sequence relationship and change trend characteristics of the multi-source operation data of the power grid regulation are analyzed, and the plurality of non-continuous time slice sets are reordered according to the time sequence relationship and change trend characteristics to form time-ordered operation data;

[0009] The primary key information of the running data is associated with the power grid regulation model data, the features of the running data are extracted, the features of the running data are dimensionally decomposed, aggregated and reorganized, and a to-be-cleaned data stream is formed;

[0010] Multiple data sources are identified for the to-be-cleaned data stream, multiple data sources are identified, the multiple data sources are screened by using a multi-source data optimal source selection algorithm, and unique source data is obtained;

[0011] Abnormal data is identified for the unique source data, the abnormal data is identified, the abnormal data is correspondingly repaired by using a data repair algorithm, and cleaned data is formed.

[0012] In a second aspect, the present application provides a data cleaning system for instant processing of power grid regulation multi-source business flow, comprising:

[0013] A data analysis and distribution module is configured to analyze the data stream of the power grid regulation multi-source running data, distribute the analyzed data according to single objects, and form multiple non-continuous time slice sets;

[0014] A data grading and sorting module is configured to analyze the time sequence relationship and change trend characteristics of the power grid regulation multi-source running data, reorder the multiple non-continuous time slice sets according to the time sequence relationship and change trend characteristics, and form time-ordered running data;

[0015] A feature extraction and reorganization module is configured to associate the primary key information of the running data with the power grid regulation model data, extract the features of the running data, dimensionally decompose, aggregate and reorganize the features of the running data, and form a to-be-cleaned data stream;

[0016] A data source identification and screening module is configured to identify multiple data sources for the to-be-cleaned data stream, identify the multiple data sources, screen the multiple data sources by using a multi-source data optimal source selection algorithm, and obtain unique source data;

[0017] An abnormal data identification and repair module is configured to identify abnormal data for the unique source data, identify the abnormal data, correspondingly repair the abnormal data by using a data repair algorithm, and form cleaned data.

[0018] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0019] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable by a processor to implement the steps of the above method.

[0020] Compared with the prior art, the present application has the following beneficial effects:

[0021] The data cleaning method for grid regulation multi-source business flow instant processing of the present application is driven by grid regulation multi-source business data flow, fuses model data with grid operation equipment as the center, and performs preliminary filtering on the grid regulation data of multi-business systems such as EMS and OMS received in real time, analyzes the original data, proposes a multi-source business flow instant processing method, performs data shunting according to single objects to form a non-continuous time slice set, fully analyzes the time sequence relationship and change trend characteristics of the grid regulation multi-source operation data in combination with the grid regulation model data, reorders, dimensionally decomposes, aggregates and reorganizes the data, forms a data flow to be cleaned, further proposes an abnormal data identification and repair method based on stream data processing, uses a data cleaning engine to select an optimal source for the data to be cleaned and identify and repair abnormal data such as abnormal defects, jump points and out-of-limit points, stores the cleaned clean data to a big data platform, and finally provides a fast, business-driven multi-dimensional interactive analysis service of regulation data, thereby providing high-quality panoramic data support for deep mining of grid data value.

[0022] Further, the present application proposes a multi-source business flow instant processing method, which performs concurrent reading, filtering, feature dimension decomposition, feature aggregation and reorganization of multi-business system data through stream partitioning technology, dynamically, intelligently and adaptively combines data features according to the needs and characteristics of different business scenarios, supports real-time processing of hundreds of millions of data, and realizes the improvement of read and write performance to the millisecond level.

[0023] Further, the present application proposes a data reordering algorithm based on window sequence alignment, which analyzes the time sequence relationship and change trend characteristics of the grid regulation multi-source operation data, performs data window sequence division, window sequence start and end time boundary determination, and window sequence time alignment, and guarantees the orderliness of the data of multiple non-continuous time slice sets.

[0024] Further, the present application analyzes the data needs of different applications, combines the dispatching agencies to which the equipment belongs, the data source level, the historical data quality of the data source, the data quality of the data source on the same day, and the selection of the historical data source of the measuring point, and proposes an optimal source selection method for multi-source data, calculates the comprehensive evaluation results of each data source by performing weight distribution on each index of the influence factor set, and realizes the fast selection of the optimal data source.

[0025] Further, the application proposes an abnormal data identification and repair method based on stream data processing, according to the demand of data cleaning business on data characteristics, dynamically selects and combines data object, data source, data type, voltage level, data volume, data value and other data stream segment key characteristic information, fuses model data with the power grid operation equipment as the center, realizes the identification and repair of abnormal data such as defects, jump points and out-of-limit, and through the reasonable integration of multiple types of abnormal problem analysis methods, the instability of single method in data identification and correction effect is made up.

[0026] The data cleaning system for instant processing of power grid regulation multi-source business stream of the application fuses model data with the power grid operation equipment as the center, fully analyzes the time sequence relationship and change trend characteristics of the power grid regulation multi-source data, through the reasonable integration of multiple types of abnormal problem analysis methods, the instability of single method in data identification and correction effect is made up, and the instant processing method of multi-source business stream is that the flowing data generated by the business system in a certain time window is not stored after arriving, but directly imported into the memory for real-time calculation, valuable information is output from the flowing and disordered data, the problems of poor timeliness, slow speed and multiple data I / O occupation in batch data processing are effectively solved, the entire business stream data processing capacity can be expanded through the form of online adding nodes, and the accuracy, timeliness and high scalability of the power grid regulation multi-source data cleaning are improved. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical scheme of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0028] Figure 1 The flow chart of the data cleaning method for instant processing of power grid regulation multi-source business stream of the application.

[0029] Figure 2 The schematic diagram of the data cleaning system for instant processing of power grid regulation multi-source business stream of the application.

[0030] Figure 3 The data cleaning framework diagram for instant processing of power grid regulation multi-source business stream of the application.

[0031] Figure 4 The data cleaning flow chart for instant processing of power grid regulation multi-source business stream of the application.

[0032] Figure 5 The data stream preprocessing flow chart of the application.

[0033] Figure 6 Flowchart of power grid regulation data cleaning process of the present application. DETAILED DESCRIPTION

[0034] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents the selected embodiments of the application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative work fall within the scope of protection of the present application.

[0036] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0037] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship when the product of the present application is used, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0038] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0039] In the description of the embodiments of the present application, it also needs to be explained that, unless explicitly specified and limited, if the terms "arrange", "install", "connect", "connect" appear, they should be understood in a broad sense, for example, they can be fixedly connected, or can be detachably connected, or integrally connected; can be mechanically connected, or can be electrically connected; can be directly connected, or can be indirectly connected through an intermediate medium, or can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0040] The present application will be further described in detail below with reference to the accompanying drawings:

[0041] Referring to Figure 1 The embodiment of the present application discloses a data cleaning method for real-time processing of power grid regulation multi-source business flow, comprising the following steps:

[0042] S1 analyzes the data stream of the power grid regulation multi-source operation data, and forms a plurality of non-continuous time slice sets by data shunting of the parsed data according to single objects, specifically as follows:

[0043] S1-1) The source data end encapsulates the collected multi-business system data into original messages, and uploads them to the regulation cloud big data platform through the message bus;

[0044] S1-2) The big data platform collects and fuses the business system messages, and performs data analysis according to the power dispatch data object structured design, and preliminarily filters the data that does not conform to the primary key specification;

[0045] S1-3) The original data is stored, and at the same time, the original data is encapsulated into raw data messages for driving cleaning business analysis, and is sent to a data preprocessing engine. The raw data message contains multiple data types, multiple objects, and multiple voltage level data, and the raw data message topic is distinguished according to the data type;

[0046] S1-4) The data preprocessing engine receives the raw data message, triggers the data stream processing and scheduling program, analyzes the raw data message, groups the real-time data according to the data object type, forms a plurality of single object time slice sets in units of messages, and triggers the stream data processing process of different data objects.

[0047] S2 analyzes the time sequence relationship and change trend characteristics of the power grid regulation multi-source operation data, and reorders the plurality of non-continuous time slice sets according to the time sequence relationship and change trend characteristics to form time-ordered operation data, specifically as follows:

[0048] S2-1) Divide the data window sequence in units of messages;

[0049] S2-2) Grouping the data in the window according to the object id, data type and data source to form a single measurement point data sequence, arranging the data sequence in ascending order according to the data service time, and determining the start time and end time of the single measurement point data sequence;

[0050] S2-3) Comparing the start and end boundaries of the newly received data window sequence with the existing data window sequence, aligning the start and end boundaries of the latter window with the boundaries of the former sequence processing window, and ensuring the time order of the overall data stream of the same measurement point.

[0051] S3 associates the primary key information of the running data with the power grid control model data, extracts the features of the running data, and performs dimension decomposition, aggregation and reorganization on the features of the running data to form a to-be-cleaned data stream, as follows:

[0052] S3-1) Extracting the features of the running data using the fused power grid control model data, obtaining the object ID code, statistical caliber code, data type code and data source code in the running data, associating the object ID code, statistical caliber code, data type code and data source code with the power grid model data, analyzing the key information, and obtaining the association relationship between the data objects of the dispatching institution and equipment, operation and maintenance institution and equipment, and equipment container and equipment;

[0053] S3-2) Statistics of the data volume information in the time-ordered running data, combination of the key information and the association relationship to form a feature set of multiple dimensions describing the data stream segment, and dynamic selection and combination of the features in the feature set of multiple dimensions using the data cleaning service to obtain the reorganized feature set of multiple dimensions, the reorganized dimension information as the primary key description information of the data stream segment, and the to-be-cleaned data stream.

[0054] S4 performs multi-data source identification on the to-be-cleaned data stream, identifies multiple data sources, and uses a multi-source data optimal source selection algorithm to filter the multiple data sources to obtain unique source data, as follows:

[0055] S4-1) Determining the factor set F affecting data source selection:

[0056] F={f1,f2,...,f i ,...,f n}

[0057] Where f i represents the i th factor affecting data source selection, and n represents the number of influencing factors;

[0058] S4-2) Evaluating the influence weight factor W of each factor of each data source:

[0059] W={w1,w2,...,wi ,..., w n}

[0060] wherein w i represents the influence weight of the i-th influence factor f i ;

[0061] S4-3) calculating the influence score G of each factor of each data source:

[0062] G = {g1, g2,..., g i ,..., g n}

[0063] wherein g i represents the influence score of the i-th influence factor f i ;

[0064] S4-4) calculating the comprehensive evaluation result B of each data source:

[0065] B = G · W T

[0066] wherein W T represents the transpose of W;

[0067] Comparing the comprehensive evaluation results of the data sources, the data source with the highest score is considered as the optimal source, so as to select the unique data source.

[0068] S5 performing abnormal data identification on the unique source data, identifying abnormal data, and performing corresponding repair processing on the abnormal data by using a data repair algorithm to form cleaned data. The abnormal data includes defects, jump points and out-of-limit, and the repair processing includes jump point smoothing processing, defect supplement point processing and out-of-limit correction processing. The specific method is as follows:

[0069] S5-1) jump point judgment:

[0070] Comparing the absolute values of the current t time measurement point and the front and rear measurement points, if the absolute values are all greater than or less than the threshold value, the point is determined as a jump point, and the calculation method is as follows:

[0071] max(|V t |, V min ) > R × max(|V t-1 |, V min ) and max(|V t |, V min ) > R × max(|V t+1 |, V min )

[0072] or

[0073] R × max(|Vt |V min )<max(|V t-1 |V min )&&R*max(|V t |V min )<max(|V t+1 |V min )

[0074] wherein, V t represents the data value at time t; V min represents the lower limit of comparison, V min max(|V t |, 30); R is the step multiple of data;

[0075] S5-2) Out-of-limit judgment:

[0076] The current time t measurement point is compared with the upper and lower limits of the reasonable range of the service, and the point outside the reasonable range of the service is determined as an out-of-limit point, and the calculation method is as follows:

[0077] F mnin ≤V t ≤F max

[0078] wherein, F min represents the lower limit of comparison; F max represents the upper limit of comparison;

[0079] S5-3) Missing point judgment:

[0080] The abnormal points are found out through the jump point and the out-of-limit judgment, and the time is marked; the missing points are discriminated, the missing points refer to the cases that the data value is empty, empty string, "NULL" string; the abnormal data is corrected, the data correction method includes linear interpolation, standard value filling or multi-source filling or historical data filling.

[0081] S5-4) Correcting the abnormal data, specifically as follows:

[0082] (1) If the measurement point data corresponding to the abnormal data is a single data source, and the data type of the measurement point data is frequency or voltage, the standard value is calculated through the formula to process the abnormal data; if the measurement point data is of other data types, and the two data points before and after the abnormal data in the daily data can be found, then the linear interpolation is used for correction, otherwise the historical data filling method is used for processing;

[0083] (2) If there are multiple data sources for the measurement point data, firstly determine whether the data of other data sources at the corresponding time is abnormal, if the other data sources are abnormal, process the measurement point data as a single data source; otherwise, select the data of the replaceable source through the weight method and the consistency analysis of the data trend of the previous day and the next day to fill in the missing data;

[0084] Linear interpolation is to find the previous and next two data points of the missing point or abnormal point at the nearest time, find the change trend of the data by calculating the slope, and take the value at the corresponding time for interpolation filling; standard value filling is to fill in the data beyond the business range by calculating the standard value; multi-source filling is to select the data of the replaceable source through weight analysis to fill in the missing data, and the data obtained by multi-source selection needs to analyze the data of the previous day and the next day; historical data filling is to select the corresponding time data that can be replaced through historical data analysis to fill in the missing data, and the data obtained by historical data selection needs to analyze the change trend of the data.

[0085] As shown in Figure 2 The embodiment of the application discloses a data cleaning system for real-time processing of power grid regulation multi-source business flow, comprising:

[0086] A data analysis and distribution module is configured to analyze the data stream of the power grid regulation multi-source operation data, and distribute the analyzed data according to single objects to form a plurality of non-continuous time slice sets.

[0087] A data grading and sorting module is configured to analyze the time sequence relationship and change trend characteristics of the power grid regulation multi-source operation data, and reorder the plurality of non-continuous time slice sets according to the time sequence relationship and change trend characteristics to form time-ordered operation data.

[0088] A feature extraction and reorganization module is configured to associate the primary key information of the operation data with the power grid regulation model data, extract the features of the operation data, and perform dimension decomposition and aggregation reorganization on the features of the operation data to form a to-be-cleaned data stream.

[0089] A data source identification and screening module is configured to identify multiple data sources from the to-be-cleaned data stream, and screen the multiple data sources by using a multi-source data optimal source selection algorithm to obtain unique source data.

[0090] An abnormal data identification and repair module is configured to identify abnormal data from the unique source data, and perform corresponding repair processing on the abnormal data by using a data repair algorithm to form cleaned data.

[0091] Embodiment

[0092] The embodiment provides a data cleaning method for real-time processing of power grid regulation multi-source business flow, comprising the following steps:

[0093] Step 1, the data stream of the multi-source operation data of the multi-system and multi-type mass power grid regulation is parsed, and the data is shunted according to a single object to form a plurality of non-continuous time slice sets.

[0094] The source data end encapsulates the multi-service system data such as EMS, OMS, PMS, DMS, WAMS and user power consumption information collection system collected into original messages, and uploads them to the regulation cloud big data platform through the message bus. The big data platform collects and fuses the messages of each service system, analyzes the data according to the structured design of the power dispatch data object, and preliminarily filters the data that does not meet the primary key specification. Then, on the one hand, the original data is stored, and on the other hand, the original data is encapsulated into raw data messages for driving cleaning business analysis, and sent to the data preprocessing engine. The raw data messages cover multi-type, multi-object, multi-voltage level data, and the multi-type data is distinguished according to the message topic. The data preprocessing engine receives the raw data messages, triggers the data flow processing and scheduling program, analyzes the raw data messages, and groups the real-time data according to the data object types such as power grid, generator and power plant, forms a plurality of single-object time slice sets in units of messages, and triggers the flow data processing process of different data objects. In the single-object shunting process, due to the different arrival times of multiple time slice data, the same time slice data set is ordered, and multiple time slice sets are often disordered.

[0095] Step 2, analyze the time sequence relationship and change trend characteristics of the multi-source operation data of the power grid regulation, reorder the non-continuous time slice set to form a time-ordered power grid regulation operation data set.

[0096] For power grid regulation data analysis, it is often necessary to analyze the spatial and temporal relationship of the data by combining the historical data curve change trend, and the orderliness of the data to be analyzed needs to be maintained. In order to effectively change the multiple non-continuous time slice set data into a time-ordered power grid regulation operation data set, a data reordering algorithm based on window sequence alignment is proposed, and the algorithm implementation process is as follows:

[0097] Step 2-1, divide the data window sequence. Since the data of the same object id, the same data type and the same data source in a message is ordered, the present application divides the window in units of messages, which can maximize the preservation of the original upload details.

[0098] Step 2-2, determine the start and end time boundaries of the window sequence. After grouping the data in the window according to the object id, data type and data source, a single measuring point data sequence is formed, which is arranged in ascending order according to the data service time, and the start time and end time of the single measuring point data sequence are determined.

[0099] Step 2-3 window sequence time alignment, compare the start and end boundaries of the newly received window sequence with the existing window sequence, align the start and end boundaries of the latter window with the boundaries of the previous sequence processing window, and ensure the time order of the overall data stream of the same measuring point.

[0100] In the power grid regulation system, different types of data often have different collection frequencies. The collection frequencies of common types of data are shown in Table 1. By analyzing the spatio-temporal relationship of historical data curves, the window size of the reordering algorithm can be adjusted according to the sampling frequency of different types of data.

[0101] Table 1 Collection frequencies of common types of data

[0102] Serial number Data type Collection frequency Daily sampling points 1 Electric power 1 minute 1 point 1440 2 Frequency 1 second 1 point 86400 3 Electric quantity 1 day 1 point 1 4 Plan 15 minutes 1 point 96 5 Prediction 15 minutes 1 point 96

[0103] Step 3, fusion of power grid regulation model data to extract operation data features, dimension decomposition, aggregation and reorganization of operation data features to form the to-be-cleaned data stream.

[0104] First, fuse the power grid regulation model data to extract the operation data features, and perform dimension decomposition on the features of the data stream. By extracting the object ID code, statistical caliber code, data type code, and data source code in the data stream, and associating them with the power grid model data, the corresponding device voltage level, belonging station, and belonging station type are analyzed to obtain key information for application analysis and mining. At the same time, the dispatch management relationship between the dispatching agency and the device, the operation and maintenance management relationship between the operation and maintenance agency and the device, and the subordinate relationship between the device container and the device can be obtained. Then, the data amount information in the time slice can be counted to form a feature set describing the data source, data type, voltage level, data amount, and time of the data stream segment. Then, according to the data cleaning business requirements for data features, the data feature dynamic selection and combination are performed, and the reorganized dimension information is used as the primary key description information of the data stream segment and transmitted to the data cleaning engine.

[0105] For power grid regulation data processing, prediction, management, and mining, by analyzing the requirements and characteristics of different business scenarios, the data feature combination method can be dynamically, intelligently, and adaptively adjusted. The reorganized feature dimension information and data are transmitted to the corresponding application service system to meet the data requirements of diversified power grid regulation business calculation and improve the efficiency of horizontal association and vertical penetration of device subjects and multiple types of data.

[0106] Step 4, multiple data source identification of the to-be-cleaned data stream, using the multi-source data optimal source selection algorithm to filter multiple data sources to obtain unique source data.

[0107] The number of data sources for the data to be cleaned can be identified by the data source dimension feature information in step 3. Data with more than 1 data source is considered to be multi-source data. During the cleaning process, it is necessary to select the optimal source for the multi-source data to obtain a unique data source.

[0108] In the multi-source data optimal source selection algorithm, factors such as the scheduling agency to which the equipment belongs, the level of the data source, the historical data quality of the data source, the current data quality of the data source, and the historical data source selection status of the measurement point are all important factors affecting the selection of the data source. Based on the data requirements analysis of different applications, the influence weight factors of each factor are evaluated, and then the weights of each indicator in the set of influencing factors are assigned to calculate the comprehensive evaluation results of each data source, thereby selecting the optimal data source. The algorithm implementation process is as follows:

[0109] Step 4-1: Determine the set of factors influencing the selection of the data source: F = {f1, f2, ..., f...} i , ..., f n}, where f i This represents the i-th factor influencing the selection of the data source, and n represents the number of influencing factors.

[0110] Step 4-2: Evaluate the influence of factors from different data sources using the weighting factor W = {w1, w2, ..., w...} i , ..., w n}, where w i f represents the i-th influencing factor i The influence weight.

[0111] Step 4-3 Calculate the influence score of each factor for each data source G = {g1, g2, ..., g...} i , ..., g n}, where g i f represents the i-th influencing factor i Impact rating.

[0112] Step 4-4 Calculate the comprehensive evaluation result B = G·W for each data source. T The comprehensive evaluation results of each data source are compared, and the data source with the highest score is selected as the optimal source, thus selecting a unique data source.

[0113] Step 5: Identify abnormal data such as defects, jump points, and exceeding limits in the unique source data, and use data repair algorithms to smooth jump points, fill in defects, and correct exceeding limits to form cleaned data.

[0114] First, the abnormal point data is identified, including jump points and out-of-limit data. The jump point often refers to a single data that suddenly increases or decreases due to problems such as source data sending program. The out-of-limit often refers to points that exceed the reasonable range of frequency and voltage. For the problem of abnormal points, an abnormal data identification method suitable for multi-object data is proposed, and the specific implementation is as follows:

[0115] Step 5-1 jump point judgment: compare the absolute values of the current t time measurement point and the front and rear measurement points, and determine the point as a jump point if the absolute values are greater than or less than the threshold value. The calculation formula is:

[0116] max(|V t |, V min ) > R x max(|V t-1 |, V min ) && max(|V t |, V min ) > R x max(|V t+1 |, V min )

[0117] or

[0118] R x max(|V t |, V min ) < max(|V t-1 |, V min ) && R x max(|V t |, V min ) < max(|V t+1 |, V min )

[0119] Wherein, V t represents the data value at t time; V min represents the lower limit of comparison, in order to avoid the influence of the multiple of the value near 0, V min takes max(|V t |, 30); R is the step multiple of data, which is different for different objects or types of data.

[0120] Step 5-2 out-of-limit judgment: compare the current t time measurement point with the upper and lower limits of the reasonable range of business, and determine the point as out-of-limit point if it is not within the reasonable range of business. The calculation formula is

[0121] F min ≤ V t ≤ F max

[0122] Wherein, V t represents the data value at t time; F min represents the lower limit of comparison; F maxIndicates the upper limit of comparison. In the process of power grid operation, the frequency takes 50 Hz as the reference frequency, and the voltage often takes the rated voltage of the equipment as the reference voltage.

[0123] Step 5-3 Abnormal point marking: Find out the abnormal point through the jump point and out-of-limit judgment, and mark the time, which is convenient for subsequent correction.

[0124] Then the missing point is judged. The missing point refers to the case that the data value is empty, an empty string, or a "NULL" string. For different data sampling frequencies, different step sizes need to be set for judgment. For example, for 5-minute / point data, the judgment step size is set to 5; for 1-minute / point data, the judgment step size is set to 1. The data acquisition frequencies of various common types can be seen in Table 1.

[0125] Finally, the abnormal data is corrected. The data correction methods include linear interpolation, standard value interpolation, multi-source interpolation, and historical data interpolation. The appropriate correction method is selected according to the data problem and data situation. For data that exceeds the business range of frequency and voltage, the standard value is calculated by formula to fill in the problem data. Linear interpolation correction is to find the two data points before and after the nearest time of the missing point or abnormal point, calculate the slope to find the trend of data change, and take the corresponding time value for interpolation. For multi-source data, the data that is verified to be normal by other data sources is preferred, and the data that can be replaced by the source is selected for filling by weight analysis. The data obtained by multi-source selection needs to be further analyzed for the data of the previous day and the next day. For the case where there is no multi-source data, appropriate historical data can be selected for interpolation. Historical data interpolation is to select the corresponding time data that can be replaced by historical data analysis for filling. The data obtained by historical data selection needs to be analyzed for the trend of data change. The abnormal data recognition method for multi-object data does not need to scan and traverse the data of the day for multiple times to identify the missing point and abnormal point of the unique source data stream. The time correlation and internal dependence of the data stream are used for real-time identification and analysis.

[0126] The specific method for correcting abnormal data is as follows:

[0127] Step 5-3-1: If there are multiple data sources for the data, first determine whether the data of the corresponding time of other data sources is abnormal. If it is not abnormal, select the data that can be replaced by the source by weight method and consistency analysis of the data trend of the previous day and the next day. If the data of other data sources is abnormal, do not use the multi-source interpolation method to process, and process according to the single data source of the data.

[0128] Step 5-3-2: If the data is a single data source, and the data type of the data is frequency or voltage, the standard value is calculated by formula to process abnormal data; if the data is of other data types, and the two data points before and after the nearest time of the abnormal data in the daily data can be found, linear interpolation is used for correction, otherwise, the problem data is processed by historical data interpolation;

[0129] The linear interpolation is to find the two data points before and after the nearest time of the missing point or the abnormal point, find the change trend of the data by calculating the slope, and take the value at the corresponding time for interpolation filling;

[0130] The standard value interpolation is to fill in the problem data by calculating the standard value for the data beyond the business range;

[0131] The multi-source interpolation is to select the data that can replace the source by weight analysis for filling, and the data obtained by multi-source selection needs to analyze the data of the previous and next days;

[0132] The historical data interpolation is to select the corresponding time data that can be replaced by historical data analysis for filling, and the data obtained by historical data selection needs to analyze the change trend of the data.

[0133] Step 6, store the clean data after cleaning according to the form meeting the business requirements, and provide fast, business-driven power grid regulation data multi-dimensional interactive analysis service.

[0134] The big data platform provides cross-time and space, cross-business, cross-scheduling data services through standard data interface services, can query the cleaned data for annual, monthly and daily curves, and supports comparison with multi-source raw data and annotation display of correction points. In order to query the cleaned data faster, multi-dimensional indexes of data are established according to business time, data source, data type and other key information, reasonable storage primary keys are constructed, and millisecond-level query of daily TB-level data is realized, which provides high-quality panoramic data support for deep mining of power grid data value.

[0135] Principle of the application

[0136] The power grid regulation business type is basically based on dynamic data flow, and the data flow is difficult to control and track due to its real-time, flow and changeability. Therefore, historical data needs to be extracted continuously, and historical data characteristics are combined to pre-process, identify abnormal data, repair data and other operations on incremental data. Therefore, after the static storage of the flow data, repeated reading and writing of the data is easy to occur, the data I / O becomes a bottleneck, the data processing efficiency is greatly reduced, and the timeliness requirement of massive real-time data analysis and mining cannot be met.

[0137] Data cleaning is the key to improve data quality and realize efficient management of data. At present, many data cleaning methods are applied to power data, but with the rapid expansion of the service range of power grid control system, the power dispatching data has exploded, and the data types are more diversified. Many traditional data cleaning methods cannot meet the needs of the era of power big data. The existing data cleaning methods are not ideal for processing a large number of power grid control time series data, the algorithm complexity is high or the algorithm is designed for specific data, and there is no unified data cleaning method for power grid control time series operation data. At the same time, the flow and variability characteristics of time series data are not fully considered, and there are problems of poor timeliness and slow speed in cleaning massive time series data. The present application fully analyzes the time series relationship and trend characteristics of multi-source data of power grid control, reasonably integrates multiple types of abnormal problem analysis methods to make up for the instability of single method in data identification and correction effect, and then uses multi-source business flow real-time processing method to obtain valuable information output from flowing and disordered data in real time, effectively solving the problems of poor timeliness, slow speed and multiple data I / O occupation in batch data processing.

[0138] As shown in Figure 3 , Figure 3 The data cleaning framework for real-time processing of multi-source business flow of power grid control.

[0139] 1) The data access module obtains the EMS, OMS, PMS, DMS, WAMS, and other multi-business system messages generated by multi-level power grid dispatching through the message bus, parses the messages according to the power dispatching data object structured design, combines the model data centered on power grid operation equipment, realizes the data collection and fusion of each business system, and simultaneously performs preliminary filtering on the real-time received data, records the key information of the data, and simultaneously sends the key information and data to the data flow processing and cleaning module, triggering the data preprocessing engine to analyze the original data.

[0140] 2) The data flow processing and cleaning module pre-processes and cleans the data, including data preprocessing engine and data cleaning engine. The data processing engine divides the original data into non-continuous time slice sets according to single objects, analyzes the time series relationship and trend characteristics of power grid control multi-source operation data in combination with power grid control model data, reorders, dimensionally decomposes, aggregates and reorganizes the data to form a data stream to be cleaned. The data cleaning engine selects the optimal source for the data to be cleaned, and identifies and repairs abnormal data such as abnormal defects, jump points and out-of-limit points to obtain clean data after cleaning.

[0141] 3) The cleaning data storage module establishes multi-dimensional index of data through business time, data source, data type and other key information, constructs a reasonable Hbase storage primary key for storage of cleaning data, and realizes reasonable utilization of storage space through data compression.

[0142] 4) Multi-dimensional interactive visualization analysis module is used for multi-dynamic interactive display and analysis of data before and after cleaning, including multi-dimensional interactive query service of regulation data and year, month and day curve query of cleaned data, and can be compared with multi-source raw data and the display of marked correction points.

[0143] As shown in Figure 4 , the data cleaning process for the power grid regulation multi-source business flow instant processing of the application is as follows: Figure 4

[0144] 1) The big data platform collects and fuses data of various business systems, encapsulates the real-time received power grid regulation data into raw data messages after preliminary filtering, and sends them to the big data cluster message bus.

[0145] 2) The data preprocessing engine analyzes the raw data messages, performs data shunting according to single objects, extracts data features in combination with model data, and performs dimension decomposition, aggregation and reorganization on the data to generate a data stream to be cleaned.

[0146] 3) The data cleaning engine selects multi-source data for the data to be cleaned, then identifies and repairs abnormal data such as missing data, jump points and out-of-limit data for unique source data, and obtains cleaned data.

[0147] 4) The cleaned data is stored in the big data platform in a form meeting the business requirements, and a fast and business-driven multi-dimensional interactive analysis service of regulation data is provided.

[0148] As shown in Figure 5 , the data flow preprocessing process for the power grid regulation data of the application is as follows: Figure 5

[0149] 1) The real-time power grid regulation data flow is parsed to decompose key information such as data type, object, voltage level, data source, business time and data value.

[0150] 2) The real-time data is grouped according to data object types such as power grid, generator and power plant.

[0151] 3) In combination with model data, a plurality of single object time slice sets are formed in units of messages, and the unordered time slices are rearranged through a data rearrangement algorithm based on window sequence alignment to form a non-continuous ordered time slice data set.

[0152] 4) The single object time slice set is decomposed in feature dimensions, including but not limited to data source, data type, voltage level, data volume, time, etc.

[0153] ​​5) Dynamic selection and combination of features for data characteristics required by data cleaning services, and the recombined dimension information is used as the primary key description information of the data stream segment.

[0154] 6) Transmission of the aggregated recombined data set to the data cleaning engine to provide data and feature information support for subsequent data cleaning.

[0155] As Figure 6 shown, Figure 6 the power grid regulation data cleaning process of the application is as follows:

[0156] 1) Calculate the number of data sources of the measurement point data, identify the multiple data sources of the data to be cleaned, if the number of data sources is greater than 1, consider that the measurement point is a multi-source data, then perform step 2) to select the multi-source, otherwise, perform step 3) to identify abnormal data.

[0157] 2) Select a unique data source, use the multi-source data optimal source selection algorithm to filter multiple data sources to obtain unique source data.

[0158] 3) Identify abnormal data, use the multi-object data abnormal data identification method to identify jump points and out-of-limit data, if the data is an abnormal point, mark the abnormal point, if the data is a normal point, perform step 4).

[0159] 4) Identify missing points, use the missing point identification algorithm to identify missing points, if the data is a missing point, mark the missing point, perform step 5), if it is a normal point, store it as a mature data.

[0160] 5) Abnormal data correction, use linear interpolation, standard value interpolation, multi-source interpolation, historical data interpolation and other data correction methods to correct jump, out-of-limit, missing and other data, and form mature data stored in the HBase database of the big data platform.

[0161] The computer device provided by the embodiment of the application. The computer device of the embodiment comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the steps in each of the above method embodiments. Alternatively, the processor executes the computer program to implement the functions of each module / unit in each of the above device embodiments.

[0162] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the application.

[0163] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can include, but is not limited to, a processor and a memory.

[0164] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.

[0165] The memory can be used to store the computer program and / or modules, and the processor can realize various functions of the computer device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory.

[0166] The modules / units integrated in the computer device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0167] The above merely describes the preferred embodiments of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data cleaning method for real-time processing of multi-source business flows in power grid control, characterized in that, Includes the following steps: The data stream of multi-source operation data of power grid regulation is parsed, and the parsed data is split into multiple sets of non-continuous time slices according to single objects. The temporal relationships and trends of multi-source power grid control operation data are analyzed, and multiple non-continuous time slice sets are reordered to form time-ordered operation data based on the temporal relationships and trends. The primary key information of the operational data is associated with the power grid control model data, the features of the operational data are extracted, and the features of the operational data are decomposed, aggregated and recombined in a dimensional manner to form a data stream to be cleaned. The data stream to be cleaned is subjected to multi-source identification, and multiple data sources are identified. The multi-source data optimal source selection algorithm is used to filter the multiple data sources to obtain a unique source data. The unique source data is subjected to anomaly identification. The identified anomalies are then repaired using data repair algorithms to produce cleaned data.

2. The data cleaning method for real-time processing of multi-source business flows in power grid regulation according to claim 1, characterized in that, The process involves parsing the data stream of multi-source operation data for power grid regulation, and then splitting the parsed data into multiple non-continuous time slice sets based on single objects, including: 1) Obtain the raw message encapsulated from the collected data from multiple service systems at the source data end; 2) Parse each original message to obtain the original data corresponding to each original message, filter each original data to obtain the original data that conforms to the primary key specification; 3) Store the raw data and encapsulate it into raw data messages used for driving the cleaning business analysis, and send them to the data processing center; the raw data messages contain data of multiple data types, multiple objects, and multiple voltage levels, and the raw data message subject is distinguished according to data type; 4) After receiving the raw data message, the data processing center parses the raw data message and groups the real-time data according to the data object type in the raw data message, forming multiple single-object time slice sets by message.

3. The data cleaning method for real-time processing of multi-source business flows in power grid regulation according to claim 1, characterized in that, The analysis examines the temporal relationships and trends of multi-source power grid control operation data. Based on these temporal relationships and trends, multiple discontinuous time slice sets are reordered to form time-ordered operation data, including: 1) Divide the data of the non-continuous time slice set into units of messages to obtain a data window sequence; 2) Group the data in the window according to object ID, data type, and data source to form a single measurement point data sequence, sort it in ascending order according to data business time, and determine the start and end time of the single measurement point data sequence; 3) Compare the start and end times of the newly received data window sequence with the existing data window sequence, and align the start time of the next window with the end time of the previous processing window to ensure the temporal order of the overall data stream at the same measurement point.

4. The data cleaning method for real-time processing of multi-source business flows in power grid regulation according to claim 1, characterized in that, The process of associating the primary key information of the operational data with the power grid control model data, extracting the features of the operational data, and performing dimensional decomposition, aggregation, and recombination of the features of the operational data to form a data stream to be cleaned includes: 1) Extract features of operational data from integrated power grid control model data. Extract the features of operational data to obtain object ID code, statistical caliber code, data type code, and data source code in the operational data. Associate the object ID code, statistical caliber code, data type code, and data source code with the power grid model data to analyze key information. At the same time, obtain the association relationship between data objects of dispatching agencies and equipment, operation and maintenance agencies and equipment, and equipment containers and equipment. 2) Collect data volume information from the time-ordered running data, and combine key information and relationships to form a multi-dimensional feature set describing the data stream segment; then use data cleaning to dynamically select and combine features from the multi-dimensional feature set to obtain a reorganized multi-dimensional feature set, and use the reorganized dimensional information as the primary key description information of the data stream segment to obtain the data stream to be cleaned.

5. The data cleaning method for real-time processing of multi-source business flows in power grid regulation according to claim 1, characterized in that, The process involves identifying multiple data sources from the data stream to be cleaned, and then using a multi-source optimal source selection algorithm to filter these data sources and obtain a unique source data, including: 1) Determine the set of factors influencing the selection of data sources. : in, This indicates the first factor influencing the choice of data source. One factor, Indicates the number of influencing factors; 2) Evaluate the weighting factors of each factor in each data source. : in, Indicates the first One influencing factor Influence weight; 3) Calculate the impact score of each factor for each data source. : in, Indicates the first One influencing factor Impact score; 4) Calculate the comprehensive evaluation result for each data source. : in, express transpose; The comprehensive evaluation results of each data source are compared, and the data source with the highest score is selected as the optimal source, thus selecting a unique data source.

6. The data cleaning method for real-time processing of multi-source business flows in power grid regulation according to claim 1, characterized in that, The abnormal data includes missing points, jump points, and limit-crossing points.

7. The data cleaning method for real-time processing of multi-source business flows in power grid regulation according to claim 6, characterized in that, The process involves identifying anomalous data from the unique source data, identifying anomalous data, and then using data repair algorithms to repair the anomalous data, resulting in cleaned data, including: 1) Jump point determination: Based on the current unique source data Data values ​​of measurement points at any time, current The data values ​​of the measurement points before and after a given time are determined using the following formula. Is the time measurement point a jump point? or in, express Data values ​​at any given time; Indicates the lower bound of the comparison. Pick ; The step factor of the data; 2) Limit Exceeding Point Judgment: current The data values ​​of the time measurement points are compared with the upper and lower limits of the reasonable business range. Points that are outside the reasonable business range are determined to be out-of-limit points. The calculation method is as follows: in, Indicates the lower limit of the comparison; Indicates the upper limit of the comparison; 3) Missing point identification: In the present The data value at each time measurement point is either empty, an empty string, or the string "NULL", representing the current situation. The time measurement point is missing; 4) Correct outlier data. Data correction methods include linear interpolation, standard value interpolation, multi-source interpolation, or historical data interpolation, as detailed below: (1) If the measurement point data corresponding to the abnormal data is a single data source and the data type of the measurement point data is frequency or voltage, the standard value is calculated by formula to process the abnormal data; if the measurement point data is of other data types and the two data points before and after the most recent moment of the abnormal data in the data of the day can be found, the correction is performed by linear interpolation; otherwise, the data is processed by historical data supplementation. (2) If there are multiple data sources for the measurement point, first determine whether there are any abnormalities in the data at the corresponding time of other data sources. If there are abnormalities in other data sources, process the measurement point data as a single data source; otherwise, select alternative data sources to fill the gaps by using the weighting method and the consistency analysis of the trend of the data before and after the day. The linear interpolation involves finding the two data points before and after the most recent moment of a missing or outlier point, calculating the slope to find the trend of data change, and then using the value at the corresponding moment for interpolation to fill in the missing or outlier point. The standard value supplement is used to fill and correct problematic data by calculating a standard value using a formula for data that exceeds the business scope. The multi-source compensation is achieved by selecting data that can replace the source through weight analysis. The data obtained from the multi-source selection needs to be analyzed from the previous and next day's data. The historical data supplementation is achieved by selecting corresponding time data that can be substituted through historical data analysis. The historical data selected needs to have its changing trends analyzed.

8. A data cleaning system for real-time processing of multi-source business flows in power grid control, characterized in that, include: The data parsing and splitting module is used to parse the data stream of multi-source operation data of power grid regulation, and split the parsed data into multiple non-continuous time slice sets according to single objects; The data classification and sorting module is used to analyze the temporal relationship and trend characteristics of multi-source operation data of power grid control, and to reorder multiple non-continuous time slice sets to form time-ordered operation data based on the temporal relationship and trend characteristics. The feature extraction and recombination module is used to associate the primary key information of the operating data with the power grid control model data, extract the features of the operating data, and perform dimensional decomposition, aggregation and recombination of the features of the operating data to form a data stream to be cleaned. The data source identification and filtering module is used to identify multiple data sources in the data stream to be cleaned. Multiple data sources are identified, and the optimal source selection algorithm for multiple data sources is used to filter the multiple data sources to obtain a unique source data. The abnormal data identification and repair module is used to identify abnormal data from the unique source data. Once abnormal data is identified, data repair algorithms are used to repair the abnormal data accordingly, resulting in cleaned data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images