A power distribution network mass data quality improvement system

Abstract

The present application relates to a kind of distribution network mass data quality promotion system, the system includes: physical system, repair system, data quality evaluation system, data service system, business application system, data visualization system, management and control layer system.The present application supports mass data repair in different scenarios of offline analysis and online analysis, multidimensional, multilevel data quality evaluation system, can effectively improve the quality and reliability of distribution network data, solve the common key technical problems in the field of distribution data, realize the dynamic management and data repair to different life cycle data;For a variety of business systems provide data interface that meets business application needs, can realize the integration and extension of typical application modules such as distribution network load prediction technology, reactive power optimization, energy saving and loss reduction.

Description

Technical field

[0001] The present invention belongs to the field of power distribution networks, and in particular relates to a system for improving the quality of massive data in power distribution networks. [Background Technology]

[0002] The distribution network is at the end of the power system and has the distinctive characteristics of wide geographical distribution, large grid scale, many types of equipment, and diverse network connections. With the increasing level of informationization of the distribution network, various information systems in the distribution network provide rich data for the operation status and business applications of the distribution network, including internal data such as distribution automation, dispatching automation, and production management systems, as well as external data such as natural environment and social economy, and involves multiple links such as distribution network planning, operation and maintenance. The distribution network data has the following characteristics: (1) a lot of data is collected, and each collection point collects relatively fixed types of data, and is distributed within each voltage level; (2) the sampling time scale of different collection points is different, and the data section is different; (3) the data is not complete, and there are errors and omissions in data collection; (4) the data is scattered in different application systems. The distribution network data collection volume is large, the time and space scales are different, the data is not complete, the data structure is diverse, and there are errors and omissions. Therefore, the data quality problem of the distribution network is a common and prominent typical problem. Although major breakthroughs have been made in the research of data-intensive basic analysis, intelligent decision-making support, and high-performance parallel optimization computing in distribution networks, the problem of "weakness in the middle" of data transmission in distribution networks has not been solved. How to select and adapt algorithms and strategies for data, further improve data quality, achieve accurate data repair, and enhance the level of business applications in the distribution network field is a current research focus. Therefore, there is an urgent need for a new system for improving the quality of massive data in distribution networks. The present invention supports offline analysis and online analysis of massive data repair in different scenarios, a multi-dimensional, multi-level data quality assessment system, which can effectively improve the quality and credibility of distribution network data, solve common key technical problems in the field of distribution data, and realize dynamic governance and data repair of data in different life cycles; provide data interfaces that meet business application requirements for various business systems, and realize the integration and expansion of typical application modules such as distribution network load forecasting technology, reactive power optimization, energy saving and loss reduction, etc. [Summary of the invention]

[0003] In order to solve the above problems in the prior art, the present invention proposes a distribution network massive data quality improvement system, which includes: a physical system, a repair system, a data quality assessment system, a data service system, a business application system, a data visualization system, and a management and control layer system.

[0004] Furthermore, the repair system includes: an access layer module, a conversion layer module, a cleaning layer module and an offline layer module; the repair system is used to perform data monitoring extraction and data repair; wherein: data monitoring extraction includes the extraction of real-time online data and static historical data, and realizes dynamic data capture according to pre-defined rules and metadata; static historical data extraction can be fully extracted or incrementally extracted according to the availability of hardware resources; it is also used to build a full information database based on the distribution network public information model; the data repair is a full-process repair, which is used to realize data measurement adjustment, data packet repair, rule base verification and multi-business system data integration for the different characteristics and different stages of abnormal data in all modules in the physical system, and at the same time perform distributed parallel cleaning on massive static data.

[0005] Furthermore, the physical system includes: an acquisition layer module, a transmission layer module, a network layer module, a business layer module and a storage layer module; the physical system is used to provide distribution network services, distribution network data transmission, calculation and provision.

[0006] Furthermore, the acquisition layer module includes various distribution network terminals and other distribution network secondary equipment, which is used to perform secondary power distribution to various terminal devices and collect various distribution data.

[0007] Furthermore, the transport layer module includes communication equipment for real-time data transmission; wherein, the communication equipment includes a wireless communication public network and a dedicated communication line.

[0008] Furthermore, the network layer module includes network layer equipment for providing network data transmission; the network layer equipment is used to connect to each distribution network business server; wherein: the network layer equipment includes physical isolation equipment, information bus, channel cabinet, firewall and encryption authentication equipment.

[0009] Furthermore, the business layer module is used to provide services and is the carrier of the distribution network business information systems and their databases that provide services; the storage layer module includes various storage devices and databases built on the storage devices, which are used to provide data storage.

[0010] Furthermore, the access layer module corresponds to the transmission layer module and the collection layer module. Specifically, the access layer module uses a front-end cluster to perform cluster online cleaning on data measurement errors in the collection layer module, and data omissions, mistransmissions, error codes, and garbled codes in the transmission layer module.

[0011] Furthermore, the conversion layer module corresponds to the network layer module, specifically: the conversion layer module parses, verifies and corrects data packets based on the application layer gateway for data packet mismatch, misordering, packet loss and other phenomena that occur in the network layer module through the heartbeat sampling mechanism and network protocol verification and correction mechanism.

[0012] Furthermore, the cleaning layer module corresponds to the business layer module. Specifically, the cleaning layer module performs rule-based cleaning and data integration on the business logic in the business layer module based on the middleware server.

[0013] The offline layer module corresponds to the storage layer module. Specifically, the offline layer module performs a full scan on the data in the storage layer based on the ETL server to complete logical investigation and clarification. Preferably, the ETL server is a distributed server, and the full scan is a distributed parallel operation.

[0014] The beneficial effects of the present invention include: supporting offline analysis and online analysis of massive data repair in different scenarios, multi-dimensional and multi-level data quality assessment system, which can effectively improve the quality and credibility of distribution network data, solve common key technical problems in the field of distribution data, and realize dynamic governance and data repair of data with different life cycles; provide data interfaces that meet business application requirements for various business systems, and realize the integration and expansion of typical application modules such as distribution network load forecasting technology, reactive power optimization, energy saving and loss reduction; carry out dynamic governance and data repair problems for data with different life cycles, establish a multi-dimensional and multi-level data quality assessment system, design the system architecture to support the distribution network data repair prototype system, propose the distribution network multi-source heterogeneous data exchange format and standard, design the massive data quality improvement and data repair system, and realize the improvement of the quality of massive data in the distribution network.

Brief Description of the Drawings

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application, but do not constitute an improper limitation of the present invention. In the drawings:

[0016] Figure 1 It is a schematic diagram of the architecture of the distribution network massive data quality improvement system of the present invention.

[0017] Figure 2 It is a structural diagram of the data quality assessment system of the present invention.

[0018] Figure 3 Schematic diagram of the data visualization system of the present invention. [Specific implementation method]

[0019] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0020] like Figure 1As shown, a distribution network massive data quality improvement system applied by the present invention is described in detail, the system includes: a physical system, a repair system, a data quality assessment system, a data service system, a business application system, a data visualization system, and a management and control layer system;

[0021] The repair system includes: an access layer module, a conversion layer module, a cleaning layer module, and an offline layer module; the repair system is used to perform data monitoring extraction and data repair; wherein: data monitoring extraction includes the extraction of real-time online data and static historical data, and realizes dynamic data capture according to pre-defined rules and metadata; static historical data extraction can be full or incremental extraction based on the availability of hardware resources; it is also used to build a full information database based on the distribution network public information model; the data repair is a full-process repair, which is used to implement data measurement adjustment, data packet repair, rule base verification and multi-business system data integration for the different characteristics and different stages of abnormal data in all modules of the physical system, and simultaneously perform distributed parallel cleaning on massive static data;

[0022] The physical system includes: an acquisition layer module, a transmission layer module, a network layer module, a business layer module and a storage layer module; the physical system is used to provide distribution network services, distribution network data transmission, calculation and provision;

[0023] Preferably: the acquisition layer module includes various distribution network terminals and other distribution network secondary equipment, which is used to perform secondary power distribution to various terminal devices and collect various distribution data;

[0024] Preferably: the transport layer module includes a communication device for real-time data transmission; wherein the communication device includes a wireless communication public network and a dedicated communication line;

[0025] Preferably: the network layer module includes a network layer device for providing network data transmission;

[0026] The network layer equipment is used to connect to each distribution network service server; wherein: the network layer equipment includes physical isolation equipment, information bus, channel cabinet, firewall and encryption authentication equipment;

[0027] The business layer module is used to provide services and is the carrier of the business information systems and databases of the distribution network that provide services;

[0028] The storage layer module includes various storage devices and a database built on the storage devices, which is used to provide data storage;

[0029] The access layer module corresponds to the transport layer module and the collection layer module. Specifically, the access layer module uses a front-end cluster to perform cluster online cleaning on data measurement errors in the collection layer module and data omissions, mistransmissions, error codes, and garbled codes in the transport layer module.

[0030] The conversion layer module corresponds to the network layer module, specifically: the conversion layer module performs data packet analysis, verification and error correction based on the application layer gateway for data packet mismatch, misorder, packet loss and other phenomena that occur in the network layer module through the heartbeat sampling mechanism and the network protocol verification and error correction mechanism;

[0031] The cleaning layer module corresponds to the business layer module. Specifically, the cleaning layer module performs rule-based cleaning and data integration on the business logic in the business layer module based on the middleware server.

[0032] The offline layer module corresponds to the storage layer module. Specifically, the offline layer module performs a full scan of the data in the storage layer based on the ETL server to complete logical investigation and clarification. Preferably, the ETL server is a distributed server, and the full scan is a distributed parallel operation.

[0033] The data quality assessment system is built on a physical system and is used to analyze and classify the causes of possible quality issues for structured, semi-structured, and unstructured data in the distribution network's massive data quality improvement system. For data anomalies, missing data, attribute redundancy, inconsistent precision, and irregular formats, it uses data detection methods based on parsing, rule bases, and statistics, distance, density, and association to implement distribution network-oriented outlier detection. Suspected bad data sets are obtained through confidence hypothesis testing, and estimation and identification methods are used to estimate the noise of the suspected bad data sets, identify and correct implicit bad data, and implement data variance homogeneity and error normality tests.

[0034] Preferably, the structured data is structured data in PMS and GIS; the semi-structured data is semi-structured data in system logs; the unstructured data is unstructured data in video monitoring, customer service audio, etc.

[0035] The data service system is built on the physical system and is used to provide data services such as self-service data analysis, data retrieval, data market, application interface and data monitoring based on the physical system data;

[0036] The business application system is built on the physical system and is used to provide business applications such as load forecasting, reactive power optimization, grid optimization, economic operation, energy saving and loss reduction;

[0037] As attached Figure 3As shown, the data visualization system is built on the physical system, business application system, data service system, and data quality assessment system. It is used to visualize data based on dynamic rendering algorithms and provides a human-computer interaction interface to facilitate user data access.

[0038] The control layer system is built on the physical system and includes a data security management module and a data public model module; it is used to provide data security management and modeling for third parties;

[0039] As attached Figure 2 As shown, the data quality assessment system includes a data quality problem module, a basic data assessment module, a deep data assessment module, a distribution network data quality assessment indicator system construction module, and an indicator assessment module;

[0040] The data quality assessment system uses adaptive sliding window online monitoring technology to evaluate data to obtain the current evaluation value. Specifically: the previous data evaluation time point, the current data evaluation time point, and the previous evaluation value are obtained; based on the sliding window, the data at the previous data evaluation time point and the current data evaluation time point are evaluated in windows to obtain an evaluation value sequence (VALi), where VALi is the evaluation value obtained by the i-th sliding window in the evaluation value sequence; the current evaluation value CurVAL is calculated based on the evaluation value sequence (VALi) and the previous evaluation value OldVAL;

[0041] And W1>W2····>W n , Where W1 is the weight value of the evaluation value corresponding to the sliding window with the closest time, W n is the weight of the evaluation value corresponding to the sliding window with the longest time; where:

[0042] Preferably, the sliding window size is set according to the size of the computing resources in the data quality assessment system; when the computing resources are more, the sliding window value is set larger, and vice versa;

[0043] The data quality problem module is used to classify the causes of possible data quality problems, including data anomalies, missing values, redundant attributes, irregular formats, inconsistent precision, inconsistent dimensions, missing time scales, data contradictions, etc.

[0044] The basic data evaluation module is used to perform rule base checking, normative analysis, default checking, and outlier detection. Specifically, it adopts data detection methods based on analysis, rule base, statistics, distance, density, and association to realize outlier detection for distribution network.

[0045] The data depth assessment module is used to perform variance homogeneity test, error normality test, data noise estimation and confidence hypothesis test; specifically: through the confidence hypothesis test, a suspected bad data set is obtained, and the estimation and identification method is used to estimate the noise of the suspected bad data set, identify and correct implicit bad data, and implement the data variance homogeneity test and error normality test;

[0046] The distribution network data quality assessment index system construction module is used to quantitatively calculate the integrity I, validity V, reliability R and consistency U of the data to obtain a four-dimensional evaluation value; DQ = {I, V, R, U};

[0047] The indicator evaluation module is used to perform indicator expectation calculation to determine whether each evaluation value reaches the expected value. If the expected value is reached, the weight of each evaluation value is determined, and finally the evaluation value itself is evaluated through hierarchical analysis and fuzzy comprehensive evaluation to obtain the indicator evaluation value; when the four-dimensional evaluation value is presented through the data visualization module, the indicator evaluation value of the four-dimensional evaluation value itself is presented to the user together;

[0048] In the several embodiments provided by the present invention, it should be understood that the disclosed methods and terminals can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0049] In addition, the technical solutions in the above embodiments can be combined and replaced with each other if no contradiction occurs.

[0050] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0051] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0052] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims be included in the present invention. Any accompanying figure marks in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple modules or devices stated in the system claims may also be implemented by one module or device through software or hardware. Words such as first, second, etc. are used to indicate names and do not indicate any particular order.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A system for improving the quality of massive data in a distribution network, characterized in that: The system includes: physical system, repair system, data quality assessment system, data service system, business application system, data visualization system, and management and control layer system; The control layer system is built on the physical system and includes a data security management module and a data public model module; it is used to provide data security management and modeling for third parties; The physical system includes: an acquisition layer module, a transmission layer module, a network layer module, a business layer module and a storage layer module; the physical system is used to provide distribution network services, distribution network data transmission, calculation and provision; The data quality assessment system uses adaptive sliding window online monitoring technology to evaluate data and obtain the assessment value. Specifically, it includes: Obtain the previous data evaluation time point, the current data evaluation time point, and the previous evaluation value; perform window-by-window evaluation on the data in the previous data evaluation time point and the current data evaluation time point based on the sliding window to obtain an evaluation value sequence (VALi), where VALi is the evaluation value obtained by the i-th sliding window in the evaluation value sequence; calculate the current evaluation value CurVAL based on the evaluation value sequence (VALi) and the previous evaluation value OldVAL; ； in, is the weight value of the evaluation value corresponding to the sliding window with the latest time, is the weight value of the evaluation value corresponding to the sliding window with the longest time, , ;in: .

2. The distribution network mass data quality improvement system according to claim 1, characterized in that: The repair system includes: an access layer module, a conversion layer module, a cleaning layer module and an offline layer module; the repair system is used to perform data monitoring extraction and data repair; wherein: data monitoring extraction includes the extraction of real-time online data and static historical data, and realizes dynamic data capture according to pre-defined rules and metadata; static historical data extraction is carried out in full or incremental extraction according to the availability of hardware resources; it is also used to build a full information database based on the distribution network public information model; the data repair is a full-process repair, which is used to realize data measurement adjustment, data packet repair, rule base verification and multi-business system data integration for the different characteristics and different stages of abnormal data in all modules in the physical system, and at the same time perform distributed parallel cleaning on massive static data.

3. The distribution network mass data quality improvement system according to claim 2, characterized in that: The acquisition layer module includes various types of power distribution network secondary equipment, which is used to perform secondary power distribution to various types of terminal devices and collect various types of power distribution data.

4. The distribution network mass data quality improvement system according to claim 3, characterized in that: The transport layer module includes communication equipment for real-time data transmission; wherein, the communication equipment includes a wireless communication public network and a dedicated communication line.

5. The distribution network mass data quality improvement system according to claim 4, characterized in that: The network layer module includes network layer equipment for providing network data transmission; the network layer equipment is used to connect to each distribution network business server; wherein: the network layer equipment includes physical isolation equipment, information bus, channel cabinet, firewall and encryption authentication equipment.

6. The distribution network mass data quality improvement system according to claim 5, characterized in that: The business layer module is used to provide services and is the carrier of the distribution network business information systems and their databases that provide services; the storage layer module includes various storage devices and databases built on the storage devices, which are used to provide data storage.

7. The distribution network mass data quality improvement system according to claim 6, characterized in that: The access layer module corresponds to the transport layer module and the collection layer module. Specifically, the access layer module uses a front-end cluster to perform cluster online cleaning on data measurement errors in the collection layer module and data omissions, errors, and garbled codes in the transport layer module.

8. The distribution network mass data quality improvement system according to claim 7, characterized in that: The conversion layer module corresponds to the network layer module, specifically: the conversion layer module parses, verifies and corrects data packets based on the application layer gateway for data packet mismatch, misordering and packet loss phenomena that occur in the network layer module through a heartbeat sampling mechanism and a network protocol verification and correction mechanism.

9. The distribution network mass data quality improvement system according to claim 8, characterized in that: The cleaning layer module corresponds to the business layer module. Specifically, the cleaning layer module performs rule-based cleaning and data integration on the business logic in the business layer module based on the middleware server. The offline layer module corresponds to the storage layer module. Specifically, the offline layer module performs a full scan on the data in the storage layer based on the ETL server to complete logical investigation and clarification. The ETL server is a distributed server, and the full scan is a distributed parallel operation.

Images