Model-based data quality checking method and device

Abstract

The invention relates to the technical field of electric power, in particular to a data quality checking method and device based on a model, and the method comprises the following steps: S1, summarizing target metadata, building a target database, and recording transformer data information, metering box data information, electric energy meter data information and user data information of property rights belonging to users; s2, establishing a data checking model based on each piece of data configuration information of the target database; and S3, establishing a background data management module, and connecting the background data management module with the target database. Different from the prior art, the data quality checking method and device based on the model are configured based on the metadata corresponding to the target database, and a corresponding data checking model is established based on the obtained checking configuration information; therefore, dynamic creation of the data checking model is realized, personalized requirements of data checking are met, a large amount of data can be processed and bills can be checked without manual screening, and the problems of high error rate and repeated screening of manual screening are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power, and specifically to a method and device for data quality verification based on a model. Background Art

[0002] The power supply reliability index is an important index for evaluating the power quality of the power supply system of power grid enterprises, which reflects the satisfaction degree of the power industry with the power demand of the national economy. In the application of power grid management, it is often necessary to provide decision support for the development planning and operation of the power grid based on relevant data of power grid operation.

[0003] At present, there are still deficiencies in aspects such as the operation efficiency of power grid services and the accuracy of data support. In actual work, data quality problems such as non-standard, non-specification, missing filling, missing, and non-uniform basic information occur in data verification, which cannot meet the requirements of timely extraction and sharing of power grid companies. Moreover, the accuracy of some production business data is poor, the measurement units, data precision, and names are inaccurate, and the abnormal trigger rate caused by data problems is relatively high, which has a greater impact on system resources, subsequent bills, and verification workload. Therefore, we propose a method and device for data quality verification based on a model. Summary of the Invention

[0004] To solve the above technical problems, an embodiment of the present application provides a method for data quality verification based on a model, including the following steps:

[0005] S1: Summarize the target metadata and establish a target database, and record the transformer data information, metering box data information, electric energy meter data information, and user data information whose property rights belong to users;

[0006] S2: Establish a data verification model based on the data configuration information of each data in the target database;

[0007] S3: Establish a background data management module, connect it to the target database, and be used to verify each metadata of the target database;

[0008] S4: Use the background data management module, and combine with the data verification rules to determine and eliminate the problem data that does not conform to the data verification rules;

[0009] S5: Use the data verification rules to restrict the data to be verified by users, and import the qualified data to be verified by users into the data verification model for comparison;

[0010] S6: Use the data verification model to output and determine the verification result corresponding to the data to be verified.

[0011] In some embodiments, the data verification rules include the basic attribute rules and attribute association rules of the device;

[0012] Among them, the basic attribute rules include the regular expression, non-emptiness, inclusion, digital type restriction, and size range restriction of the device.

[0013] The attribute association rule refers to whether the attributes of multiple associated resource items are consistent.

[0014] In some embodiments, the target database further includes power outage event data, which collects power outage events, voltage anomaly events, power load overload events, abnormal electricity charge collection events, events where the actual electricity quantity does not match the record, abnormal electricity price collection data events, incorrect electricity meter reading events, abnormal electricity charge collection events, events where the actual electricity quantity does not match the record, abnormal electricity price collection data events, and incorrect electricity meter reading events within the required time period or the required power supply range from the business management system, including obtaining the main network and / or distribution network power outage data of the power dispatching automation system within the required time period, and obtaining the system power outage user data of the metering automation system.

[0015] In some embodiments, the business system is a metering automation management system.

[0016] In some embodiments, the information of newly installed, missed, and rotated metering devices that have not been entered into the metering automation management system is verified; users who need to supplement information are found, and professional personnel are dispatched to conduct data collection and recording to form a verification bill.

[0017] In some embodiments, when verifying the data to be verified, the data to be verified is first used as column information, and then a two-dimensional data structure object associated with data by row index and column index is created.

[0018] In some embodiments, the background data management module further includes:

[0019] A query module, connected to the target database, for searching and filtering abnormal data that does not conform to the data verification rules according to the received query instruction;

[0020] A modification module, connected to the target database, for adding, deleting, or modifying abnormal metadata according to the received modification instruction.

[0021] In some embodiments, when obtaining target data, the user historical data corresponding to the target user group is used to generate a first target model, the user historical data corresponding to the target user group is input into the first target model to obtain first intermediate abnormal data, and the initial classification model is trained based on the first intermediate abnormal data and the second intermediate data to obtain a second target model.

[0022] In some embodiments, when establishing a data verification model, the first target model and the second target model are sequentially connected in series to generate a target data verification model.

[0023] A model-based data quality verification device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the above are implemented;

[0024] When the computer program is executed by the processor, the steps of the method described in any one of the above are implemented.

[0025] The present invention has at least the following beneficial effects:

[0026] Different from the prior art, by configuring based on the metadata corresponding to the target database and creating a corresponding data verification model based on the obtained verification configuration information, the dynamic creation of the data verification model is realized, meeting the personalized needs of data verification. It can process a large amount of data without manual screening, avoiding the problems of high error rate and repeated screening in manual screening. Moreover, an abnormal data list is constructed for each device to assist the staff lacking inspection and control methods to directly carry out data quality governance independently. In addition, by using the data verification model, the verification result can be obtained by only reading the target database once, without repeatedly reading the data frequently, avoiding the situation of excessive occupation of database resources due to data verification, and reducing the service pressure on the database to ensure the normal use of the database. Specific embodiments

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0028] The present invention provides a technical solution: a model-based data quality verification method, including the following steps:

[0029] S1: Summarize the target metadata and establish a target database, and record the transformer data information, metering box data information, electric energy meter data information, and user data information whose property rights belong to the user;

[0030] S2: Establish a data verification model based on the data configuration information of the target database. Specifically, by reading the metadata corresponding to the target database, all data table field information of the target database can be obtained, and all data table field information can be displayed in the configuration interface, so that users can select each data table field to be verified from the displayed all data table field information based on business requirements, and configure the corresponding data verification method for each data table field to be verified, thereby realizing the dynamic and flexible configuration of the data verification method and avoiding over-reliance on the database and data table structure, further improving the verification efficiency. Based on the verification configuration information configured by the user, the corresponding data verification model can be dynamically created, so that the verification configuration information can be solidified into the data verification model, enabling the data verification model to perform data verification on the corresponding field data in the input data to be verified based on the data verification method corresponding to each data table field to be verified in the verification configuration information, thus realizing the dynamic creation of the data verification model and meeting the personalized needs of data verification;

[0031] S3: Establish a background data management module, connect it to the target database, and use it to verify the metadata of the target database;

[0032] S4: Use the background data management module and combine it with the data verification rules to determine and eliminate the problem data that does not conform to the data verification rules. The method for generating abnormal data verification is to obtain the target group division model, obtain the user historical data, input the user historical data into the target group division model, perform group division on each user based on the target group division model to obtain the target user group; perform unsupervised training on the user historical data corresponding to the target user group to generate the first target model, input the user historical data corresponding to the target user group into the first target model to obtain the first intermediate abnormal data; obtain the second intermediate data, and train the initial classification model based on the first intermediate abnormal data and the second intermediate data to obtain the second target model; sequentially connect and combine the target group division model, the first target model, and the second target model to generate the target verification model. The target verification model is used to obtain the user abnormal category and user abnormal list based on the data to be verified of the user. Through the target group division model, perform group division on each user corresponding to the user historical data to obtain the target user group, perform unsupervised training on the user historical data corresponding to the target user group to generate the first target model, find the abnormal data in the user historical data corresponding to the target user group based on the first target model as the first intermediate abnormal data, obtain the second intermediate data, train the initial classification model based on the first intermediate abnormal data and the second intermediate data to obtain the second target model, and sequentially connect and combine the target group division model, the first target model, and the second target model to generate the target verification model, realizing the verification process of data anomalies;

[0033] S5: Use the data verification rules to restrict the user's data to be verified, and import the qualified user data to be verified into the data verification model for comparison. Specifically, after the target user data to be verified is input into the target verification model, the target group division model corresponding to the target verification model will divide the users corresponding to the target user data to be verified into groups. When the user belongs to the user abnormal category, the target user data to be verified corresponding to the user will be input into the first model corresponding to the target verification model. After the first model verifies the target user data to be verified, the first output data is obtained. The second model further verifies the first output data to obtain abnormal data with higher accuracy. The abnormal data is the abnormal data found from the target user data to be verified, that is, the target user abnormal bill;

[0034] S6: Use the data verification model to output and determine the verification result corresponding to the data to be verified.

[0035] It should be noted that the data to be verified is determined in the form of table fields, which can refer to the fields to be verified in the data table pre-configured by the user based on business requirements, and the number of fields in the data table to be verified can be one or more. In addition, the data verification method can be pre-configured to verify the data corresponding to the corresponding fields in the data table to be verified.

[0036] Specifically, the data verification method can include but is not limited to: data verification rules and verification trigger methods. Among them, the data verification rules can be represented by using data verification functions to improve the data verification efficiency.

[0037] For example, the data verification rules can include at least one data verification function, and the data verification functions can include but are not limited to: non-empty verification function, enumeration value function, and value range verification function.

[0038] If the data verification rules corresponding to the fields in the data table to be verified include two or more data verification functions, the verification order of the data verification functions can also be dynamically configured or defaulted, that is, the data verification method can also include: function verification order information, and the function verification order information can include simultaneous verification or sequential verification according to the configured order.

[0039] In some embodiments, the data verification rules include the basic attribute rules and attribute association rules of the device;

[0040] Among them, the basic attribute rules include the regular expression, non-empty, inclusion, digital type restriction, and size range restriction of the device;

[0041] The attribute association rule refers to whether the attributes of multiple associated resource items are consistent.

[0042] In some embodiments, the target database further includes power outage event data, which collects power outage events, voltage anomaly events, power load overload events, abnormal electricity charge collection events, events where the actual electricity quantity does not match the record, abnormal electricity price collection data events, incorrect electricity meter readings events, abnormal electricity charge collection events, events where the actual electricity quantity does not match the record, abnormal electricity price collection data events, incorrect electricity meter readings events within the required time period or the required power supply range from the business management system, including obtaining the main network and / or distribution network power outage data of the power dispatching automation system within the required time period, and obtaining the system power outage user data of the metering automation system. It should also be noted that relevant problem events regarding transformer power factor, variable loss, line loss, peak-valley, metering, metering point, power source, property demarcation point, hub station, pipeline pole, distribution station, transformer substation, etc. should also be taken into account. At the same time, it should be noted that although this embodiment lists relevant data events as much as possible, it does not mean that those not exemplified do not fall within the protection scope of this application.

[0043] In some embodiments, the business system is a metering automation management system.

[0044] In some embodiments, the information of newly installed, missed, and rotated metering devices that are not entered into the metering automation management system is verified; users who need to supplement information are found, and professional personnel are dispatched to carry out collection and recording.

[0045] In some embodiments, when verifying the data to be verified, first, the data to be verified is used as column information, and then a two-dimensional data structure object associated with data by row index and column index is created.

[0046] In some embodiments, the background data management module further includes:

[0047] A query module, connected to the target database, for searching and filtering abnormal data that does not conform to the data verification rules according to the received query instruction;

[0048] A modification module, connected to the target database, for adding, deleting, or modifying abnormal metadata according to the received modification instruction.

[0049] In some embodiments, when obtaining target data, the user historical data corresponding to the target user group is used to generate a first target model, the user historical data corresponding to the target user group is input into the first target model to obtain first intermediate abnormal data, and based on the first intermediate abnormal data and the second intermediate data, the initial classification model is trained to obtain a second target model;

[0050] It should be noted that there may be a lot of duplicate data in the first intermediate abnormal data. To ensure the accuracy of the initial classification model training process, it is necessary to remove the duplicate data in the first intermediate abnormal data to obtain the first target abnormal data. The set of the first target abnormal data and the second intermediate data is used as the training data for subsequent training of the initial classification model, that is, the target data.

[0051] In some embodiments, when establishing a data verification model, the first target model and the second target model are sequentially connected in series to generate a target data verification model.

[0052] A model-based data quality verification device includes a memory and a processor. The memory stores a computer program. A computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of this computer device is used to provide computing and control capabilities. The memory of this computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of this computer device is used to store user historical data. The input / output interface of this computer device is used to exchange information between the processor and external devices. The communication interface of this computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it is used to implement a data model verification.

[0053] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0054] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A model-based data quality verification method, characterized in that: S1: Summarize the target metadata and establish a target database, and record the transformer data information, metering box data information, electric energy meter data information, and user data information whose property rights belong to the user; S2: Establish a data verification model based on the data configuration information of each database in the target database; S3: Establish a background data management module, connect it to the target database, and use it to verify each metadata in the target database; S4: Use the background data management module and combine it with the data verification rules to determine and eliminate the problem data that does not conform to the data verification rules; S5: Use the data verification rules to restrict the user's data to be verified, and import the qualified user's data to be verified into the data verification model for comparison; S6: Use the data verification model to output and determine the verification result corresponding to the data to be verified.

2. The method and device for verifying data quality based on a model according to claim 1, wherein: The data verification rules include the basic attribute rules and attribute association rules of the device; Among them, the basic attribute rules include the regular expression, non-empty, inclusion, digital type restriction, and size range restriction of the device; The attribute association rule refers to whether the attributes of multiple associated resource items are consistent.

3. The method and device for verifying data quality based on a model according to claim 1, wherein: The target database also includes power outage event data, which collects power outage events, voltage abnormality events,..., power load overload events within the required time period or the required power supply range from the business management system, including obtaining the main network and / or distribution network power outage data of the power dispatching automation system within the required time period, and obtaining the system power outage user data of the metering automation system.

4. The method and device for data quality verification based on a model according to claim 3, wherein: The business system is a metering automation management system.

5. The method and device for verifying data quality based on a model according to claim 3, wherein: Verify the information of newly installed, missed, and rotated metering devices that have not been entered into the metering automation management system; find out the users who need to supplement the information, and dispatch professional personnel to carry out the collection and recording.

6. The method and device for verifying data quality based on a model according to claim 1, wherein: When verifying the data to be verified, first use the data to be verified as column information, and then create a two-dimensional data structure object associated with the data by row index and column index.

7. The method and device for verifying data quality based on a model according to claim 3, wherein: The background data management module further includes: A query module, connected to the target database, used to search and screen the abnormal data that does not conform to the data verification rules according to the received query instruction; A modification module, connected to the target database, used to add, delete, or modify the abnormal metadata according to the received modification instruction.

8. The method and device for verifying data quality based on a model according to claim 1, wherein: When obtaining the target data, generate a first target model from the user historical data corresponding to the target user group, input the user historical data corresponding to the target user group into the first target model to obtain the first intermediate abnormal data, and train the initial classification model based on the first intermediate abnormal data and the second intermediate data to obtain the second target model.

9. The method and device for verifying data quality based on a model according to claim 7, wherein: When establishing the data verification model, sequentially connect and combine the first target model and the second target model to generate the target data verification model.

10. The model-based data quality verification device according to claim 9, comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9; When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.