A method for storing and detecting multi-version metadata in a power grid data warehouse

Abstract

The present invention discloses a method for storing and detecting multi-version metadata of a power grid data warehouse, which includes the following steps: constructing a power grid Data Vault (DV) data warehouse; constructing a metamodel for a power grid DV data warehouse environment, including a DV mode and a multidimensional (MD) mode part; performing metadata consistency detection of the DV mode; performing metadata consistency detection of the MD mode; performing consistency detection of the integrity constraints of the attribute table Attributes; performing metadata consistency detection of the DV mode and the MD mode; the present invention utilizes the metamodel of the power grid metadata repository to respectively check the multi-version metadata consistency relationship between the data warehouse area, the data mart area, and the data warehouse and the data mart. The present invention can automatically discover metadata missing, duplication and conflict at the power grid metadata level, thereby achieving the purpose of improving the quality of power grid data.

Description

Technical Field

[0001] The present invention relates to the technical field of data warehouse metadata processing, and in particular to a method for storing and detecting multiple versions of metadata in a power grid data warehouse. Background Art

[0002] In the construction of power grid data warehouse, power grid enterprises have been faced with the technical problem of how to efficiently organize massive data. The existing traditional solution of implementing data warehouse with multidimensional data model and its association can no longer meet the requirements of organizing massive data in the era of big data. Therefore, power grid enterprises have begun to explore the introduction of Data Vault (DV) modeling method to meet the special requirements of power grid enterprises for big data organization.

[0003] The existing disclosure discloses a Data Vault model that automatically generates a data warehouse according to table logical relationships based on a table query device and a table creation device, as well as an automated construction method and device for completing the initialization of a data warehouse. However, its business logic relationships are very complex, and the table logical relationships are difficult to completely cover all model generation, and metadata related issues are not involved;

[0004] The existing disclosed data warehouse construction and loading method and loading system mainly include: (1) a model input module, which is used for users to input model definitions and generate corresponding models; (2) a model naming module, which is used to output the names of libraries, tables, and fields according to naming specifications; (3) a table building module, which is used to generate initialization statements for corresponding libraries and tables. This solution can realize data extraction, data loading, character analysis, and task scheduling, but does not involve related issues of metadata storage management.

[0005] In the prior art, some studies have been conducted on multi-version metadata of multi-dimensional (MD) models for schema evolution in data warehouse environments, and mapping from multi-version metadata of MD models to relational schemas and OLAP operations in standard SQL have been proposed. However, this metamodel is a storage architecture established for multi-version metadata generated by schema evolution, and only involves the MD model in the data warehouse, without involving the metadata version management problem of the DV data warehouse, nor considering the metadata consistency detection algorithm problem. In addition, a MAP table group is established in the metamodel to associate the new and old versions, and the version management is too cumbersome and complicated.

[0006] A metadata version management metamodel has also been proposed based on the version management requirements of various data areas in the DW2.0 architecture to support version evolution management. However, this metamodel only involves the MD model in the data warehouse and does not involve the DV model. Moreover, the constructed structural integrity verification algorithm mainly checks whether the structural relationship between facts, dimensions, hierarchies and attribute tables in the MD model meets the requirements of the MD model, and does not involve comprehensive detection of metadata consistency.

[0007] When constructing the data organization of the power grid data warehouse layer, three data areas are mainly involved. The first is the temporary storage area, where the source business system data is copied to. After the data is confirmed, it is loaded into the data warehouse area according to the DV model. In order to facilitate the use of end users, the data is converted from the data warehouse area to the data set city area according to the multidimensional data model. At a certain moment, multiple versions of the same data may be formed in multiple storage areas. In the power grid data warehouse built based on DV technology, it is necessary to coordinate and manage multiple data areas, multiple data sets and metadata sets, which requires solving the technical problem of multi-version metadata storage and ensuring its consistency. For multi-version metadata storage, it is necessary to design a corresponding meta model to guide the construction of the metadata repository. Furthermore, check whether the metadata stored using this meta model has inconsistencies such as missing, duplication, disconnection and conflict between metadata tables, and whether there are inconsistencies such as missing, duplication and disconnection between metadata table records and corresponding data tables.

[0008] In summary, traditional data warehouse solutions based on multidimensional data models cannot meet the needs of power grid companies for organizing massive data on a centralized, highly scalable, and highly available platform that supports full-network, cross-domain data integration, as well as dynamic on-demand supply and real-time allocation of resources. The DV modeling method can meet the above-mentioned big data organizational structure and operational efficiency requirements of power grid companies, but the existing technologies do not disclose multi-version metamodels and their consistency detection technologies for DV data warehouses. The existing multi-version metadata storage and overall structure detection methods that only rely on traditional data warehouses have too cumbersome and complicated version management of metadata, and have defects such as complex operations and low operating efficiency. Therefore, there is an urgent need for a technical solution to solve the storage and consistency detection problems of multi-version metadata for power grid data warehouses. Summary of the invention

[0009] In order to overcome the defects and shortcomings of the prior art, the present invention provides a method for storing and detecting multiple versions of metadata of a power grid data warehouse. The present invention is oriented to the power grid DV data warehouse environment, solves the problems of storing and detecting multiple versions of metadata of a data warehouse area and a data set city area, adds a DV mode, and the association between the DV mode and the MD mode in the metamodel, and simplifies the table relationship, which can meet the version management requirements of the power grid DV data warehouse and realize the storage and consistency detection of multiple versions of metadata of the power grid.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] The present invention provides a method for storing and detecting multi-version metadata of a power grid data warehouse, comprising the following steps:

[0012] Construct a power grid DV data warehouse. In the power grid DV model, three types of tables, namely, center point, link, and subsidiary, are used to store the attribute data of power grid business entities, relationships, center points or links respectively. In the power grid MD model, fact tables and dimension tables are used to store the power consumption of power grid customers and the attributes of business entities respectively.

[0013] Construct a metamodel for the power grid DV data warehouse environment, including metadata storage tables of the DV mode and the MD mode, which are used to store corresponding metadata. The metadata storage tables of the DV mode include: center point version table, link version table, link element table, subsidiary version table and subsidiary element table. The metadata storage tables of the MD mode include: fact version table, dimension version table, hierarchy version table, layer node element table and layer node version table. There are also basic tables of the common part, including: global version table, attribute table, attribute constraint table and integrity constraint table. The basic tables of the common part are shared by the DV mode and the MD mode.

[0014] Perform metadata consistency detection of the power grid data warehouse, including metadata consistency detection of the DV mode and metadata consistency detection of the MD mode. The metadata consistency detection of the DV mode includes consistency detection of the center point and its affiliated nodes, and consistency detection of the link and its affiliated nodes;

[0015] Perform consistency check on the integrity constraints of the attribute table Attributes to check whether there are corresponding integrity constraint records in the integrity constraint table;

[0016] Perform metadata consistency check on the DV mode and the MD mode, check whether there are records corresponding to the current version in the link version table, and output the consistency test results; based on each fact identifier of the current version, obtain the corresponding set of all dimension identifier records in the dimension version table, check whether there are corresponding records in the center point version table, and output the consistency test results.

[0017] As a preferred technical solution, the steps of detecting the consistency of the center point and its appendages specifically include:

[0018] Starting from the global identifier, obtain all business keys of the global identifier and the current central point version identifier in the central point version table, check whether there are corresponding central point tables that store these business keys in the data warehouse, if not, output an error prompt, check whether there are corresponding central point tables in the data warehouse and the record sources of the tables are consistent, if not, output an error prompt;

[0019] For all obtained business keys, check whether the metadata records stored in the subsidiary version table that are equal to the current version identifier and the business key foreign key have corresponding subsidiary tables in the data warehouse. If not, output an error prompt;

[0020] If there are corresponding subsidiary tables storing these business keys in the data warehouse, obtain several subsidiary element table records corresponding to the subsidiary version table records, and check whether the attribute table corresponding to each subsidiary element table record contains corresponding records. If not, output an error prompt.

[0021] As a preferred technical solution, the steps of link and its affiliated consistency detection specifically include:

[0022] Starting from the global identifier, all link identifiers of the global identifier and the current link version identifier in the link version table are obtained to detect whether the corresponding link table exists in the data warehouse and the record source of the link table is consistent. If not or the record source is inconsistent, an error prompt is output;

[0023] Based on all the link identifiers of the current link version identifier, a number of business keys and center point version identifiers are searched through the link element table to determine whether they are all stored in the center point version table. If not, an error prompt is output;

[0024] Based on all the link identifiers of the current link version identifier, check whether the metadata stored in the subsidiary version table related to the foreign keys of these two identifiers has corresponding subsidiary tables in the data warehouse. If not, output an error prompt;

[0025] If there is a corresponding link table in the data warehouse, several subsidiary feature table records corresponding to the subsidiary version table records are obtained, and it is detected whether the corresponding attribute table contains corresponding records. If not, an error prompt is output.

[0026] As a preferred technical solution, the metadata consistency detection of the MD mode specifically includes the following steps:

[0027] Starting from the global identifier, obtain all fact identifiers of the global identifier and the current fact version identifier in the fact version table, and check whether the corresponding fact table exists in the data mart. If not, output an error prompt;

[0028] For all the obtained fact identifiers, check whether each dimension version table that is equal to the current fact version identifier and the fact identifier foreign key has a corresponding dimension table in the data mart. If not, output an error prompt;

[0029] If there is a corresponding dimension table in the data mart, obtain several hierarchical structures corresponding to the dimension version table, and then obtain several layer node element table records corresponding to the hierarchical structure records, and check whether the parent-child relationship is established. If not, output an error prompt, and check whether there is a corresponding layer node identification record. If not, output an error prompt;

[0030] For the layer node identification record, the layer node identification foreign key in the attribute table is used to detect whether the corresponding attribute table contains the corresponding record. If not, an error prompt is output;

[0031] For the fact version identifier and fact identifier foreign key in the attribute table, check whether the corresponding fact version table contains corresponding records. If not, output a warning prompt. Use the fact version identifier and fact identifier to check whether the corresponding fact table exists in the data mart. If not, output an error prompt.

[0032] As a preferred technical solution, the consistency check of the integrity constraint of the attribute table Attributes includes the following specific steps:

[0033] For each attribute identifier in the attribute table, obtain several integrity constraint records corresponding to the attribute identifier, and check whether there are related records in the integrity constraint table. If not, output an error prompt.

[0034] As a preferred technical solution, the metadata consistency detection of the DV mode and the MD mode specifically includes the following steps:

[0035] Starting from the global identifier, check the link identifier and version identifier foreign key of each record of the current version in the fact version table to see if there is a corresponding record in the link version table. If not, output a warning prompt;

[0036] Starting from the global identifier, use each fact identifier of the current version in the fact version table to obtain the corresponding set of all dimension identifier records in the dimension version table, and check the business key and fact version identifier foreign key in each record to see if there is a corresponding record in the central point version table. If not, a warning prompt is output.

[0037] As a preferred technical solution, when a certain metadata changes, the version update is limited to a part of the DV mode or the MD mode, and a new version identifier is generated correspondingly regarding the center point version identifier, the link version identifier or the fact version identifier.

[0038] As a preferred technical solution, metadata is updated in local batches using DV or MD.

[0039] As a preferred technical solution, there is a corresponding association relationship between the link version table and the fact version table, and the fact table in the MD mode is generated by 0 or 1 link table in the DV mode; there is a corresponding association relationship between the center point version table and the dimension version table, and the dimension table in the MD mode is generated by 0 or 1 center point table in the DV mode.

[0040] The present invention also provides a power grid data warehouse multi-version metadata storage and consistency detection system, including: a power grid DV data warehouse construction module, a meta-model construction module, a power grid data warehouse metadata consistency module, an integrity constraint consistency detection module, and a DV mode and MD mode metadata consistency detection module;

[0041] The power grid DV data warehouse construction module is used to construct the power grid DV data warehouse. In the power grid DV mode, three types of tables, namely, center point, link, and subsidiary, are used to store the attribute data of power grid business entities, relationships, center points or links respectively. In the power grid MD mode, fact tables and dimension tables are used to store the power consumption of power grid customers and the attributes of business entities respectively.

[0042] The metamodel construction module is used to construct a metamodel for the power grid DV data warehouse environment, including metadata storage tables of the DV mode and the MD mode, which are used to store corresponding metadata. The metadata storage table of the DV mode includes: a central point version table, a link version table, a link element table, an attached version table, and an attached element table. The metadata storage table of the MD mode includes: a fact version table, a dimension version table, a hierarchy version table, a layer node element table, and a layer node version table. There are also basic tables of the common part, including: a global version table, an attribute table, an attribute constraint table, and an integrity constraint table, wherein the attribute table is shared by the DV mode and the MD mode;

[0043] The power grid data warehouse metadata consistency module is used to perform metadata consistency detection of the power grid data warehouse, including metadata consistency detection of the DV mode and metadata consistency detection of the MD mode. The metadata consistency detection of the DV mode includes consistency detection of the center point and its affiliated parts, and consistency detection of the link and its affiliated parts.

[0044] The integrity constraint consistency detection module is used to perform consistency detection of the integrity constraint of the attribute table Attributes, and detect whether there is a corresponding integrity constraint record in the integrity constraint table;

[0045] The DV mode and MD mode metadata consistency detection module is used to perform metadata consistency detection on the DV mode and the MD mode, detect whether there is a record corresponding to the current version in the link version table, obtain the corresponding set of all dimension identification records in the dimension version table based on each fact identifier of the current version, detect whether there is a corresponding record in the center point version table, and output the consistency detection result.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] (1) The present invention is oriented to the power grid DV data warehouse environment, solves the technical problem of multi-version metadata storage in the data warehouse area and the data set city area, adds the DV mode and the association between the DV mode and the MD mode in the meta-model, and simplifies the table relationship, which can meet the version management requirements of the power grid DV data warehouse and achieve efficient storage of multi-version metadata of the power grid data warehouse.

[0048] (2) The present invention is oriented to the power grid DV data warehouse environment, adopts the above-mentioned meta-model to store the corresponding multi-version metadata, and solves the technical problem of metadata consistency detection based on the definition of metadata types and their relationship characteristics, thereby achieving automatic consistency verification of multi-version metadata of the power grid, and can automatically discover data missing, duplication and conflict at the metadata level, thereby improving the quality of power grid data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of the method for storing and detecting consistency of multiple versions of metadata of a power grid according to the present invention;

[0050] Figure 2 (a) is a schematic diagram of a fragment of a power grid DV mode in a power grid data warehouse environment of the present invention;

[0051] Figure 2 (b) is a schematic diagram of a fragment of the power grid MD model in the power grid data warehouse environment of the present invention;

[0052] Figure 3 It is a schematic diagram of the metamodel of the present invention for the power grid DV data warehouse environment;

[0053] Figure 4 It is a schematic diagram of the process of metadata consistency detection in the DV mode of the present invention;

[0054] Figure 5 It is a schematic diagram of the process of metadata consistency detection of the MD mode of the present invention;

[0055] Figure 6 It is a flow chart of consistency detection of integrity constraints of attribute table Attributes of the present invention;

[0056] Figure 7 It is a schematic diagram of the process of metadata consistency detection between DV mode and MD mode of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment provides a method for storing and checking the consistency of multi-version metadata in a power grid data warehouse, including the following steps:

[0060] S1: Build a power grid DV data warehouse;

[0061] In order to organize the massive data of the power grid, the data organization requirements of the power grid are: a centralized, highly scalable, and highly available platform that supports full-network and cross-domain data integration, as well as dynamic on-demand supply and real-time allocation of resources, and the construction of a power grid DV data warehouse, including:

[0062] 1) If Figure 2 As shown in (a), the three types of tables, namely, center point, link, and subsidiary, are used to store the attribute data of power grid business entities, relationships, and center point / link respectively; in the figure, there are the power grid customer center point table Hub_Customer (with customer subsidiary table Sat_Customer and customer address subsidiary table Sat_CustAddr), the power consumption contract center point table Hub_Contract (with contract subsidiary table Sat_Contract) and the service center point table Hub_Service (with service subsidiary table Sat_Service), as well as the customer-contract link table Lnk_Cust-Cont and the contract-service link table Lnk_Cont-Serv (with contract-service subsidiary table Sat_Cont-Serv);

[0063] 2) The central point table connects the link table and the subsidiary table based on the business key, and the link table stores the many-to-many relationship;

[0064] 3) A central point or link may have multiple attachments, each of which forms a historical record of different periods of related attributes according to the timestamp (Load_Date);

[0065] 4) The three types of tables, center point, link, and subsidiary, store the record source (Rec_Source) attribute, which can store the data sets of all source power grid business systems.

[0066] The power grid DV model is suitable for efficiently storing massive data of the power grid, but it is not conducive to user use. Therefore, in the DV data warehouse environment, the power grid MD model is also included. It mainly has a multidimensional model composed of fact tables and dimension tables to support end users to analyze data, such as Figure 2 (b), including:

[0067] 1) Use fact tables to store the electricity consumption of grid customers. There is a fact table MD_Usage, which includes two measurement attributes: electricity consumption and amount. There are foreign keys connecting four dimension tables in the primary key area.

[0068] 2) Dimension tables store the attributes of business entities, including customer dimension MD_Customer, contract dimension MD_Contract, service dimension MD_Service, and public date dimension MD_Data. The primary keys of these dimension tables jointly determine a record in the fact table, namely, electricity consumption and amount;

[0069] 3) The dimension table can contain a hierarchical structure, which is composed of layer nodes. Each layer node corresponds to an attribute. For example, the public date dimension MD_Data has three layer nodes that constitute the hierarchy "date-month-year", and the customer dimension MD_Customer has four layer nodes that constitute the hierarchy "customer name-electricity address-city-province".

[0070] The above-mentioned organization of massive power grid data meets the demand for a centralized, highly scalable, and highly available power grid data platform to the greatest extent possible, while supporting full-network, cross-domain data integration, as well as dynamic on-demand supply and real-time allocation of resources.

[0071] S2: Constructing a meta-model for the power grid DV data warehouse environment;

[0072] In this embodiment, the metamodel is used to store multi-version metadata related to the data warehouse and the data mart, wherein a metadata schema version is composed of tables of the DV schema and the MD schema.

[0073] like Figure 3 As shown in the figure, the entire metamodel is divided into three parts. The DV mode part includes 5 basic metadata storage tables: the hub version table HUB_Vers, the link version table Lnk_Vers, the link element table Lnk_Eles, the subsidiary version table Sat_Vers and the subsidiary element table Sat_Eles; the MD mode part includes 5 basic metadata storage tables: the fact version table FACT_Vers, the dimension version table Dim_Vers, the hierarchy version table Hier_Vers, the layer node element table Hier_Eles, and the layer node version table Lev_Vers; the public part includes 4 basic tables: the global version table Versions, the attribute table Attributes, the attribute constraint table Att_Cons and the integrity constraint table Int_Cons. The lines in the figure represent the connection between the tables, the dotted line represents a non-dependent relationship, the solid line represents a dependent relationship, the black dots at the end points of the lines directly connect to represent a many-to-1 relationship, and the diamond represents 0 or 1;

[0074] The metadata stored in the global version table Versions is about the global version of the power grid, mainly involving the center point, link and fact table, including a unique version identifier, name, start and end validity time, and status (whether a version is submitted or under development), etc. After a stable version is formed, when a certain metadata changes, if it only affects the DV or MD locally, the version update will be limited to the DV mode or MD mode. A new version identifier for the center point version identifier Hub_vers_id, link version identifier Lnk_vers_id or fact version identifier FV_id is generated accordingly, and the global identifier VER_id of the Versions table is not generated. In order to avoid frequent generation of global new version identifiers in the DV mode or MD mode, the local batch update of metadata is adopted here to simplify the management of multi-version metadata.

[0075] In order to save metadata related to the source system, the hub version table HUB_Vers, link version table Lnk_Vers, and subsidiary version table Sat_Vers in the DV mode contain metadata in the hub, link, and subsidiary data tables, respectively, which can be verified with the source table and data loading metadata in the DV data table, such as the record source and loading time in these tables. The metadata of all hubs is stored in the hub version table HUB_Vers, which mainly includes the business key Bus_key and the hub version identifier Hub_vers_id, such as the business keys in the Hub_Customer, Hub_Contract, and Hub_Service tables in the power grid DV mode. The hub version identifier Hub_vers_id here mainly reflects the different versions formed by the changes in its affiliated attributes. Then, the metadata of each subsidiary is stored in the subsidiary version table Sat_Vers, such as the customer subsidiary table Sat_Customer, the customer address subsidiary table Sat_CustAddr, and the contract subsidiary table Sat_Contract, etc. The metadata of the subsidiary attributes is stored in the attribute table Attributes through the subsidiary element table Sat_Eles, such as name, phone number, and email.

[0076] For the link version table Lnk_Vers, firstly, the link identifier Lnk_id, version identifier Lnk_vers_id, etc. of each link table are stored in the link version table Lnk_Vers; secondly, the link version table Lnk_Eles is connected with the hub version table HUB_Vers using the business key (such as the specific business keys Cust_key and Cont_key stored in Bus_key, etc.), and metadata such as multiple business keys of the relevant hub table are saved. Then, if the link has an attached table, the metadata of each attached table can be stored in the attached version table Sat_Vers, and the metadata of the attached attributes can be stored in the attribute table Attributes through the attached element table Sat_Eles.

[0077] According to the multidimensional analysis tasks of the end user, the MD model starts from the fact table, connects to some hierarchical tables through the dimension table, and then to the layer node table. The metadata about the fact version is stored in the fact version table FACT_Vers, including a unique fact identifier FV_id, fact version identifier FV_vers_id, fact name, fact system identifier, etc.

[0078] The metadata about the dimension version is stored in the dimension version table Dim_Vers, including a unique dimension identifier DimV_id, fact identifier FV_id, version identifier FV_vers_id, and dimension name.

[0079] The metadata describing the hierarchy and its related dimensions is stored in the hierarchy version table Hier_Vers. The hierarchy version consists of several layer node versions. The description of the layer node version is stored in the layer node version table Lev_Vers, including the layer node identifier LV_id, version identifiers FV_id and FV_vers_id, name, layer node type, etc. The contact information of multiple layer nodes in the hierarchy is stored in the layer node element table Hier_Eles, which contains the parent and child identifier attributes of the layer node.

[0080] Attribute metadata belongs to the public part of the meta-model, and the attribute table Attributes can be shared by the DV mode and the MD mode. It has been clarified before that the attribute table Attributes can store the subsidiary attribute metadata of the DV mode. Similarly, in the MD mode, each fact version and layer node version contains its own attribute set, namely measurement attributes and dimension attributes, and is stored in the attribute table Attributes. The integrity constraints on the attributes are stored in the attribute constraint table Att_Cons through the integrity constraint table Int_Cons. The integrity constraint table Int_Cons stores the integrity constraint name, type and definition.

[0081] In the meta-model, the version issues of the business key Bus_key and the attribute Attribute are not considered.

[0082] The above respectively describes the storage of multi-version metadata in the DV mode and the MD mode. There is also an association relationship between the DV mode and the MD mode: ① Through the relationship between the link version table Lnk_Vers and the fact version table FACT_Vers, that is, the link table and its subordinate attributes actually have a corresponding association relationship with the relevant attributes of the fact table, which indicates that a certain fact table in the MD mode is generated by 0 or 1 link table in the DV mode; ② Through the relationship between the hub version table HUB_Vers and the dimension version table Dim_Vers, that is, the hub and its subordinate attributes actually have a corresponding association relationship with the relevant attributes of the dimension table, which indicates that a certain dimension table in the MD mode is generated by 0 or 1 hub table in the DV mode. The two dotted non-dependent association relationships can reflect the metadata correspondence between the data warehouse and data mart areas.

[0083] S3: Perform metadata consistency check on the power grid data warehouse, including:

[0084] S31: Perform metadata consistency check in DV mode, such as Figure 4 As shown, it specifically includes consistency detection of the center point and its attached parts and consistency detection of the link and its attached parts;

[0085] The consistency detection steps of the center point and its attached points specifically include:

[0086] ① Starting from the global identifier VER_id, obtain all the business keys Bus_key of the global identifier VER_id and the current center point version identifier Hub_vers_id in the center point version table HUB_Vers, and check whether there are corresponding center point tables that store these business keys in the data warehouse (there is a reference table of business keys, center point table names, and subsidiary table names in the power grid DV data warehouse, which can be used to determine whether there is a corresponding center point table in the data warehouse). If not, output an error, and then check whether there is a corresponding center point table in the data warehouse and the record source of the table is Rec_Source (find the center point table according to the above method, and there are business keys and Rec_Source fields in the corresponding records of the center table. It should be confirmed that the two record sources match). If not, output an error;

[0087] ② For all the business keys Bus_key obtained in ①, check whether the metadata records stored in the subsidiary version table Sat_Vers that are equal to the current version identifier Hub_vers_id and the business key Bus_key foreign key exist in the data warehouse (using the reference table described in ①, the judgment method is the same), if not, output an error;

[0088] ③ If there are corresponding subsidiary tables storing these business keys in the data warehouse, obtain several subsidiary element table Sat_Eles records corresponding to the subsidiary version table Sat_Vers records, and check whether the attribute table Attributes corresponding to each subsidiary element table Sat_Eles record contains corresponding records. If not, output an error;

[0089] The consistency check steps for links and their dependencies specifically include:

[0090] ④ Starting from the global identifier VER_id, obtain all the link identifiers Lnk_id of the global identifier VER_id and the current link version identifier Lnk_vers_id in the link version table Lnk_Vers, and check whether there is a corresponding link table in the data warehouse and the record source of the link table is Rec_Source (there is a reference table of Lnk_id, link table name, and subsidiary table name in the power grid DV data warehouse, which can be used to determine whether there is a corresponding link table in the data warehouse. In addition, Rec_Source is also saved in the link table of the power grid DV data warehouse, and it should be confirmed that the two record sources are consistent). If there is no or the record source does not match, an error is output;

[0091] ⑤Use all the link identifiers Lnk_id of the current link version identifier Lnk_vers_id in ④, find several business keys Bus_key and hub version identifiers Hub_vers_id through the link element table Lnk_Eles, and determine whether they are all stored in the hub version table HUB_Vers. If not, output an error;

[0092] ⑥Use all the link identifiers Lnk_id of the current link version identifier Lnk_vers_id in ④ to detect whether the metadata stored in the subsidiary version table Sat_Vers related to the two identifier foreign keys have corresponding subsidiary tables in the data warehouse (using the reference table described in ④, the judgment method is the same), if not, output an error;

[0093] ⑦ If there is a corresponding link table in the data warehouse, obtain several subsidiary element table Sat_Eles records corresponding to the subsidiary version table Sat_Vers record, and check whether the corresponding attribute table Attributes contains the corresponding record. If not, output an error.

[0094] S32: Figure 5 As shown, perform metadata consistency check in MD mode:

[0095] ① Start with the same global identifier VER_id mentioned above, obtain all fact identifiers FV_id of the global identifier VER_id and the current fact version identifier FV_vers_id in the fact version table FACT_Vers, and check whether there are corresponding fact tables in the data mart (using the reference table of identifiers and table names in the DV data warehouse, which will be described in detail below). If not, output an error;

[0096] ② For all the fact identifiers FV_id obtained in ①, check whether each dimension version table DimV_id that is equal to the current fact version identifier FV_vers_id and the fact identifier FV_id foreign key has a corresponding dimension table in the data mart. If not, output an error;

[0097] ③ If there is a corresponding dimension table in the data mart, obtain several hierarchical structures corresponding to the dimension version table Dim_Vers, that is, Hier_id, and then obtain several layer node element table Hier_Eles records corresponding to the Hier_id record. First, check whether the parent-child relationship is established. If not, output an error; then check whether there is a corresponding layer node identifier LV_id record. If not, output an error;

[0098] ④For the layer node identifier LV_id record in ③, through the layer node identifier LV_id foreign key in the attribute table Attributes, check whether the corresponding attribute table Attributes contains the corresponding record. If not, output an error;

[0099] ⑤For the fact version identifier FV_vers_id and the fact identifier FV_id foreign key in the attribute table Attributes, check whether the corresponding fact version table FACT_Vers contains the corresponding record. If not, output an error; then use the fact version identifier FV_vers_id and the fact identifier FV_id to check whether the corresponding fact table exists in the data mart. If not, output an error;

[0100] S33: Figure 6 As shown, the consistency check of the integrity constraint of the attribute table Attributes is performed;

[0101] For each attribute identifier attr_id in the attribute table Attributes, obtain several integrity constraint ic_id records corresponding to the attribute identifier attr_id, and check whether there are related records in the integrity constraint table Int_Cons. If not, output an error.

[0102] S34: Figure 7 As shown, metadata consistency detection is performed between DV mode and MD mode;

[0103] This embodiment mainly detects whether the data mart table established in the MD mode is derived from the corresponding data table in the DV mode. Considering that part of the model may be changed at the request of the end user when building the MD mode, there may be data table metadata in the MD mode that does not correspond to the DV mode.

[0104] ① Starting from the global identifier VER_id, check the link identifier Lnk_id and version identifier Lnk_vers_id foreign key of each record of the current version in the fact version table FACT_Vers to see if there is a corresponding record in the link version table Lnk_Vers. If not, output a possible error message;

[0105] ② Starting from the global identifier VER_id, use each fact identifier FV_id of the current version in the fact version table FACT_Vers to obtain the corresponding record set of all dimension identifiers DimV_id in the dimension version table Dim_Vers, and check the business key Bus_key and fact version identifier FV_vers_id foreign key in each record to see if there is a corresponding record in the hub version table HUB_Vers. If not, output a possible error message.

[0106] Through the above detection steps, the metadata stored in the metamodel is tested for consistency. The metadata stored in the three basic tables in the metamodel are tested for their interrelationships one by one according to the current version, thereby ensuring that the multi-version metadata stored therein corresponds one-to-one with the specific data stored in the data warehouse and data mart, effectively improving the data quality of the power grid DV data warehouse environment.

[0107] In the meta-model of this embodiment, it is not considered to establish multiple versions for the business key Bus_key in the power grid DV model, nor is it considered to establish multiple versions for the attribute table Attributes.

[0108] This embodiment is for the metamodel and metadata consistency detection of the power grid DV data warehouse environment, solves the technical problems of building a metadata repository, and lays the foundation for improving the infrastructure of the existing power grid DV data warehouse environment and building a basic guarantee platform for data quality.

[0109] (1) Metamodel: This provides a storage structure for multi-version metadata and solves the technical problem of building an information model for the metadata repository in the power grid DV data warehouse environment.

[0110] (2) Metadata consistency detection algorithm: To address the quality issues of metadata stored in the power grid metadata repository, the algorithm checks whether the metadata in multiple tables of the repository are missing, duplicated, conflicting, and inconsistent, and solves technical problems in the quality assurance of the stored metadata.

[0111] Embodiment 2:

[0112] This embodiment provides a power grid data warehouse multi-version metadata storage and consistency detection system, including: a power grid DV data warehouse construction module, a meta-model construction module, a power grid data warehouse metadata consistency module, an integrity constraint consistency detection module, and a DV mode and MD mode metadata consistency detection module;

[0113] In this embodiment, the power grid DV data warehouse construction module is used to construct the power grid DV data warehouse. In the power grid DV mode, three types of tables, namely, center point, link, and subsidiary, are used to store the attribute data of power grid business entities, relationships, center points or links respectively. In the power grid MD mode, fact tables and dimension tables are used to store the power consumption of power grid customers and the attributes of business entities respectively.

[0114] In this embodiment, the metamodel construction module is used to construct a metamodel for the power grid DV data warehouse environment, including metadata storage tables of the DV mode and the MD mode, which are used to store corresponding metadata. The metadata storage tables of the DV mode include: a central point version table, a link version table, a link element table, an attached version table, and an attached element table. The metadata storage tables of the MD mode include: a fact version table, a dimension version table, a hierarchy version table, a layer node element table, and a layer node version table. There are also basic tables of the common part, including: a global version table, an attribute table, an attribute constraint table, and an integrity constraint table. The basic tables of the common part are shared by the DV mode and the MD mode.

[0115] In this embodiment, the power grid data warehouse metadata consistency module is used to perform metadata consistency detection of the power grid data warehouse, including metadata consistency detection of the DV mode and metadata consistency detection of the MD mode. The metadata consistency detection of the DV mode includes consistency detection of the center point and its affiliated parts, and consistency detection of the link and its affiliated parts.

[0116] In this embodiment, the integrity constraint consistency detection module is used to perform consistency detection of the integrity constraint of the attribute table Attributes, and detect whether there is a corresponding integrity constraint record in the integrity constraint table;

[0117] In this embodiment, the DV mode and MD mode metadata consistency detection module is used to perform metadata consistency detection on the DV mode and MD mode, detect whether there is a record corresponding to the current version in the link version table, obtain the corresponding set of all dimension identification records in the dimension version table based on each fact identifier of the current version, detect whether there is a corresponding record in the center point version table, and output the consistency detection result.

[0118] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for storing and detecting multi-version metadata in a power grid data warehouse. It is characterized in that The steps include: Construct a power grid DV data warehouse. In the power grid DV model, three types of tables, namely, center point, link, and subsidiary, are used to store the attribute data of power grid business entities, relationships, center points or links respectively. In the power grid MD model, fact tables and dimension tables are used to store the power consumption of power grid customers and the attributes of business entities respectively. Construct a metamodel for the power grid DV data warehouse environment, including metadata storage tables of the DV mode and the MD mode, which are used to store corresponding metadata. The metadata storage tables of the DV mode include: center point version table, link version table, link element table, subsidiary version table and subsidiary element table. The metadata storage tables of the MD mode include: fact version table, dimension version table, hierarchy version table, layer node element table and layer node version table. There are also basic tables of the common part, including: global version table, attribute table, attribute constraint table and integrity constraint table. The basic tables of the common part are shared by the DV mode and the MD mode. Perform metadata consistency detection of the power grid data warehouse, including metadata consistency detection of the DV mode and metadata consistency detection of the MD mode. The metadata consistency detection of the DV mode includes consistency detection of the center point and its affiliated nodes, and consistency detection of the link and its affiliated nodes; Perform consistency check on the integrity constraints of the attribute table Attributes to check whether there are corresponding integrity constraint records in the integrity constraint table; Perform metadata consistency check between the DV mode and the MD mode, check whether there are records corresponding to the current version in the link version table, and output the consistency test results; based on each fact identifier of the current version, obtain the corresponding set of all dimension identifier records in the dimension version table, check whether there are corresponding records in the central point version table, and output the consistency test results; The metadata consistency detection of the DV mode and the MD mode specifically includes the following steps: Starting from the global identifier, check the link identifier and version identifier foreign key of each record of the current version in the fact version table to see if there is a corresponding record in the link version table. If not, output a warning prompt; Starting from the global identifier, use each fact identifier of the current version in the fact version table to obtain the corresponding set of all dimension identifier records in the dimension version table, and check the business key and fact version identifier foreign key in each record to see if there is a corresponding record in the central point version table. If not, a warning prompt is output.

2. The method for storing and detecting multi-version metadata of a power grid data warehouse according to claim 1, It is characterized in that The steps of consistency detection of the center point and its attached parts specifically include: Starting from the global identifier, obtain all business keys of the global identifier and the current central point version identifier in the central point version table, check whether there are corresponding central point tables that store these business keys in the data warehouse, if not, output an error prompt, check whether there are corresponding central point tables in the data warehouse and the record sources of the tables are consistent, if not, output an error prompt; For all obtained business keys, check whether the metadata records stored in the subsidiary version table that are equal to the current central point version identifier and the business key foreign key have corresponding subsidiary tables in the data warehouse. If not, output an error prompt; If there are corresponding subsidiary tables storing these business keys in the data warehouse, obtain several subsidiary element table records corresponding to the subsidiary version table records, and check whether the attribute table corresponding to each subsidiary element table record contains corresponding records. If not, output an error prompt.

3. The method for storing and detecting multi-version metadata of a power grid data warehouse according to claim 1, It is characterized in that The steps of link and its affiliated consistency detection specifically include: Starting from the global identifier, all link identifiers of the global identifier and the current link version identifier in the link version table are obtained to detect whether the corresponding link table exists in the data warehouse and the record source of the link table is consistent. If not or the record source is inconsistent, an error prompt is output; Based on all the link identifiers of the current link version identifier, a number of business keys and center point version identifiers are searched through the link element table to determine whether they are all stored in the center point version table. If not, an error prompt is output; Based on all the link identifiers of the current link version identifier, check whether the metadata stored in the subsidiary version table related to the foreign keys of these two identifiers has corresponding subsidiary tables in the data warehouse. If not, output an error prompt; If there is a corresponding link table in the data warehouse, several subsidiary feature table records corresponding to the subsidiary version table records are obtained, and it is detected whether the corresponding attribute table contains corresponding records. If not, an error prompt is output.

4. The method for storing and detecting multi-version metadata of a power grid data warehouse according to claim 1, It is characterized in that The metadata consistency detection of the MD mode specifically includes the following steps: Starting from the global identifier, obtain all fact identifiers of the global identifier and the current fact version identifier in the fact version table, and check whether the corresponding fact table exists in the data mart. If not, output an error prompt; For all the obtained fact identifiers, check whether each dimension version table that is equal to the current fact version identifier and the fact identifier foreign key has a corresponding dimension table in the data mart. If not, output an error prompt; If there is a corresponding dimension table in the data mart, obtain several hierarchical structures corresponding to the dimension version table, and then obtain several layer node element table records corresponding to the hierarchical structure records, and check whether the parent-child relationship is established. If not, output an error prompt, and check whether there is a corresponding layer node identification record. If not, output an error prompt; For the layer node identification record, the layer node identification foreign key in the attribute table is used to detect whether the corresponding attribute table contains the corresponding record. If not, an error prompt is output; For the fact version identifier and fact identifier foreign key in the attribute table, check whether the corresponding fact version table contains corresponding records. If not, output a warning prompt. Use the fact version identifier and fact identifier to check whether the corresponding fact table exists in the data mart. If not, output an error prompt.

5. The method for storing and detecting multi-version metadata of a power grid data warehouse according to claim 1, It is characterized in that The consistency check of the integrity constraint of the attribute table Attributes specifically includes the following steps: For each attribute identifier in the attribute table, obtain several integrity constraint records corresponding to the attribute identifier, and check whether there are related records in the integrity constraint table. If not, output an error prompt.

6. The method for storing and detecting multi-version metadata of a power grid data warehouse according to claim 1, It is characterized in that When a certain metadata changes, the version update is limited to the DV mode or the local MD mode, and a new version identifier is generated corresponding to the central point version identifier, link version identifier or fact version identifier.

7. The method for storing and detecting multi-version metadata of a power grid data warehouse according to claim 1, It is characterized in that The metadata is updated in partial batches using DV or MD.

8. The method for storing and detecting multi-version metadata of a power grid data warehouse according to claim 1, It is characterized in that There is a corresponding association relationship between the link version table and the fact version table. The fact table in the MD mode is generated by 0 or 1 link table in the DV mode; There is a corresponding association relationship between the center point version table and the dimension version table. The dimension table in the MD mode is generated by 0 or 1 center point table in the DV mode.

9. A multi-version metadata storage and consistency detection system for power grid data warehouse, It is characterized in that A method for storing and detecting the multi-version metadata of a power grid data warehouse according to any one of claims 1 to 8, comprising: a power grid DV data warehouse construction module, a metamodel construction module, a power grid data warehouse metadata consistency module, an integrity constraint consistency detection module, and a DV mode and MD mode metadata consistency detection module; The power grid DV data warehouse construction module is used to construct the power grid DV data warehouse. In the power grid DV mode, three types of tables, namely, center point, link, and subsidiary, are used to store the attribute data of power grid business entities, relationships, center points or links respectively. In the power grid MD mode, fact tables and dimension tables are used to store the power consumption of power grid customers and the attributes of business entities respectively. The metamodel construction module is used to construct a metamodel for the power grid DV data warehouse environment, including metadata storage tables of the DV mode and the MD mode, which are used to store corresponding metadata. The metadata storage table of the DV mode includes: a central point version table, a link version table, a link element table, an attached version table, and an attached element table. The metadata storage table of the MD mode includes: a fact version table, a dimension version table, a hierarchy version table, a layer node element table, and a layer node version table. There are also basic tables of the common part, including: a global version table, an attribute table, an attribute constraint table, and an integrity constraint table, wherein the attribute table is shared by the DV mode and the MD mode; The power grid data warehouse metadata consistency module is used to perform metadata consistency detection of the power grid data warehouse, including metadata consistency detection of the DV mode and metadata consistency detection of the MD mode. The metadata consistency detection of the DV mode includes consistency detection of the center point and its affiliated parts, and consistency detection of the link and its affiliated parts. The integrity constraint consistency detection module is used to perform consistency detection of the integrity constraint of the attribute table Attributes, and detect whether there is a corresponding integrity constraint record in the integrity constraint table; The DV mode and MD mode metadata consistency detection module is used to perform metadata consistency detection on the DV mode and the MD mode, detect whether there is a record corresponding to the current version in the link version table, obtain the corresponding set of all dimension identification records in the dimension version table based on each fact identifier of the current version, detect whether there is a corresponding record in the center point version table, and output the consistency detection result.

Images