Data management method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a data management method and device, electronic equipment, a storage medium and a program product. The method comprises the steps that target metadata of multiple data sources is collected; determining and storing a meta-model of the target metadata according to the target metadata of each data source and a pre-constructed meta-model template; determining a data quality rule corresponding to the target metadata according to the meta-model of each target metadata; and performing quality detection on the to-be-detected data corresponding to the target metadata according to the data quality rule, and determining the to-be-detected data which does not conform to the data quality rule as quality non-compliance data. According to the scheme, on one hand, the situation that only metadata management can be carried out on the data of one data source when a single data model is adopted is avoided, so that the management efficiency of the data is improved, and the use efficiency of the data is improved. On the other hand, the situation that the data quality rule is not suitable for the to-be-detected data of a certain data source is avoided, and the accuracy of data quality detection is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data management method, device, electronic device, storage medium and program product. Background Art

[0002] With the development of the information age, data is widely used in corporate decision-making, business development, and operational management. However, the large amount of data makes data management increasingly difficult. In order to ensure the effective use and accuracy of data, data governance has emerged.

[0003] Data governance mainly includes two aspects: metadata management and data quality management. Metadata management is to store information such as data attributes and sources to facilitate effective use of data. The goal of data quality management is to ensure the accuracy and integrity of data.

[0004] However, traditional metadata management usually adopts a single data model, so metadata management can only be performed on data from one data source, and it is impossible to manage multiple heterogeneous data sources, thereby reducing data management efficiency and usage efficiency. Summary of the invention

[0005] The present application provides a data governance method, device, electronic device, storage medium and program product to solve the problem that traditional metadata management in the prior art usually adopts a single data model, so metadata management can only be performed on data of one data source, and multiple heterogeneous data sources cannot be managed, thereby reducing data management efficiency and usage efficiency.

[0006] In a first aspect, the present application provides a data governance method, the method comprising:

[0007] Collect target metadata from multiple data sources;

[0008] According to the target metadata of each data source and the pre-built metamodel template, determine and store the metamodel of the target metadata; wherein the metamodel template includes a plurality of attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata;

[0009] Determining data quality rules corresponding to the target metadata according to the metamodel of each target metadata;

[0010] The quality of the data to be tested corresponding to the target metadata is checked according to the data quality rules, and the data to be tested that does not meet the data quality rules is determined as quality non-compliant data.

[0011] In a second aspect, the present application provides a data management device, the device comprising:

[0012] A collection module, used to collect target metadata from multiple data sources;

[0013] A metamodel determination module, used to determine and store a metamodel of the target metadata according to the target metadata of each data source and a pre-built metamodel template; wherein the metamodel template includes a plurality of attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata;

[0014] A rule determination module, used to determine the data quality rule corresponding to the target metadata according to the meta-model of each target metadata;

[0015] The quality detection module is used to perform quality detection on the data to be tested corresponding to the target metadata according to the data quality rules, and determine the data to be tested that does not meet the data quality rules as quality non-compliant data.

[0016] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data governance method as described in any embodiment of the present application is implemented.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data governance method as described in any embodiment of the present application.

[0018] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data governance method as described in any embodiment of the present application.

[0019] The scheme of the present application collects target metadata of multiple data sources; determines and stores the metamodel of the target metadata according to the target metadata of each data source and a pre-built metamodel template; wherein the metamodel template includes multiple attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata; determines the data quality rules corresponding to the target metadata according to the metamodel of each target metadata; performs quality inspection on the data to be tested corresponding to the target metadata according to the data quality rules, and determines the data to be tested that does not meet the data quality rules as quality non-compliant data. That is, the scheme of the present application, on the one hand, determines and stores the metamodel of the target metadata through the pre-built metamodel template and the target metadata of each data source in the multiple data sources, thereby completing the metadata management of multiple data sources, avoiding the situation that only metadata management can be performed on the data of one data source when a single data model is used, thereby improving the management efficiency of the data, and, when using the data, because the metadata of multiple data sources are stored, the data of multiple data sources can be selected, thereby improving the use efficiency of the data. On the other hand, the data quality rules corresponding to the target metadata are determined according to the metamodel of the target metadata, and the quality of the data to be tested corresponding to the target metadata is tested according to the data quality rules, so that the data quality test is based on the metamodel corresponding to the data, avoiding the situation where the data quality rules are not applicable to the data to be tested of a certain data source, thereby improving the accuracy of data quality testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flowchart of the data governance method provided by this application;

[0022] Figure 2a is another flow chart of the data governance method provided by this application;

[0023] Figure 2b It is an exemplary structural diagram of a metadata management platform of the data governance method provided by this application;

[0024] Figure 2c It is an exemplary structural diagram of a data quality platform of the data governance method provided by this application;

[0025] Figure 3 It is a structural diagram of the data management device provided by this application;

[0026] Figure 4 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Before introducing the solutions of the present invention, relevant terms involved in the embodiments of the present invention are first explained.

[0030] Data governance: Data governance is the process of managing data by storing metadata to store information such as data attributes and sources, improving data utilization, and managing data quality to enhance data integrity. Therefore, the data governance process includes metadata storage and data quality management.

[0031] Figure 1 The present invention provides a flow chart of a data management method, which can be executed by a data management device, which can be implemented in software and / or hardware. In a specific embodiment, the device can be applied to an electronic device, which can be a computer. The following embodiments will be described by taking the application of the device in an electronic device as an example, referring to Figure 1 , the method may specifically include the following steps:

[0032] Step 101: collect target metadata from multiple data sources.

[0033] Specifically, the data source is the source of data that the electronic device executing this embodiment needs to collect. The data source can be any object that can be described by data in the data field such as data development framework, database, data warehouse tool and data processing system. For example, the data source can be Hive, Hbase and Kafka. Metadata is data used to describe data, that is, information that can describe the attributes of data. For example, metadata is information that can describe the organization of data, data domain and its relationship. For example, the metadata of the database can describe the basic information of the database object and the meaning of the data. By collecting target metadata of multiple data sources, the attribute information of multiple data sources can be obtained. For example, the metadata collection method can include push mode and pull mode. When the metadata is collected in pull mode, the electronic device executing this embodiment can collect metadata once every preset time interval by scheduling the DolphinScheduler scheduling engine, so as to obtain the target metadata. When the metadata is collected in push mode, the electronic device executing this embodiment can monitor and obtain the target metadata through the Hook function. After collecting the metadata, the electronic device executing this embodiment can also upload the metadata to the message system plug-in, and the message system plug-in can be Pulsar.

[0034] Step 102: Determine and store a metamodel of the target metadata according to the target metadata of each data source and a pre-built metamodel template.

[0035] The metamodel template includes a plurality of attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata.

[0036] Specifically, a metamodel is a model that can describe the attributes of metadata, that is, it can define the attributes of metadata, so multiple attributes of metadata are metamodels. The metamodel template includes multiple attribute templates of metadata. According to the attributes of the target metadata of each data source, multiple attribute templates in the metamodel template can describe the attributes of the target metadata of each data source, and the metamodel of the target metadata can be obtained. After obtaining the metamodel of the target metadata, the metamodel of the target metadata is stored, for example, in a storage unit of the electronic device that executes this embodiment.

[0037] Optionally, according to the target metadata of each data source and a pre-built metamodel template, determining and storing the metamodel of the target metadata may be implemented through steps 1021 to 1022 .

[0038] Step 1021 : determining multiple attributes of target metadata according to the target metadata of each data source.

[0039] Specifically, metadata includes multiple attributes, for example, metadata attributes may be a metadata name field, metadata length, metadata description, metadata modification time, metadata type, metadata creation time, etc. Based on the collected target metadata of each data source, multiple attributes of the target metadata are determined.

[0040] Step 1022: replace multiple attribute templates in the meta-model template with multiple target metadata attributes of the target metadata, obtain the meta-model of the target metadata, and store the meta-model of the target metadata.

[0041] Specifically, after determining multiple attributes of the target metadata, multiple attribute templates in the metamodel template are replaced with multiple target metadata attributes of the target metadata, so that the metamodel includes the attributes of the target metadata of each data source, that is, the metamodel of the target metadata is obtained. After obtaining the metamodel of the target metadata, the metamodel of the target metadata is stored.

[0042] Exemplarily, the metamodel template includes a name field template, a length template, a description template, a modification time template, a type template, and a creation time template. Multiple attribute templates in the metamodel template are replaced with multiple target metadata attributes of the target metadata, even if the metamodel template includes the metadata name field, metadata length, metadata description, metadata modification time, metadata type, and metadata creation time of the metadata, after the replacement, the metamodel of the target metadata can be obtained.

[0043] Optionally, after determining and storing the meta-model of the target metadata according to the target metadata of each data source and the pre-built meta-model template, step 1023 may be performed.

[0044] Step 1023, taking each attribute value of the target metadata in the meta-model of the target metadata as a data point, and taking the relationship between the attribute values ​​of the target metadata in the meta-model as an edge, to establish graph data information.

[0045] Specifically, graph data is data composed of nodes and edges, and the nodes and edges in the graph are correlated. Each attribute value of the target metadata is used as a data point, and the relationship between the attribute values ​​of the target metadata in the metamodel is used as an edge to obtain graph data information. After storage, the generated graph data information can further characterize the relationship between metadata attributes, facilitate data processing, and provide a data basis for the subsequent determination of data quality rules corresponding to the target metadata. The graph data information can also be associated with a search library with Elastic Stack as the core to facilitate searching for the graph data information of the required metadata.

[0046] For example, Figure 2b1 is an exemplary structural diagram of a metadata management platform of the data governance method provided by the present application. The metadata management platform installed in the electronic device executing the present embodiment is used to implement the metadata storage step in the data governance process. Figure 2b As shown, when metadata is collected in pull mode, the metadata of the data source is collected at preset intervals by scheduling the DolphinScheduler scheduling engine. When metadata is collected in push mode, the metadata of the data source is collected through the Message-plugins interface or the application programming interface (Application ProgramInterface, API) that follows the REST architecture specification. After collecting the metadata, the metadata is stored in the MetaData Store storage area in the form of a metadata metamodel for subsequent application. In addition, the process of storing metadata can be automatically stored through the DB-plugins plug-in. Subsequent applications of metadata may include data search, data discovery, tag management, asset catalog management, data map setting, metadata lineage establishment, data tracking, and data monitoring. Among them, establishing a data map can make data resource information detailed, unified, and transparent, improve the efficiency of data query, and provide support for data use. Establishing metadata lineage can quickly locate the source of data and the processing process of data, and can help staff quickly locate data problems.

[0047] Step 103: determine the data quality rules corresponding to the target metadata according to the meta-model of each target metadata.

[0048] Specifically, after determining and storing the metamodel of the target metadata, the quality qualification range of the data in the data source corresponding to the target metadata is determined according to the metamodel of each target metadata, and the data quality rules corresponding to the target metadata are determined according to the quality qualification range of the data in the data source.

[0049] Exemplarily, according to the metamodel of the target metadata, it is determined that the qualified quality range of the data in the data source is a string length of 16 bits, then according to the qualified quality range of the data in the data source, the data quality rule corresponding to the target metadata is determined as the data quality is qualified when the string length is 16 bits.

[0050] Optionally, after executing step 1023 , determining data quality rules corresponding to the target metadata according to the meta-model of each target metadata may be implemented through step 1031 .

[0051] Step 1031, determining the data quality rules corresponding to the target metadata according to the graph data information corresponding to each target metadata.

[0052] Specifically, after establishing the graph data information, the quality qualified range of the data in the data source corresponding to the target metadata is determined according to the graph data information corresponding to each target metadata, and the data quality rules corresponding to the target metadata are determined according to the quality qualified range of the data in the data source.

[0053] Optionally, determining the data quality rules corresponding to the target metadata according to the meta-model of each target metadata may also be implemented through steps 1032 to 1033.

[0054] Step 1032: Determine data standard information of the target metadata according to the meta-model of each target metadata.

[0055] The data standard information is used to represent the standard information of the governed data in the data source corresponding to the target metadata.

[0056] Specifically, the governed data is the data in the data source that is determined to be compliant with data quality. According to the metamodel of each target metadata, the standard information of the governed data in the data source corresponding to the target metadata is determined, that is, the data standard information of the target metadata.

[0057] Exemplarily, the data standard information of the target metadata is determined according to the metadata meta-model as a character string length of 16 bits.

[0058] Step 1033, determining data quality rules according to the data standard information.

[0059] Specifically, after obtaining the data standard information, the data standard information is used as the data quality rule. For example, the data standard information is that the length of the character string is 16 bits, and the data quality rule is that the data quality is qualified when the length of the character string is 16 bits.

[0060] Step 104 , performing quality inspection on the data to be tested corresponding to the target metadata according to the data quality rules, and determining the data to be tested that does not meet the data quality rules as quality non-compliant data.

[0061] Specifically, the data to be tested is the data in the database corresponding to the target metadata that has not been quality tested. After the data to be tested is quality tested, data governance is completed. According to the data quality rules, the data to be tested corresponding to the target metadata is quality tested, and the data to be tested that meets the data quality rules is determined as quality compliant data, and the data to be tested that does not meet the data quality rules is determined as quality non-compliant data.

[0062] Exemplarily, the data quality rule is that the data quality is qualified when the string length is 16 bits, then the data with a data string length of 16 bits in the tested data is quality-compliant data, and the data with a data string length not of 16 bits is quality-incompliant data.

[0063] The scheme of the present application collects target metadata of multiple data sources; determines and stores the metamodel of the target metadata according to the target metadata of each data source and a pre-built metamodel template; wherein the metamodel template includes multiple attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata; determines the data quality rules corresponding to the target metadata according to the metamodel of each target metadata; performs quality inspection on the data to be tested corresponding to the target metadata according to the data quality rules, and determines the data to be tested that does not meet the data quality rules as quality non-compliant data. That is, the scheme of the present application, on the one hand, determines and stores the metamodel of the target metadata through the pre-built metamodel template and the target metadata of each data source in the multiple data sources, thereby completing the metadata management of multiple data sources, avoiding the situation that only metadata management can be performed on the data of one data source when a single data model is used, thereby improving the management efficiency of the data, and, when using the data, because the metadata of multiple data sources are stored, the data of multiple data sources can be selected, thereby improving the use efficiency of the data. On the other hand, the data quality rules corresponding to the target metadata are determined according to the metamodel of the target metadata, and the quality of the data to be tested corresponding to the target metadata is tested according to the data quality rules, so that the data quality test is based on the metamodel corresponding to the data, avoiding the situation where the data quality rules are not applicable to the data to be tested of a certain data source, thereby improving the accuracy of data quality testing.

[0064] Figure 2a is another flow chart of the data governance method provided by the present application. Figure 1 Based on the illustrated embodiment and various optional implementation schemes, the steps of performing quality inspection on the data to be tested corresponding to the target metadata according to the data quality rules are described in detail, and the steps after determining the quality non-compliant data are described in detail. Figure 2a As shown, the method may include the following steps:

[0065] Step 201: collect target metadata from multiple data sources.

[0066] Step 202: Determine and store a metamodel of the target metadata according to the target metadata of each data source and a pre-built metamodel template.

[0067] Step 203: determine the data quality rules corresponding to the target metadata according to the meta-model of each target metadata.

[0068] Step 204 , performing quality inspection on the data to be tested corresponding to the target metadata according to the data quality rules, and determining the data to be tested that does not meet the data quality rules as quality non-compliant data.

[0069] Optionally, the data quality rule includes a plurality of parameter qualified ranges. Performing quality detection on the data to be tested corresponding to the target metadata according to the data quality rule can be implemented through steps 2041 to 2043.

[0070] Step 2041 , compare the plurality of test data with the qualified parameter ranges corresponding to the test data.

[0071] The parameter qualified range corresponding to the data to be tested is one of multiple parameter qualified ranges.

[0072] Specifically, one of the multiple data to be tested can be data at a moment, or multiple data within a period of time, and this application does not limit this. The multiple data to be tested can be data from different data sources, and the data from different data sources correspond to different parameter qualified ranges. Therefore, the multiple data to be tested correspond to different parameter qualified ranges. For example, the parameter qualified range corresponding to the test data A is a string length less than 16 bits, and the parameter qualified range corresponding to the test data B is capital letters. The data quality rule includes multiple parameter qualified ranges, and the multiple data to be tested are compared with the parameter qualified range corresponding to each test data.

[0073] Step 2042: If the data to be tested is outside the qualified range of the parameters corresponding to the data to be tested, it is determined that the data to be tested does not meet the data quality rules.

[0074] Exemplarily, the parameter qualified range corresponding to the test data A is that the string length is less than 16 bits, and the parameter qualified range corresponding to the test data B is uppercase letters. If the string length of the test data A is greater than 16 bits, it is determined that the test data A is outside the parameter qualified range corresponding to the test data, and therefore it is determined that the test data A does not meet the data quality rules. If the letters of the test data B are lowercase, it is determined that the test data B is outside the parameter qualified range corresponding to the test data, and therefore it is determined that the test data B does not meet the data quality rules.

[0075] Step 2043: if the data to be tested is within the qualified parameter range corresponding to the data to be tested, it is determined that the data to be tested meets the data quality rules, and information of the data to be tested that meets the data quality rules and information of the data that does not meet the quality rules are stored in a preset storage unit.

[0076] Specifically, the preset storage unit is a unit capable of storing data information. For example, the preset storage unit may be a StarRocks database pre-set in a storage module in an electronic device that executes this embodiment. The information of the data includes information such as the data itself, the amount of data, and the pass rate of the data. If the data to be tested is within the qualified range of the parameters corresponding to the data to be tested, it is determined that the data to be tested complies with the data quality rules, and the information of the data to be tested that complies with the data quality rules and the information of the data that does not meet the quality standards are stored in the preset storage unit.

[0077] Step 205: Generate a data quality report based on the information of the data that does not meet the quality requirements and the information of the data to be tested that meets the data quality rules.

[0078] Specifically, after obtaining the data that does not meet the quality standards, a data quality report is generated based on the data information of the data that does not meet the quality standards and the data to be tested that meets the data quality rules, so as to display the difference information between the data that meets the quality standards and the data that does not meet the quality standards in the data to be tested, such as displaying the data delay, output, illegal partition, null value rate, processing time, heterogeneous synchronization difference, and abnormal value information. For example, the data quality report can display the ratio between the data that meets the quality standards and the data that does not meet the quality standards, so that the staff can view it easily.

[0079] Optionally, generating a data quality report according to the information of the data not meeting the quality standards and the information of the data to be tested that meets the data quality rules may be achieved through steps 2051 to 2053 .

[0080] Step 2051, obtain the target historical duration.

[0081] Specifically, the historical duration is the duration from any historical moment when the data quality test was performed to the current moment. The target historical duration can be the target historical duration set by the user collected by the electronic device executing this embodiment, or it can be the target historical duration input by the user through the interactive interface and sent to the electronic device executing this embodiment. This solution does not limit this.

[0082] Step 2052: In the target storage unit, information about historical quality non-compliant data within the target historical time period and information about historical data to be tested that complies with the data quality rules are determined.

[0083] Specifically, after obtaining the target historical duration, information on historical quality non-compliant data stored in the target storage unit within the target historical duration and information on historical data to be tested that complies with data quality rules stored in the target storage unit within the target historical duration are determined.

[0084] Step 2053, generating a data quality report based on the information of historical quality non-compliant data, the information of historical data to be tested that complies with data quality rules, the information of quality non-compliant data, and the information of data to be tested that complies with data quality rules.

[0085] Specifically, after determining the information of historical quality non-compliant data within the target historical time period and the information of historical data to be tested that complies with data quality rules, the difference between the data with qualified quality and the data with unqualified quality within the target historical time period is determined based on the information of historical quality non-compliant data, the information of historical data to be tested that complies with data quality rules, the information of quality non-compliant data, and the information of data to be tested that complies with data quality rules, and a data quality report is generated.

[0086] Exemplarily, the number of historical quality non-compliant data is added to the number of quality non-compliant data, and then the number of historical test data that complies with data quality rules is added to the number of test data that complies with data quality rules. A data quality report is generated based on the ratio of the two to display the difference between qualified quality data and unqualified quality data within the target historical time period.

[0087] Step 206: Send the data quality report to the user equipment corresponding to the preset storage unit.

[0088] Specifically, the user device corresponding to the preset storage unit may be a user device of a staff member. After the data quality report is generated, the data quality report is sent to the user device corresponding to the preset storage unit for easy viewing by the staff member.

[0089] For example, Figure 2c This is an exemplary structural diagram of a data quality platform of the data governance method provided by this application. The data quality platform installed in the electronic device executing this embodiment is used to implement the steps of data quality management in the data governance process. Figure 2c As shown, after determining the data quality rules according to the stored metadata, the data to be tested is obtained by scheduling the DolphinScheduler scheduling engine, or the data to be tested in the data source input by the user is received, and the data to be tested is sent to the message middleware Pulsar. Then, the distributed computing engine Flink is called to implement the step of using data quality rules to detect the data in Pulsar, where Flink obtains data quality rules through asynchronous input / output caching, and stores the data information of the data to be tested that meets the data quality rules and the data information of the data that does not meet the quality standards in the StarRocks database, and generates a data quality report, which is sent to the display module for display so that the staff can view it.

[0090] The solution of the present application performs quality inspection on the data to be tested corresponding to the target metadata according to the data quality rules, obtains information on data that does not meet the quality requirements and information on data to be tested that meets the data quality rules, and provides a data basis for the subsequent generation of data quality reports. After the data to be tested corresponding to the target metadata is quality inspected according to the data quality rules, a data quality report is generated, which is convenient for staff to view in order to understand the data quality of the data source, improves the data management efficiency of the staff, and thus improves the accuracy of the data. In addition, a data quality report on the data to be tested within the target historical time period can also be generated based on the information on historical data that does not meet the quality requirements and the information on historical data to be tested that meets the data quality rules stored in the preset storage unit, which is more convenient for staff to view, and further improves the data management efficiency of staff.

[0091] Figure 3 Schematic diagram of a data management device provided by the present application, which is suitable for executing the data management method provided by the present application. Figure 3 As shown, the device may specifically include:

[0092] A collection module 301 is used to collect target metadata from multiple data sources;

[0093] The metamodel determination module 302 is used to determine and store the metamodel of the target metadata according to the target metadata of each data source and the pre-built metamodel template; wherein the metamodel template includes a plurality of attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata;

[0094] A rule determination module 303, configured to determine a data quality rule corresponding to the target metadata according to a meta-model of each target metadata;

[0095] The quality detection module 304 is used to perform quality detection on the data to be tested corresponding to the target metadata according to the data quality rules, and determine the data to be tested that does not meet the data quality rules as quality non-compliant data.

[0096] In one embodiment, the metamodel determination module 302 is specifically used to: determine multiple attributes of the target metadata based on the target metadata of each of the data sources; replace multiple attribute templates in the metamodel template with multiple target metadata attributes of the target metadata to obtain the metamodel of the target metadata, and store the metamodel of the target metadata.

[0097] In one embodiment, the device also includes: an information establishment module, which is used to establish graph data information by taking each attribute value of the target metadata in the metamodel of the target metadata as a data point and the relationship between the attribute values ​​of the target metadata in the metamodel as an edge after the metamodel determination module 302 determines and stores the metamodel of the target metadata; the rule determination module 303 is specifically used to determine the data quality rules corresponding to the target metadata according to the graph data information corresponding to each of the target metadata.

[0098] In one embodiment, the rule determination module 303 is specifically used to: determine the data standard information of the target metadata based on the metamodel of each target metadata; wherein the data standard information is used to characterize the standard information of the governed data in the data source corresponding to the target metadata; and determine the data quality rules based on the data standard information.

[0099] In one embodiment, the data quality rules include multiple parameter qualified ranges; the quality detection module 304, in terms of performing quality detection on the data to be tested corresponding to the target metadata according to the data quality rules, is specifically used to: compare the multiple data to be tested with the parameter qualified ranges corresponding to the data to be tested; wherein the parameter qualified range corresponding to the data to be tested is one of the multiple parameter qualified ranges; if the data to be tested is outside the parameter qualified range corresponding to the data to be tested, it is determined that the data to be tested does not comply with the data quality rules; if the data to be tested is within the parameter qualified range corresponding to the data to be tested, it is determined that the data to be tested complies with the data quality rules, and the information of the data to be tested that complies with the data quality rules and the information of the quality non-compliant data are stored in a preset storage unit.

[0100] In one embodiment, the device also includes: a quality report generation module, which is used to generate a data quality report based on the information of the quality-noncompliant data and the information of the data to be tested that complies with the data quality rules; and a sending module, which is used to send the data quality report to a user device corresponding to the preset storage unit.

[0101] In one embodiment, the quality report generation module is specifically used to: obtain a target historical duration; determine, in the target storage unit, information on historical quality non-compliant data within the target historical duration and information on historical data to be tested that complies with the data quality rules; generate the data quality report based on the information on historical quality non-compliant data, information on historical data to be tested that complies with the data quality rules, information on quality non-compliant data, and information on data to be tested that complies with the data quality rules.

[0102] The device of the present application collects target metadata of multiple data sources; determines and stores the metamodel of the target metadata according to the target metadata of each data source and a pre-built metamodel template; wherein the metamodel template includes multiple attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata; determines the data quality rules corresponding to the target metadata according to the metamodel of each target metadata; performs quality inspection on the data to be tested corresponding to the target metadata according to the data quality rules, and determines the data to be tested that does not meet the data quality rules as quality non-compliant data. That is, the scheme of the present application, on the one hand, determines and stores the metamodel of the target metadata through the pre-built metamodel template and the target metadata of each data source in the multiple data sources, thereby completing the metadata management of the multiple data sources, avoiding the situation that only the metadata management of the data of one data source can be performed when a single data model is used, thereby improving the management efficiency of the data, and, when using the data, because the metadata of multiple data sources are stored, the data of multiple data sources can be selected, thereby improving the use efficiency of the data. On the other hand, the data quality rules corresponding to the target metadata are determined according to the metamodel of the target metadata, and the quality of the data to be tested corresponding to the target metadata is tested according to the data quality rules, so that the data quality test is based on the metamodel corresponding to the data, avoiding the situation where the data quality rules are not applicable to the data to be tested of a certain data source, thereby improving the accuracy of data quality testing.

[0103] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data governance method provided in any of the above embodiments is implemented.

[0104] The present application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data governance method provided in any of the above embodiments.

[0105] Reference below Figure 4 , which shows a structural schematic diagram of an electronic device 400 suitable for implementing the present application. Figure 4 The electronic device shown is only an example and should not bring any limitation to the function and scope of use of the present application.

[0106] like Figure 4As shown, the electronic device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0107] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0108] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-mentioned functions defined in the system of the present application are executed.

[0109] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The modules and / or units described in this application may be implemented in software or hardware. The modules and / or units described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, a metamodel determination module, a rule determination module, and a quality detection module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0112] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device performs the following operations:

[0113] Collect target metadata from multiple data sources; determine and store a metamodel of the target metadata based on the target metadata of each data source and a pre-built metamodel template; wherein the metamodel template includes multiple attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata; determine data quality rules corresponding to the target metadata based on the metamodel of each target metadata; perform quality inspection on the data to be tested corresponding to the target metadata based on the data quality rules, and determine the data to be tested that does not meet the data quality rules as quality non-compliant data.

[0114] According to the technical solution of the present invention, target metadata of multiple data sources are collected; according to the target metadata of each data source and the pre-built metamodel template, the metamodel of the target metadata is determined and stored; wherein the metamodel template includes multiple attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata; according to the metamodel of each target metadata, the data quality rules corresponding to the target metadata are determined; according to the data quality rules, the quality of the data to be tested corresponding to the target metadata is tested, and the data to be tested that does not meet the data quality rules is determined as quality non-compliant data. That is, the solution of the present application, on the one hand, determines and stores the metamodel of the target metadata through the pre-built metamodel template and the target metadata of each data source in the multiple data sources, thereby completing the metadata management of multiple data sources, avoiding the situation that only metadata management can be performed on the data of one data source when a single data model is used, thereby improving the management efficiency of the data, and, when using the data, because the metadata of multiple data sources are stored, the data of multiple data sources can be selected, thereby improving the use efficiency of the data. On the other hand, the data quality rules corresponding to the target metadata are determined according to the metamodel of the target metadata, and the quality of the data to be tested corresponding to the target metadata is tested according to the data quality rules, so that the data quality test is based on the metamodel corresponding to the data, avoiding the situation where the data quality rules are not applicable to the data to be tested of a certain data source, thereby improving the accuracy of data quality testing.

[0115] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements a data governance method as provided in any embodiment of the present application.

[0116] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present application, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data governance method, characterized in that: The method comprises: Collect target metadata from multiple data sources; According to the target metadata of each data source and the pre-built metamodel template, determine and store the metamodel of the target metadata; wherein the metamodel template includes a plurality of attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata; Determining data quality rules corresponding to the target metadata according to the metamodel of each target metadata; The quality of the data to be tested corresponding to the target metadata is checked according to the data quality rules, and the data to be tested that does not meet the data quality rules is determined as quality non-compliant data.

2. The method according to claim 1, characterized in that According to the target metadata of each data source and the pre-built meta-model template, determining and storing the meta-model of the target metadata includes: Determine a plurality of attributes of the target metadata according to the target metadata of each of the data sources; The multiple attribute templates in the meta-model template are replaced with multiple target metadata attributes of the target metadata to obtain the meta-model of the target metadata, and the meta-model of the target metadata is stored.

3. The method according to claim 1, characterized in that: After determining and storing the meta-model of the target metadata, the method further includes: Taking each attribute value of the target metadata in the meta-model of the target metadata as a data point, and taking the relationship between the attribute values ​​of the target metadata in the meta-model as an edge, to establish graph data information; The step of determining the data quality rules corresponding to the target metadata according to the meta-model of each target metadata includes: The data quality rules corresponding to the target metadata are determined according to the graph data information corresponding to each of the target metadata.

4. The method according to claim 1, characterized in that: The step of determining the data quality rules corresponding to the target metadata according to the meta-model of each target metadata includes: Determine data standard information of the target metadata according to the metamodel of each target metadata; wherein the data standard information is used to characterize standard information of governed data in the data source corresponding to the target metadata; The data quality rule is determined according to the data standard information.

5. The method according to claim 1, characterized in that The data quality rules include a plurality of parameter qualified ranges; The performing quality detection on the data to be tested corresponding to the target metadata according to the data quality rule includes: Compare the plurality of the test data with the parameter qualified ranges corresponding to the test data respectively; wherein the parameter qualified range corresponding to the test data is one of the plurality of parameter qualified ranges; If the data to be tested is outside the qualified range of the parameter corresponding to the data to be tested, it is determined that the data to be tested does not meet the data quality rule; If the data to be tested is within the qualified parameter range corresponding to the data to be tested, it is determined that the data to be tested meets the data quality rule, and information of the data to be tested that meets the data quality rule and information of the data not meeting the quality requirements are stored in a preset storage unit.

6. The method according to claim 5, characterized in that The method further comprises: Generate a data quality report based on the information of the data that does not meet the quality requirements and the information of the data to be tested that meets the data quality rules; The data quality report is sent to a user device corresponding to the preset storage unit.

7. The method according to claim 6, characterized in that The generating of a data quality report according to the information of the quality non-compliant data and the information of the tested data that complies with the data quality rules includes: Get the target history duration; In the target storage unit, determining information of historical quality non-compliant data within the target historical duration and information of historical data to be tested that meets the data quality rules; The data quality report is generated according to the information of the historical quality non-compliant data, the information of the historical data to be tested that complies with the data quality rules, the information of the quality non-compliant data, and the information of the data to be tested that complies with the data quality rules.

8. A data management device, characterized in that: The device comprises: A collection module, used to collect target metadata from multiple data sources; A metamodel determination module, used to determine and store a metamodel of the target metadata according to the target metadata of each data source and a pre-built metamodel template; wherein the metamodel template includes a plurality of attribute templates of metadata, and the metamodel of the target metadata is used to characterize the attributes of the target metadata; A rule determination module, used to determine the data quality rule corresponding to the target metadata according to the meta-model of each target metadata; The quality detection module is used to perform quality detection on the data to be tested corresponding to the target metadata according to the data quality rules, and determine the data to be tested that does not meet the data quality rules as quality non-compliant data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the data management method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data management method as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the data governance method as described in any one of claims 1 to 7.

Images