A data quality assessment method, apparatus, electronic device, and medium

Abstract

The application relates to the technical field of data processing, in particular to a data quality evaluation method and device, electronic equipment and medium, which are used to solve the problem that the data quality in a database is affected by unreasonable data blood relationship in the prior art. The method comprises the following steps: acquiring a plurality of data tables included in a database; determining a data dependency relationship corresponding to each data table according to the table name of each data table, and obtaining a plurality of data dependency relationships; the data dependency relationship corresponding to any data table comprises an upstream data layer of any data table and a current data layer of any data table; the upstream data layer of any data table is used for indicating the source of any data table; determining the data dependency quality category of each data dependency relationship according to a mapping relationship between a pre-stored data dependency quality category and a data dependency relationship; and determining the data dependency quality of the database according to the data dependency quality category of each data dependency relationship and the weight of each data dependency quality category.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data quality assessment method, apparatus, electronic device, and medium. Background Technology

[0002] With the development of computer and network technologies, all industries are gradually becoming information-based, and the world has entered the era of big data. In the process of data development, from data generation, extraction-transformation-loading (ETL) processing, processing, fusion, and flow to final presentation, relationships naturally form between data points—this is called data lineage. Data lineage is an important concept in data governance, referring to finding the connections between related data during data tracing. However, in practice, different developers may have different approaches to calculate the same set of data, such as choosing different data tables and performing different operations, thus creating different data lineages. Therefore, as work progresses, some unreasonable data lineages inevitably emerge, often affecting the quality of data in the database. Summary of the Invention

[0003] This application provides a data quality assessment method, apparatus, device, and medium to address the problem in the prior art where unreasonable data lineage relationships affect data quality in databases.

[0004] In a first aspect, embodiments of this application provide a data quality assessment method, the method comprising:

[0005] The process involves: acquiring multiple data tables from the database; determining the data dependencies corresponding to each data table based on its name; obtaining multiple data dependencies; the data dependency for any given data table includes its upstream data layer and its current data layer; the upstream data layer represents the source of the given data table; determining the data dependency quality category of each data dependency based on a pre-stored mapping between data dependency quality categories and data dependencies; and determining the data dependency quality of the database based on the data dependency quality categories of each data dependency and their preset weights.

[0006] Based on the above scheme, by classifying the various data dependencies included in the system and determining the data dependency quality of the database according to the weight of each data dependency quality category, the data quality of the database can be evaluated. This evaluation of database data quality can provide optimization directions for database data governance and also provide a basis for horizontal comparisons of multiple databases.

[0007] In one possible implementation, before obtaining the multiple data tables included in the database, the method further includes: performing layered processing on the data tables in the database according to the process from generation to application of the multiple data tables, resulting in multiple data layers; the multiple data layers sequentially include: a raw data layer, a normalized data layer, a thematic data layer, a dimensional data layer, a summary data layer, and an application data layer; the raw data layer is used to describe the level where no data processing is performed on the data in the data tables, the normalized data layer is used to describe the level where data in the data tables is normalized without data association, the thematic data layer is used to describe the level where data in the data tables is associated according to business themes, the dimensional data layer is used to describe the level where the data tables include the characteristics of fact records in the fact data tables, the summary data layer is used to describe the level where the data tables include general dimensions and indicators, and the application data layer is used to describe the level where the data tables are applied to actual usage scenarios; for any data table in the database, the table name of the data table is set according to the current data layer and the upstream data layer of the data table.

[0008] Based on the above scheme, according to the process sequence of data from generation to application, the data table is divided into multiple data layers, and the table name is set according to the data layer to which the data table belongs. This can clearly determine the data dependencies, allowing relevant staff to easily evaluate the data quality based on the data dependencies.

[0009] In one possible embodiment, the data dependency quality categories include a first category, a second category, and a third category; the third category has a higher optimization level than the second category, and the second category has a higher optimization level than the first category.

[0010] Based on the above scheme, data dependencies can be classified according to their optimization level, thereby determining the data dependency quality of the database based on the category of each data dependency.

[0011] In one possible embodiment, the first category has a mapping relationship with at least one of the following data dependencies: the upstream data layer of the data dependency is the original data layer, and the current data layer is the canonical data layer; the upstream data layer of the data dependency is the canonical data layer, and the current data layer is a topic data layer or a dimension data layer; the upstream data layer of the data dependency is the topic data layer or a dimension data layer, and the current data layer is a summary data layer or an application data layer; the upstream data layer of the data dependency is the summary data layer, and the current data layer is an application data layer.

[0012] Based on the above scheme, it can be determined whether a data dependency belongs to the first category by judging whether the data dependency has a mapping relationship with the first category.

[0013] In one possible embodiment, the second category has a mapping relationship with the following data dependency: the upstream data layer of the data dependency is a canonical data layer, and the current data layer is a summary data layer or an application data layer.

[0014] Based on the above scheme, it can be determined whether a data dependency belongs to the second category by judging whether the data dependency has a mapping relationship with the second category.

[0015] In one possible embodiment, the third category has a mapping relationship with at least one of the following data dependencies: the upstream data layer of the data dependency is the original data layer, and the current data layer is any layer other than the canonical data layer; the level of the upstream data layer of the data dependency is located after the level of the current data layer.

[0016] Based on the above scheme, it can be determined whether a data dependency belongs to the third category by judging whether the data dependency has a mapping relationship with the third category.

[0017] In one possible embodiment, determining the data dependency quality of the database based on the data dependency quality categories of each data dependency relationship and the preset weights of each data dependency quality category includes: determining a set of data dependency quality categories for the plurality of data dependencies based on the data dependency quality categories of each data dependency relationship; determining the proportion of each data dependency quality category in the set of data dependency quality categories; and determining the data dependency quality of the database based on the proportion of each data dependency quality category and the weight corresponding to each data dependency quality category.

[0018] Based on the above scheme, the data dependency quality of the database can be determined by the weight of the data dependency quality category to which each data dependency belongs and the proportion of the data dependency quality category, thereby evaluating the data quality in the database.

[0019] Secondly, this application provides a data quality assessment device, the device comprising:

[0020] The retrieval unit is used to retrieve multiple data tables included in the database;

[0021] The processing unit is configured to perform the following processes: determining the data dependencies corresponding to each data table based on the table name, thereby obtaining multiple data dependencies; the data dependency corresponding to any data table includes the upstream data layer and the current data layer of any data table; the upstream data layer of any data table represents the source of any data table; determining the data dependency quality category of each data dependency based on the pre-stored mapping relationship between data dependency quality categories and data dependencies; and determining the data dependency quality of the database based on the data dependency quality categories of each data dependency and the preset weights of each data dependency quality category.

[0022] In one possible embodiment, before the acquisition unit acquires the multiple data tables included in the database, the processing unit is further configured to: perform layered processing on the data tables in the database according to the process from generation to application of the multiple data tables, to obtain multiple data layers; the multiple data layers sequentially include: a raw data layer, a normalized data layer, a thematic data layer, a dimensional data layer, a summary data layer, and an application data layer; the raw data layer is used to describe the level where no processing is performed on the data in the data table, the normalized data layer is used to describe the level where normalization processing is performed on the data in the data table without data association, the thematic data layer is used to describe the level where data in the data table is associated according to business themes, the dimensional data layer is used to describe the level where the data table includes the characteristics of fact records in the fact data table, the summary data layer is used to describe the level where the data table includes general dimensions and indicators, and the application data layer is used to describe the level where the data table is applied to actual use scenarios; for any data table in the database, the table name of the data table is set according to the current data layer and the upstream data layer of the data table.

[0023] In one possible embodiment, the data dependency quality categories include a first category, a second category, and a third category; the third category has a higher optimization level than the second category, and the second category has a higher optimization level than the first category.

[0024] In one possible embodiment, the first category has a mapping relationship with at least one of the following data dependencies: the upstream data layer of the data dependency is the original data layer, and the current data layer is the canonical data layer; the upstream data layer of the data dependency is the canonical data layer, and the current data layer is a topic data layer or a dimension data layer; the upstream data layer of the data dependency is the topic data layer or a dimension data layer, and the current data layer is a summary data layer or an application data layer; the upstream data layer of the data dependency is the summary data layer, and the current data layer is an application data layer.

[0025] In one possible embodiment, the second category has a mapping relationship with the following data dependency: the upstream data layer of the data dependency is a canonical data layer, and the current data layer is a summary data layer or an application data layer.

[0026] In one possible embodiment, the third category has a mapping relationship with at least one of the following data dependencies: the upstream data layer of the data dependency is the original data layer, and the current data layer is any layer other than the canonical data layer; the level of the upstream data layer of the data dependency is located after the level of the current data layer.

[0027] In one possible embodiment, when the processing unit determines the data dependency quality of the database based on the data dependency quality categories of each data dependency relationship and the preset weights of each data dependency quality category, it is specifically configured to: determine a set of data dependency quality categories for the plurality of data dependencies based on the data dependency quality categories of each data dependency relationship; determine the proportion of each data dependency quality category in the set of data dependency quality categories; and determine the data dependency quality of the database based on the proportion of each data dependency quality category and the weight corresponding to each data dependency quality category.

[0028] Thirdly, this application provides an electronic device, comprising:

[0029] Memory, used to store program instructions;

[0030] A processor is configured to invoke program instructions stored in the memory and execute the steps of the method described in any one of the first aspects according to the obtained program instructions.

[0031] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in any one of the first aspects.

[0032] Fifthly, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects. Attached Figure Description

[0033] Figure 1 A schematic diagram of the system structure for applying a data quality assessment method according to an embodiment of this application;

[0034] Figure 2 A flowchart of a data quality assessment method provided in this application embodiment;

[0035] Figure 3 A structural diagram of a data quality assessment device provided in an embodiment of this application;

[0036] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0038] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0039] The data collection, dissemination, and use in this application all comply with relevant national laws and regulations.

[0040] Before introducing the data quality assessment method provided in the embodiments of this application, for ease of understanding, the technical background of the embodiments of this application will be described in detail below.

[0041] Currently, in the data development process, from data generation, ETL processing, processing, fusion, and flow to final presentation, data naturally forms data lineage relationships. Data lineage is an important concept in data governance, referring to finding the connections between related data during data tracing. However, in practice, different developers may have different approaches to calculate the same set of data, such as choosing different data tables and performing different operations, thus creating different data lineage relationships. Therefore, as the work progresses, some unreasonable data lineage relationships inevitably emerge, often affecting the quality of data in the database.

[0042] In view of this, embodiments of this application provide a data quality assessment method. This method determines multiple data dependencies existing in the database based on the table names of each data table. These data dependencies may include data lineage. Then, by using the mapping relationship between data dependency quality categories and data dependencies, the data dependency quality category to which each data dependency belongs is determined. Finally, based on the weights corresponding to the data dependency quality categories, the data dependency quality of the database is determined, thereby achieving an assessment of the database's data quality. This method can provide optimization directions for database data governance and also provides a basis for horizontal comparisons of multiple databases.

[0043] See Figure 1 This is a schematic diagram of the system architecture for the data quality assessment method provided in this application embodiment. The system can be a server 100, including: a memory 110, a processor 120, and a communication interface 130. The communication interface 130 can be used to retrieve multiple data tables included in the database. The processor 120 is the control center of the server 100, connecting various parts of the server 100 through various interfaces and lines. It executes various functions of the server 100 and processes data by running or executing software programs or modules stored in the memory 110 and calling data stored in the memory 110. Optionally, the processor 120 may include one or more processing units. The memory 110 can be a high-speed random access memory or a non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. It should be noted that the above... Figure 1 The structure shown is merely an example, and the embodiments of this application are not limited thereto.

[0044] See Figure 2 A flowchart of a data quality assessment method provided for embodiments of this application may include the following steps:

[0045] S201, retrieve multiple data tables included in the database.

[0046] In some embodiments, different databases can be built for different business operations or different projects. That is, database A includes the various data tables for project A, and database B includes the various data tables for project B.

[0047] In other embodiments, the database may include data tables for multiple businesses or multiple projects, that is, database C may include both data tables for project A and data tables for project B.

[0048] In one possible implementation, before retrieving the multiple data tables included in the database, the data tables in the database can be layered according to the process from generation to application, resulting in multiple data layers. Then, for any data table in the database, the table name can be set according to the data layers corresponding to the current and upstream data layers. The multiple data layers, in the order of the data generation to application process, include the following a) to f):

[0049] a) Raw data layer, which describes the layer where the data in the data table is not processed. Data is usually stored in the form of data snapshots at this layer.

[0050] b) Normalized data layer: This layer describes the normalization process performed on the data in the data table without data association. Normalization may include encoding conversion, cleaning, and data anonymization.

[0051] c) Thematic data layer, used to describe the hierarchy in a data table where data is related according to a business theme. For example, data tables belonging to the generalized data layer can be managed according to a business theme, forming a wide table where indicators, dimensions, and attributes related to the business theme are linked together. The current data layer of this wide table is the thematic data layer.

[0052] d) Dimensional data layer, which describes the characteristics of the data table, including the fact records in the fact data table. It can be seen as a window for users to analyze the data.

[0053] e) Summary data layer, which describes the hierarchy of data tables including general dimensions and metrics. This layer of data tables can be used to support business requirement development.

[0054] f) Application data layer, used to describe the hierarchy of data surfaces to actual usage scenarios, such as scenarios for reports, dashboards, and applications.

[0055] In one example, when setting the table name based on the current and upstream data layers, you can first determine the pre-defined identifiers for each data layer. Then, you can set the table name to the identifier corresponding to the upstream data layer, followed by the identifier corresponding to the current data layer, and finally the identifier corresponding to the relevant data table information. This relevant data table information can include data content, data type, and other information.

[0056] For example, suppose a data table is currently in the canonical data layer and its upstream data layer is the raw data layer. Furthermore, the identifier for this data table is 'test', the identifier for the canonical data layer is 'gf', and the identifier for the raw data layer is 'ys', then the table name can be set to 'gf_ys_test'.

[0057] It should be noted that the above method of setting the table name is exemplary, and other methods can also be used to set the table name so that it can reflect the current data layer and the upstream data layer of the data table. This application does not limit this.

[0058] S202, determine the data dependencies corresponding to each data table based on the table name, and obtain multiple data dependencies.

[0059] The data dependencies for any given data table include its upstream data layer and its current data layer. The upstream data layer represents the source of the data table. For example, if the upstream data layer of data table A is the original data layer, then data table A is obtained from a data table in a certain original data layer.

[0060] S203, determine the data dependency quality category of each data dependency relationship based on the pre-stored mapping relationship between data dependency quality categories and data dependency relationships.

[0061] In one possible implementation, data dependency quality categories include three categories: Category 1, Category 2, and Category 3. Category 1 represents healthy data dependencies that can be retained long-term. Category 2 represents sub-healthy data dependencies that cannot be retained long-term but can be temporarily retained. Category 3 represents unhealthy data dependencies that require optimization. In other words, the optimization level of Category 3 is higher than that of Category 2, and the optimization level of Category 2 is higher than that of Category 1.

[0062] In some embodiments, the mapping relationship between data dependency quality categories and data dependency relationships can be as shown in Table 1.

[0063] Table 1. Mapping Relationship between Data Dependency Quality Categories and Data Dependency Relationships

[0064]

[0065] In this data dependency relationship, since application data layer tables that do not require aggregation can directly use data tables from the thematic data layer, the relationship can be classified as belonging to the first category when the current data layer is the application data layer and the upstream data layer is the thematic data layer. Similarly, data tables in the summary data layer can temporarily use data tables from the normalized data layer, but the processing logic needs to be integrated into the thematic data layer after the data stabilizes. Therefore, when the current data layer is the normalized data layer and the upstream data layer is the summary data layer, this data dependency relationship can be classified as belonging to the second category. Finally, since application data layer tables that do not require aggregation can temporarily use data tables from the normalized data layer, but the processing logic needs to be integrated into the thematic data layer after the data stabilizes, this data dependency relationship can also be classified as belonging to the second category.

[0066] Since only tables in the canonical data layer can use tables in the original data layer, when the upstream data layer in a data dependency relationship is the original data layer and the current data layer is any other than the canonical data layer, this data dependency relationship can be identified as belonging to the third category. Because the layering of data tables in the database follows the sequence from data generation to application, all reverse data dependencies—that is, when the upstream data layer is located after the current data layer—can be identified as belonging to the third category, which are unhealthy data dependencies and urgently need optimization.

[0067] In one example, after obtaining multiple data dependencies in S202, the upstream data layer and current data layer in each data dependency can be determined. Then, based on the upstream and current data layers, the data dependency quality category of each data dependency is determined. For example, the obtained data dependencies include: Relation 1, Relation 2, and Relation 3. In Relation 1, the upstream data layer is the original data layer, and the current data layer is the canonical data layer. In Relation 2, the upstream data layer is the original data layer, and the current data layer is the topic data layer. In Relation 3, the upstream data layer is the canonical data layer, and the current data layer is the application data layer. Based on the mapping relationship between data dependency quality categories and data dependencies shown in Table 1, Relation 1 belongs to the first category, Relation 2 belongs to the third category, and Relation 3 belongs to the second category.

[0068] In some embodiments, different data dependency quality categories can be identified by color. For example, green can indicate the first category, yellow the second, and red the third. Using a prominent red color to identify data dependencies that urgently need optimization makes the direction of optimization immediately clear, facilitating subsequent optimization and management of the data in the database by relevant personnel.

[0069] S204. Determine the data dependency quality of the database based on the data dependency quality categories of each data dependency relationship and the preset weights of each data dependency quality category.

[0070] In one possible implementation, a set of data dependency quality categories for multiple data dependencies can be determined based on the data dependency quality categories of each data dependency. The proportion of each data dependency quality category in the set of data dependency quality categories is then determined. Based on the proportion of each data dependency quality category and the weight corresponding to each data dependency quality category, the data dependency quality of the database is determined, where the data dependency quality of the database can satisfy formula (1).

[0071] W = a% * w1 + b% * w2 + c% * w3 (Formula 1)

[0072] In the formula, W represents the data dependency quality of the database. a represents the proportion of the first category, b represents the proportion of the second category, and c represents the proportion of the third category. w1 represents the weight of the first category, w2 represents the weight of the second category, and w3 represents the weight of the third category.

[0073] For example, suppose the obtained data dependencies include: Relation 1 to Relation 5. In S203, the data dependency quality category of each data dependency can be determined: Relation 1 belongs to Category 1, Relation 2 belongs to Category 1, Relation 3 belongs to Category 3, Relation 4 belongs to Category 2, and Relation 5 belongs to Category 1. Then the set of data dependency quality categories for multiple data dependencies can include 3 Category 1 categories, 1 Category 2 category, and 1 Category 3 category. The proportion of Category 1 is 60%, Category 2 is 20%, and Category 3 is 20%. Assuming the weight of Category 1 is 1, the weight of Category 2 is 0.8, and the weight of Category 3 is 0.6, then the data dependency quality of the database is: 1*60% + 0.8*20% + 0.6*20% = 0.88.

[0074] In one example, the initial score of the database can be set to a preset score. Then, based on the proportion of each data dependency quality category and the weight of each data dependency quality category, it can be determined whether the score corresponding to the data dependency quality of the database can satisfy formula (2).

[0075] W = S + a% * w1 + b% * w2 + c% * w3 (Formula 2)

[0076] In the formula, S is the initial score of the database.

[0077] For example, a preset score of 100 points can be set, with the weight of the first category being 0 points, the weight of the second category being -5 points, and the weight of the third category being -10 points. Assume the obtained data dependencies include: Relationship 1 to Relationship 5. And in S203, the data dependency quality category of each data dependency is determined: Relationship 1 belongs to the first category, Relationship 2 belongs to the first category, Relationship 3 belongs to the third category, Relationship 4 belongs to the second category, and Relationship 5 belongs to the first category. Then the set of data dependency quality categories for multiple data dependencies can include 3 first categories, 1 second category, and 1 third category. The proportion of the first category is 60%, the second category is 20%, and the third category is 20%. Therefore, the database's data dependency quality is 100 + 0 * 60% + (-5) * 20% + (-10) * 20% = 97.

[0078] It should be noted that the initial scores and the weights of each data-dependent quality category can be preset based on experience or actual conditions, and this application does not impose any restrictions on this.

[0079] Based on the above scheme, by classifying the various data dependencies included in the system and determining the data dependency quality of the database according to the weight of each data dependency quality category, the data quality of the database can be evaluated. This evaluation of database data quality can provide optimization directions for database data governance and also provide a basis for horizontal comparisons of multiple databases.

[0080] Based on the same inventive concept, embodiments of this application provide a data quality assessment device, please refer to... Figure 3 The device includes:

[0081] The acquisition unit 301 is used to acquire multiple data tables included in the database;

[0082] Processing unit 302 is configured to perform the following processes: determining the data dependency relationships corresponding to each data table based on the table name of each data table, thereby obtaining multiple data dependency relationships; the data dependency relationship corresponding to any data table includes the upstream data layer of any data table and the current data layer of any data table; the upstream data layer of any data table is used to represent the source of any data table; determining the data dependency quality category of each data dependency relationship based on the pre-stored mapping relationship between data dependency quality categories and data dependency relationships; and determining the data dependency quality of the database based on the data dependency quality categories of each data dependency relationship and the preset weights of each data dependency quality category.

[0083] In one possible embodiment, before the acquisition unit 301 acquires the multiple data tables included in the database, the processing unit 302 is further configured to: perform layered processing on the data tables in the database according to the process from generation to application of the multiple data tables, to obtain multiple data layers; the multiple data layers sequentially include: a raw data layer, a normalized data layer, a thematic data layer, a dimensional data layer, a summary data layer, and an application data layer; the raw data layer is used to describe the level where no processing is performed on the data in the data table, the normalized data layer is used to describe the level where normalization processing is performed on the data in the data table without data association, the thematic data layer is used to describe the level where data in the data table is associated according to business themes, the dimensional data layer is used to describe the level where the data table includes the characteristics of fact records in the fact data table, the summary data layer is used to describe the level where the data table includes general dimensions and indicators, and the application data layer is used to describe the level where the data table is applied to actual use scenarios; for any data table in the database, the table name of the data table is set according to the current data layer and the upstream data layer of the data table.

[0084] In one possible embodiment, the data dependency quality categories include a first category, a second category, and a third category; the third category has a higher optimization level than the second category, and the second category has a higher optimization level than the first category.

[0085] In one possible embodiment, the first category has a mapping relationship with at least one of the following data dependencies: the upstream data layer of the data dependency is the original data layer, and the current data layer is the canonical data layer; the upstream data layer of the data dependency is the canonical data layer, and the current data layer is a topic data layer or a dimension data layer; the upstream data layer of the data dependency is the topic data layer or a dimension data layer, and the current data layer is a summary data layer or an application data layer; the upstream data layer of the data dependency is the summary data layer, and the current data layer is an application data layer.

[0086] In one possible embodiment, the second category has a mapping relationship with the following data dependency: the upstream data layer of the data dependency is a canonical data layer, and the current data layer is a summary data layer or an application data layer.

[0087] In one possible embodiment, the third category has a mapping relationship with at least one of the following data dependencies: the upstream data layer of the data dependency is the original data layer, and the current data layer is any layer other than the canonical data layer; the level of the upstream data layer of the data dependency is located after the level of the current data layer.

[0088] In one possible embodiment, when the processing unit 302 determines the data dependency quality of the database based on the data dependency quality categories of each data dependency relationship and the preset weights of each data dependency quality category, it is specifically configured to: determine a set of data dependency quality categories for the plurality of data dependencies based on the data dependency quality categories of each data dependency relationship; determine the proportion of each data dependency quality category in the set of data dependency quality categories; and determine the data dependency quality of the database based on the proportion of each data dependency quality category and the weight corresponding to each data dependency quality category.

[0089] Based on the same concept as the above method, see [link to relevant documentation]. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes at least one processor 402 and a memory 401 connected or coupled to the at least one processor 402. In addition, the electronic device may also include a communication interface 403. The electronic device can interact with other devices through the communication interface 403.

[0090] For example, the communication interface 403 can be a transceiver, circuit, bus, module, pin, or other type of communication interface. When the electronic device is a chip-based device or circuit, the communication interface 403 in the electronic device can also be an input / output circuit, which can input information (or receive information) and output information (or send information). The processor can be an integrated processor, a microprocessor, an integrated circuit, or a logic circuit, and the processor can determine the output information based on the input information.

[0091] The coupling in this application embodiment is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. The processor 402 may operate in conjunction with the memory 401 and the communication interface 403. This application does not limit the specific connection medium between the processor 402, the memory 401, and the communication interface 403.

[0092] Optional, see Figure 4 The processor 402, the memory 401, and the communication interface 403 are interconnected via a bus. This bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0093] In this embodiment, memory 401, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 401 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 401 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Memory 401 in this embodiment may also be a circuit or any other device capable of implementing storage functions for storing instructions, computer programs, and / or data.

[0094] In this embodiment, the processor 402 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the data quality assessment method disclosed in this embodiment can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0095] By designing and programming the processor 402, the code corresponding to the data quality assessment method described in the foregoing embodiments can be embedded into the chip, so that the chip can execute the steps of the aforementioned data quality assessment method when running. How to design and program the processor 402 is a well-known technique to those skilled in the art, and will not be described in detail here.

[0096] In one or more embodiments, memory 401 stores instructions that can be executed by at least one processor 402, which can implement the steps of any of the above methods by calling the instructions or computer programs stored in memory 401.

[0097] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium. The computer program product includes computer program code, which, when executed on a computer, causes the computer to perform any of the data quality assessment methods discussed above. Since the principle by which the above-described computer-readable storage medium solves the problem is similar to that of the data quality assessment methods, the implementation of the above-described computer-readable storage medium can be referred to the implementation of the method; repeated details will not be elaborated further.

[0098] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the data quality assessment methods discussed above. Since the principle by which the above-described computer program product solves the problem is similar to that of the data quality assessment method, the implementation of the above-described computer program product can be referred to the implementation of the method, and repeated details will not be described again.

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of user-operated steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of data quality assessment, characterized by, The method comprises: acquiring a plurality of data tables included in a database; determining a data dependency relationship corresponding to each of the data tables according to a table name of the data table, to obtain a plurality of data dependency relationships; the data dependency relationship corresponding to any data table comprises an upstream data layer of the any data table and a current data layer of the any data table; the upstream data layer of the any data table is used to indicate a source of the any data table; determining a data dependency quality category of each of the data dependency relationships according to a pre-stored mapping relationship between a data dependency quality category and a data dependency relationship; the data dependency quality category comprises a first category, a second category and a third category; the first category is a healthy data dependency relationship that can be retained for a long time, the second category is a sub-healthy data dependency relationship that can be temporarily retained, and the third category is an unhealthy data dependency relationship that needs to be optimized; the optimization level of the third category is higher than that of the second category, and the optimization level of the second category is higher than that of the first category; determining a data dependency quality of the database according to the data dependency quality category of each of the data dependency relationships and a preset weight of each of the data dependency quality categories.

2. The method of claim 1, wherein, Before the acquiring a plurality of data tables included in a database, the method further comprises: performing hierarchical processing on the data tables in the database according to a process from generation to application of the plurality of data tables, to obtain a plurality of data layers; the plurality of data layers comprise, in sequence, an original data layer, a standardized data layer, a subject data layer, a dimension data layer, a summary data layer and an application data layer; the original data layer is used to describe a level in which data in the data tables is not processed, the standardized data layer is used to describe a level in which data in the data tables is standardized without data association, the subject data layer is used to describe a level in which data in the data tables is associated according to a business subject, the dimension data layer is used to describe a level in which a fact record in a fact data table is included, the summary data layer is used to describe a level in which a general dimension and an index are included, and the application data layer is used to describe a level in which the data tables are used for actual use scenarios; for any data table of the database, setting a table name of the data table according to a current data layer and an upstream data layer of the data table.

3. The method of claim 2, wherein, The first category has a mapping relationship with at least one of the following data dependency relationships: the upstream data layer of the data dependency relationship is the original data layer, and the current data layer is the standardized data layer; the upstream data layer of the data dependency relationship is the standardized data layer, and the current data layer is the subject data layer or the dimension data layer; the upstream data layer of the data dependency relationship is the subject data layer or the dimension data layer, and the current data layer is the summary data layer or the application data layer; the upstream data layer of the data dependency relationship is the summary data layer, and the current data layer is the application data layer.

4. The method of claim 2, wherein, The second category has a mapping relationship with the following data dependency relationship: the upstream data layer of the data dependency relationship is the standardized data layer, and the current data layer is the summary data layer or the application data layer.

5. The method of claim 2, wherein, The third category has a mapping relationship with at least one of the following data dependency relationships: An upstream data layer of the data dependency relationship is an original data layer, and a current data layer is any layer except a specification data layer; The upstream data layer of the data dependency relationship is located after a level of the current data layer.

6. The method according to any one of claims 1 to 5, wherein, The data dependency quality of the database is determined according to the data dependency quality category of each data dependency relationship and a preset weight of each data dependency quality category, including: A data dependency quality category set of the plurality of data dependency relationships is determined according to the data dependency quality category of each data dependency relationship; A proportion of each data dependency quality category in the data dependency quality category set is determined; The data dependency quality of the database is determined according to the proportion of each data dependency quality category and the weight corresponding to each data dependency quality category.

7. A data quality assessment apparatus, characterized by, The apparatus includes: An acquisition unit configured to acquire a plurality of data tables included in a database; A processing unit configured to perform the following processing: A data dependency relationship corresponding to each data table is determined according to a table name of the data table, and a plurality of data dependency relationships are obtained; a data dependency relationship of any data table includes an upstream data layer of the any data table and a current data layer of the any data table; the upstream data layer of the any data table is used to indicate a source of the any data table; A data dependency quality category of each data dependency relationship is determined according to a preset mapping relationship between a data dependency quality category and a data dependency relationship; the data dependency quality category includes a first category, a second category, and a third category; the first category is a healthy and long-term preserved data dependency relationship, the second category is a sub-healthy and temporarily preserved data dependency relationship, and the third category is an unhealthy and optimized data dependency relationship; an optimization level of the third category is higher than that of the second category, and an optimization level of the second category is higher than that of the first category; The data dependency quality of the database is determined according to the data dependency quality category of each data dependency relationship and a preset weight of each data dependency quality category.

8. The apparatus of claim 7, wherein, Before the acquisition unit acquires the plurality of data tables included in the database, the processing unit is further configured to: According to a flow from generation to application of the plurality of data tables, the data tables in the database are processed in layers to obtain a plurality of data layers; the plurality of data layers include in sequence: an original data layer, a specification data layer, a subject data layer, a dimension data layer, a summary data layer, and an application data layer; the original data layer is used to describe a layer level without processing data in the data tables; the specification data layer is used to describe a layer level with normalized processing of data in the data tables without data association; the subject data layer is used to describe a layer level with association of data in the data tables according to business subjects; the dimension data layer is used to describe a layer level including characteristics of fact records in fact data tables; the summary data layer is used to describe a layer level including general dimensions and indexes; and the application data layer is used to describe a layer level facing actual use scenarios; For any data table of the database, a table name of the data table is set according to a current data layer and an upstream data layer of the data table.

9. The apparatus of claim 8, wherein, The first category has a mapping relationship with at least one of the following data dependency relationships: The upstream data layer of the data dependency relationship is the original data layer, and the current data layer is the specification data layer; The upstream data layer of the data dependency relationship is the specification data layer, and the current data layer is the subject data layer or the dimension data layer; The upstream data layer of the data dependency relationship is the subject data layer or the dimension data layer, and the current data layer is the summary data layer or the application data layer; The upstream data layer of the data dependency relationship is the summary data layer, and the current data layer is the application data layer.

10. The apparatus of claim 8, wherein, The second category has a mapping relationship with the following data dependency relationship: The upstream data layer of the data dependency relationship is the specification data layer, and the current data layer is the summary data layer or the application data layer.

11. The apparatus of claim 8, wherein, The third category has a mapping relationship with at least one of the following data dependency relationships: The upstream data layer of the data dependency relationship is the original data layer, and the current data layer is any layer except the specification data layer; The upstream data layer is located after the current data layer in the layer level.

12. The apparatus of any one of claims 7-11, wherein, When determining the data dependency quality of the database according to the data dependency quality categories of the respective data dependency relationships and the weights of the respective data dependency quality categories, the processing unit is specifically configured to: determine a data dependency quality category set of the plurality of data dependency relationships according to the data dependency quality categories of the respective data dependency relationships; determine a proportion of each data dependency quality category in the data dependency quality category set; determine the data dependency quality of the database according to the proportion of each data dependency quality category and the weight corresponding to each data dependency quality category.

13. An electronic device, comprising: Comprise: a memory for storing program instructions; a processor for calling the program instructions stored in the memory and executing the steps included in the method of any one of claims 1-6 according to the obtained program instructions.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-6.

15. A computer program product, characterised in that, The computer program product comprises computer program code which, when run on a computer, causes the computer to perform the method of any one of claims 1-6.

Images