Method and apparatus for detecting quality of data warehouse

Abstract

The one or more embodiments of the specification provide a quality detection method and device of a data warehouse, wherein the method comprises: collecting metadata of the data warehouse, the metadata comprising: running data generated by the data warehouse in a running process; obtaining a running state parameter of the data warehouse according to the metadata of the data warehouse, the running state parameter being used to represent a data usage state of the data warehouse in the running process; and determining quality result data of the data warehouse based on a gap between the running state parameter and a target value of a running state of the data warehouse.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of data processing technology, and in particular to a method and apparatus for quality inspection of a data warehouse. Background Technology

[0002] With the development of internet technology, many enterprises, especially large enterprises, generate massive amounts of data in their daily business activities. This data is usually scattered across different business systems in various forms. To effectively organize and manage this data in a unified manner, enabling enterprises to easily and quickly understand and use it, and to realize its value, a data warehouse can be built. A data warehouse can systematically analyze and organize large amounts of data accumulated over the years through the data storage architecture unique to data warehouse theory. This facilitates various analytical methods (such as Online Analytical Query (OLAP) and data mining), thereby providing data support for decision-makers' decision-making process.

[0003] In related technologies, expert scoring can be used to evaluate the quality of data warehouse construction. However, this method is often influenced by human factors such as the expert's technical level and experience, subjectivity, objectivity, impartiality, and personal technical preferences. Different experts also have different evaluation dimensions. Therefore, the data warehouse quality evaluation results obtained through expert scoring are often greatly affected by the evaluator's subjectivity, making it difficult to objectively reflect the true state of the data warehouse. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a data warehouse quality inspection method and apparatus to obtain data warehouse quality data more accurately and objectively.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data warehouse quality inspection method is proposed, the method comprising:

[0007] Collect metadata of the data warehouse, including operational data generated during the operation of the data warehouse;

[0008] Based on the metadata of the data warehouse, obtain the running status parameters of the data warehouse, which are used to characterize the data usage status of the data warehouse during operation;

[0009] The quality result data of the data warehouse is determined based on the difference between the operating status parameters and the target operating status values ​​of the data warehouse.

[0010] According to a second aspect of one or more embodiments of this specification, a data warehouse quality inspection apparatus is provided, the apparatus comprising:

[0011] The data acquisition module is used to collect metadata of the data warehouse, including operational data generated by the data warehouse during operation.

[0012] The parameter calculation module is used to obtain the running status parameters of the data warehouse based on the metadata of the data warehouse. The running status parameters are used to characterize the data usage status of the data warehouse during operation.

[0013] The quality determination module is used to determine the quality result data of the data warehouse based on the difference between the operating status parameters and the target value of the operating status of the data warehouse.

[0014] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, the electronic device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor executes the executable instructions to implement the method described in any embodiment of this specification.

[0015] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the methods described in any embodiment of this specification.

[0016] The data warehouse quality inspection method described in this specification uses data warehouse metadata to obtain more accurate and objective data on data warehouse quality. This metadata includes operational data generated during the data warehouse's operation, such as monitoring changes to model tables as listed above, or monitoring the data output process of model tables. The operational status parameters calculated based on this data reflect the actual usage status of the data warehouse, providing relatively objective data. Furthermore, this method also measures data warehouse quality by considering the difference between operational status parameters and corresponding operational status target values. On one hand, as mentioned above, the operational status target value represents the theoretically optimal state of the data warehouse, providing relatively objective data that is not subjective. On the other hand, measuring quality by the difference between operational status parameters reflecting the actual usage status of the data warehouse and the operational status target value indicates better data warehouse quality, as the closer the parameters are to the target value. This approach is both objective and accurate, providing a better evaluation of data warehouse quality. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in one or more embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in one or more embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a data warehouse quality inspection method provided in an exemplary embodiment.

[0019] Figure 2 This is a schematic diagram of the structure of a data warehouse provided in an exemplary embodiment.

[0020] Figure 3 This is an exemplary embodiment of a flowchart for calculating quality result data.

[0021] Figure 4 This is a schematic diagram of the structure of a data warehouse quality inspection device provided in an exemplary embodiment.

[0022] Figure 5 This is a block diagram of an electronic device provided in an exemplary embodiment. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0024] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0025] This specification provides a method for quality inspection of a data warehouse, which can be used to evaluate the quality of a data warehouse. Figure 1 This is a flowchart illustrating a data warehouse quality inspection method provided in an exemplary embodiment, such as... Figure 1 As shown, the process may include the following steps:

[0026] In step 100, metadata of the data warehouse is collected, including operational data generated by the data warehouse during its operation.

[0027] In this embodiment, the metadata of the data warehouse may include, but is not limited to, the following two parts:

[0028] 1) Operational data generated during the operation of the data warehouse.

[0029] Operational data refers to data monitored during the operation of the data warehouse. This operational data can be understood as dynamic data generated during the data warehouse's operation and can include various aspects of the data.

[0030] For example, the operational data could be which model table a certain model table in the data warehouse depends on for its output, or it could be the frequency of data access in the data warehouse, or it could be the number of times DDL (Data Definition Language) operations are executed on the model table, or it could be the consumption of various data in the data warehouse, etc.

[0031] 2) Data generated during the development of a data warehouse that describes the data warehouse itself.

[0032] For example, node naming, model table naming, and field comments in a data warehouse. This data is generated during the construction of the data warehouse, and it is relatively static, remaining largely unchanged during the operation of the data warehouse.

[0033] In this step, the metadata of the data warehouse can be collected as operational data generated during the operation of the data warehouse, or it can include static data generated during the development of the data warehouse.

[0034] In step 102, the running status parameters of the data warehouse are obtained based on the metadata of the data warehouse. The running status parameters are used to characterize the data usage status of the data warehouse during operation.

[0035] In this step, the operational status parameters of the data warehouse can be obtained based on the metadata collected in step 100. These operational status parameters can be used to characterize the data usage status of the data warehouse during its operation.

[0036] The data usage status mentioned may include, but is not limited to, the following: whether cross-layer data references occur when the data warehouse processes external data query requests or executes internal tasks; whether reverse data dependencies occur during task execution; the frequency of model table changes; or the compatibility rate of data in the data warehouse with various business operations, etc.

[0037] That is, the data usage status represented by the running status parameters can reflect the relationship between data, whether the data itself has changed, and whether the data in the data warehouse can meet business needs, etc. This embodiment is not limited to the situations listed above.

[0038] One way to obtain the operational status parameters of a data warehouse based on the metadata is by statistically analyzing and calculating the metadata. This embodiment can obtain multiple operational status parameters, each reflecting the data usage status of the data warehouse during operation from different dimensions.

[0039] In step 104, the quality result data of the data warehouse is determined based on the difference between the operating status parameters and the target value of the operating status of the data warehouse.

[0040] In this embodiment, the target value of the data warehouse's operational status corresponds to the ideal, better state of the data warehouse. This target value can be a theoretically optimal state value, and this value is usually objective and does not vary from person to person.

[0041] For example, suppose one of the operational status parameters reflects the completeness of a data warehouse. Ideally, if a data warehouse is well-constructed, external data queries should be able to retrieve data directly from the aggregated data layer (assuming the target value for this operational status parameter is 100%). However, if the query data needs to be retrieved from the raw data layer, it indicates that the data warehouse is not yet fully developed (assuming the parameter value is 40%). A significant difference between these two values ​​reflects insufficient quality in terms of data warehouse completeness.

[0042] The above example uses completeness as an example. In actual implementation, if multiple operational status parameters can be obtained in step 102, these parameters can reflect the data usage status of the data warehouse from different dimensions (including but not limited to "completeness"). Each operational status parameter can have a corresponding operational status target value. This embodiment can measure the quality of the data warehouse based on the difference between the operational status parameters and their corresponding operational status target values, and ultimately obtain quality result data.

[0043] The data warehouse quality inspection method in this embodiment uses data warehouse metadata to obtain more accurate and objective data on data warehouse quality. This includes collecting operational data generated during the data warehouse's operation, such as monitoring changes to model tables as listed above, or monitoring the data output process of model tables. The operational status parameters calculated based on this operational data can reflect the actual usage status of the data warehouse, which is relatively objective data. Furthermore, this method also combines the difference between the operational status parameters and the corresponding operational status target values ​​to measure data warehouse quality. On the one hand, as mentioned above, the operational status target value is the theoretical optimal state value of the data warehouse, which is relatively objective data and will not vary from person to person; this will be explained again in later embodiments. On the other hand, by measuring the quality through the difference between the operational status parameters, which reflect the actual usage status of the data warehouse, and the operational status target value, the closer the difference is to the operational status target value, the better the quality of the data warehouse. This method is both objective and accurate, and can better evaluate the quality of the data warehouse.

[0044] Furthermore, the data warehouse quality inspection method in this embodiment has the following advantages: the quality result data of the data warehouse is obtained by utilizing the operational data generated during the operation of the data warehouse, thus reflecting the actual operational effect of the data warehouse. Specifically, the purpose of building a data warehouse is to support enterprises in using data and realizing its value. Therefore, the quality of a data warehouse needs to be judged by its degree of support for the enterprise's business. For example, if the model tables of a data warehouse are changed frequently, it indicates that the original model tables of the data warehouse cannot meet the enterprise's business needs, hence the frequent changes. In the expert scoring mode of existing technologies, experts usually do not consider the actual usage of the data warehouse, so the evaluation results of the expert scoring mode are not accurate and objective enough. This embodiment, however, collects operational data during the operation of the data warehouse, which can reflect the actual usage effect of the data warehouse. Moreover, the use of the target value of the data warehouse's operational status is equivalent to providing a standard for quality evaluation, making the quality evaluation of the data warehouse more objective.

[0045] The following embodiments will list several operational status parameters and explain how to obtain quality results data from the data warehouse in conjunction with these parameters. However, it should be understood that this embodiment is not limited to the operational status parameters listed below, and other types of parameters may also be used.

[0046] First, to facilitate the subsequent method description, let's take a data warehouse structure as an example. Figure 2 This is a schematic diagram of the structure of a data warehouse provided in an exemplary embodiment, such as... Figure 2 As shown, the data warehouse has a layered structure, which may include: ODS layer (Operation Data Store), DWD layer (data warehouse details), DWS layer (Data Warehouse Service), and ADS layer (Application Data Service).

[0047] Data from data sources (e.g., product systems, transaction systems, etc.) is processed through ETL (Extract-Transform-Load, which describes the process of extracting, transforming, and loading data from the source to the destination) before entering the ODS layer for centralized storage.

[0048] Then, the data in the ODS layer is further categorized and standardized using techniques such as business abstraction and dimensional modeling to obtain the DWD layer model tables. The DWD layer can also perform data cleaning and standardization operations on the ODS data, such as removing empty data, dirty data, and outliers. The DWS layer stores commonly used dimension and summary indicator data for use by the ADS layer. The ADS layer contains pre-calculated summary data, provided for data products and data analysis, directly serving business applications. The data in the DWS layer is processed from the DWD layer via ETL, while the data in the ADS layer is processed from either the DWD or DWS layer via ETL.

[0049] Next, in conjunction with the above Figure 2 The data warehouse structure shown in this example defines the following operational status parameters to be calculated and obtained:

[0050] Table 1 List of Operating Status Parameters

[0051]

[0052]

[0053] As shown in Table 1 above, this embodiment can obtain multiple operational status parameters, and these multiple operational status parameters belong to multiple dimensions, with each dimension including at least one of the operational status parameters. For example, the "data link clarity" dimension may include two operational status parameters: "average link length and reverse dependency ratio". As another example, the "stability / robustness" dimension may include two operational status parameters: "model table change frequency and latency output rate".

[0054] The calculation method for each running status parameter will be explained below:

[0055] (1) Cross-level citation rate:

[0056] The system can retrieve the number of target objects in the data warehouse's metadata. These target objects are identified as objects that have cross-layer data references based on the data warehouse's operational data. The cross-layer reference rate of the data warehouse is then determined by the proportion of these target objects to the total number of objects.

[0057] For example, for Figure 2 In the example data warehouse's ADS layer, for data query requests from external applications, if the query data requires retrieval across layers, such as the DWD layer or ODS layer, then it can be considered cross-layer retrieval, and the data query request can be referred to as the target object. If the query data is retrieved from the ADS layer or DWS layer, it is not considered cross-layer retrieval.

[0058] For example, regarding Figure 2 In the example data warehouse's DWD layer, if the data output from the DWS or ADS layer directly references data from the ODS layer, this can be considered a cross-layer data reference. The task nodes in the DWS or ADS layers that exhibit cross-layer data references can be called target objects. That is, in this case, for data tasks in the data warehouse, the task nodes that produce data by referencing data across layers can be called target objects.

[0059] After identifying the target objects, the cross-level referencing rate of the data warehouse can be determined based on the proportion of target objects to the total number of objects. The cross-level referencing rate can be directly calculated as the proportion of target objects to the total number of objects. Alternatively, other calculations can be performed based on this proportion to obtain the cross-level referencing rate. For example, the proportion of target objects to the total number of objects can be multiplied by a coefficient to obtain the cross-level referencing rate.

[0060] The following is a formula for calculating the cross-layer reference rate of a data warehouse:

[0061] R=λ1*qp+λ2*tp…………………….(1)

[0062] The meanings of each term in formula (1) above are explained as follows:

[0063] R represents the "overall cross-layer citation rate of the data warehouse";

[0064] QP represents the cross-layer referencing rate that occurs when processing data query requests initiated by external applications to the data warehouse. For example, QP can be calculated as the ratio of the number of cross-layer data query requests to the total number of query requests received by the data warehouse.

[0065] tp represents the cross-layer reference rate that occurs when various data tasks in the data warehouse are executed. For example, tp can be calculated as the ratio of the number of task nodes that have cross-layer data references to the total number of task nodes in the data warehouse.

[0066] λ1 and λ2 can be the weighted weights of qp and tp mentioned above.

[0067] (2) Average link length:

[0068] A data warehouse can contain multiple routine data tasks. These tasks process and manipulate data obtained from transaction systems, product systems, etc., ultimately producing data that serves upper-layer applications or modules. Each data task can include multiple task nodes, each responsible for a sub-task in the data production process.

[0069] For example, suppose we want to generate a data model table to serve a specific topic analysis at a higher level. This could involve four steps: "raw data collection -> building a categorical dimensional model table -> preprocessing commonly used data -> application-oriented data aggregation." Each of these four steps can be executed by a task node. For instance, a task node in the ODS layer collects raw data, a task node in the DWD layer generates the dimensional model table, a task node in the DWS layer preprocesses the data, and finally, a task node in the ADS layer generates the application data aggregation.

[0070] Furthermore, data flows in a unidirectional manner, such as along the path from ODS layer to DWD layer to DWS layer to ADS layer. The node executing a task first can be called the upstream node, and the node executing a task later can be called the downstream node. Generally, the downstream node's task depends on the completion of the upstream node's task. For example, in the above example, the task node in the DWD layer can be called the downstream node of the task node in the ODS layer, and the task node in the ODS layer is the upstream node. The task node in the DWD layer begins execution after the task node in the ODS layer has finished executing. Similarly, the DWS layer is upstream of the ADS layer, and the task in the ADS layer begins execution after the task in the DWS layer has finished executing.

[0071] In this embodiment, the output process of a data task can be referred to as a data link, and the number of task nodes included in the link can be called the link length. For example, in the example above, the data link includes 4 task nodes, so the link length is 4. In the actual metadata collection process, a task without downstream nodes can be identified as the end node task of a data link. The link is then traced upstream along the upstream tasks it depends on until it reaches the root node without upstream nodes. The total number of task nodes included in the entire data link can be used as the link length.

[0072] The average link length can be obtained by averaging the link lengths of each data link in the data warehouse.

[0073] (3) Proportion of reverse dependency:

[0074] As mentioned above, the data output process of a data link can usually be a unidirectional flow, such as flowing along the direction of ODS layer -> DWD layer -> DWS layer -> ADS layer, which makes the link relatively clear. However, sometimes reverse dependencies occur, which means that the data output of a certain task node depends on the task node in the opposite direction. For example, the data output of a task node in the DWD layer depends on the data in the ADS layer.

[0075] In this embodiment, the proportion of task nodes with reverse data dependencies to the total number of task nodes can be used as the reverse dependency ratio of the data warehouse.

[0076] The average link length and the reverse dependency ratio mentioned above can both be called data link factors. That is, the data link factor can be either the average link length or the reverse dependency ratio.

[0077] (4) Model table reference coefficients:

[0078] The model table reference coefficient can be used to represent the reusability of data. Higher reusability indicates more useful data and a more rationally constructed data warehouse. Usage information of the model tables in the data warehouse can be obtained from the data warehouse's operational data. This usage information may include the number of objects served by the model table or the frequency of access to the model table. The objects served by the model table can be downstream model tables or upper-layer applications. The model table reference coefficient of the data warehouse can be determined based on the usage information of the model tables.

[0079] For model tables in the DWD or DWS layers of a data warehouse, the reusability of model tables in the DWD or DWS layers can be measured by the average number of times a model table is read and used to generate downstream model tables. For example, if one model table in the DWD layer is read and used to generate 5 downstream model tables, and one model table in the DWS layer is read and used to generate 2 downstream model tables, these are examples of model table reuse.

[0080] For the Adaptive Data System (ADS) layer of a data warehouse, metrics used to measure the reusability of model tables in that ADS layer can be the frequency of data access to the model tables, and / or the number of upper-layer applications or modules served by the model tables. For example, the data in a model table in the ADS layer may be accessed M times in a week (M is a natural number), or the model table may serve 3 upper-layer applications.

[0081] Formula (2) below illustrates one formula for calculating the model reference coefficient of a data warehouse (but is not limited to this):

[0082] Y=λ3*n1+λ4*(f+n2)…………………….(2)

[0083] The meanings of each term in formula (2) above are explained as follows:

[0084] Y represents the model reference coefficient of the data warehouse.

[0085] n1 represents the average number of model tables in the ODS / DWD / DWS layers of the data warehouse that are read and used to produce downstream model tables. f represents the access frequency of the model tables in the ADS layer, and n2 represents the number of upper-layer applications served by the ADS layer model tables in the data warehouse. λ3 and λ4 represent the weighting factors.

[0086] (5) Data quality dimension:

[0087] The evaluation dimensions for data quality can include two operational parameters: DQC (Data Quality Control) coverage and average DQC count. When performing data tasks in a data warehouse, each task is typically equipped with DQC, which monitors the data quality produced by that task to ensure the quality of the data output.

[0088] In this embodiment, data quality monitoring tasks (DQC) can be detected in the data warehouse, and it can be determined whether each data task is equipped with a DQC. For example, assuming there are 10 data tasks in the data warehouse, 8 of which have DQC to monitor the quality of their output data, and the other 2 data tasks do not have DQC, then the proportion of tasks monitored by the DQC task is "8 / 10". That is, the ratio of the number of data tasks equipped with DQC to the total number of tasks in the data warehouse can be used as the DQC coverage rate.

[0089] Furthermore, the average DQC count refers to the number of Data Quality Control (DQC) tasks per unit node. For example, the average DQC count can be calculated as the ratio of the total number of DQCs in the data warehouse to the total number of task nodes. For instance, a task executed on a task node might be monitored by three DQCs, each performing quality monitoring in a different manner.

[0090] The aforementioned DQC coverage or average DQC count can both be referred to as data quality measurement factors.

[0091] (6) Data compatibility:

[0092] It can obtain the number of newly added first nodes upstream and the number of newly added second nodes downstream in the data warehouse, and determine the data compatibility rate of the data warehouse based on the relationship between the number of first nodes and second nodes.

[0093] For example, the data output of the ADS layer can be obtained from data in the DWD or DWS layers through ETL processing. Suppose that to support newly added applications in the upper layers, three task nodes are added to the application-oriented ADS layer to produce summary data for the applications. If the DWD or DWS layers can meet the data requirements of the three new ADS task nodes without needing to add new task nodes, then the data warehouse has good data compatibility. However, if the DWD or DWS layers need to add two task nodes to produce new model table data to meet the new data requirements of the ADS, then the data warehouse has poor data compatibility.

[0094] Based on the above principles, the data compatibility rate of this embodiment can be calculated as follows:

[0095] Data compatibility rate = Number of new nodes added in the ADS layer / Total number of new nodes added in the DWD and DWS layers;

[0096] This can be done by setting a time period, for example, from January 1st of each year to the current time when the data compatibility rate is calculated, including the number of new nodes added to the ADS layer and the total number of new nodes added to the DWD and DWS layers during this period. Of course, other time periods can also be used, and this embodiment does not impose any restrictions.

[0097] (7) Frequency of model table changes:

[0098] The model table change frequency can be used to characterize the change rate of the model table.

[0099] For example, the following calculation method can be used:

[0100] Model table change frequency = number of times the model table has been changed / total number of model tables.

[0101] The model table changes mentioned above can be DDL (Data Definition Language).

[0102] (8) Delayed output rate:

[0103] Among them, the delayed output rate can be used to characterize the proportion of delayed tasks in a data warehouse.

[0104] For example, the following calculation method can be used:

[0105] Delayed output rate = Number of days of delayed output / Total number of operating days;

[0106] For example, assuming the total number of running days is only 1 day, the following calculation can be made: if there are 100 data tasks in that 1 day, and 50 of them are delayed, then the delay output rate can be 50 / 100.

[0107] Assuming a total runtime of 100 days, and all 100 data tasks ran for 100 days, but only on one day did 50 data tasks experience a delay, while the other 99 days saw no delays. In this case, the delay output rate can be calculated as: 50 / (100*100) = 0.5%.

[0108] (9) Average resource consumption:

[0109] The average resource consumption can be used to characterize the resource consumption of a unit of data in the data warehouse.

[0110] For example, it could be the amount of computing resources required per GB of data.

[0111] (10) Resource utilization rate:

[0112] The resource utilization rate can be calculated as follows:

[0113] Active data can be retrieved from the data warehouse, which is data accessed at a preset frequency. The resource utilization rate of the data warehouse is determined based on the proportion of active data storage to the total data storage volume of the data warehouse.

[0114] For example, data with more than 3 daily accesses can be considered active data. This access count can include both external queries and access from downstream task nodes. The ratio of storage usage for active data to the total data storage capacity of the data warehouse can be used as the resource utilization rate.

[0115] (11) Normalization dimension:

[0116] The metadata of a data warehouse may also include the following data: object names and / or object annotations in the data warehouse; the objects can be nodes, fields, or model tables. Based on this metadata, multiple operational status parameters of the following standardization dimensions can be calculated: node naming standardization rate, table naming standardization rate, field naming standardization rate, and field annotation completeness rate.

[0117] For example, the calculation methods for each operating status parameter can be as follows:

[0118] Node naming standardization rate = Number of nodes with standard naming / Total number of nodes;

[0119] Table naming standardization rate = Number of tables with standardized naming / Total number of tables;

[0120] Field naming standardization rate = Number of fields with standardized naming / Total number of fields;

[0121] Field annotation completeness rate = number of fields with annotations / total number of fields.

[0122] As mentioned above, in the process of obtaining various operational status parameters, operational data generated during the operation of the data warehouse is used. For example, this operational data may include, but is not limited to: the frequency of data access, the amount of data resource consumption, whether data output is delayed, whether the model table has been changed, whether there are new task nodes in each layer of the data warehouse, whether DQC has been executed, and the data output process.

[0123] For each of the operating state parameters, this embodiment also sets an optimal and a worst-case value for each operating state parameter. These optimal / worst values ​​are theoretically optimal / worst values. Please refer to Table 2 below, which illustrates the optimal or worst-case values ​​for each operating state parameter.

[0124] Table 2 shows the theoretical optimal and theoretical worst values ​​for each operating state parameter.

[0125]

[0126] As shown in Table 2 above, the theoretical optimal / worst values ​​for each operational parameter are relatively objective. For example, a higher proportion of active data indicates a more rational data warehouse with no wasted resources. Similarly, a lower model change frequency suggests a more rational data warehouse model tables that don't require modification. Furthermore, a lower cross-layer referencing rate indicates a more complete data warehouse structure across its layers, capable of meeting data requirements. In short, these optimal / worst values ​​are not subjective and can be considered generally accepted theoretical values.

[0127] This embodiment will calculate the data warehouse quality results based on the differences between the obtained operational status parameters and their respective corresponding data warehouse operational status target values ​​(the theoretical optimal values ​​mentioned above). To perform the following calculations, this embodiment also sets weights for each operational status parameter, as shown in the example in Table 3:

[0128] Table 3 shows the weights corresponding to each operating status parameter.

[0129]

[0130] The weights in Table 3 above include "indicator weights" and "dimension weights." Indicator weights correspond to the weights of each operational state parameter, while dimension weights correspond to the weights of each evaluation dimension. The "indicator weights" and "dimension weights" satisfy the following relationship: m represents the total number of dimensions, and n represents the number of operational state parameters in each dimension.

[0131]

[0132]

[0133] Based on the above formulas (3) and (4), if the weight of each indicator is placed in the overall evaluation of the data warehouse, its weight should be Li as shown in formula (5):

[0134] L i =W i ×w m_i ………………(5)

[0135] Furthermore, the determination of the aforementioned "indicator weights" and "dimensional weights" can be done manually by the user or based on the Analytic Hierarchy Process (AHP), which will not be elaborated further.

[0136] Next, TOPSIS (Technique for Order Preference by Similarity to an Ideal Solution) can be used to calculate the dimensional score of each dimension of the data warehouse, as well as the overall quality results of the data warehouse.

[0137] First, the various operating status parameters are positiveized and converted into extremely large indicator data.

[0138] Table 4 below illustrates the usual index classifications, which can be divided into 4 categories; Table 5 illustrates the index classifications to which each operating status parameter in this embodiment belongs.

[0139] Table 4 Indicator Classification

[0140] Indicator Type Indicator characteristics example Extremely large indicators The bigger the better Performance, corporate profits Minimal Indicators The smaller the better Costs, expenses, defect rate intermediate indicators The closer to a certain value, the better. pH value during water quality assessment Interval indicators It is best if it falls within a certain range body temperature

[0141] Table 5. Index Classification of Each Operating Status Parameter

[0142]

[0143] 1) For extremely small indicators, the following formula is used for positive transformation:

[0144]

[0145] Where, x i These are the running status parameters that need to be forwarded.

[0146] 2) For interval-type indicators, the following formula is used for positive transformation:

[0147] If the optimal interval is [a, b], we can take M = max{a - min(x)} i ),max(x i Then proceed as follows:

[0148] When x i When <a,

[0149] When a < x i When <b,

[0150] When x i When >b,

[0151] The operational parameters that have already been normalized are then standardized to eliminate the influence of dimensions.

[0152] Next, after normalizing each running status parameter, we can begin calculating the dimension scores.

[0153] Figure 3 This is an exemplary embodiment providing a flowchart of the calculation of quality result data, such as... Figure 3 As shown, the following processing can be included:

[0154] In step 300, for each dimension, based on the various operating state parameters included in the dimension, the optimal value and the worst value corresponding to the operating state parameters, and the index weights corresponding to the operating state parameters, the positive distance between the operating state parameters included in the dimension and the optimal value, and the negative distance between the operating state parameters included in the dimension and the worst value are determined.

[0155] In this step, please refer to the following formula (7).

[0156]

[0157] Where i represents the i-th dimension, j represents the j-th dimension, and there are a total of n running state parameters in the i-th dimension. ij This represents the j-th running state parameter of the i-th dimension, and this parameter is a standardized parameter. This indicates the running status parameter z ij The corresponding theoretical optimal value. w i_j This indicates the running status parameter z ij The corresponding indicator weights. This represents the positive distance between the n running state parameters in the i-th dimension and the optimal value.

[0158] See formula (8) below:

[0159]

[0160] Should This represents the negative distance between the n running state parameters in the i-th dimension and the worst value; Indicates the running status parameter z ij The corresponding theoretical worst value. For the meaning of other parameters, please refer to the explanation of the parameter meaning of formula (7) above.

[0161] In step 302, the dimension score corresponding to the dimension is obtained based on the positive and negative distances.

[0162] In this step, the dimensional scores for each dimension can be calculated, as shown in the following formula (9):

[0163]

[0164] Among them, Si It is the dimension score of the i-th dimension.

[0165] In step 304, based on the dimension scores of each dimension and the dimension weights corresponding to each dimension, the scores of each dimension are combined to obtain the quality result data of the data warehouse.

[0166] This step will combine the dimensional scores of each dimension to obtain the overall quality result data of the data warehouse.

[0167] For example, see the following formula (10):

[0168]

[0169] Here, R represents the quality outcome data of the data warehouse, which can be a score. W i It is the dimensional weight, S i is the dimension score, where i represents the i-th dimension, and there are a total of m dimensions.

[0170] A higher score for the quality results data indicates a better quality data warehouse; conversely, a lower score indicates a worse quality data warehouse.

[0171] To implement the data warehouse quality inspection method described in the embodiments of this specification, Figure 4 A schematic diagram of the structure of a data warehouse quality inspection device of an exemplary embodiment is provided, such as... Figure 4 As shown, the device may include: a data acquisition module 41, a parameter calculation module 42, and a quality determination module 43.

[0172] The data acquisition module 41 is used to collect metadata of the data warehouse, including operational data generated during the operation of the data warehouse.

[0173] The parameter calculation module 42 is used to obtain the running status parameters of the data warehouse based on the metadata of the data warehouse. The running status parameters are used to characterize the data usage status of the data warehouse during operation.

[0174] The quality determination module 43 is used to determine the quality result data of the data warehouse based on the difference between the operating status parameters and the target value of the operating status of the data warehouse.

[0175] In one example, the running status parameters include: cross-layer reference rate; when the parameter calculation module 42 obtains the running status parameters of the data warehouse based on the metadata of the data warehouse, it includes: obtaining the number of target objects in the metadata of the data warehouse, wherein the target objects are objects that are determined to have cross-layer data references based on the running data; and determining the cross-layer reference rate of the data warehouse based on the proportion of the number of target objects to the total number of objects.

[0176] In one example, the target object includes at least one of the following: a data query request initiated by a business system to the data warehouse, wherein the data query request is a request to obtain query data across layers in the data warehouse based on the running data; or, a task node in the data warehouse, wherein the data output of the task node depends on cross-layer data references.

[0177] In one example, the operational status parameters include a data link factor. When the parameter calculation module 42 obtains the operational status parameters of the data warehouse based on the data warehouse's metadata, it includes: obtaining the data links of each data task in the data warehouse based on the operational data, wherein each data link includes multiple task nodes; determining the link length or reverse dependency ratio in the data warehouse based on the data links, wherein the reverse dependency ratio is the proportion of nodes with reverse data dependencies to the total number of nodes; and determining the data link factor of the data warehouse based on the link length or reverse dependency ratio.

[0178] In one example, the running status parameters include: model table reference coefficients; when the parameter calculation module 42 obtains the running status parameters of the data warehouse based on the metadata of the data warehouse, it includes: obtaining the usage information of the model tables in the data warehouse based on the running data, the usage information including the number of objects served by the model tables or the frequency of access to the model tables; and determining the model table reference coefficients of the data warehouse based on the usage information of the model tables.

[0179] In one example, the operational status parameters include: data compatibility rate. When the parameter calculation module 42 obtains the operational status parameters of the data warehouse based on the data warehouse's metadata, it includes: obtaining the first newly added upstream node and the second newly added downstream node in the data warehouse; and determining the data compatibility rate of the data warehouse based on the quantitative relationship between the first node and the second node.

[0180] In one example, the running status parameters include at least one of the following: model table change frequency, used to characterize the change rate of the model table; delayed output rate, used to characterize the task delay ratio in the data warehouse; and average resource consumption, used to characterize the resource consumption per unit of data in the data warehouse.

[0181] In one example, the operating status parameters include resource utilization. When the parameter calculation module 42 obtains the operating status parameters of the data warehouse based on the data warehouse's metadata, it includes: obtaining active data in the data warehouse, where active data is data whose access frequency reaches a preset value; and determining the resource utilization of the data warehouse based on the proportion of the storage volume of the active data to the total data storage volume of the data warehouse.

[0182] In one example, the operational status parameters include a data quality measurement factor. When the parameter calculation module 42 obtains the operational status parameters of the data warehouse based on the data warehouse's metadata, it includes: obtaining data quality monitoring tasks in the data warehouse; and determining the data quality measurement factor of the data warehouse based on the proportion of tasks performing quality monitoring through the data quality monitoring tasks and / or the number of data quality monitoring tasks per unit node.

[0183] In one example, there are multiple operational status parameters, each belonging to multiple dimensions, and each dimension includes at least one operational status parameter. The operational status target value of the data warehouse includes the optimal and worst values ​​corresponding to each operational status parameter. The quality determination module 43, when determining the quality result data of the data warehouse based on the difference between the operational status parameters and the operational status target value of the data warehouse, includes: for each dimension, determining the positive distance between the operational status parameters included in the dimension and the optimal value, and the negative distance between the operational status parameters included in the dimension and the worst value, based on the operational status parameters included in the dimension, the optimal value and the worst value corresponding to the operational status parameters, and the indicator weights corresponding to the operational status parameters; obtaining the dimension score corresponding to the dimension based on the positive and negative distances; and combining the dimension scores of each dimension and the dimension weights corresponding to each dimension to obtain the quality result data of the data warehouse.

[0184] Figure 5 This is a block diagram of an electronic device provided in an exemplary embodiment. Please refer to... Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other hardware required for business operations. One or more embodiments of this specification can be implemented in software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0185] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0186] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0187] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0188] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0189] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0190] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0191] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0192] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0193] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

Claims

1. A method of quality detection of a data warehouse, characterized in that, The method includes: Collect metadata of the data warehouse, including operational data generated during the operation of the data warehouse; The operational status parameters of the data warehouse are calculated based on the metadata of the data warehouse, and the operational status parameters are used to characterize the data usage status of the data warehouse during operation. Based on the difference between the operating status parameters and the target value of the operating status of the data warehouse, the quality result data of the data warehouse is determined. The target value of the operating status is used to characterize the data usage status of the data warehouse under ideal conditions, and the quality result data is used to evaluate the construction quality of the data warehouse. The number of the running status parameters is multiple, and the multiple running status parameters belong to multiple dimensions. Each dimension includes at least one running status parameter. The running status parameters of each dimension include: cross-layer reference rate, average link length and reverse dependency ratio, model table reference coefficient, DQC coverage and average DQC number, data compatibility rate, model table change frequency and latency output rate, average resource consumption and resource utilization rate. The target values ​​for the operational status of the data warehouse include: the optimal value and the worst value corresponding to each operational status parameter, respectively. The step of determining the quality result data of the data warehouse based on the difference between the operating status parameters and the target value of the operating status of the data warehouse includes: For each dimension, based on the various operating state parameters included in the dimension, the optimal value and the worst value corresponding to the operating state parameters, and the index weights corresponding to the operating state parameters, the positive distance between the operating state parameters included in the dimension and the optimal value, and the negative distance between the operating state parameters included in the dimension and the worst value are determined. Based on the positive and negative distances, the dimension score corresponding to the dimension is obtained; Based on the dimensional scores of each dimension and the corresponding dimensional weights, the scores of each dimension are combined to obtain the quality result data of the data warehouse.

2. The method of claim 1, wherein, The step of calculating the operational status parameters of the data warehouse based on its metadata includes: Obtain the number of target objects in the metadata of the data warehouse, wherein the target objects are objects that are determined to have cross-layer data references based on the runtime data; The cross-layer reference rate of the data warehouse is determined based on the proportion of the number of target objects to the total number of objects.

3. The method of claim 2, wherein, The target object includes at least one of the following: A data query request initiated by a business system to the data warehouse, wherein the data query request is a request to obtain query data across layers in the data warehouse based on the operational data; Alternatively, the task nodes in the data warehouse may have data outputs that depend on cross-layer data references.

4. The method of claim 1, wherein, The step of calculating the operational status parameters of the data warehouse based on its metadata includes: The data links for each data task in the data warehouse are obtained based on the operational data, and the data links include multiple task nodes; The average link length is obtained by averaging the link lengths of each data link in the data warehouse. The proportion of task nodes with reverse data dependencies to the total number of task nodes is taken as the reverse dependency ratio of the data warehouse.

5. The method of claim 1, wherein, The step of calculating the operational status parameters of the data warehouse based on its metadata includes: Based on the operational data, the usage information of the model table in the data warehouse is obtained, including the number of objects served by the model table or the frequency of access to the model table; Based on the usage information of the model table, determine the model table reference coefficient of the data warehouse.

6. The method of claim 1, wherein, The step of calculating the operational status parameters of the data warehouse based on its metadata includes: Obtain the first newly added upstream node and the second newly added downstream node in the data warehouse; The data compatibility rate of the data warehouse is determined based on the quantity relationship between the first node and the second node.

7. The method of claim 1, wherein, The step of calculating the operational status parameters of the data warehouse based on its metadata includes: Obtain active data from the data warehouse, wherein the active data is data whose access frequency reaches a preset value; The resource utilization rate of the data warehouse is determined based on the proportion of the active data storage to the total data storage of the data warehouse.

8. The method of claim 1, wherein, The step of calculating the operational status parameters of the data warehouse based on its metadata includes: Obtain data quality monitoring tasks from the data warehouse; The proportion of tasks that perform quality monitoring through the aforementioned data quality monitoring tasks is used as the DQC coverage rate, and / or the number of data quality monitoring tasks per unit node is used as the average DQC count.

9. A data warehouse quality detection apparatus, characterized by comprising: The device includes: The data acquisition module is used to collect metadata of the data warehouse, including operational data generated by the data warehouse during operation. The parameter calculation module is used to calculate the operating status parameters of the data warehouse based on the metadata of the data warehouse. The operating status parameters are used to characterize the data usage status of the data warehouse during operation. The quality determination module is used to determine the quality result data of the data warehouse based on the difference between the operating status parameters and the target value of the operating status of the data warehouse. The target value of the operating status is used to characterize the data usage status of the data warehouse under ideal conditions, and the quality result data is used to evaluate the construction quality of the data warehouse. The number of the running status parameters is multiple, and the multiple running status parameters belong to multiple dimensions. Each dimension includes at least one running status parameter. The running status parameters of each dimension include: cross-layer reference rate, average link length and reverse dependency ratio, model table reference coefficient, DQC coverage and average DQC number, data compatibility rate, model table change frequency and latency output rate, average resource consumption and resource utilization rate. The target values ​​for the operational status of the data warehouse include: the optimal value and the worst value corresponding to each operational status parameter, respectively. The specific steps by which the quality determination module determines the quality result data of the data warehouse based on the difference between the operating status parameters and the target value of the operating status of the data warehouse include: For each dimension, based on the various operating state parameters included in the dimension, the optimal value and the worst value corresponding to the operating state parameters, and the index weights corresponding to the operating state parameters, the positive distance between the operating state parameters included in the dimension and the optimal value, and the negative distance between the operating state parameters included in the dimension and the worst value are determined. Based on the positive and negative distances, the dimension score corresponding to the dimension is obtained; Based on the dimensional scores of each dimension and the corresponding dimensional weights, the scores of each dimension are combined to obtain the quality result data of the data warehouse.

10. An electronic device, comprising: include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-8 by executing the executable instructions.

11. A computer readable storage medium having stored thereon computer instructions, wherein, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-8.

Images