Data Warehouse Quality Evaluation Method, Device, Electronic Device and Storage Medium

Abstract

The present invention relates to the technical field of data processing, and provides a method, apparatus, electronic device and storage medium for evaluating the quality of a data warehouse. Compared with the current situation where the rationality judgment of the data warehouse architecture can only rely on the subjective evaluation of data warehouse designers, the present invention improves the efficiency of quality evaluation by obtaining target code segments of the data warehouse to be evaluated. The target code segments have a smaller data volume compared to the entire code file. Then, multiple preset index calculation models calculate multiple index values based on multiple data tables parsed from the target code segments and the level to which each data table belongs. Based on the index reference evaluation values corresponding to each index value, it is judged whether the corresponding index value meets the requirements, and multiple judgment results are obtained. Finally, the evaluation result of the data warehouse to be evaluated is output according to the multiple judgment results, avoiding the subjective influence of manual judgment and making the evaluation result more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, device, electronic device and storage medium for evaluating the quality of a data warehouse. Background Art

[0002] A data warehouse (DW) is a subject-oriented, integrated, relatively stable, and time-variant data collection for supporting management decision-making. The data warehouse can provide services for enterprises, for example, guiding the improvement of business processes.

[0003] If the quality of the data warehouse is poor, for example, the metadata of the data stored in the data warehouse is not standardized or inaccurate, it will lead to difficulties in data mining and analysis, waste of resources, decision-making errors, and a decrease in the value density of the data warehouse, making the services provided by the data warehouse for enterprises inaccurate or time-consuming. Therefore, it is particularly important to detect the quality of the data warehouse. Summary of the Invention

[0004] In view of the above, it is necessary to provide a method, device, electronic device and storage medium for evaluating the quality of a data warehouse, which can improve the accuracy of the quality evaluation of the data warehouse.

[0005] The first aspect of the present invention provides a method for evaluating the quality of a data warehouse, the method comprising:

[0006] Obtaining a target code segment of the data warehouse to be evaluated;

[0007] Parsing the target code segment to obtain multiple data tables included in the data warehouse and the level to which each data table belongs;

[0008] Calculating multiple index values based on the multiple data tables and the level to which each data table belongs through multiple preset index calculation models;

[0009] Obtaining an index reference evaluation value corresponding to each index value, and judging whether the corresponding index value meets the requirements based on the index reference evaluation value to obtain multiple judgment results;

[0010] Outputting an evaluation result of the data warehouse to be evaluated according to the multiple judgment results.

[0011] In an optional embodiment, the obtaining a target code segment of the data warehouse to be evaluated includes:

[0012] Scan the code files of the data warehouse to be evaluated in the preset state;

[0013] Determine the scanned code files as the target code files;

[0014] Split the target code files into the target code segments.

[0015] In an alternative embodiment, the parsing of the target code segments to obtain the multiple data tables included in the data warehouse and the level to which each data table belongs includes:

[0016] Split the target code segments according to preset keywords to obtain multiple target code sub-segments, where each target code sub-segment contains only one insert statement and one query statement;

[0017] Parse the insert statement to obtain a target table and the first level to which the target table belongs;

[0018] Parse the query statement to obtain multiple source tables and the second level to which each source table belongs.

[0019] In an alternative embodiment, the calculation of multiple metric values based on the multiple data tables and the level to which each data table belongs by multiple preset metric calculation models includes:

[0020] Obtain the first quantity of target tables across levels in the levels of the multiple source tables;

[0021] Obtain the second quantity of the target tables;

[0022] Calculate a first metric value through a preset first metric calculation model based on the first quantity and the second quantity;

[0023] Obtain the third quantity of the target tables corresponding to each source table;

[0024] Calculate a second metric value through a preset second metric calculation model based on the third quantity;

[0025] Obtain the fourth quantity of the data tables corresponding to each level;

[0026] Calculate a third metric value through a preset third metric calculation model based on the fourth quantity of the data tables corresponding to each level.

[0027] In an alternative embodiment, the obtaining of the metric reference evaluation value corresponding to each metric includes:

[0028] Obtain the historical code segments of the historical data warehouse in the preset state;

[0029] Parse the historical code snippets to obtain multiple historical data tables included in the historical data warehouse and the level to which each historical data table belongs;

[0030] Calculate multiple historical index values based on the multiple historical data tables and the level to which each historical data table belongs through the multiple preset index calculation models;

[0031] Obtain the corresponding index reference evaluation value according to the distribution of the multiple historical index values and the corresponding historical evaluation results.

[0032] In an alternative embodiment, the obtaining the corresponding index reference evaluation value according to the distribution of the multiple historical index values and the corresponding historical evaluation results includes:

[0033] Obtain the target historical evaluation result in the historical evaluation results corresponding to the same historical index value;

[0034] Obtain the target historical index value corresponding to the target historical evaluation result;

[0035] Remove the outliers in the target historical index value;

[0036] Obtain the corresponding index reference evaluation value according to the target historical index value after removing the outliers.

[0037] In an alternative embodiment, the method further includes:

[0038] Obtain the target judgment result that does not meet the requirements among the multiple judgment results;

[0039] Visually display the target judgment result.

[0040] The second aspect of the present invention provides a data warehouse quality evaluation device, the device includes:

[0041] An acquisition module, configured to acquire the target code snippet of the data warehouse to be evaluated;

[0042] A parsing module, configured to parse the target code snippet to obtain multiple data tables included in the data warehouse and the level to which each data table belongs;

[0043] A calculation module, configured to calculate multiple index values based on the multiple data tables and the level to which each data table belongs through multiple preset index calculation models;

[0044] A judgment module, configured to obtain the corresponding index reference evaluation value for each index value, and judge whether the corresponding index value meets the requirements based on the index reference evaluation value, and obtain multiple judgment results;

[0045] An evaluation module, configured to output an evaluation result of the data warehouse to be evaluated according to the multiple judgment results.

[0046] A third aspect of the present invention provides an electronic device, which includes a processor, and the processor is configured to implement the data warehouse quality evaluation method when executing a computer program stored in a memory.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the data warehouse quality evaluation method when executed by a processor.

[0048] In summary, for the data warehouse quality evaluation method, device, electronic device and storage medium of the present invention,

[0049] Compared with the current situation where the rationality judgment of the data warehouse architecture can only rely on the subjective evaluation of data warehouse designers, the present invention improves the efficiency of quality evaluation by obtaining the target code segment of the data warehouse to be evaluated. The target code segment has a smaller data volume compared to the entire code file. Then, multiple index values are calculated through multiple preset index calculation models based on multiple data tables parsed from the target code segment and the level to which each data table belongs. Based on the index reference evaluation values corresponding to each index value, it is judged whether the corresponding index value meets the requirements, and multiple judgment results are obtained. Finally, the evaluation result of the data warehouse to be evaluated is output according to the multiple judgment results, avoiding the subjective influence of manual judgment and making the evaluation result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flowchart of the data warehouse quality evaluation method provided in Embodiment 1 of the present invention.

[0051] Figure 2 is a structural diagram of the data warehouse quality evaluation device provided in Embodiment 2 of the present invention.

[0052] Figure 3 is a schematic structural diagram of the electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this invention belongs. The terms used in the description of the present invention herein are for the purpose of describing embodiments in an alternative embodiment only and are not intended to limit the present invention.

[0055] The data warehouse quality evaluation method provided by the embodiments of the present invention is executed by an electronic device. Correspondingly, the data warehouse quality evaluation device runs in the electronic device.

[0056] Embodiment 1

[0057] Figure 1 is a flowchart of the data warehouse quality evaluation method provided by Embodiment 1 of the present invention. The data warehouse quality evaluation method specifically includes the following steps. According to different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0058] S11, obtain the target code segment of the data warehouse to be evaluated.

[0059] The data warehouse to be evaluated refers to a data set that needs to be evaluated for its architecture quality. The data warehouse is usually connected to the business system to store and process the business data generated by the business system during the operation process. When developing the data warehouse, developers will establish a complete set of development specifications, thus forming a set of logical code files.

[0060] Since the number of developed code files is relatively large, if the data warehouse to be evaluated is evaluated through such a large number of code files, it will inevitably lead to more computational overhead and lower efficiency. Therefore, by obtaining the target code segment, the data warehouse to be evaluated is evaluated based on the target code segment.

[0061] In an alternative embodiment, the obtaining of the target code segment of the data warehouse to be evaluated includes:

[0062] Scan the code files of the data warehouse to be evaluated in a preset state;

[0063] Determine the scanned code files as the target code files;

[0064] Split the target code files into the target code segments.

[0065] There are two states that the data warehouse will appear during operation, one is normal operation and the other is abnormal operation. Among them, the preset state is the state where the data warehouse is operating normally. After scanning the code files in the preset state, the code files can be split into code segments through a text processing script, and the split code segments are called target code segments.

[0066] In the above optional implementation, by scanning the code files of the data warehouse to be evaluated in a preset state and splitting them, target code segments can be obtained, which can effectively ensure that the logic of the target code segments is normal, thus laying a foundation for the quality evaluation of the data warehouse to be evaluated; moreover, compared with the entire code file, the data volume of the target code segments is smaller, so based on the target code segments for quality evaluation, the efficiency of quality evaluation can be improved.

[0067] S12. Parse the target code segment to obtain multiple data tables included in the data warehouse and the level to which each data table belongs.

[0068] After parsing the target code segment, multiple data tables included in the data warehouse and the level to which each data table belongs can be obtained. Among them, the levels can include: access layer (ODS), data warehouse layer (DW), mart layer (DM), and application layer (APP). The upstream and downstream relationships of each level are: access layer (ODS) -> data warehouse layer (DW) -> mart layer (DM) -> application layer (APP).

[0069] Exemplarily, the following shows multiple data tables obtained by parsing the target code segment and the level of each data table.

[0070]

[0071] It should be noted that in a data warehouse, there is only one target table under one task, but one target table can correspond to multiple source tables. And the level to which each data table belongs is fixed, that is, each data table can only belong to one of the access layer, data warehouse layer, mart layer, or application layer.

[0072] In an optional implementation, the parsing the target code segment to obtain multiple data tables included in the data warehouse and the level to which each data table belongs includes:

[0073] Split the target code segment according to preset keywords to obtain multiple target code sub - segments, where each target code sub - segment contains only one insert statement and one query statement;

[0074] Parse the insert statement to obtain a target table and the first level to which the target table belongs;

[0075] Parse the query statement to obtain multiple source tables and the second level to which each source table belongs.

[0076] The target code segment may include: target table, target table level, source table, source table level, calculation logic, etc.

[0077] Among them, the preset keyword can be "insert" or ";", that is, using the preset keyword as a delimiter to split the target code segment into multiple target code sub-segments, so that each target code sub-segment contains and only contains one insert statement and one query statement.

[0078] An example of the target code sub-segment after splitting is as follows:

[0079] JOB_A:

[0080] Insert overwrite table DM.TABLE_A;

[0081] Select * from DW.TABLE_B left join ODS.TABLE_B_1.

[0082] The parsing process is as follows:

[0083] Task name: The name corresponding to the code file;

[0084] Target table: In the target code sub-segment after splitting, a string after Insert overwrite table, that is, the target table name DM.TABLE_A;

[0085] Target table level: The previous part of the target table name DM.TABLE_A is the target table level, that is, DM;

[0086] Source table: In the target code sub-segment after splitting, a string after from and join, that is, the target table name DW.TABLE_B and the target table name ODS.TABLE_B_1;

[0087] Source table level: The previous part of the source table name DW.TABLE_B is the level of the source table TABLE_B, that is, DW, and the previous part of the source table name ODS.TABLE_B_1 is the level of the source table TABLE_B_1, that is, ODS.

[0088] S13. Calculate multiple metric values through multiple preset metric calculation models based on the multiple data tables and the level to which each data table belongs.

[0089] Multiple metric calculation models can be defined in advance. After parsing to obtain multiple data tables and the level to which each data table belongs, multiple metric values can be calculated through the defined multiple metric calculation models.

[0090] In an alternative embodiment, the calculating multiple metric values through multiple preset metric calculation models based on the multiple data tables and the level to which each data table belongs includes:

[0091] Obtain the first quantity of the target tables across levels in the hierarchy of the multiple source tables;

[0092] Obtain the second quantity of the target tables;

[0093] Calculate a first metric value based on the first quantity and the second quantity through a preset first metric calculation model;

[0094] Obtain the third quantity of the target tables corresponding to each of the source tables;

[0095] Calculate a second metric value based on the third quantity through a preset second metric calculation model;

[0096] Obtain the fourth quantity of the data tables corresponding to each level;

[0097] Calculate a third metric value based on the fourth quantity of the data tables corresponding to each level through a preset third metric calculation model.

[0098] The first metric calculation model can be a coverage calculation model, the second metric calculation model can be a support degree calculation model, and the third metric calculation model can be a hierarchical ratio calculation model. Coverage refers to the coverage degree of the upper-level tables to the lower-level tables, support degree refers to how many target tables are supported by a source table, and hierarchical ratio refers to the ratio of the number of tables in each level.

[0099] After parsing out the levels to which each data table belongs, the hierarchical relationship of each data table can be determined according to the levels, and then the rationality of the data warehouse architecture can be evaluated according to the defined three metric calculation models.

[0100] Theoretically, for a reasonable data warehouse architecture, cross-level access should be as few as possible, and the level of the target tables should not exceed two levels or more of the source table levels, that is, the source table level corresponding to the data tables at the DM / APP level should not be ODS.

[0101] In an alternative embodiment, the preset first metric calculation model can be expressed by the following formula: T1 = (X2 - X1) / X2, where X1 represents the first quantity and X2 represents the second quantity.

[0102] A source table should support as many target tables as possible to avoid chimney-style development.

[0103] In an alternative embodiment, the preset second metric calculation model can be expressed by the following formula: T2 = X3, where X3 represents the third quantity.

[0104] For a reasonable data warehouse architecture, the number of data tables corresponding to each level of DW / DM / APP should conform to a certain ratio.

[0105] In an alternative embodiment, the preset third indicator calculation model can be expressed by the following formula: T3 = X41:X42:X43, where X41 represents the fourth quantity of data tables in the DW layer, X42 represents the fourth quantity of data tables in the DM layer, and X43 represents the fourth quantity of data tables in the APP layer.

[0106] S14. Obtain the indicator reference evaluation value corresponding to each of the indicator values, and based on the indicator reference evaluation value, determine whether the corresponding indicator value meets the requirements, obtaining a plurality of judgment results.

[0107] Different indicator values represent different meanings, so different indicator values correspond to different indicator reference evaluation values. The indicator reference evaluation value is used to measure whether the corresponding indicator value meets the expected requirements.

[0108] The first indicator value corresponds to the first indicator reference evaluation value. Based on the first indicator reference evaluation value, determine whether the first indicator value meets the requirements, obtaining the first judgment result. The first judgment result includes: the first indicator value meets the requirements, the first indicator value does not meet the requirements.

[0109] The second indicator value corresponds to the second indicator reference evaluation value. Based on the second indicator reference evaluation value, determine whether the second indicator value meets the requirements, obtaining the second judgment result. The second judgment result includes: the second indicator value meets the requirements, the second indicator value does not meet the requirements.

[0110] The third indicator value corresponds to the third indicator reference evaluation value. Based on the third indicator reference evaluation value, determine whether the third indicator value meets the requirements, obtaining the third judgment result. The third judgment result includes: the third indicator value meets the requirements, the third indicator value does not meet the requirements.

[0111] In an alternative embodiment, the obtaining the indicator reference evaluation value corresponding to each of the indicators includes:

[0112] Obtain the historical code segments of the historical data warehouse in the preset state;

[0113] Parse the historical code segments to obtain a plurality of historical data tables included in the historical data warehouse and the level to which each historical data table belongs;

[0114] Based on the plurality of historical data tables and the level to which each historical data table belongs, calculate a plurality of historical indicator values through the plurality of preset indicator calculation models;

[0115] Obtain the corresponding index reference evaluation value according to the distribution of the multiple historical index values and the corresponding historical evaluation results.

[0116] The historical data warehouse is in contrast to the data warehouse to be evaluated and refers to a data warehouse that has been run several times. The code file corresponding to the historical data warehouse is called the historical code file.

[0117] First, determine multiple historical data warehouses. For each historical data warehouse, by scanning the historical code file of the historical data warehouse in the preset state and performing segmentation, obtain historical code segments, and segment the historical code segments according to the preset keywords to obtain multiple historical code sub - segments. Among them, each historical code sub - segment contains only one historical insert statement and one historical query statement. Parse the historical insert statement to obtain a historical target table and the first level to which the historical target table belongs. Parse the historical query statement to obtain multiple historical source tables and the second level to which each historical source table belongs.

[0118] Experts in building data warehouses can score the architecture rationality of each historical data warehouse and determine whether the architecture of the historical data warehouse is reasonable according to the score. For example, compare the mean value of the scores with a preset threshold. If the mean value of the scores is greater than the preset threshold, determine that the historical evaluation result is that the architecture of the historical data warehouse is reasonable. If the mean value of the scores is not greater than the preset threshold, determine that the historical evaluation result is that the architecture of the historical data warehouse is unreasonable.

[0119] Through a large number of experiments on multiple historical data warehouses, it is found that: from the distribution of multiple historical data warehouses on the same historical index value, the data difference distributions of different historical evaluation results are obvious and regular. Thus, the architecture of the data warehouse can be quantitatively evaluated through the historical index value.

[0120] In an optional implementation manner, the obtaining the corresponding index reference evaluation value according to the distribution of the multiple historical index values and the corresponding historical evaluation results includes:

[0121] Obtain the target historical evaluation result in the historical evaluation results corresponding to the same historical index value;

[0122] Obtain the target historical index value corresponding to the target historical evaluation result;

[0123] Remove the abnormal points in the target historical index value;

[0124] Obtain the corresponding index reference evaluation value according to the target historical index value after removing the abnormal points.

[0125] Exemplarily, the historical data warehouse D1 corresponds to the first index value T11, the second index value T12, and the third index value T13. The historical data warehouse D2 corresponds to the first index value T21, the second index value T22, and the third index value T23. The historical data warehouse D3 corresponds to the first index value T31, the second index value T32, and the third index value T33. The first index values T11, T21, and T31 are the same historical index value. The second index values T12, T22, and T32 are the same historical index value. The third index values T13, T23, and T33 are the same historical index value.

[0126] The target historical evaluation result refers to the result that the architecture of the historical data warehouse is reasonable. Obtain the target historical index value corresponding to the target historical evaluation result in the same historical index value (for example, the first index value T11, the first index value T21, the first index value T31), and then remove the abnormal points to obtain the remaining target historical index value. Then determine the corresponding index reference evaluation value according to the remaining target historical index value. Specifically, the remaining target historical index values can be sorted to obtain the target historical range, and the target historical range can be determined as the corresponding index reference evaluation value, or the center point of the target historical range can be determined as the corresponding index reference evaluation value, or the minimum value of the remaining target historical index values can be determined as the corresponding index reference evaluation value. The present invention does not make any restrictions on this.

[0127] S15. Output the evaluation result of the data warehouse to be evaluated according to the multiple judgment results.

[0128] Obtain the evaluation result of the data warehouse to be evaluated by synthesizing multiple judgment results. Among them, the evaluation result may include: the architecture of the data warehouse to be evaluated is reasonable, and the architecture of the data warehouse to be evaluated is unreasonable.

[0129] When each of the multiple judgment results is that the corresponding index value meets the requirements, the evaluation result is: the architecture of the data warehouse to be evaluated is reasonable. If at least one of the multiple judgment results is that the corresponding index value does not meet the requirements, the evaluation result is: the architecture of the data warehouse to be evaluated is unreasonable.

[0130] Exemplarily, assume that the first index reference evaluation value is 100%, the second index reference evaluation value > 3, and the third index reference evaluation value is 3:5:2. If the first index value is equal to 100%, the second index value is equal to 4, and the third index value is 3:5:2, then synthesizing the first index value, the second index value, and the third index value indicates that the first index value, the second index value, and the third index value all meet the requirements, and the output evaluation result is: the architecture of the data warehouse to be evaluated is reasonable.

[0131] The reasonable architecture of the data warehouse to be evaluated indicates that the quality of the data warehouse to be evaluated is better, and the reasonable architecture of the data warehouse to be evaluated indicates that the quality of the data warehouse to be evaluated is worse.

[0132] In an alternative embodiment, the method further includes:

[0133] Obtain the target judgment results that do not meet the requirements among the multiple judgment results;

[0134] Visually display the target judgment results.

[0135] In the above alternative embodiment, by visually displaying the target judgment results that do not meet the requirements, developers can quickly locate where the unreasonable parts are in the data warehouse to be evaluated, so that corresponding measures can be quickly specified for improvement.

[0136] Compared with the current judgment of the rationality of the data warehouse architecture, which can only rely on the subjective evaluation of data warehouse designers, the method described in the embodiments of the present invention scans the target code segments of the data warehouse, analyzes the hierarchical relationships between tables, and quantitatively evaluates the architecture quality of the data warehouse from three dimensions: model coverage rate, model support rate, and model layering ratio, avoiding the subjective influence of manual judgment, reducing the input of manpower at the same time, and the evaluation results are more accurate.

[0137] Embodiment 2

[0138] Figure 2 It is a structural diagram of the data warehouse quality evaluation device provided in Embodiment 2 of the present invention.

[0139] In some embodiments, the data warehouse quality evaluation device 20 may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the data warehouse quality evaluation device 20 can be stored in the memory of the electronic device and executed by at least one processor to execute (see details in Figure 1 description) the functions of data warehouse quality evaluation.

[0140] In this embodiment, the data warehouse quality evaluation device 20 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an acquisition module 201, an analysis module 202, a calculation module 203, a judgment module 204, an evaluation module 205, and a display module 206. The module referred to in the present invention means a series of computer program segments that can be executed by at least one processor and can complete fixed functions, and are stored in the memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0141] The obtaining module 201 is configured to obtain target code segments of a data warehouse to be evaluated.

[0142] The data warehouse to be evaluated refers to a data set that needs to be evaluated for its architecture quality. A data warehouse is usually connected to a business system to store and process business data generated by the business system during its operation. When developers develop a data warehouse, they will establish a complete set of development specifications, thus forming a set of logical code files.

[0143] Since the number of developed code files is relatively large, if the data warehouse to be evaluated is evaluated through such a large number of code files, it will inevitably lead to more computational overhead and lower efficiency. Therefore, by obtaining target code segments, the evaluation of the data warehouse to be evaluated is based on the target code segments.

[0144] In an optional implementation manner, the obtaining module 201 obtaining the target code segments of the data warehouse to be evaluated includes:

[0145] Scanning the code files of the data warehouse to be evaluated in a preset state;

[0146] Determining the scanned code files as the target code files;

[0147] Dividing the target code files into the target code segments.

[0148] During the operation of the data warehouse, there will be two states, one is normal operation and the other is abnormal operation. Among them, the preset state is the state where the data warehouse is operating normally. After scanning the code files in the preset state, the code files can be divided into code segments through a text processing script, and the divided code segments are called target code segments.

[0149] In the above optional implementation manner, by scanning the code files of the data warehouse to be evaluated in a preset state and dividing them to obtain target code segments, it can effectively ensure that the logic of the target code segments is normal, thus laying a foundation for the quality evaluation of the data warehouse to be evaluated; moreover, compared with the entire code file, the amount of data of the target code segments is smaller. Therefore, based on the target code segments for quality evaluation can improve the efficiency of quality evaluation.

[0150] The parsing module 202 is configured to parse the target code segments to obtain multiple data tables included in the data warehouse and the level to which each data table belongs.

[0151] After parsing the target code snippet, multiple data tables included in the data warehouse and the level to which each data table belongs can be obtained. Among them, the levels can include: the access layer (ODS), the data warehouse layer (DW), the mart layer (DM), and the application layer (APP). The upstream and downstream relationships of each level are: access layer (ODS) -> data warehouse layer (DW) -> mart layer (DM) -> application layer (APP).

[0152] Exemplarily, the following shows the parsing of the target code snippet to obtain multiple data tables and the level of each data table.

[0153]

[0154] It should be noted that in a data warehouse, there is only one target table under one task, but one target table can correspond to multiple source tables. And the level to which each data table belongs is fixed, that is, each data table can only belong to one of the access layer, the data warehouse layer, the mart layer, or the application layer.

[0155] In an optional implementation manner, the parsing module 202 parses the target code snippet to obtain multiple data tables included in the data warehouse and the level to which each data table belongs, including:

[0156] Segment the target code snippet according to a preset keyword to obtain multiple target code sub-snippets, where each target code sub-snippet contains only one insert statement and one query statement;

[0157] Parse the insert statement to obtain a target table and the first level to which the target table belongs;

[0158] Parse the query statement to obtain multiple source tables and the second level to which each source table belongs.

[0159] The target code snippet can include: target table, target table level, source table, source table level, calculation logic, etc.

[0160] Among them, the preset keyword can be "insert" or ";", that is, the target code snippet is segmented into multiple target code sub-snippets with the preset keyword as the delimiter, so that each target code sub-snippet contains and only contains one insert statement and one query statement.

[0161] An example of the segmented target code sub-snippet is as follows:

[0162] JOB_A:

[0163] Insert overwrite table DM.TABLE_A;

[0164] Select*from DW.TABLE_B left join ODS.TABLE_B_1。

[0165] The parsing process is as follows:

[0166] Task name: The name corresponding to the code file;

[0167] Target table: In the segmented target code snippet, the string after Insert overwrite table, which is the target table name DM.TABLE_A;

[0168] Target table level: The part before the target table name DM.TABLE_A is the target table level, which is DM;

[0169] Source tables: In the segmented target code snippet, the strings after from and join, which are the target table names DW.TABLE_B and ODS.TABLE_B_1;

[0170] Source table level: The part before the source table name DW.TABLE_B is the level of the source table TABLE_B, which is DW. The part before the source table name ODS.TABLE_B_1 is the level of the source table TABLE_B_1, which is ODS.

[0171] The calculation module 203 is configured to calculate multiple metric values based on the multiple data tables and the level to which each data table belongs through multiple preset metric calculation models.

[0172] Multiple metric calculation models can be defined in advance. After parsing the multiple data tables and the level to which each data table belongs, multiple metric values can be calculated through the defined multiple metric calculation models.

[0173] In an alternative embodiment, the calculation module 203 calculating multiple metric values based on the multiple data tables and the level to which each data table belongs through multiple preset metric calculation models includes:

[0174] Obtaining a first quantity of target tables that span levels in the levels of the multiple source tables;

[0175] Obtaining a second quantity of the target tables;

[0176] Calculating a first metric value through a preset first metric calculation model based on the first quantity and the second quantity;

[0177] Obtaining a third quantity of the target tables corresponding to each source table;

[0178] Calculate the second index value based on the third quantity through a preset second index calculation model;

[0179] Obtain the fourth quantity of the data table corresponding to each level;

[0180] Calculate the third index value based on the fourth quantity of the data table corresponding to each level through a preset third index calculation model.

[0181] The first index calculation model can be a coverage calculation model, the second index calculation model can be a support degree calculation model, and the third index calculation model can be a hierarchical ratio calculation model. Coverage refers to the coverage degree of the upper-level table to the lower-level table, support degree refers to how many target tables a source table supports, and hierarchical ratio refers to the table quantity ratio of each level.

[0182] After parsing out the levels to which each data table belongs, the hierarchical relationship of each data table can be determined according to the levels, and then the rationality of the data warehouse architecture can be evaluated according to the defined three index calculation models.

[0183] Theoretically, for a reasonable data warehouse architecture, cross-level access should be as few as possible, and the level of the target table should not exceed two levels or more of the source table level, that is, the source table level corresponding to the data table at the DM / APP level should not be ODS.

[0184] In an alternative embodiment, the preset first index calculation model can be expressed by the following formula: T1 = (X2 - X1) / X2, where X1 represents the first quantity and X2 represents the second quantity.

[0185] A source table should support as many target tables as possible to avoid chimney-style development.

[0186] In an alternative embodiment, the preset second index calculation model can be expressed by the following formula: T2 = X3, where X3 represents the third quantity.

[0187] For a reasonable data warehouse architecture, the quantities of the data tables corresponding to each level of DW / DM / APP should conform to a certain ratio.

[0188] In an alternative embodiment, the preset third index calculation model can be expressed by the following formula: T3 = X41:X42:X43, where X41 represents the fourth quantity of the data table in the DW layer, X42 represents the fourth quantity of the data table in the DM layer, and X43 represents the fourth quantity of the data table in the APP layer.

[0189] The determination module 204 is configured to obtain the reference evaluation values corresponding to each of the metric values, and determine whether the corresponding metric values meet the requirements based on the reference evaluation values of the metrics, so as to obtain a plurality of determination results.

[0190] Different metric values represent different meanings, so different metric values correspond to different reference evaluation values of the metrics. The reference evaluation values of the metrics are used to measure whether the corresponding metric values meet the expected requirements.

[0191] The first metric value corresponds to the first reference evaluation value of the metric. Based on the first reference evaluation value of the metric, it is determined whether the first metric value meets the requirements, and a first determination result is obtained. The first determination result includes: the first metric value meets the requirements, and the first metric value does not meet the requirements.

[0192] The second metric value corresponds to the second reference evaluation value of the metric. Based on the second reference evaluation value of the metric, it is determined whether the second metric value meets the requirements, and a second determination result is obtained. The second determination result includes: the second metric value meets the requirements, and the second metric value does not meet the requirements.

[0193] The third metric value corresponds to the third reference evaluation value of the metric. Based on the third reference evaluation value of the metric, it is determined whether the third metric value meets the requirements, and a third determination result is obtained. The third determination result includes: the third metric value meets the requirements, and the third metric value does not meet the requirements.

[0194] In an alternative embodiment, the determination module 204 obtaining the reference evaluation values corresponding to each of the metrics includes:

[0195] Obtaining historical code snippets in the historical data warehouse in the preset state;

[0196] Parsing the historical code snippets to obtain a plurality of historical data tables included in the historical data warehouse and the level to which each historical data table belongs;

[0197] Calculating a plurality of historical metric values based on the plurality of historical data tables and the level to which each historical data table belongs through the plurality of preset metric calculation models;

[0198] Obtaining the corresponding reference evaluation value of the metric according to the distribution of the plurality of historical metric values and the corresponding historical evaluation results.

[0199] The historical data warehouse is in contrast to the data warehouse to be evaluated and refers to a data warehouse that has been run several times. The code file corresponding to the historical data warehouse is called the historical code file.

[0200] First, determine multiple historical data warehouses. For each historical data warehouse, scan the historical code files in the preset state of the historical data warehouse and perform segmentation to obtain historical code segments. Then, segment the historical code segments according to the preset keywords to obtain multiple historical code sub-segments, where each historical code sub-segment contains only one historical insert statement and one historical query statement. Parse the historical insert statement to obtain a historical target table and the first level to which the historical target table belongs. Parse the historical query statement to obtain multiple historical source tables and the second level to which each historical source table belongs.

[0201] Experts in building data warehouses can score the architecture rationality of each historical data warehouse and determine whether the architecture of the historical data warehouse is reasonable according to the score. For example, compare the average value of the scores with a preset threshold. If the average value of the scores is greater than the preset threshold, determine that the historical evaluation result is that the architecture of the historical data warehouse is reasonable. If the average value of the scores is not greater than the preset threshold, determine that the historical evaluation result is that the architecture of the historical data warehouse is unreasonable.

[0202] Through a large number of experiments on multiple historical data warehouses, it is found that: from the distribution of multiple historical data warehouses at the same historical indicator value, the data difference distributions of different historical evaluation results are obvious and regular. Therefore, the architecture of the data warehouse can be quantitatively evaluated through the historical indicator value.

[0203] In an alternative embodiment, the obtaining of the corresponding index reference evaluation value according to the distribution of the multiple historical indicator values and the corresponding historical evaluation results includes:

[0204] Obtain the target historical evaluation result in the historical evaluation results corresponding to the same historical indicator value;

[0205] Obtain the target historical indicator value corresponding to the target historical evaluation result;

[0206] Remove the abnormal points in the target historical indicator value;

[0207] Obtain the corresponding index reference evaluation value according to the target historical indicator value after removing the abnormal points.

[0208] Exemplarily, the historical data warehouse D1 corresponds to the first index value T11, the second index value T12, and the third index value T13. The historical data warehouse D2 corresponds to the first index value T21, the second index value T22, and the third index value T23. The historical data warehouse D3 corresponds to the first index value T31, the second index value T32, and the third index value T33. The first index values T11, T21, and T31 are the same historical index value. The second index values T12, T22, and T32 are the same historical index value. The third index values T13, T23, and T33 are the same historical index value.

[0209] The target historical evaluation result refers to the result that the architecture of the historical data warehouse is reasonable. Obtain the target historical index value corresponding to the target historical evaluation result in the same historical index value (for example, the first index value T11, the first index value T21, the first index value T31), and then remove the abnormal points to obtain the remaining target historical index value. Then determine the corresponding index reference evaluation value according to the remaining target historical index value. Specifically, the remaining target historical index values can be sorted to obtain the target historical range, and the target historical range can be determined as the corresponding index reference evaluation value, or the center point of the target historical range can be determined as the corresponding index reference evaluation value, or the minimum value of the remaining target historical index values can be determined as the corresponding index reference evaluation value. The present invention does not make any restrictions on this.

[0210] The evaluation module 205 outputs the evaluation result of the data warehouse to be evaluated according to the multiple judgment results.

[0211] The evaluation result of the data warehouse to be evaluated is obtained by synthesizing multiple judgment results. Among them, the evaluation result may include: the architecture of the data warehouse to be evaluated is reasonable, and the architecture of the data warehouse to be evaluated is unreasonable.

[0212] When each of the multiple judgment results is that the corresponding index value meets the requirements, the evaluation result is: the architecture of the data warehouse to be evaluated is reasonable. If at least one of the multiple judgment results is that the corresponding index value does not meet the requirements, the evaluation result is: the architecture of the data warehouse to be evaluated is unreasonable.

[0213] Exemplarily, assume that the first index reference evaluation value is 100%, the second index reference evaluation value > 3, and the third index reference evaluation value is 3:5:2. If the first index value is equal to 100%, the second index value is equal to 4, and the third index value is 3:5:2, then synthesizing the first index value, the second index value, and the third index value indicates that the first index value, the second index value, and the third index value all meet the requirements, and the output evaluation result is: the architecture of the data warehouse to be evaluated is reasonable.

[0214] The reasonable architecture of the data warehouse to be evaluated indicates that the quality of the data warehouse to be evaluated is better, and the reasonable architecture of the data warehouse to be evaluated indicates that the quality of the data warehouse to be evaluated is worse.

[0215] In an alternative embodiment, the display module 206 is configured to:

[0216] Obtain the target judgment results that do not meet the requirements among the multiple judgment results;

[0217] Visually display the target judgment results.

[0218] In the above alternative embodiment, by visually displaying the target judgment results that do not meet the requirements, developers can quickly locate the unreasonable parts in the data warehouse to be evaluated, so that corresponding measures can be quickly specified for improvement.

[0219] Compared with the current judgment of the rationality of the data warehouse architecture, which can only rely on the subjective evaluation of data warehouse designers, the device described in the embodiments of the present invention scans the target code segments of the data warehouse, analyzes the hierarchical relationships between tables, and quantitatively evaluates the architecture quality of the data warehouse from three dimensions: model coverage rate, model support rate, and model layering ratio, avoiding the subjective influence of manual judgment, reducing the input of manpower at the same time, and making the evaluation results more accurate.

[0220] Embodiment III

[0221] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the embodiments of the above data warehouse quality evaluation method are implemented, such as Figure 1 the S11-S15 shown:

[0222] S11, obtain the target code segments of the data warehouse to be evaluated;

[0223] S12, parse the target code segments to obtain multiple data tables included in the data warehouse and the level to which each data table belongs;

[0224] S13, calculate multiple index values based on the multiple data tables and the level to which each data table belongs through a plurality of preset index calculation models;

[0225] S14, obtain the index reference evaluation values corresponding to each index value, and judge whether the corresponding index value meets the requirements based on the index reference evaluation values to obtain multiple judgment results;

[0226] S15, output the evaluation result of the data warehouse to be evaluated according to the multiple judgment results.

[0227] Alternatively, when the computer program is executed by a processor, it implements the functions of each module / unit in the above device embodiments. For example Figure 2 the modules 201-205 in

[0228] The obtaining module 201 is configured to obtain a target code segment of a data warehouse to be evaluated;

[0229] The parsing module 202 is configured to parse the target code segment to obtain a plurality of data tables included in the data warehouse and the level to which each data table belongs;

[0230] The calculating module 203 is configured to calculate a plurality of index values based on the plurality of data tables and the level to which each data table belongs through a plurality of preset index calculation models;

[0231] The judging module 204 is configured to obtain an index reference evaluation value corresponding to each index value, and judge whether the corresponding index value meets the requirements based on the index reference evaluation value, so as to obtain a plurality of judgment results;

[0232] The evaluating module 205 is configured to output an evaluation result of the data warehouse to be evaluated according to the plurality of judgment results.

[0233] When the computer program is executed by a processor, it also implements the display module 206 in the above device embodiments.

[0234] Embodiment 4

[0235] Refer to Figure 3 As shown, it is a schematic structural diagram of an electronic device provided in Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0236] Those skilled in the art should understand that Figure 3 the structure of the electronic device shown does not constitute a limitation on the embodiments of the present invention. It can be a bus structure or a star structure. The electronic device 3 may further include more or fewer other hardware or software than shown, or different component arrangements.

[0237] In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit, a programmable gate array, a digital signal processor, and an embedded device, etc. The electronic device 3 may further include a client device, and the client device includes, but is not limited to, any electronic product that can perform human-computer interaction with a client through a keyboard, a mouse, a remote control, a touchpad, a voice control device, etc. For example, a personal computer, a tablet computer, a smart phone, a digital camera, etc.

[0238] It should be noted that the electronic device 3 is only an example, and other existing or future possible electronic products that can be adapted to the present invention should also be included within the protection scope of the present invention and are included herein by reference.

[0239] In some embodiments, a computer program is stored in the memory 31, and when the computer program is executed by the at least one processor 32, all or part of the steps in the data warehouse quality evaluation method as described above are implemented. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc memories, magnetic disk memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0240] Furthermore, the computer-readable storage medium mainly includes a storage program area and a storage data area. Among them, the storage program area can store an operating system, application programs required for at least one function, etc.; the storage data area can store data created according to the use of the blockchain node, etc.

[0241] The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithms. Blockchain, in essence, is a decentralized database, a series of data blocks generated by using cryptographic methods. Each data block contains information on a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block. The blockchain can include the blockchain underlying platform, the platform product service layer, and the application service layer, etc.

[0242] In some embodiments, the at least one processor 32 is the control core (Control Unit) of the electronic device 3, connecting various components of the entire electronic device 3 through various interfaces and lines. By running or executing programs or modules stored in the memory 31, and by invoking data stored in the memory 31, it performs various functions of the electronic device 3 and processes data. For example, when the at least one processor 32 executes the computer program stored in the memory, it implements all or part of the steps of the data warehouse quality evaluation method described in the embodiments of the present invention; or implements all or part of the functions of the data warehouse quality evaluation device. The at least one processor 32 can be composed of integrated circuits. For example, it can be composed of a single packaged integrated circuit, or can be composed of multiple integrated circuits with the same or different functions packaged together, including a combination of one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors, and various control chips, etc.

[0243] In some embodiments, the at least one communication bus 33 is configured to achieve connection communication between the memory 31 and the at least one processor 32, etc.

[0244] Although not shown, the electronic device 3 may further include a power supply (such as a battery) for powering each component. Preferably, the power supply can be logically connected to the at least one processor 32 through a power management device, so as to implement functions such as management of charging, discharging, and power consumption management through the power management device. The power supply may further include any components such as one or more DC or AC power supplies, a recharge device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 3 may further include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0245] The integrated units implemented in the form of software functional modules as described above can be stored in a computer-readable storage medium. The above software functional modules are stored in a storage medium and include several instructions for causing a computer device (which may be a personal computer, an electronic device, or a network device, etc.) or a processor to execute a part of the methods described in various embodiments of the present invention.

[0246] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical functional division, and there may be other division methods in actual implementation.

[0247] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units. They can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0248] In addition, in various embodiments of the present invention, the functional modules can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0249] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, it is intended to cover all changes within the meaning and scope of the equivalent elements of the claims in the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights. In addition, obviously, the word "including" does not exclude other units, and the singular does not exclude the plural. The multiple units or devices described in the specification can also be implemented by one unit or device through software or hardware. The words such as "first" and "second" are used to represent names and do not represent any specific order.

[0250] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for evaluating the quality of a data warehouse, characterized in that, The method includes: Obtaining a target code segment of the data warehouse to be evaluated. After scanning the code file when the data warehouse is in a normal operating state, the code file is segmented into code segments through a text processing script, and the segmented code segments are called the target code segments; Parsing the target code segment to obtain multiple data tables included in the data warehouse and the level to which each data table belongs, including: segmenting the target code segment according to a preset keyword to obtain multiple target code sub-segments, where each target code sub-segment contains only one insert statement and one query statement; parsing the insert statement to obtain a target table and the first level to which the target table belongs; parsing the query statement to obtain multiple source tables and the second level to which each source table belongs; Calculating multiple index values based on the multiple data tables and the level to which each data table belongs through multiple preset index calculation models, where the multiple preset index calculation models include: A first index calculation model for calculating a first index value based on the first quantity of target tables across levels in the levels of the multiple source tables and the second quantity of the target tables; A second index calculation model for calculating a second index value based on the third quantity of target tables corresponding to each source table; A third index calculation model for calculating a third index value based on the fourth quantity of data tables corresponding to each level; Obtaining the index reference evaluation value corresponding to each index value, and judging whether the corresponding index value meets the requirements based on the index reference evaluation value to obtain multiple judgment results; Outputting the evaluation result of the data warehouse to be evaluated according to the multiple judgment results.

2. The data warehouse quality evaluation method according to claim 1, wherein The obtaining of the target code segment of the data warehouse to be evaluated includes: Scanning the code file of the data warehouse to be evaluated in a preset state; Determining the scanned code file as the target code file; Segmenting the target code file into the target code segments.

3. The data warehouse quality evaluation method according to claim 2, wherein, The calculating of multiple index values based on the multiple data tables and the level to which each data table belongs through multiple preset index calculation models includes: Obtaining the first quantity of target tables across levels in the levels of the multiple source tables; Obtaining the second quantity of the target tables; Calculating the first index value based on the first quantity and the second quantity through the preset first index calculation model; Obtaining the third quantity of target tables corresponding to each source table; Calculating the second index value based on the third quantity through the preset second index calculation model; Obtaining the fourth quantity of data tables corresponding to each level; Calculating the third index value based on the fourth quantity of data tables corresponding to each level through the preset third index calculation model.

4. The data warehouse quality evaluation method according to claim 2, wherein The obtaining of the index reference evaluation value corresponding to each index includes: Obtaining the historical code segments of the historical data warehouse in the preset state; Parsing the historical code segments to obtain multiple historical data tables included in the historical data warehouse and the level to which each historical data table belongs; Based on the multiple historical data tables and the level to which each historical data table belongs, multiple historical index values are calculated through the multiple preset index calculation models; According to the distribution of the multiple historical index values and the corresponding historical evaluation results, corresponding index reference evaluation values are obtained.

5. The data warehouse quality evaluation method according to claim 4, wherein The obtaining of the corresponding index reference evaluation value according to the distribution of the multiple historical index values and the corresponding historical evaluation results includes: Obtaining the target historical evaluation result in the historical evaluation results corresponding to the same historical index value; Obtaining the target historical index value corresponding to the target historical evaluation result; Removing the abnormal points in the target historical index value; According to the target historical index value after removing the abnormal points, the corresponding index reference evaluation value is obtained.

6. The data warehouse quality evaluation method according to any one of claims 1 to 3, characterized in that The method further includes: Obtaining the target judgment result that does not meet the requirements among the multiple judgment results; Visualizing the target judgment result.

7. A data warehouse quality evaluation device, characterized in that, The device includes: An obtaining module, configured to obtain a target code segment of a data warehouse to be evaluated. After scanning the code file when the data warehouse is in a normal operation state, the code file is segmented into code segments through a text processing script, and the segmented code segments are called the target code segments; An analysis module, configured to analyze the target code segment to obtain multiple data tables included in the data warehouse and the level to which each data table belongs, including: segmenting the target code segment according to a preset keyword to obtain multiple target code sub-segments, where each target code sub-segment only contains one insert statement and one query statement; analyzing the insert statement to obtain a target table and the first level to which the target table belongs; analyzing the query statement to obtain multiple source tables and the second level to which each source table belongs; A calculation module, configured to calculate multiple index values based on the multiple data tables and the level to which each data table belongs through multiple preset index calculation models, where the multiple preset index calculation models include: a first index calculation model, configured to calculate a first index value based on the first quantity of target tables across levels in the levels of the multiple source tables and the second quantity of the target tables; a second index calculation model, configured to calculate a second index value based on the third quantity of target tables corresponding to each source table; a third index calculation model, configured to calculate a third index value based on the fourth quantity of data tables corresponding to each level; A judgment module, configured to obtain the index reference evaluation value corresponding to each index value, and judge whether the corresponding index value meets the requirements based on the index reference evaluation value, to obtain multiple judgment results; An evaluation module, configured to output an evaluation result of the data warehouse to be evaluated according to the multiple judgment results.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory. When the processor executes a computer program stored in the memory, it implements the data warehouse quality evaluation method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data warehouse quality evaluation method according to any one of claims 1 to 6.

Images