Method and device for mining quality checking rules through multi-dimensional modeling of indexes

Abstract

This invention provides a method and apparatus for mining quality inspection rules through multi-dimensional modeling of indicators. The method includes: extracting multi-dimensional data from a pair of indicators to be judged, wherein the multi-dimensional data is related to time, data lineage, and data meaning; determining the similarity of the pair of indicators to be judged in each dimension based on the multi-dimensional data; calling a pre-built target model to process the multi-dimensional data and the similarity to obtain a judgment result characterizing whether the pair of indicators to be judged is a valid association pair; and generating quality inspection rules based on the indicator pairs whose judgment result is a valid association pair. The method described in this embodiment can quickly and accurately generate effective quality inspection rules.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method and apparatus for mining quality inspection rules through multi-dimensional modeling of indicators. Background Technology

[0002] With the rapid development of various digital technologies in the era of big data, data quality has become the lifeline for the survival and development of various industries. To ensure and improve data quality, industries such as finance have established data verification rule bases and regularly invoke and execute them to identify data problems in advance, thereby better supporting upper-level applications such as business analysis and regulatory reporting.

[0003] Currently, the addition and supplementation of data quality inspection rules mainly rely on: (1) formulation based on pure manual experience, (2) extraction through interface specification documents, and (3) mining inspection rules based on a single dimension such as the correlation of indicator experience data. Among the above-mentioned current solutions, the formulation of inspection rules based on pure manual experience requires a large amount of labor costs, and is limited by the limitations of human experience, resulting in limited and easily overlooked results; while the methods of extracting inspection rules through interface specification documents or mining inspection rules through a single dimension have the disadvantage of low overall recall rate due to the single dimension they focus on. Summary of the Invention

[0004] This invention provides a method and apparatus for mining quality inspection rules through multi-dimensional modeling of indicators, which can efficiently and accurately identify and generate quality inspection rules.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for mining quality inspection rules through multi-dimensional modeling of indicators, including: Extract multi-dimensional data from the indicator pair to be judged, wherein the multi-dimensional data is related to time, data lineage, and data meaning; The similarity of the pairs of indicators to be judged in each dimension is determined based on the multi-dimensional data. The pre-built target model is invoked to process the multi-dimensional data and similarity to obtain a judgment result characterizing whether the index pair to be judged is a valid association pair; Quality check rules are generated based on the indicator pairs whose judgment results are valid association pairs.

[0006] In one embodiment, the extraction of multi-dimensional data from the pair of indicators to be determined includes: Extract the time dimension information, technical lineage information, and business meaning of each indicator in the pair of indicators to be judged. The technical lineage information includes all fields involved in the process from the source of the indicator to its processing.

[0007] In one embodiment, determining the similarity of the pair of indicators to be judged across each dimension based on the multi-dimensional data includes: The multi-dimensional data is deduplicated. Based on the deduplicated multidimensional data, the temporal correlation coefficient, bloodline similarity, and indicator meaning similarity of the pair of indicators to be judged are calculated, and the temporal correlation coefficient matches the temporal similarity of the pair of indicators to be judged.

[0008] In one embodiment, calculating the time-series correlation coefficient of the pair of indicators to be determined includes: Based on the same time dimension, the indicator information in the pair of indicators to be processed is sorted by statistical time according to the time dimension information to obtain two indicator information sequences. Calculate the Pearson correlation coefficient for the two index information sequences.

[0009] In one embodiment, calculating the bloodline similarity of the pair of indicators to be determined includes: The technical lineage information of the pairs of indicators to be determined is deduplicated; Based on the deduplication process of the technical lineage information, the number of common elements in the pairs of indicators to be processed is combined with the total number of all elements to calculate the lineage similarity.

[0010] In one embodiment, calculating the similarity of the meanings of the indicator pairs to be determined includes: The meaning of the indicators in the pair of indicators to be determined is processed to remove stop words; Based on the processed meaning of the indicator, the similarity of the meaning of the indicator is calculated by comparing the number of identical words in the Chinese text with the total number of words.

[0011] In one embodiment, constructing the target model includes: Extract multi-dimensional data from historical indicator data and define it as historical multi-dimensional data. The historical multi-dimensional data is related to time, data lineage, and data meaning. Based on the aforementioned historical multi-dimensional data, the similarity of historical indicator data across each dimension is determined; Based on the historical multi-dimensional data and similarity training, a preset model is obtained to obtain the target model, which includes a binary classification model.

[0012] In one embodiment, generating quality check rules based on the indicator pairs whose determination results are valid association pairs includes: The indicator pairs that are determined to be valid association pairs are added to the watchlist according to a preset format, which is related to the table name, indicator fields, and indicator business name. The metrics recorded in the watchlist are combined with the consistency rule template to automatically generate callable and executable SQL-type quality check rules.

[0013] In one embodiment, the method further includes: Add the quality inspection rules to the rule base.

[0014] Another embodiment of the present invention also provides a device for mining quality inspection rules through multi-dimensional modeling of indicators, comprising: The first extraction module is used to extract multi-dimensional data from the indicator pair to be judged, wherein the multi-dimensional data is related to time, blood relationship and data meaning; The first determining module is used to determine the similarity of the pair of indicators to be judged in each dimension based on the multi-dimensional data; The calling module is used to call the pre-built target model to process the multi-dimensional data and similarity, and obtain the judgment result representing whether the index pair to be judged is a valid association pair; The generation module is used to generate quality check rules based on the indicator pairs whose judgment results are valid association pairs.

[0015] Based on the above, the beneficial effects of this embodiment include the automation of the entire data verification rule mining process. By preprocessing the data and constructing a target model, rule detection and generation can be achieved, greatly reducing the manual cost of formulating and supplementing quality verification rules, i.e., reducing rule generation costs, while improving the accuracy and efficiency of rule generation. On the other hand, by comprehensively considering the multi-dimensional attributes of indicators, the recall rate of the mined rule results is improved, resulting in more effective rules, thus providing greater assurance for early warning and discovery of data problems and improving data quality. Moreover, the mined multi-dimensional comprehensive similarity indicator pairs also lay a data foundation for constructing higher-level analytical applications such as visualized knowledge graphs.

[0016] Other features and advantages of this application will be set forth in the following description. The objectives and other advantages of this application can be realized and obtained through the structures particularly pointed out in the written description and drawings.

[0017] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the method for mining quality inspection rules through multi-dimensional modeling of indicators in an embodiment of the present invention.

[0020] Figure 2 This is a flowchart illustrating the method for mining quality inspection rules through multi-dimensional modeling of indicators in an application embodiment of the present invention.

[0021] Figure 3 This is a flowchart illustrating a method for mining quality inspection rules through multi-dimensional modeling of indicators in another embodiment of the present invention.

[0022] Figure 4 This is a structural block diagram of the device for mining quality inspection rules through multi-dimensional modeling of indicators in an embodiment of the present invention. Detailed Implementation

[0023] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but these are not intended to limit the scope of the invention.

[0024] It should be understood that various modifications can be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope of this disclosure will be apparent to those skilled in the art.

[0025] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0026] These and other features of the invention will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0027] It should also be understood that although the invention has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of the invention, which have the features described in the claims and are therefore all within the scope of protection defined herein.

[0028] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0029] Specific embodiments of the present disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure and can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the present disclosure. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely to serve as the basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of substantially any suitable detailed structures.

[0030] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.

[0031] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] like Figure 1 As shown, this embodiment of the invention provides a method for mining quality inspection rules through multi-dimensional modeling of indicators, including: S1: Extract multi-dimensional data from the pair of indicators to be judged, wherein the multi-dimensional data is related to time, data lineage, and data meaning; S2: Determine the similarity of the pairs of indicators to be judged in each dimension based on the multi-dimensional data; S3: Call the pre-built target model to process the multi-dimensional data and similarity to obtain a judgment result characterizing whether the index pair to be judged is a valid association pair; S4: Generate quality check rules based on the indicator pairs whose judgment results are valid association pairs.

[0033] The method in this embodiment can be applied to any inspection technology field, such as financial inspection technology field, product inspection technology field, news media information inspection technology field, etc., and is not unique. In this embodiment, for the indicator pair to be judged, multi-dimensional data is extracted, including data in the time dimension, data lineage dimension, and data meaning dimension. Of course, data in other dimensions can also be extracted at the same time, and there is no specific limitation. After obtaining the multi-dimensional data, the similarity of the data in the indicator pair to be judged in each dimension is determined based on the multi-dimensional data, or in other words, the similarity of the indicator pair to be judged in each dimension. After determining the similarity, the similarity and multi-dimensional data are input into the pre-constructed target model for processing. The target model is a classification model, which is used to determine whether the corresponding indicator pair is a valid and related indicator pair based on the similarity and multi-dimensional data. Its output result can be, but is not limited to, yes or no. According to the judgment result output by the target model, the system filters out the valid and related effective associated indicator pairs, and generates quality inspection rules based on the indicators in the indicator pair.

[0034] The method described in this embodiment can quickly and accurately identify and generate valid quality inspection rules, greatly simplifying the manual processing workflow and significantly improving rule generation efficiency. In other words, it significantly improves the automation level and recall rate of indicator inspection rule mining, reduces labor costs, and simultaneously increases the number of valid rules mined along with the increased efficiency.

[0035] Combination Figure 2 As shown, the extraction of multi-dimensional data from the indicator pair to be judged includes: S101: Extract the time dimension information, technical lineage information, and business meaning of each indicator in the pair of indicators to be judged. The technical lineage information includes all fields involved in the process from the source of the indicator to its processing.

[0036] In this embodiment, the technical lineage information of each indicator to be determined has a corresponding data table, meaning that each piece of data can be traced back to its corresponding data table. The extracted technical lineage data includes extracting the corresponding source table and fields. For business meaning, this can be the business meaning of the indicator extracted or compiled from the corresponding interface specification. For example, the extracted multi-dimensional data includes: T1 Historical Data Table for Indicators (Indicator ID, Statistical Date, Indicator Value); T2 Indicator Lineage Information Table (Indicator ID, Processing Lineage Information); T3 Indicator Meaning Information (Indicator ID, Indicator Name, Indicator Business Meaning); The processing lineage information in the T2 indicator lineage information table records all fields involved in the process from the source table to the indicator processing, in the following format:<T1,F2> ,<T2,F4> ,...,<TN,F5> , where TN represents the table involved and FN represents the table field involved.

[0037] After extracting the multi-dimensional data, determining the similarity of the pair of indicators to be judged in each dimension based on the multi-dimensional data includes: S201: Perform deduplication processing on the multi-dimensional data; S202: Calculate the temporal correlation coefficient, bloodline similarity, and indicator meaning similarity of the pair of indicators to be judged based on the deduplicated multi-dimensional data, and the temporal correlation coefficient matches the temporal similarity of the pair of indicators to be judged.

[0038] That is, first deduplication is performed on the multi-dimensional data, and then the similarity of each dimension is calculated.

[0039] Specifically, calculating the time-series correlation coefficient of the pair of indicators to be determined includes: S203: Based on the same time dimension, the indicator information in the pair of indicators to be processed is sorted by statistical time according to the time dimension information to obtain two indicator information sequences. S204: Calculate the Pearson correlation coefficient for the two index information sequences.

[0040] For example, assuming the time dimension is the same, the information from the two indicators in the indicator pair is sorted according to statistical time to obtain two series. Then, the Pearson correlation coefficient between the two series is calculated to obtain the time-series correlation coefficient. Calculating the bloodline similarity of the pair of indicators to be determined includes: S205: Perform deduplication processing on the technical lineage information of the pair of indicators to be determined; S206: Calculate the lineage similarity based on the number of elements common to the index pairs to be processed and the total number of all elements in the deduplicated lineage information.

[0041] For example, as can be seen from the aforementioned embodiments, the lineage data of the indicators includes all data tables and fields from the source table to the completion of the processing. When calculating the lineage similarity, the system calculates the sequence similarity based on the processed lineage data sequences of the two indicators. The calculation formula is: the number of elements in common between the two lineage sequences / the number of elements after removing duplicates and taking the union of the two lineage sequences.

[0042] Calculating the similarity of the meanings of the indicator pairs to be judged includes: S207: Perform stop word removal processing on the meaning of the indicators of the pair of indicators to be determined; S208: Based on the processed meaning of the indicator, calculate the similarity of the meaning of the indicator by the number of words that are the same in the Chinese text and the total number of words.

[0043] For example, after segmenting the meaning of the indicators in the two indicators of an indicator pair (removing stop words), the text similarity is calculated using the formula: the number of identical words in the two texts / the number of words in the union of the two texts after removing duplicates (total number of words).

[0044] After determining the various similarities, the system will call the target model for processing. In this embodiment, the target model needs to be pre-built, and building the target model includes: S5: Extract multi-dimensional data from historical indicator data and define it as historical multi-dimensional data. The historical multi-dimensional data is related to time, data lineage, and data meaning. S6: Determine the similarity of historical indicator data across each dimension based on the aforementioned historical multi-dimensional data; S7: Based on the historical multi-dimensional data and similarity, train the preset model to obtain the target model, wherein the preset model includes a binary classification model.

[0045] Continue to combine Figure 2 As shown, for example, when pre-constructing a target model specific to a certain business domain, historical indicator data within the specified business domain can be selected as the basic training data. If a universal target model is to be constructed, the business domain can be omitted, and historical indicator data from the entire domain can be used as the basic training data. When constructing training data, the system extracts multi-dimensional data from the historical indicator data, obtaining historical multi-dimensional data. The dimensions involved in this historical multi-dimensional data are time, lineage, and indicator meaning. After obtaining the historical multi-dimensional data, the system calculates its similarity, obtaining the similarity regarding time, lineage, and indicator meaning. Then, the system labels the historical multi-dimensional data and uses the labeled historical multi-dimensional data and the corresponding similarities as training data to train the preset model, obtaining the target model. In this embodiment, a machine learning logistic regression algorithm is used to repeatedly train the preset binary classification model until the comprehensive F-score (considering both recall and precision) of the evaluation model reaches a relatively optimal level, thus obtaining the target model.

[0046] In one embodiment, after the trained target model is deployed, it processes the multi-dimensional data and similarity of the input indicator pairs to be judged, and then outputs yes or no to characterize whether the indicator pairs to be judged are valid association pairs. Combined with... Figure 3 As shown, the generation of quality check rules based on the indicator pairs whose judgment results are valid association pairs includes: S401: Add the indicator pairs that are determined to be valid association pairs to the watchlist according to a preset format, wherein the preset format is related to the table name, indicator field, and indicator business name; S402: Automatically generate callable and executable SQL-type quality check rules by combining the indicators recorded in the attention list with the consistency rule template.

[0047] For example, indicators with a "yes" result are added to the watchlist in a preset format, such as <Indicator 1 Table Name, Indicator 1 Field Name, Indicator 1 Business Name, Indicator 2 Table Name, Indicator 2 Field Name, Indicator 2 Business Name>. Then, based on the indicator data recorded in the watchlist, and combined with the consistency rule template, callable SOL-type quality check rules are automatically generated.

[0048] The template is as follows: --Extracting indicator 1 for the current period WITH IDX1_CUR_VALUE AS (SELECT ${F1} AS IDX_VALUE FROM ${T1} WHERE PERIOD = ${CUR_PERIOD}), --Extract the previous period value of indicator 1 IDX1_LAST_VALUE AS (SELECT ${F1} AS IDX_VALUE FROM ${T1} WHERE PERIOD = ${LAST_PERIOD}), --Extract indicator 2 for the current period value IDX2_CUR_VALUE AS (SELECT ${F2} AS IDX_VALUE FROM ${T2} WHERE PERIOD = ${CUR_PERIOD}), --Extract the previous period value of indicator 2 IDX2_LAST_VALUE AS (SELECT ${F2} AS IDX_VALUE FROM ${T2} WHERE PERIOD = ${LAST_PERIOD}) SELECT 'The difference in current period changes between Indicator 1: ${IDX_NAME1} and Indicator 2: ${IDX_NAME2} has exceeded the set tolerance value. Please pay attention.' AS ERRMSG FROM IDX1_CUR_VALUE icv1, IDX1_LAST_VALUE ilv1, IDX2_CUR_VALUE icv2, IDX2_LAST_VALUE ilv2 WHERE ABS((icv1.IDX_VALUE - ilv1.IDX_VALUE) / ilv1.IDX_VALUE -(icv2.IDX_VALUE - ilv2.IDX_VALUE) / ilv2.IDX_VALUE)>0.1 -- Tolerance value for the difference in changes between the two indicators AND ilv1.IDX_VALUE<>0 AND ilv2.IDX_VALUE<>0 Where <${T1},${F1},${IDX_NAME1}> and <${T2},${F2},${IDX_NAME2}> are the <table name, field name, and indicator name> of the two indicators returned in the watchlist from the previous step, respectively; ${CUR_PERIOD} and ${LAST_PERIOD} are the current period and the previous period, respectively, and are passed in uniformly when executing batch calls; the tolerance value for the difference in changes between the two indicators in the current period is set to 0.1 by default, and can be adjusted according to actual usage.

[0049] After generating the quality inspection rules, the method further includes: S8: Add the quality inspection rules to the rule base.

[0050] When applied, the system can execute rules from the rule base in batches according to different preset time dimensions, so as to issue early warnings after abnormal data is detected, thereby improving data quality.

[0051] like Figure 4 As shown, another embodiment of the present invention also provides a device for mining quality inspection rules through multi-dimensional modeling of indicators, including: The first extraction module is used to extract multi-dimensional data from the indicator pair to be judged, wherein the multi-dimensional data is related to time, blood relationship and data meaning; The first determining module is used to determine the similarity of the pair of indicators to be judged in each dimension based on the multi-dimensional data; The calling module is used to call the pre-built target model to process the multi-dimensional data and similarity, and obtain the judgment result representing whether the index pair to be judged is a valid association pair; The generation module is used to generate quality check rules based on the indicator pairs whose judgment results are valid association pairs.

[0052] In one embodiment, the extraction of multi-dimensional data from the pair of indicators to be determined includes: Extract the time dimension information, technical lineage information, and business meaning of each indicator in the pair of indicators to be judged. The technical lineage information includes all fields involved in the process from the source of the indicator to its processing.

[0053] In one embodiment, determining the similarity of the pair of indicators to be judged across each dimension based on the multi-dimensional data includes: The multi-dimensional data is deduplicated. Based on the deduplicated multidimensional data, the temporal correlation coefficient, bloodline similarity, and indicator meaning similarity of the pair of indicators to be judged are calculated, and the temporal correlation coefficient matches the temporal similarity of the pair of indicators to be judged.

[0054] In one embodiment, calculating the time-series correlation coefficient of the pair of indicators to be determined includes: Based on the same time dimension, the indicator information in the pair of indicators to be processed is sorted by statistical time according to the time dimension information to obtain two indicator information sequences. Calculate the Pearson correlation coefficient for the two index information sequences.

[0055] In one embodiment, calculating the bloodline similarity of the pair of indicators to be determined includes: The technical lineage information of the pairs of indicators to be determined is deduplicated; Based on the deduplication process of the technical lineage information, the number of common elements in the pairs of indicators to be processed is combined with the total number of all elements to calculate the lineage similarity.

[0056] In one embodiment, calculating the similarity of the meanings of the indicator pairs to be determined includes: The meaning of the indicators in the pair of indicators to be determined is processed to remove stop words; Based on the processed meaning of the indicator, the similarity of the meaning of the indicator is calculated by comparing the number of identical words in the Chinese text with the total number of words.

[0057] In one embodiment, constructing the target model includes: Extract multi-dimensional data from historical indicator data and define it as historical multi-dimensional data. The historical multi-dimensional data is related to time, data lineage, and data meaning. Based on the aforementioned historical multi-dimensional data, the similarity of historical indicator data across each dimension is determined; Based on the historical multi-dimensional data and similarity training, a preset model is obtained to obtain the target model, which includes a binary classification model.

[0058] In one embodiment, generating quality check rules based on the indicator pairs whose determination results are valid association pairs includes: The indicator pairs that are determined to be valid association pairs are added to the watchlist according to a preset format, which is related to the table name, indicator fields, and indicator business name. The metrics recorded in the watchlist are combined with the consistency rule template to automatically generate callable and executable SQL-type quality check rules.

[0059] In one embodiment, the device further includes: An add module is used to add the quality inspection rules to the rule base.

[0060] Another embodiment of the present invention also provides an electronic device, comprising: One or more processors; Memory, configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for mining quality check rules through multi-dimensional modeling of indicators as described above.

[0061] Furthermore, one embodiment of the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the method described above for mining quality inspection rules through multi-dimensional modeling of indicators. It should be understood that the various solutions in this embodiment have the corresponding technical effects in the above-described method embodiments, and will not be repeated here.

[0062] Furthermore, embodiments of the present invention also provide a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions, which, when executed, cause at least one processor to perform a method for mining quality check rules through multi-dimensional modeling of indicators, as described in the embodiments above.

[0063] It should be noted that the computer storage medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program configured for use by or in connection with an instruction execution system, system, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, antenna, optical fiber, RF, etc., or any suitable combination thereof.

[0064] Furthermore, those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

Claims

1. A method for mining quality inspection rules through multi-dimensional modeling of indicators, characterized in that, include: Extract multi-dimensional data from the indicator pair to be judged, wherein the multi-dimensional data is related to time, data lineage, and data meaning; The similarity of the pairs of indicators to be judged in each dimension is determined based on the multi-dimensional data. The pre-built target model is invoked to process the multi-dimensional data and similarity to obtain a judgment result characterizing whether the index pair to be judged is a valid association pair; Quality check rules are generated based on the indicator pairs whose judgment results are valid association pairs.

2. The method for mining quality inspection rules through multi-dimensional modeling of indicators according to claim 1, characterized in that, The extraction of multi-dimensional data from the indicator pairs to be judged includes: Extract the time dimension information, technical lineage information, and business meaning of each indicator in the pair of indicators to be judged. The technical lineage information includes all fields involved in the process from the source of the indicator to its processing.

3. The method for mining quality inspection rules through multi-dimensional modeling of indicators according to claim 2, characterized in that, Determining the similarity of the pairs of indicators to be judged across each dimension based on the multi-dimensional data includes: The multi-dimensional data is deduplicated. Based on the deduplicated multidimensional data, the temporal correlation coefficient, bloodline similarity, and indicator meaning similarity of the pair of indicators to be judged are calculated, and the temporal correlation coefficient matches the temporal similarity of the pair of indicators to be judged.

4. The method for mining quality inspection rules through multi-dimensional modeling of indicators according to claim 3, characterized in that, Calculating the time-series correlation coefficient of the pair of indicators to be determined includes: Based on the same time dimension, the indicator information in the pair of indicators to be processed is sorted by statistical time according to the time dimension information to obtain two indicator information sequences. Calculate the Pearson correlation coefficient for the two index information sequences.

5. The method for mining quality inspection rules through multi-dimensional modeling of indicators according to claim 3, characterized in that, Calculating the bloodline similarity of the pair of indicators to be determined includes: The technical lineage information of the pairs of indicators to be determined is deduplicated; Based on the deduplication process of the technical lineage information, the number of common elements in the pairs of indicators to be processed is combined with the total number of all elements to calculate the lineage similarity.

6. The method for mining quality inspection rules through multi-dimensional modeling of indicators according to claim 3, characterized in that, Calculating the similarity of the meanings of the indicator pairs to be judged includes: The meaning of the indicators in the pair of indicators to be determined is processed to remove stop words; Based on the processed meaning of the indicator, the similarity of the meaning of the indicator is calculated by comparing the number of identical words in the Chinese text with the total number of words.

7. The method for mining quality inspection rules through multi-dimensional modeling of indicators according to claim 1, characterized in that, Constructing the target model includes: Extract multi-dimensional data from historical indicator data and define it as historical multi-dimensional data. The historical multi-dimensional data is related to time, data lineage, and data meaning. Based on the aforementioned historical multi-dimensional data, the similarity of historical indicator data across each dimension is determined; Based on the historical multi-dimensional data and similarity training, a preset model is obtained to obtain the target model, which includes a binary classification model.

8. The method for mining quality inspection rules through multi-dimensional modeling of indicators according to claim 1, characterized in that, The generation of quality check rules based on the indicator pairs whose determination results are valid association pairs includes: The indicator pairs that are determined to be valid association pairs are added to the watchlist according to a preset format, which is related to the table name, indicator fields, and indicator business name. The metrics recorded in the watchlist are combined with the consistency rule template to automatically generate callable and executable SQL-type quality check rules.

9. The method for mining quality inspection rules through multi-dimensional modeling of indicators according to claim 1, characterized in that, The method further includes: Add the quality inspection rules to the rule base.

10. A device for mining quality inspection rules through multi-dimensional modeling of indicators, characterized in that, include: The first extraction module is used to extract multi-dimensional data from the indicator pair to be judged, wherein the multi-dimensional data is related to time, blood relationship and data meaning; The first determining module is used to determine the similarity of the pair of indicators to be judged in each dimension based on the multi-dimensional data; The calling module is used to call the pre-built target model to process the multi-dimensional data and similarity, and obtain the judgment result representing whether the index pair to be judged is a valid association pair; The generation module is used to generate quality check rules based on the indicator pairs whose judgment results are valid association pairs.

Images