A method, device, medium and equipment for screening rules for data quality inspection

Abstract

The present invention relates to the field of data processing, and in particular to a rule screening method, device, medium and equipment for data quality inspection, including: performing quality inspection on verification data in a verification data table according to candidate quality inspection rules, obtaining target quality inspection results and target quality inspection labels of the verification data in the verification data table relative to the candidate quality inspection rules, and then obtaining the verification data quality score of the verification data table relative to the candidate quality inspection rules, judging the quality inspection validity of the candidate quality inspection rules for data corresponding to each target data source according to a comparison between the verification data quality score and the quality score range, and obtaining a target quality inspection model for quality inspection of a data table to be quality inspected, performing quality inspection on each data table to be quality inspected through the corresponding target quality inspection model, and obtaining the data quality level corresponding to each quality inspection data, thereby improving the richness of the quality inspection rules of the target quality inspection model and the accuracy of the quality inspection results.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a rule screening method, device, medium and equipment for data quality inspection. Background Art

[0002] Data quality inspection can measure the health of data in multiple dimensions such as accuracy, completeness, consistency, and reliability, and provide strong support for tasks such as data trend analysis, comparative analysis, classification statistics, traceability analysis, and relationship analysis.

[0003] In the prior art, the quality inspection method for big data is usually to upload the data table to be inspected to a data quality inspection platform, and sequentially inspect the data in the uploaded data table to be inspected through multiple quality inspection rules bound in the data quality inspection platform, find out the problematic data and mark them, and finally return the data quality inspection results to characterize the health status of the data in the data table to be inspected.

[0004] Different quality inspection rules can perform quality inspection on data in different dimensions. With the continuous development of data forms and data needs, new quality inspection rules have emerged. However, the new quality inspection rules are not applicable to all data tables and all data sources. If the new quality inspection rules are incorrectly applied in the data quality inspection platform, the accuracy of the quality inspection results of the quality inspection data will be reduced.

[0005] Therefore, how to improve the accuracy of quality inspection results while improving the richness of quality inspection rules in the data quality inspection platform has become an urgent problem to be solved. Summary of the invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is a rule screening method for data quality inspection, and the rule screening method for data quality inspection comprises the following steps:

[0007] S100, obtain N data tables to be quality inspected, N verification data tables, a preset quality score range, a preset data quality inspection platform and Q candidate quality inspection rules to be screened in the data quality inspection platform corresponding to N target data sources, wherein the data quality inspection platform includes N preset quality inspection models, and the i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, i=1, 2..., N, N refers to the total number of data tables to be quality inspected.

[0008] S200, perform quality inspection on the vth verification data in the i-th verification data table according to the qth candidate quality inspection rule, and obtain the target quality inspection result W of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule. iv q, where v = 1, 2, ..., V(i), V(i) refers to the total amount of data in the i-th verification data table, and q = 1, 2, ..., Q, Q refers to the total number of preset quality inspection rules.

[0009] S300, according to the target quality inspection result W iv q , get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q .

[0010] S400, traverse v=1, 2, ..., V(i), and obtain V(i) target quality inspection labels of the i-th verification data table corresponding to the q-th candidate quality inspection rule.

[0011] S500: Based on V(i) target quality inspection labels, obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule. i q .

[0012] S600, if H i q Within the quality score range, the qth candidate quality inspection rule is added to the ith preset quality inspection model, and the ith target quality inspection model is obtained by traversing q=1, 2..., Q.

[0013] S700, performing quality inspection on the i-th data table to be inspected by using the i-th target quality inspection model, and obtaining a data quality level corresponding to the i-th data table to be inspected.

[0014] The present invention also provides a rule screening device for data quality inspection, the rule screening device for data quality inspection comprises:

[0015] The data acquisition module is used to obtain N data tables to be quality inspected, N verification data tables, a preset quality score range, a preset data quality inspection platform and Q candidate quality inspection rules to be screened in the data quality inspection platform corresponding to N target data sources, wherein the data quality inspection platform includes N preset quality inspection models, and the i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, i=1, 2..., N, N refers to the total number of data tables to be quality inspected.

[0016] The target quality inspection result acquisition module is used to perform quality inspection on the vth verification data in the ith verification data table according to the qth candidate quality inspection rule, and obtain the target quality inspection result W of the vth verification data in the ith verification data table relative to the qth candidate quality inspection rule. iv q, where v = 1, 2, ..., V(i), V(i) refers to the total amount of data in the i-th verification data table, and q = 1, 2, ..., Q, Q refers to the total number of preset quality inspection rules.

[0017] The target quality inspection label acquisition module is used to obtain the target quality inspection label based on the target quality inspection result W iv q , get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q .

[0018] The data table traversal module is used to traverse v=1, 2..., V(i) to obtain V(i) target quality inspection labels of the i-th verification data table relative to the q-th candidate quality inspection rule.

[0019] The verification data quality scoring module is used to obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule based on V(i) target quality inspection labels. i q .

[0020] Target quality inspection model acquisition module, used for H i q Within the quality score range, the qth candidate quality inspection rule is added to the ith preset quality inspection model, and the ith target quality inspection model is obtained by traversing q=1, 2..., Q.

[0021] The data quality inspection module is used to perform quality inspection on the i-th data table to be inspected through the i-th target quality inspection model, and obtain the data quality level corresponding to the i-th data table to be inspected.

[0022] The present invention also provides a non-transitory computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned rule screening method for data quality inspection.

[0023] The present invention also provides an electronic device, comprising a processor and the above-mentioned non-transitory computer-readable storage medium.

[0024] The present invention has at least the following beneficial effects: quality inspection is performed on the vth verification data in the ith verification data table according to the qth candidate quality inspection rule, and the target quality inspection result W of the vth verification data in the ith verification data table relative to the qth candidate quality inspection rule is obtained. iv q , according to the target quality inspection result W iv q, get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q , traverse v = 1, 2 ..., V (i), obtain V (i) target quality inspection labels of the i-th verification data table relative to the q-th candidate quality inspection rule, and the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule i q , which is used to characterize the quality inspection status of the i-th verification data table relative to the q-th candidate quality inspection rule, and provides a data basis for verifying the quality inspection validity of the candidate quality inspection rules for the data corresponding to each target data source; according to H i q By comparing with the quality score range, the effectiveness of the candidate quality inspection rules for the data corresponding to each target data source is judged, and the target quality inspection model is obtained for the quality inspection of the data table to be inspected. The i-th data table to be inspected is quality inspected by the i-th target quality inspection model, and the data quality level corresponding to the i-th data table to be inspected is obtained, which improves the richness of the quality inspection rules of the target quality inspection model and the accuracy of the quality inspection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flowchart of a rule screening method for data quality inspection provided in the first embodiment of the present invention;

[0027] Figure 2 A flow chart of a data quality inspection method based on data sampling provided in Embodiment 2 of the present invention;

[0028] Figure 3 A flowchart of a data quality inspection method provided in Embodiment 3 of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a rule screening device for data quality inspection provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] This embodiment 1 provides a rule screening method for data quality inspection, and the rule screening method for data quality inspection includes the following steps: Figure 1 As shown:

[0034] S100, obtain N data tables to be quality inspected, N verification data tables, a preset quality score range, a preset data quality inspection platform and Q candidate quality inspection rules to be screened in the data quality inspection platform corresponding to N target data sources, wherein the data quality inspection platform includes N preset quality inspection models, and the i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, i=1, 2..., N, N refers to the total number of data tables to be quality inspected.

[0035] The data table to be inspected refers to a data table that needs to be quality inspected and is provided by the target data source. The data table to be inspected may include structured data or unstructured data. The target data source may be selected based on actual conditions.

[0036] For example, the target data source may refer to the comprehensive police application platforms at provincial, ministerial and other levels, and the data table to be inspected may refer to the data collected by each comprehensive police application platform. By conducting quality inspection on the data collected by each comprehensive police application platform, comprehensive information analysis support such as case analysis, personnel analysis, early warning analysis, data collision, comprehensive query, trajectory analysis, frequency analysis, and full-text retrieval can be provided for each type of police.

[0037] The preset data quality inspection platform has a built-in preset quality inspection model, which is bound to a number of preset quality inspection rules for performing quality inspection on the input data in multiple dimensions, identifying problematic data, and finally outputting a corresponding list of problematic data.

[0038] The candidate quality inspection rules to be screened are newly added quality inspection rules for data quality inspection. Before adding the candidate quality inspection rules to the data quality inspection platform, it is necessary to measure the quality inspection effects of the candidate quality inspection rules on the data corresponding to each target data source, and determine the target data sources to which the candidate quality inspection rules can be added to the quality inspection models of the data. This can improve the richness of the quality inspection rules of the data quality inspection platform while improving the quality inspection accuracy of the data quality inspection platform.

[0039] The verification data table may refer to a data table that has been quality inspected by manual methods or other methods. The quality inspection results corresponding to the verification data table may be obtained, and then the corresponding quality score range may be obtained to verify the quality inspection effectiveness of the candidate quality inspection rules for the data corresponding to each target data source.

[0040] Among them, specific quality inspection rules can be set by the implementer according to actual conditions. For example, the quality inspection rules may include integrity rule checks such as field missing check, data null value check, data source check, and format rule checks such as ID card number format check, mobile phone number format check, date, time format check, numeric format check, number format check, and value domain rule checks such as dictionary value range, dictionary name range, date type range, numeric type range, and normative format checks such as date of birth and ID card check, gender and ID card check, age and ID card check.

[0041] S200, perform quality inspection on the vth verification data in the i-th verification data table according to the qth candidate quality inspection rule, and obtain the target quality inspection result W of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule. iv q , where v = 1, 2, ..., V(i), V(i) refers to the total amount of data in the i-th verification data table, and q = 1, 2, ..., Q, Q refers to the total number of preset quality inspection rules.

[0042] Among them, the target quality inspection results include whether there are problems with the data or not.

[0043] In the above, the i-th verification data table is quality inspected by the q-th candidate quality inspection rule, and the target quality inspection result of each verification data in the i-th verification data table relative to the q-th candidate quality inspection rule is obtained, which provides a data basis for verifying the quality inspection validity of the candidate quality inspection rules for the data corresponding to each target data source.

[0044] S300, according to the target quality inspection result W iv q , get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q .

[0045] In a specific implementation manner, the target quality inspection result includes data having problems and data not having problems, and S300 further includes the following steps:

[0046] S310, if the target quality inspection result is W iv q If there is a problem with the data, the corresponding target quality inspection label R iv q is equal to the first preset value.

[0047] S320, if the target quality inspection result is W iv q If there is no problem with the data, the corresponding target quality inspection label R iv q is equal to the second preset value.

[0048] The specific values ​​of the first preset value and the second preset value can be set by the implementer according to the actual situation. For example, in this embodiment, the first preset value can be set to 1 and the second preset value can be set to 0.

[0049] S400, traverse v=1, 2, ..., V(i), and obtain V(i) target quality inspection labels of the i-th verification data table corresponding to the q-th candidate quality inspection rule.

[0050] S500: Based on V(i) target quality inspection labels, obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule. i q .

[0051] In a specific implementation, S500 further includes the following steps:

[0052] S510, according to V(i) target quality inspection labels, obtain a third quantity F corresponding to the first preset value in V(i) target quality inspection labels 3 i q The fourth quantity F corresponding to the second preset value 4 i q .

[0053] S520, according to the third quantity F 3 i q and the fourth quantity F 4i q , obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule i q =F 4 i q / (F 3 i q +F 4 i q ).

[0054] Among them, the fourth quantity F corresponding to the second preset value 4 i q The larger the proportion of , the smaller the proportion of problematic data in the i-th verification data table relative to the q-th candidate quality inspection rule, which means that the verification data quality score of the i-th verification data table relative to the q-th candidate quality inspection rule is higher.

[0055] As mentioned above, based on V(i) target quality inspection labels, the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule is obtained. i q , which is used to characterize the quality inspection status of the i-th verification data table relative to the q-th candidate quality inspection rule, and serves as the basis for verifying the quality inspection effectiveness of the candidate quality inspection rules for the data corresponding to each target data source.

[0056] S600, if H i q Within the quality score range, the qth candidate quality inspection rule is added to the ith preset quality inspection model, and the ith target quality inspection model is obtained by traversing q=1, 2..., Q.

[0057] In a specific embodiment, the i-th preset quality inspection model includes M i The preset quality inspection rules, S600 also includes the following steps:

[0058] S610, the M in the i-th preset quality inspection model i A preset quality inspection rule is determined as the target quality inspection rule in the i-th target quality inspection model.

[0059] S620, if H i q Within the quality score range, the qth candidate quality inspection rule is determined as the target quality inspection rule in the ith target quality inspection model.

[0060] S630, traverse q=1, 2, ..., Q, and obtain the i-th target quality inspection model, wherein the i-th target quality inspection model includes Φ(i) target quality inspection rules.

[0061] Among them, the quality score range includes the upper limit and lower limit of the quality score. i q Less than or equal to the upper limit of the quality score and greater than or equal to the lower limit of the quality score, it means H i q Within the quality score range, it means that the quality inspection situation of the i-th verification data table relative to the q-th candidate quality inspection rule meets the preset situation, then the q-th candidate quality inspection rule is determined as the target quality inspection rule in the i-th target quality inspection model, which is used to perform quality inspection on the i-th data table to be quality inspected.

[0062] According to H i q By comparing with the quality score range, the effectiveness of the candidate quality inspection rules for the data corresponding to each target data source is judged, and the target quality inspection model is obtained for the quality inspection of the data table to be inspected, which improves the richness of the quality inspection rules of the target quality inspection model and the accuracy of the quality inspection results.

[0063] S700, performing quality inspection on the i-th data table to be inspected by using the i-th target quality inspection model, and obtaining a data quality level corresponding to the i-th data table to be inspected.

[0064] In a specific implementation, S700 further includes the following steps:

[0065] S710, according to the φth target quality inspection rule in the i-th target quality inspection model, perform quality inspection on the u-th data to be inspected in the i-th data table to be inspected, and obtain a third quality inspection result E of the u-th data to be inspected in the i-th data table to be inspected relative to the φth target quality inspection rule in the i-th target quality inspection model. 3 iu φ , where u = 1, 2…, U(i), U(i) refers to the total amount of data in the i-th data table to be quality inspected, and φ = 1, 2…, Ф(i), Ф(i) refers to the total number of target quality inspection rules in the i-th target quality inspection model.

[0066] S720, if the third quality inspection result is E 3 iu φ If there is a problem with the data, the corresponding third quality inspection label R 3 iu φ is equal to the first preset value.

[0067] S730, if the third quality inspection result is E3 iu φ If there is no problem with the data, the corresponding third quality inspection label R 3 iu φ is equal to the second preset value.

[0068] S740, traverse u=1, 2..., U(i) and φ=1, 2..., Φ(i), and obtain the fifth quantity F of the first preset value corresponding to the i-th data table to be inspected 5 i and the corresponding sixth number F of the second preset value 6 i .

[0069] S750, according to F 5 i 、F 6 i and the preset quality level range, and obtain the data quality level corresponding to the i-th data table to be inspected.

[0070] In a specific implementation, S750 further includes the following steps:

[0071] S751, obtain α preset quality level ranges and the quality level corresponding to each preset quality level range, wherein the data quality level corresponding to the βth preset quality level range is β, β=1, 2..., α, α refers to the total number of preset quality level ranges.

[0072] S752, if F 6 i / (F 5 i +F 6 i ) meets the β-th preset quality level range, then the data quality level corresponding to the i-th data table to be inspected is determined to be β.

[0073] The specific value of α and each preset quality level range can be set by the implementer according to actual conditions.

[0074] For example, in this embodiment, α can be set to 4, and the first quality level range is [0, 10), the second quality level range is [10, 60), the first quality level range is [60, 90), and the second quality level range is [90, 100].

[0075] In the above, according to the fifth quantity F of the first preset value corresponding to the i-th to-be-checked data table 5 i , the sixth number F of the corresponding second preset value 6 iand the preset quality level range, the data quality level corresponding to the i-th data table to be inspected is obtained, thereby improving the accuracy of the data quality level.

[0076] In a specific implementation, S700 further includes the following steps:

[0077] S760, traverse u=1, 2, ..., U(i), and obtain the seventh quantity F of the first preset value of the φth target quality inspection rule in the i-th target quality inspection model for the i-th data table to be inspected 7 i φ and the eighth number F of the second preset value 8 i φ .

[0078] S770, according to the seventh quantity F 7 i φ and the eighth quantity F 8 i φ , obtain the first target data quality score of the i-th data table to be inspected relative to the φ-th target quality inspection rule in the i-th target quality inspection model Ж i φ =F 8 i φ / (F 7 i φ +F 8 i φ ).

[0079] S780, traverse φ=1, 2, ..., Φ(i), and obtain all first target data quality scores of the i-th data table to be quality inspected relative to the i-th target quality inspection model.

[0080] S790, obtaining the preset importance corresponding to each target quality inspection rule, and obtaining the second target data quality score σ corresponding to the i-th data table to be quality inspected according to all the first target data quality scores corresponding to the i-th data table to be quality inspected and the preset importance corresponding to each target quality inspection rule. i , where σ i Meet the following conditions:

[0081] σ i =Σ φ Ф(i) (γ i φ × i φ ), where γ i φIt refers to the preset importance level corresponding to the φth target quality inspection rule in the ith target quality inspection model, i φ It refers to the target data quality score of the i-th data table to be quality inspected relative to the φ-th target quality inspection rule in the i-th target quality inspection model.

[0082] S791, according to σ i and the preset quality level range, and obtain the data quality level corresponding to the i-th data table to be inspected.

[0083] Among them, the preset importance corresponding to each target quality inspection rule can be determined according to the quality inspection rule type corresponding to the target quality inspection rule. For example, the quality inspection rule type can include red line problem rules, common problem rules and general problem rules, and the corresponding importance during data quality inspection is from high to low.

[0084] In the above, combined with the first target data quality score of the i-th data table to be inspected relative to all target quality inspection rules in the i-th target quality inspection model, and the preset importance corresponding to each target quality inspection rule, the second target data quality score σ corresponding to the i-th data table to be inspected is obtained. i , which improves the accuracy of the second target data quality score, thereby improving the accuracy of the data quality grade.

[0085] In the above, the vth verification data in the ith verification data table is quality inspected according to the qth candidate quality inspection rule, and the target quality inspection result W of the vth verification data in the ith verification data table relative to the qth candidate quality inspection rule is obtained. iv q , according to the target quality inspection result W iv q , get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q , traverse v = 1, 2 ..., V (i), obtain V (i) target quality inspection labels of the i-th verification data table relative to the q-th candidate quality inspection rule, and the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule i q , which is used to characterize the quality inspection status of the i-th verification data table relative to the q-th candidate quality inspection rule, and provides a data basis for verifying the quality inspection validity of the candidate quality inspection rules for the data corresponding to each target data source; according to H i qBy comparing with the quality score range, the effectiveness of the candidate quality inspection rules for the data corresponding to each target data source is judged, and the target quality inspection model is obtained for the quality inspection of the data table to be inspected. The i-th data table to be inspected is quality inspected by the i-th target quality inspection model, and the data quality level corresponding to the i-th data table to be inspected is obtained, which improves the richness of the quality inspection rules of the target quality inspection model and the accuracy of the quality inspection results.

[0086] Embodiment 2

[0087] The second embodiment provides a data quality inspection method based on data sampling, and the data quality inspection method based on data sampling includes: Figure 2 As shown:

[0088] S10, obtaining N data tables to be inspected corresponding to N target data sources, the historical data quality level corresponding to each data table to be inspected, and a preset data quality score threshold, wherein N refers to the total number of data tables to be inspected.

[0089] The quality level of historical data can be obtained by statistics and analysis of the quality inspection results of the historical quality inspection data of the corresponding target data source. Correspondingly, the fewer the problematic data in the historical quality inspection data, the higher the quality level of the historical data corresponding to the to-be-inspected data table corresponding to the target data source.

[0090] The specific value of the preset data quality score threshold can be set by the implementer according to actual conditions.

[0091] S20, sampling each data table to be inspected, and obtaining the i-th sampling data table corresponding to the i-th data table to be inspected, where i=1, 2, ..., N.

[0092] In a specific implementation, S20 further includes the following steps:

[0093] S201, according to the historical data quality level D corresponding to the i-th data table to be inspected i and the preset initial sampling ratio L 0 , obtain the target sampling ratio L corresponding to the i-th quality inspection data table i =L 0 ×((2 / (1+e^(-D i )))-1).

[0094] S202, obtaining the data volume of the ith data table to be inspected, and performing a sampling operation based on the data volume of the ith data table to be inspected and the corresponding target sampling ratio L i , obtain the amount of sampled data corresponding to the i-th data table to be inspected.

[0095] S203, extracting data from the i-th data table to be inspected according to the amount of sampling data corresponding to the i-th data table to be inspected, to form an i-th sampling data table corresponding to the i-th data table to be inspected.

[0096] Among them, the historical data quality level D corresponding to the i-th data table to be inspected i The larger the value is, the more problematic data there may be in the i-th data table to be inspected. In order to better measure the problem status of the i-th data table to be inspected, more data needs to be extracted for quality inspection. Therefore, the historical data quality level D corresponding to the i-th data table to be inspected is i and the target sampling ratio L i There is a positive correlation.

[0097] The preset initial sampling ratio L 0 The specific value can be set by the implementer according to the actual situation.

[0098] According to the amount of sampling data corresponding to the i-th data table to be inspected, data can be randomly selected from the i-th data table to be inspected to form the i-th sampling data table corresponding to the i-th data table to be inspected.

[0099] In the above, each data table to be inspected is sampled to obtain the i-th sampling data table corresponding to the i-th data table to be inspected, which provides a data basis for measuring the data quality of the i-th data table to be inspected and improves the efficiency of measuring the data quality of the i-th data table to be inspected.

[0100] S30, uploading the i-th sampling data table to the preset data quality inspection platform, performing quality inspection on the i-th sampling data table through the i-th preset quality inspection model in the data quality inspection platform, and obtaining the i-th sampling quality inspection result table corresponding to the i-th sampling data table.

[0101] In a specific implementation, the data quality inspection platform includes N preset quality inspection models, wherein the i-th preset quality inspection model includes M i The i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, and S30 further includes the following steps:

[0102] S301, according to the jth preset quality inspection rule in the i-th preset quality inspection model, perform quality inspection on the kth sampled data in the i-th sampled data table, and obtain a first quality inspection result E of the kth sampled data in the i-th sampled data table relative to the jth preset quality inspection rule in the i-th preset quality inspection model 1 ik j , where k = 1, 2, ..., K(i), K(i) refers to the total amount of data in the i-th sampling data table, j = 1, 2, ..., M i , Mi It refers to the total number of preset quality inspection rules in the i-th preset quality inspection model.

[0103] S302, according to the first quality inspection result E 1 ik j , obtain the first quality inspection label R of the kth sample data in the i-th sample data table relative to the j-th preset quality inspection rule in the i-th preset quality inspection model 1 ik j .

[0104] S303, the first quality inspection label R 1 ik j As the data value of the kth row and jth column in the i-th sampling quality inspection result table.

[0105] S304, traverse k=1, 2..., K(i) and j=1, 2..., M i , forming the i-th sampling quality inspection result table.

[0106] In a specific implementation, S302 further includes the following steps:

[0107] S3021, if the first quality inspection result is E 1 ik j If there is a problem with the data, the corresponding first quality inspection label R 1 ik j is equal to the first preset value.

[0108] S3022, if the first quality inspection result is E 1 ik j If there is no problem with the data, the corresponding first quality inspection label R 1 ik j is equal to the second preset value.

[0109] In the above, the i-th sampling data table is quality inspected through the i-th preset quality inspection model in the data quality inspection platform, and the i-th sampling quality inspection result table corresponding to the i-th sampling data table is obtained, which provides a data basis for measuring the data quality of the i-th data table to be quality inspected, and improves the accuracy of measuring the data quality of the i-th data table to be quality inspected.

[0110] S40, obtaining the i-th data quality score corresponding to the i-th data table to be quality inspected according to the historical data quality level corresponding to the i-th data table to be quality inspected and the i-th sampling quality inspection result table.

[0111] In a specific implementation, S40 further includes the following steps:

[0112] S401: According to the i-th sampling quality inspection result table, obtain the first quantity F corresponding to the first preset value in the i-th sampling quality inspection result table. 1 i The second number F corresponding to the second preset value 2 i .

[0113] S402, according to the second quantity F 2 i and the first quantity F 1 i , obtain the sampling data quality score P corresponding to the i-th data table to be inspected 1 i =F 2 i / (F 1 i +F 2 i ).

[0114] S403, according to the historical data quality level D corresponding to the i-th data table to be inspected i and the sampling data quality score P corresponding to the i-th data table to be inspected 1 i , obtain the i-th data quality score P corresponding to the i-th data table to be inspected 2 i =P 1 i ×((2 / (1+e^(-D i )))-1).

[0115] The second number F corresponding to the second preset value 2 i The larger the proportion of , the smaller the proportion of problematic data in the i-th sampling data table, and further the smaller the proportion of problematic data in the i-th data table to be inspected. Correspondingly, the higher the sampling data quality score corresponding to the i-th data table to be inspected.

[0116] Above, through historical data quality level D i The data quality of the target data source corresponding to the i-th data table to be inspected is represented by the sample data quality score P 1 i Characterize the actual data quality of the i-th data table to be inspected, and combine it with the historical data quality level D i and sampling data quality score P 1 i Get the data quality score P corresponding to the i-th data table to be inspected 2 i, improving the data quality score P 2 i accuracy.

[0117] S50: judging whether the i-th data table to be quality inspected meets the preset quality inspection standard according to the i-th data quality score and the data quality score threshold.

[0118] Among them, if the i-th data quality score is greater than the data quality score threshold, then the i-th data table to be inspected is judged to meet the preset quality inspection standard. If the i-th data quality score is less than or equal to the data quality score threshold, then the i-th data table to be inspected is judged to not meet the preset quality inspection standard.

[0119] The specific value of the data quality score threshold can be set by the implementer according to actual conditions.

[0120] S60: If the i-th data table to be inspected does not meet the preset quality inspection standard, the i-th data table to be inspected is returned to the corresponding i-th target data source.

[0121] Among them, the target data source can manage problems on the returned data table to be inspected, and upload the managed data table to the data quality inspection platform again as the data table to be inspected, so as to conduct another sampling quality inspection on the managed data table, obtain the corresponding data quality score, and again judge whether the managed data table meets the preset quality inspection standards, and perform corresponding operations based on the judgment results.

[0122] S70, if the i-th data table to be quality inspected meets the preset quality inspection standard, the i-th data table to be quality inspected is uploaded to the data quality inspection platform, and the i-th data table to be quality inspected is quality inspected by the i-th preset quality inspection model, and the quality inspection result table corresponding to the i-th data table to be quality inspected is obtained.

[0123] In a specific implementation, S70 further includes the following steps:

[0124] S701, according to the jth preset quality inspection rule in the i-th preset quality inspection model, perform quality inspection on the u-th data to be inspected in the i-th data table to be inspected, and obtain the second quality inspection result E of the u-th data to be inspected in the i-th data table to be inspected relative to the j-th preset quality inspection rule in the i-th preset quality inspection model 2 iu j , where u = 1, 2, ..., U(i), U(i) refers to the total amount of data in the i-th data table to be inspected.

[0125] S702, if the second quality inspection result is E 2 iu j If there is a problem with the data, the corresponding second quality inspection label R 2iu j is equal to the first preset value.

[0126] S703, if the second quality inspection result is E 2 iu j If there is no problem with the data, the corresponding second quality inspection label R 2 iu j is equal to the second preset value.

[0127] S704, the second quality inspection label R 2 iu j As the data value of the u-th row and j-th column in the i-th quality inspection result table.

[0128] S705, traverse u=1, 2, ..., U(i) and j=1, 2, ..., M i , forming the i-th quality inspection result table corresponding to the i-th data table to be inspected.

[0129] In the above, each data table to be inspected is sampled, and the i-th sampling data table corresponding to the i-th data table to be inspected is obtained, which provides a data basis for measuring the data quality of the i-th data table to be inspected, and improves the efficiency of measuring the data quality of the i-th data table to be inspected; the i-th sampling data table is quality inspected by the i-th preset quality inspection model in the data quality inspection platform, and the i-th sampling quality inspection result table corresponding to the i-th sampling data table is obtained, which provides a data basis for measuring the data quality of the i-th data table to be inspected, and improves the accuracy of measuring the data quality of the i-th data table to be inspected; through the historical data quality level D i Characterize the data quality of the target data source corresponding to the i-th data table to be inspected, characterize the data quality of the i-th data table to be inspected actually corresponding to the i-th data table to be inspected through the i-th sampling quality inspection result, and combine it with the historical data quality level D i The data quality score corresponding to the ith data table to be inspected is obtained from the ith sampling quality inspection result table, thereby improving the accuracy of the data quality score; if the ith data table to be inspected does not meet the preset quality inspection standard, the ith data table to be inspected is returned to the corresponding ith target data source; if the ith data table to be inspected meets the preset quality inspection standard, the ith data table to be inspected is uploaded to the data quality inspection platform, and the ith data table to be inspected is quality inspected by the ith preset quality inspection model, and the quality inspection result table corresponding to the ith data table to be inspected is obtained. The data tables to be inspected are screened by the data quality score, low-quality data are reworked, and high-quality data are quality inspected, thereby reducing the quality inspection pressure of the data quality inspection platform and improving the quality inspection efficiency of the data quality inspection platform.

[0130] Embodiment 3

[0131] This embodiment 3 provides a data quality inspection method, which includes: Figure 3 As shown:

[0132] S1, obtain N data tables to be inspected, the predicted inspection time of each data table to be inspected, and the historical data quality level corresponding to each data table to be inspected, and upload the i-th data table to be inspected according to the preset upload time T i Input into the preset data quality inspection platform, i=1, 2..., N, N refers to the total number of data tables to be inspected.

[0133] Among them, the predicted quality inspection time refers to the predicted quality inspection time required when the data quality inspection platform conducts quality inspection on each data table to be inspected. The quality inspection time can be predicted manually or through a trained prediction model, which will not be elaborated here.

[0134] In a specific implementation, obtaining the predicted quality inspection time of each data table to be inspected further includes the following steps:

[0135] S01, obtaining the data volume of each data table to be quality inspected and the quality inspection speed of the data quality inspection platform.

[0136] S02: Determine the ratio of the data volume and the quality inspection speed of each data table to be quality inspected as the predicted quality inspection time of the corresponding data table to be quality inspected.

[0137] Among them, the quality inspection speed of the data quality inspection platform can be obtained by counting and calculating the historical quality inspection data volume and historical quality inspection time of the data quality inspection platform.

[0138] S2, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0 , obtain the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, where W0=0, and the quality inspection time conflict result includes quality inspection time conflict and quality inspection time non-conflict.

[0139] In a specific implementation, S2 further includes the following steps:

[0140] S21, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspectedi , get the predicted quality inspection completion time C of the i-th quality inspection data table i =W i-1 +T i +Y i .

[0141] S22, according to the upload time T of the i+1th quality inspection data table i+1 and the predicted quality inspection completion time C of the i-th quality inspection data table i , obtain the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) =T i+1 -C i .

[0142] S23, if G i(i+1) >T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection time does not conflict.

[0143] S24, if G i(i+1) ≤T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as the quality inspection time conflict.

[0144] Among them, T 0 The specific value can be set by the implementer according to the actual situation.

[0145] As mentioned above, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0 , determine whether there is a quality inspection time conflict between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, and obtain the quality inspection time conflict result, which provides a data basis for adjusting the quality inspection order of the data sets to be quality inspected.

[0146] S3. If the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be quality inspected and the historical data quality level of the i+1-th data table to be quality inspected, the quality inspection order replacement result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained, wherein the quality inspection order replacement result includes replacing the quality inspection order and not replacing the quality inspection order.

[0147] Among them, when the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, if the historical data quality level of the i-th data to be quality inspected is lower than the historical data quality level of the i+1-th data table to be quality inspected, it is necessary to determine whether the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected needs to be exchanged, and obtain the quality inspection order exchange result as the basis for quality inspection of the data table to be quality inspected through the data quality inspection platform.

[0148] In a specific implementation, S3 further includes the following steps:

[0149] S32, according to the historical data quality level D of the i-th data table to be inspected i and the quality level D of the historical data of the i+1th data table to be inspected i+1 , obtain the second quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 2 i(i+1) =(2 / (1+e^(D i -D i+1 )))-1.

[0150] S33, if B 2 i(i+1) >B 2 , then the quality inspection order exchange result between the i-th quality inspection data table and the i+1-th quality inspection data table is obtained as the quality inspection order exchange, where B 2 Refers to the preset second quality inspection order replacement degree threshold.

[0151] S34, if B 2 i(i+1) ≤B 2 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged.

[0152] Among them, the degree of the second quality inspection order exchange between the i-th data table to be inspected and the i+1-th data table to be inspected is related to the historical data quality level D of the i+1-th data table to be inspected. i+1 The quality level D of the historical data of the i-th data table to be inspected i The difference between i+1 -D i Positively correlated.

[0153] As mentioned above, based on the difference between the quality levels of historical data, the second quality inspection order replacement degree B is obtained. 1 i(i+1), to characterize the degree to which the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected needs to be swapped, which serves as the basis for adjusting the quality inspection order of the data tables to be quality inspected, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform.

[0154] In a specific implementation, the data quality inspection method further includes the following steps:

[0155] S31, if the quality inspection time conflict result between the i-th data table to be inspected and the i+1-th data table to be inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be inspected, the historical data quality level of the i+1-th data table to be inspected, the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) , preset time interval threshold T 0 The preset first quality inspection sequence exchange degree threshold B 1 , obtain the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected.

[0156] In a specific implementation, S31 further includes the following steps:

[0157] S311, according to the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) , and the preset time interval threshold T 0 , obtain the quality inspection time overlap degree A between the i-th quality inspection data table and the i+1-th quality inspection data table i(i+1) =(T 0 -G i(i+1) ) / T 0 .

[0158] S312, based on the historical data quality level D of the i-th data table to be inspected i 、The quality level of historical data in the i+1th data table to be inspected is D i+1 Overlap degree with quality inspection time A i(i+1) , obtain the first quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 1 i(i+1) =A i(i+1) ×((2 / (1+e^(D i -D i+1 )))-1).

[0159] S313, if B 1 i(i+1) >B 1 , then the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as the exchanged quality inspection order.

[0160] S314, if B 1 i(i+1) ≤B 1 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged.

[0161] Among them, based on the quality level gap of historical data, combined with the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) and the preset time interval threshold T 0 , the degree of overlap of the quality inspection time between the i-th data table to be inspected and the i+1-th data table to be inspected is measured, then the degree of exchange of the first quality inspection order between the i-th data table to be inspected and the i+1-th data table to be inspected is related to the historical data quality level D of the i+1-th data table to be inspected i+1 The quality level D of the historical data of the i-th data table to be inspected i The difference between i+1 -D i Positively correlated with the quality inspection time overlap degree A i(i+1) is positively correlated, thereby obtaining the first quality inspection order replacement degree B 1 i(i+1) , to represent the degree to which the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected needs to be swapped.

[0162] The above-mentioned, comprehensive quality inspection time overlap degree and the difference between the historical data quality levels, obtain the first quality inspection sequence replacement degree B 1 i(i+1) , to characterize the degree to which the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected needs to be swapped, which serves as the basis for adjusting the quality inspection order of the data tables to be quality inspected, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform.

[0163] S4: According to the quality inspection order swap result between the i-th data table to be inspected and the i+1-th data table to be inspected, the data quality inspection platform performs quality inspection on the i-th data table to be inspected and the data table to be inspected that is earlier in the order of the i+1-th data table to be inspected, and obtains the problem data list of the i-th data table to be inspected and the data table to be inspected that is earlier in the order of the i+1-th data table to be inspected, as well as the quality inspection completion time W of the first i data tables to be inspected. i , update i=i+1.

[0164] The specific content of the problem data list can be set by the implementer according to the actual situation. For example, the problem data list can include the problem data detected by quality inspection, and can also include the problem data, the business table corresponding to the problem data, the business component corresponding to the problem data, the quality inspection rules for detecting the problem data, and other dimensional contents.

[0165] When the quality inspection of the i-th data table to be inspected and the i+1-th data table to be inspected that is the previous one in sequence is completed, the corresponding completion time is obtained as the quality inspection completion time W of the first i data tables to be inspected i , and update i=i+1, thereby serving as the data basis for obtaining the quality inspection time conflict result between the i+1th data table to be quality inspected and the i+2th data table to be quality inspected.

[0166] In a specific implementation, S4 further includes the following steps:

[0167] S41, if the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is to exchange the quality inspection order, then determine the i+1-th data table to be quality inspected as the data table to be quality inspected that comes earlier in order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected.

[0168] S42, if the result of exchanging the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is not to change the quality inspection order, then determine that the i-th data table to be quality inspected is the data table to be quality inspected that comes earlier in order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected.

[0169] In the above, the quality inspection of the data to be inspected is performed according to the quality inspection order exchange result between the i-th data table to be inspected and the i+1-th data table to be inspected, and the quality inspection completion time W of the first i data tables to be inspected is obtained. i , combined with the update method of i=i+1, it provides a data basis for obtaining the quality inspection time conflict results between the i+1th data table to be quality inspected and the i+2th data table to be quality inspected, realizes the continuous adjustment of the quality inspection order of the data tables to be quality inspected and the continuous quality inspection of the data tables to be quality inspected, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform.

[0170] S5, repeat S2-S4 until i=N is updated, and the N-1th data table to be inspected and the subsequent data table to be inspected in the Nth data table to be inspected are quality inspected through the data quality inspection platform to obtain N problem data lists corresponding to the N data tables to be inspected.

[0171] In the above, by repeatedly executing S2-S4 until i=N is updated, the quality inspection order of N-1 data tables to be quality inspected is adjusted in sequence, and the quality inspection of N-1 data tables to be quality inspected is completed. Further quality inspection is performed on the N-1 data table to be quality inspected and the data table to be quality inspected that is later in the N data table to be quality inspected, and the quality inspection of N data tables to be quality inspected is completed. N problem data lists corresponding to the N data tables to be quality inspected are obtained as the data basis for tasks such as data trend analysis, comparative analysis, classification statistics, traceability analysis, and relationship analysis.

[0172] In the above, the i-th data table to be inspected is uploaded according to the preset upload time T i Input into the preset data quality inspection platform, according to the quality inspection completion time W of the first i-1 data tables to be inspected i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0 , determine whether there is a conflict in quality inspection time between the i-th data table to be inspected and the i+1-th data table to be inspected, and obtain the quality inspection time conflict result, which provides a data basis for adjusting the quality inspection order of the data set to be inspected; according to the difference between the quality levels of historical data, obtain the second quality inspection order replacement degree B 1 i(i+1) , to characterize the degree to which the quality inspection order between the i-th data table to be inspected and the i+1-th data table to be inspected needs to be swapped, as the basis for adjusting the quality inspection order of the data table to be inspected, improving the quality inspection efficiency and real-time quality inspection of the data quality inspection platform; according to the quality inspection order swap result between the i-th data table to be inspected and the i+1-th data table to be inspected, the quality inspection of the data to be inspected is performed, and the quality inspection completion time W of the first i data tables to be inspected is obtained i , combined with the update method of i=i+1, a data basis is provided for obtaining the quality inspection time conflict result between the i+1th data table to be inspected and the i+2th data table to be inspected, and the continuous adjustment of the quality inspection order of the data tables to be inspected and the continuous quality inspection of the data tables to be inspected are realized, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform; by repeatedly executing S2-S4 until i=N is updated, the quality inspection order of N-1 data tables to be inspected is adjusted in sequence, and the quality inspection of N-1 data tables to be inspected is completed, and the quality inspection of the N-1 data tables to be inspected and the data tables to be inspected that are later in the order of the N-1 data tables to be inspected are further quality inspected, and the quality inspection of the N data tables to be inspected is completed, and the quality inspection of the N data tables to be inspected is completed, and the N problem data lists corresponding to the N data tables to be inspected are obtained, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform.

[0173] Embodiment 4

[0174] The fourth embodiment provides a rule screening device for data quality inspection, and the rule screening device for data quality inspection includes: Figure 4 As shown:

[0175] The data acquisition module 41 is used to obtain N data tables to be quality inspected, N verification data tables, a preset quality score range, a preset data quality inspection platform and Q candidate quality inspection rules to be screened in the data quality inspection platform corresponding to N target data sources, wherein the data quality inspection platform includes N preset quality inspection models, and the i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, i=1, 2..., N, N refers to the total number of data tables to be quality inspected.

[0176] The target quality inspection result acquisition module 42 is used to perform quality inspection on the vth verification data in the i-th verification data table according to the qth candidate quality inspection rule, and obtain the target quality inspection result W of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule. iv q , where v = 1, 2, ..., V(i), V(i) refers to the total amount of data in the i-th verification data table, and q = 1, 2, ..., Q, Q refers to the total number of preset quality inspection rules.

[0177] The target quality inspection label acquisition module 43 is used to obtain the target quality inspection label according to the target quality inspection result W iv q , get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q .

[0178] The data table traversal module 44 is used to traverse v=1, 2, ..., V(i) to obtain V(i) target quality inspection labels of the i-th verification data table relative to the q-th candidate quality inspection rule.

[0179] The verification data quality scoring module 45 is used to obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule according to V(i) target quality inspection labels. i q .

[0180] The target quality inspection model acquisition module 46 is used for i q Within the quality score range, the qth candidate quality inspection rule is added to the ith preset quality inspection model, and the ith target quality inspection model is obtained by traversing q=1, 2..., Q.

[0181] The data quality inspection module 47 is used to perform quality inspection on the i-th data table to be inspected by using the i-th target quality inspection model, and obtain the data quality level corresponding to the i-th data table to be inspected.

[0182] In a specific implementation, the target quality inspection result includes data having problems and data not having problems, and the target quality inspection label acquisition module 43 further includes:

[0183] The first label acquisition submodule is used to obtain the target quality inspection result W iv q If there is a problem with the data, the corresponding target quality inspection label R iv q is equal to the first preset value.

[0184] The second label acquisition submodule is used to obtain the target quality inspection result W iv q If there is no problem with the data, the corresponding target quality inspection label R iv q is equal to the second preset value.

[0185] In a specific embodiment, the verification data quality scoring module 45 further includes:

[0186] The first quantity acquisition submodule is used to obtain the third quantity F corresponding to the first preset value in the V(i) target quality inspection labels according to the V(i) target quality inspection labels. 3 i q The fourth quantity F corresponding to the second preset value 4 i q .

[0187] The validation data quality scoring submodule is used to evaluate the quality of the data according to the third quantity F 3 i q and the fourth quantity F 4 i q , obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule i q =F 4 i q / (F 3 i q +F 4 i q ).

[0188] In a specific implementation, the target quality inspection model acquisition module 46 further includes:

[0189] The first target quality inspection rule acquisition submodule is used to convert M in the i-th preset quality inspection model i A preset quality inspection rule is determined as the target quality inspection rule in the i-th target quality inspection model.

[0190] The second target quality inspection rule acquisition submodule is used if H i q Within the quality score range, the qth candidate quality inspection rule is determined as the target quality inspection rule in the ith target quality inspection model.

[0191] The target quality inspection model acquisition submodule is used to traverse q=1, 2..., Q to obtain the i-th target quality inspection model, wherein the i-th target quality inspection model includes Φ(i) target quality inspection rules.

[0192] In a specific implementation, the data quality inspection module 47 further includes:

[0193] The third quality inspection result acquisition submodule is used to perform quality inspection on the u-th data to be inspected in the i-th data table to be inspected according to the φ-th target quality inspection rule in the i-th target quality inspection model, and obtain the third quality inspection result E of the u-th data to be inspected in the i-th data table to be inspected relative to the φ-th target quality inspection rule in the i-th target quality inspection model. 3 iu φ , where u = 1, 2…, U(i), U(i) refers to the total amount of data in the i-th data table to be quality inspected, and φ = 1, 2…, Ф(i), Ф(i) refers to the total number of target quality inspection rules in the i-th target quality inspection model.

[0194] The first quality inspection label acquisition submodule is used to obtain the third quality inspection result E 3 iu φ If there is a problem with the data, the corresponding third quality inspection label R 3 iu φ is equal to the first preset value.

[0195] The second quality inspection label acquisition submodule is used to obtain the third quality inspection result E 3 iu φ If there is no problem with the data, the corresponding third quality inspection label R 3 iu φ is equal to the second preset value.

[0196] The second quantity acquisition submodule is used to traverse u=1, 2..., U(i) and φ=1, 2..., Φ(i) to obtain the fifth quantity F of the first preset value corresponding to the i-th quality inspection data table.5 i and the corresponding sixth number F of the second preset value 6 i .

[0197] The first data quality level acquisition submodule is used to obtain the data quality level according to F 5 i 、F 6 i and the preset quality level range, and obtain the data quality level corresponding to the i-th data table to be inspected.

[0198] In a specific implementation, the first data quality level acquisition submodule further includes:

[0199] The data acquisition unit is used to acquire α preset quality level ranges and the quality level corresponding to each preset quality level range, wherein the data quality level corresponding to the βth preset quality level range is β, β=1, 2..., α, α refers to the total number of preset quality level ranges.

[0200] The first data quality level acquisition unit is used to obtain the quality level of the data. 6 i / (F 5 i +F 6 i ) meets the β-th preset quality level range, then the data quality level corresponding to the i-th data table to be inspected is determined to be β.

[0201] In a specific implementation, the data quality inspection module 47 further includes:

[0202] The third quantity acquisition submodule is used to traverse u=1, 2..., U(i) to obtain the seventh quantity F of the first preset value of the φth target quality inspection rule in the i-th target quality inspection model for the i-th quality inspection data table. 7 i φ and the eighth number F of the second preset value 8 i φ .

[0203] The first quality scoring submodule is used to score the quality of the 7 i φ and the eighth quantity F 8 i φ , obtain the first target data quality score of the i-th data table to be inspected relative to the φ-th target quality inspection rule in the i-th target quality inspection model Ж i φ =F 8i φ / (F 7 i φ +F 8 i φ ).

[0204] The data table traversal submodule is used to traverse φ=1, 2..., Φ(i) to obtain all first target data quality scores of the i-th data table to be quality inspected relative to the i-th target quality inspection model.

[0205] The second quality scoring submodule is used to obtain the preset importance level corresponding to each target quality inspection rule, and obtain the second target data quality score σ corresponding to the i-th data table to be inspected according to all the first target data quality scores corresponding to the i-th data table to be inspected and the preset importance level corresponding to each target quality inspection rule. i , where σ i Meet the following conditions:

[0206] σ i =Σ φ Ф(i) (γ i φ × i φ ), where γ i φ It refers to the preset importance level corresponding to the φth target quality inspection rule in the ith target quality inspection model, i φ It refers to the target data quality score of the i-th data table to be quality inspected relative to the φ-th target quality inspection rule in the i-th target quality inspection model.

[0207] The data quality level acquisition submodule is used to obtain the data quality level according to σ i and the preset quality level range, and obtain the data quality level corresponding to the i-th data table to be inspected.

[0208] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0209] Embodiment 5

[0210] Embodiment 5 of the present invention provides a non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the steps:

[0211] S100, obtain N data tables to be quality inspected, N verification data tables, a preset quality score range, a preset data quality inspection platform and Q candidate quality inspection rules to be screened in the data quality inspection platform corresponding to N target data sources, wherein the data quality inspection platform includes N preset quality inspection models, and the i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, i=1, 2..., N, N refers to the total number of data tables to be quality inspected.

[0212] S200, perform quality inspection on the vth verification data in the i-th verification data table according to the qth candidate quality inspection rule, and obtain the target quality inspection result W of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule. iv q , where v = 1, 2, ..., V(i), V(i) refers to the total amount of data in the i-th verification data table, and q = 1, 2, ..., Q, Q refers to the total number of preset quality inspection rules.

[0213] S300, according to the target quality inspection result W iv q , get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q .

[0214] S400, traverse v=1, 2, ..., V(i), and obtain V(i) target quality inspection labels of the i-th verification data table corresponding to the q-th candidate quality inspection rule.

[0215] S500: Based on V(i) target quality inspection labels, obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule. i q .

[0216] S600, if H i q Within the quality score range, the qth candidate quality inspection rule is added to the ith preset quality inspection model, and the ith target quality inspection model is obtained by traversing q=1, 2..., Q.

[0217] S700, performing quality inspection on the i-th data table to be inspected by using the i-th target quality inspection model, and obtaining a data quality level corresponding to the i-th data table to be inspected.

[0218] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and volatile memory.

[0219] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0220] Embodiment 6

[0221] A sixth embodiment of the present invention provides an electronic device, which includes a processor and the non-transitory computer-readable storage medium in the fifth embodiment of the present invention.

[0222] The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A rule screening method for data quality inspection, characterized in that: The rule screening method for data quality inspection comprises the following steps: S100, obtaining N data tables to be quality inspected, N verification data tables, a preset quality score range, a preset data quality inspection platform, and Q candidate quality inspection rules to be screened in the data quality inspection platform corresponding to N target data sources, wherein the data quality inspection platform includes N preset quality inspection models, and the i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, i=1, 2..., N, where N refers to the total number of data tables to be quality inspected; S200, performing quality inspection on the vth verification data in the i-th verification data table according to the qth candidate quality inspection rule, and obtaining a target quality inspection result W of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule. iv q , where v=1, 2..., V(i), V(i) refers to the total amount of data in the i-th verification data table, and q=1, 2..., Q, Q refers to the total number of candidate quality inspection rules; S300, according to the target quality inspection result W iv q , obtain the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q ; S400, traverse v=1, 2, ..., V(i), and obtain V(i) target quality inspection labels of the i-th verification data table relative to the q-th candidate quality inspection rule; S500, based on the V(i) target quality inspection labels, obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule. i q ; S600, if H i q Within the preset quality score range, the qth candidate quality inspection rule is added to the ith preset quality inspection model, and the ith target quality inspection model is obtained by traversing q=1, 2..., Q; S700 , performing quality inspection on the i th data table to be inspected by using the i th target quality inspection model, and obtaining a data quality level corresponding to the i th data table to be inspected.

2. The rule screening method for data quality inspection according to claim 1, characterized in that: The target quality inspection result includes data having problems and data not having problems. S300 also includes the following steps: S310, if the target quality inspection result is W iv q If there is a problem with the data, the corresponding target quality inspection label R iv q is equal to a first preset value; S320, if the target quality inspection result is W iv q If there is no problem with the data, the corresponding target quality inspection label R iv q is equal to the second preset value.

3. The rule screening method for data quality inspection according to claim 2, characterized in that: S500 also includes the following steps: S510, according to the V(i) target quality inspection labels, obtain a third quantity F corresponding to the first preset value in the V(i) target quality inspection labels 3 i q The fourth quantity F corresponding to the second preset value 4 i q ; S520, according to the third quantity F 3 i q and the fourth quantity F 4 i q , obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule i q =F 4 i q / (F 3 i q +F 4 i q ).

4. The rule screening method for data quality inspection according to claim 1, characterized in that: The i-th preset quality inspection model includes M i The preset quality inspection rules, S600 also includes the following steps: S610: M in the i-th preset quality inspection model i a preset quality inspection rule, and determining it as the target quality inspection rule in the i-th target quality inspection model; S620, if H i q Within the quality score range, the qth candidate quality inspection rule is determined as the target quality inspection rule in the i-th target quality inspection model; S630, traverse q=1, 2, ..., Q, and obtain the i-th target quality inspection model, wherein the i-th target quality inspection model includes Φ(i) target quality inspection rules.

5. The rule screening method for data quality inspection according to claim 4, characterized in that: S700 also includes the following steps: S710, according to the φth target quality inspection rule in the i-th target quality inspection model, perform quality inspection on the u-th data to be quality inspected in the i-th data table to be quality inspected, and obtain a third quality inspection result E of the u-th data to be quality inspected in the i-th data table to be quality inspected relative to the φth target quality inspection rule in the i-th target quality inspection model. 3 iu φ , where u=1, 2..., U(i), U(i) refers to the total amount of data in the i-th data table to be inspected, φ=1, 2..., Ф(i), Ф(i) refers to the total number of target quality inspection rules in the i-th target quality inspection model; S720, if the third quality inspection result E 3 iu φ If there is a problem with the data, the corresponding third quality inspection label R 3 iu φ is equal to a first preset value; S730, if the third quality inspection result E 3 iu φ If there is no problem with the data, the corresponding third quality inspection label R 3 iu φ is equal to a second preset value; S740, traverse u=1, 2..., U(i) and φ=1, 2..., Φ(i), and obtain the fifth quantity F of the first preset value corresponding to the i-th data table to be inspected 5 i and the corresponding sixth number F of the second preset value 6 i ; S750, according to F 5 i 、F 6 i and the preset quality level range, and obtain the data quality level corresponding to the i-th data table to be inspected.

6. The rule screening method for data quality inspection according to claim 5, characterized in that: S750 also includes the following steps: S751, obtaining α preset quality level ranges and the quality level corresponding to each preset quality level range, wherein the data quality level corresponding to the β-th preset quality level range is β, β=1, 2, ..., α, α refers to the total number of preset quality level ranges; S752, if F 6 i / (F 5 i +F 6 i ) conforms to the β-th preset quality level range, then the data quality level corresponding to the i-th data table to be inspected is determined to be β.

7. The rule screening method for data quality inspection according to claim 5, characterized in that: S700 also includes the following steps: S760, traverse u=1, 2, ..., U(i), and obtain the seventh quantity F of the first preset value of the i-th to-be-inspected data table relative to the φ-th target quality inspection rule in the i-th target quality inspection model 7 i φ and the eighth number F of the second preset value 8 i φ ; S770, according to the seventh quantity F 7 i φ and the eighth quantity F 8 i φ , obtain the first target data quality score Ж of the i-th to-be-inspected data table relative to the φ-th target quality inspection rule in the i-th target quality inspection model i φ =F 8 i φ / (F 7 i φ +F 8 i φ ); S780, traverse φ=1, 2, ..., Φ(i), and obtain all first target data quality scores of the i-th data table to be quality inspected relative to the i-th target quality inspection model; S790, obtaining a preset importance corresponding to each target quality inspection rule, and obtaining a second target data quality score σ corresponding to the i-th data table to be quality inspected according to all first target data quality scores corresponding to the i-th data table to be quality inspected and the preset importance corresponding to each target quality inspection rule. i , where σ i Meet the following conditions: σ i =Σ φ Ф(i) (γ i φ × i φ ), where γ i φ It refers to the preset importance level corresponding to the φth target quality inspection rule in the ith target quality inspection model, i φ It refers to the target data quality score of the i-th data table to be quality inspected relative to the φ-th target quality inspection rule in the i-th target quality inspection model; S791, according to σ i and the preset quality level range, and obtain the data quality level corresponding to the i-th data table to be inspected.

8. A rule screening device for data quality inspection, characterized in that: The rule screening device for data quality inspection comprises: A data acquisition module is used to acquire N data tables to be inspected, N verification data tables, a preset quality score range, a preset data quality inspection platform, and Q candidate quality inspection rules to be screened in the data quality inspection platform corresponding to N target data sources, wherein the data quality inspection platform includes N preset quality inspection models, and the i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, i=1, 2..., N, N refers to the total number of data tables to be inspected; The target quality inspection result acquisition module is used to perform quality inspection on the vth verification data in the ith verification data table according to the qth candidate quality inspection rule, and obtain the target quality inspection result W of the vth verification data in the ith verification data table relative to the qth candidate quality inspection rule. iv q , where v=1, 2..., V(i), V(i) refers to the total amount of data in the i-th verification data table, and q=1, 2..., Q, Q refers to the total number of candidate quality inspection rules; The target quality inspection label acquisition module is used to obtain the target quality inspection label based on the target quality inspection result W iv q , obtain the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q ; A data table traversal module, used to traverse v=1, 2..., V(i), and obtain V(i) target quality inspection labels of the i-th verification data table relative to the q-th candidate quality inspection rule; A verification data quality scoring module is used to obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule according to the V(i) target quality inspection labels. i q ; Target quality inspection model acquisition module, used for H i q Within the preset quality score range, the qth candidate quality inspection rule is added to the ith preset quality inspection model, and the ith target quality inspection model is obtained by traversing q=1, 2..., Q; The data quality inspection module is used to perform quality inspection on the i-th data table to be inspected through the i-th target quality inspection model, and obtain the data quality level corresponding to the i-th data table to be inspected.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the rule screening method for data quality inspection as described in any one of claims 1-7.

10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images