Data quality inspection method, device, medium and equipment

Abstract

The present invention relates to the field of data processing, and in particular to a data quality inspection method, device, medium and equipment, comprising: inputting a data table to be inspected into a data quality inspection platform, judging whether there is a quality inspection time conflict between the data tables to be inspected according to the quality inspection completion time, predicted quality inspection time, upload time and a preset time interval threshold of the data table to be inspected, and obtaining a quality inspection time conflict result; if the quality inspection time conflict result is a quality inspection time conflict, obtaining a quality inspection sequence replacement result to perform quality inspection on the data to be inspected according to the difference between the quality levels of historical data, and obtaining the quality inspection completion time of the previous i data tables to be inspected, and combining with an update method of i=i+1, providing a data basis for obtaining the next quality inspection time conflict result, realizing continuous adjustment of the quality inspection sequence of the data tables to be inspected and continuous quality inspection of the data tables to be inspected, thereby improving the quality inspection efficiency and real-time quality inspection of the data quality inspection platform.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data quality inspection method, device, medium and equipment. Background Art

[0002] Data quality inspection can measure the health of data in multiple dimensions such as accuracy, completeness, consistency, and reliability, and provide strong support for tasks such as data trend analysis, comparative analysis, classification statistics, traceability analysis, and relationship analysis.

[0003] In the prior art, the quality inspection method for big data is usually to perform sequential quality inspection on the data in the uploaded data table to be inspected according to the time when the data table to be inspected is uploaded to the data quality inspection platform through multiple quality inspection rules bound in the data quality inspection platform, find out the problematic data and mark them, and finally return the data quality inspection results to characterize the health status of the data in the data table to be inspected.

[0004] However, the data quality of different data tables to be quality checked is different. When the upload time of different data tables to be quality checked is close, and the data quality of the data tables to be quality checked uploaded first is poor, the data quality inspection platform will spend too much time on low-quality data tables to be quality checked, and will not be able to perform quality inspection on high-quality data tables to be quality checked in time, resulting in low efficiency and real-time performance of the data quality inspection platform, making it difficult to meet the real-time quality inspection requirements of the data.

[0005] Therefore, how to improve the quality inspection efficiency and real-time performance of the data quality inspection platform has become an urgent problem to be solved. Summary of the invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is a data quality inspection method, which includes the following steps:

[0007] S1, obtain N data tables to be inspected, the predicted inspection time of each data table to be inspected, and the historical data quality level corresponding to each data table to be inspected, and upload the i-th data table to be inspected according to the preset upload time T i Input into the preset data quality inspection platform, i=1, 2..., N, N refers to the total number of data tables to be inspected.

[0008] S2, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0, obtain the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, where W0=0, and the quality inspection time conflict result includes quality inspection time conflict and quality inspection time non-conflict.

[0009] S3. If the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be quality inspected and the historical data quality level of the i+1-th data table to be quality inspected, the quality inspection order replacement result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained, wherein the quality inspection order replacement result includes replacing the quality inspection order and not replacing the quality inspection order.

[0010] S4: According to the quality inspection order swap result between the i-th data table to be inspected and the i+1-th data table to be inspected, the data quality inspection platform performs quality inspection on the i-th data table to be inspected and the data table to be inspected that is earlier in the order of the i+1-th data table to be inspected, and obtains the problem data list of the i-th data table to be inspected and the data table to be inspected that is earlier in the order of the i+1-th data table to be inspected, as well as the quality inspection completion time W of the first i data tables to be inspected. i , update i=i+1.

[0011] S5, repeat S2-S4 until i=N is updated, and the N-1th data table to be inspected and the subsequent data table to be inspected in the Nth data table to be inspected are quality inspected through the data quality inspection platform to obtain N problem data lists corresponding to the N data tables to be inspected.

[0012] The present invention also provides a data quality inspection device, which includes:

[0013] The first data acquisition module is used to obtain N data tables to be inspected, the predicted quality inspection time of each data table to be inspected, and the historical data quality level corresponding to each data table to be inspected, and upload the i-th data table to be inspected according to the preset upload time T i Input into the preset data quality inspection platform, i=1, 2..., N, N refers to the total number of data tables to be inspected.

[0014] The quality inspection time conflict analysis module is used to calculate the quality inspection completion time W of the first i-1 quality inspection data tables. i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0, obtain the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, where W0=0, and the quality inspection time conflict result includes quality inspection time conflict and quality inspection time non-conflict.

[0015] The first quality inspection sequence replacement module is used to obtain the quality inspection sequence replacement result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected according to the historical data quality level of the i-th data table to be quality inspected and the historical data quality level of the i+1-th data table to be quality inspected if the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, wherein the quality inspection sequence replacement result includes replacing the quality inspection sequence and not replacing the quality inspection sequence.

[0016] The first data quality inspection module is used to perform quality inspection on the i-th data table to be quality inspected and the data table to be quality inspected that is earlier in the i+1-th data table to be quality inspected according to the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected through the data quality inspection platform, and obtain the problem data list of the i-th data table to be quality inspected and the data table to be quality inspected that is earlier in the i+1-th data table to be quality inspected, as well as the quality inspection completion time W of the first i data tables to be quality inspected i , update i=i+1.

[0017] The second data quality inspection module is used to repeatedly execute S2-S4 until i=N is updated, and to perform quality inspection on the N-1th data table to be quality inspected and the data table to be quality inspected that is subsequent in sequence in the Nth data table to be quality inspected through the data quality inspection platform, and obtain N problem data lists corresponding to the N data tables to be quality inspected.

[0018] The present invention also provides a non-transitory computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned data quality inspection method.

[0019] The present invention also provides an electronic device, comprising a processor and the above-mentioned non-transitory computer-readable storage medium.

[0020] The present invention has at least the following beneficial effects: the i-th data table to be inspected is uploaded according to the preset upload time T i Input into the preset data quality inspection platform, according to the quality inspection completion time W of the first i-1 data tables to be inspected i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0, determine whether there is a conflict in quality inspection time between the i-th data table to be inspected and the i+1-th data table to be inspected, and obtain the quality inspection time conflict result, which provides a data basis for adjusting the quality inspection order of the data set to be inspected; according to the difference between the quality levels of historical data, obtain the second quality inspection order replacement degree B 1 i(i+1) , to characterize the degree to which the quality inspection order between the i-th data table to be inspected and the i+1-th data table to be inspected needs to be swapped, as the basis for adjusting the quality inspection order of the data table to be inspected, improving the quality inspection efficiency and real-time quality inspection of the data quality inspection platform; according to the quality inspection order swap result between the i-th data table to be inspected and the i+1-th data table to be inspected, the quality inspection of the data to be inspected is performed, and the quality inspection completion time W of the first i data tables to be inspected is obtained i , combined with the update method of i=i+1, a data basis is provided for obtaining the quality inspection time conflict result between the i+1th data table to be inspected and the i+2th data table to be inspected, and the continuous adjustment of the quality inspection order of the data tables to be inspected and the continuous quality inspection of the data tables to be inspected are realized, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform; by repeatedly executing S2-S4 until i=N is updated, the quality inspection order of N-1 data tables to be inspected is adjusted in sequence, and the quality inspection of N-1 data tables to be inspected is completed, and the quality inspection of the N-1 data tables to be inspected and the data tables to be inspected that are later in the order of the N-1 data tables to be inspected are further quality inspected, and the quality inspection of the N data tables to be inspected is completed, and the quality inspection of the N data tables to be inspected is completed, and the N problem data lists corresponding to the N data tables to be inspected are obtained, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of a data quality inspection method provided in Embodiment 1 of the present invention;

[0023] Figure 2 A flow chart of a data quality inspection method based on data sampling provided in Embodiment 2 of the present invention;

[0024] Figure 3 A flowchart of a rule screening method for data quality inspection provided in Embodiment 3 of the present invention;

[0025] Figure 4 A schematic diagram of the structure of a data quality inspection device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Embodiment 1

[0029] This embodiment provides a data quality inspection method, which includes the following steps: Figure 1 As shown:

[0030] S1, obtain N data tables to be inspected, the predicted inspection time of each data table to be inspected, and the historical data quality level corresponding to each data table to be inspected, and upload the i-th data table to be inspected according to the preset upload time T i Input into the preset data quality inspection platform, i=1, 2..., N, N refers to the total number of data tables to be inspected.

[0031] The data table to be inspected refers to a data table that needs to be quality inspected and is provided by the target data source. The data table to be inspected may include structured data or unstructured data. The target data source may be selected based on actual conditions.

[0032] For example, the target data source may refer to the comprehensive police application platforms at provincial, ministerial and other levels, and the data table to be inspected may refer to the data collected by each comprehensive police application platform. By conducting quality inspection on the data collected by each comprehensive police application platform, comprehensive information analysis support such as case analysis, personnel analysis, early warning analysis, data collision, comprehensive query, trajectory analysis, frequency analysis, and full-text retrieval can be provided for each type of police.

[0033] Predicting quality inspection time refers to predicting the quality inspection time required when quality inspection is performed on each data table to be inspected based on the data quality inspection platform. The quality inspection time can be predicted manually or through a trained prediction model, which will not be elaborated here.

[0034] The quality level of historical data can be obtained by statistics and analysis of the quality inspection results of the historical quality inspection data of the corresponding target data source. Correspondingly, the fewer problematic data in the historical quality inspection data, the higher the quality level of the historical data corresponding to the to-be-inspected data table corresponding to the target data source.

[0035] The data table to be quality inspected is uploaded to the preset data quality inspection platform according to the preset upload time. The data quality inspection platform has a built-in quality inspection model, and a number of quality inspection rules are bound to the quality inspection model. They are used to perform quality inspection on the input data to be quality inspected in multiple dimensions, find out the problematic data in the data table to be quality inspected, and finally output a list of problematic data corresponding to each data table to be quality inspected.

[0036] Among them, specific quality inspection rules can be set by the implementer according to actual conditions. For example, the quality inspection rules may include integrity rule checks such as field missing check, data null value check, data source check, and format rule checks such as ID card number format check, mobile phone number format check, date, time format check, numeric format check, number format check, and value domain rule checks such as dictionary value range, dictionary name range, date type range, numeric type range, and normative format checks such as date of birth and ID card check, gender and ID card check, age and ID card check.

[0037] In a specific implementation, obtaining the predicted quality inspection time of each data table to be inspected further includes the following steps:

[0038] S01, obtaining the data volume of each data table to be quality inspected and the quality inspection speed of the data quality inspection platform.

[0039] S02: Determine the ratio of the data volume and the quality inspection speed of each data table to be quality inspected as the predicted quality inspection time of the corresponding data table to be quality inspected.

[0040] Among them, the quality inspection speed of the data quality inspection platform can be obtained by counting and calculating the historical quality inspection data volume and historical quality inspection time of the data quality inspection platform.

[0041] S2, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1and the preset time interval threshold T 0 , obtain the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, where W0=0, and the quality inspection time conflict result includes quality inspection time conflict and quality inspection time non-conflict.

[0042] In a specific implementation, S2 further includes the following steps:

[0043] S21, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , get the predicted quality inspection completion time C of the i-th quality inspection data table i =W i-1 +T i +Y i .

[0044] S22, according to the upload time T of the i+1th quality inspection data table i+1 and the predicted quality inspection completion time C of the i-th quality inspection data table i , obtain the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) =T i+1 -C i .

[0045] S23, if G i(i+1) >T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection time does not conflict.

[0046] S24, if G i(i+1) ≤T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as the quality inspection time conflict.

[0047] Among them, T 0 The specific value can be set by the implementer according to the actual situation.

[0048] As mentioned above, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0, determine whether there is a quality inspection time conflict between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, and obtain the quality inspection time conflict result, which provides a data basis for adjusting the quality inspection order of the data sets to be quality inspected.

[0049] S3. If the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be quality inspected and the historical data quality level of the i+1-th data table to be quality inspected, the quality inspection order replacement result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained, wherein the quality inspection order replacement result includes replacing the quality inspection order and not replacing the quality inspection order.

[0050] Among them, when the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, if the historical data quality level of the i-th data to be quality inspected is lower than the historical data quality level of the i+1-th data table to be quality inspected, it is necessary to determine whether the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected needs to be exchanged, and obtain the quality inspection order exchange result as the basis for quality inspection of the data table to be quality inspected through the data quality inspection platform.

[0051] In a specific implementation, S3 further includes the following steps:

[0052] S32, according to the historical data quality level D of the i-th data table to be inspected i and the quality level D of the historical data of the i+1th data table to be inspected i+1 , obtain the second quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 2 i(i+1) =(2 / (1+e^(D i -D i+1 )))-1.

[0053] S33, if B 2 i(i+1) >B 2 , then the quality inspection order exchange result between the i-th quality inspection data table and the i+1-th quality inspection data table is obtained as the quality inspection order exchange, where B 2 Refers to the preset second quality inspection order replacement degree threshold.

[0054] S34, if B 2 i(i+1) ≤B 2 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged.

[0055] Among them, the degree of the second quality inspection order exchange between the i-th data table to be inspected and the i+1-th data table to be inspected is related to the historical data quality level D of the i+1-th data table to be inspected. i+1 The quality level D of the historical data of the i-th data table to be inspected i The difference between i+1 -D i Positively correlated.

[0056] As mentioned above, based on the difference between the quality levels of historical data, the second quality inspection order replacement degree B is obtained 1 i(i+1) , to characterize the degree to which the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected needs to be swapped, which serves as the basis for adjusting the quality inspection order of the data tables to be quality inspected, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform.

[0057] In a specific implementation, the data quality inspection method further includes the following steps:

[0058] S31, if the quality inspection time conflict result between the i-th data table to be inspected and the i+1-th data table to be inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be inspected, the historical data quality level of the i+1-th data table to be inspected, the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) , preset time interval threshold T 0 The preset first quality inspection sequence exchange degree threshold B 1 , obtain the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected.

[0059] In a specific implementation, S31 further includes the following steps:

[0060] S311, according to the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) , and the preset time interval threshold T 0 , obtain the degree of overlap of quality inspection time between the i-th quality inspection data table and the i+1-th quality inspection data table i(i+1) =(T 0 -G i(i+1) ) / T 0 .

[0061] S312, based on the historical data quality level D of the i-th data table to be inspected i 、The quality level of historical data in the i+1th data table to be inspected is D i+1 Overlap degree with quality inspection time A i(i+1), obtain the first quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 1 i(i+1) =A i(i+1) ×((2 / (1+e^(D i -D i+1 )))-1).

[0062] S313, if B 1 i(i+1) >B 1 , then the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as the exchanged quality inspection order.

[0063] S314, if B 1 i(i+1) ≤B 1 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged.

[0064] Among them, based on the quality level gap of historical data, combined with the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) and the preset time interval threshold T 0 , the degree of overlap of the quality inspection time between the i-th data table to be inspected and the i+1-th data table to be inspected is measured, then the degree of exchange of the first quality inspection order between the i-th data table to be inspected and the i+1-th data table to be inspected is related to the historical data quality level D of the i+1-th data table to be inspected i+1 The quality level D of the historical data of the i-th data table to be inspected i The difference between i+1 -D i Positively correlated with the quality inspection time overlap degree A i(i+1) is positively correlated, thereby obtaining the first quality inspection order replacement degree B 1 i(i+1) , to represent the degree to which the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected needs to be swapped.

[0065] The above-mentioned, comprehensive quality inspection time overlap degree and the difference between the historical data quality levels, obtain the first quality inspection sequence replacement degree B 1 i(i+1) , to characterize the degree to which the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected needs to be swapped, which serves as the basis for adjusting the quality inspection order of the data tables to be quality inspected, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform.

[0066] S4: According to the quality inspection order swap result between the i-th data table to be inspected and the i+1-th data table to be inspected, the data quality inspection platform performs quality inspection on the i-th data table to be inspected and the data table to be inspected that is earlier in the order of the i+1-th data table to be inspected, and obtains the problem data list of the i-th data table to be inspected and the data table to be inspected that is earlier in the order of the i+1-th data table to be inspected, as well as the quality inspection completion time W of the first i data tables to be inspected. i , update i=i+1.

[0067] The specific content of the problem data list can be set by the implementer according to the actual situation. For example, the problem data list can include the problem data detected by quality inspection, and can also include the problem data, the business table corresponding to the problem data, the business component corresponding to the problem data, the quality inspection rules for detecting the problem data, and other dimensional contents.

[0068] When the quality inspection of the i-th data table to be inspected and the i+1-th data table to be inspected that is the previous one in sequence is completed, the corresponding completion time is obtained as the quality inspection completion time W of the first i data tables to be inspected i , and update i=i+1, thereby serving as the data basis for obtaining the quality inspection time conflict result between the i+1th data table to be quality inspected and the i+2th data table to be quality inspected.

[0069] In a specific implementation, S4 further includes the following steps:

[0070] S41, if the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is to exchange the quality inspection order, then determine the i+1-th data table to be quality inspected as the data table to be quality inspected that comes earlier in order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected.

[0071] S42, if the result of exchanging the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is not to change the quality inspection order, then determine that the i-th data table to be quality inspected is the data table to be quality inspected that comes earlier in order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected.

[0072] In the above, the quality inspection of the data to be inspected is performed according to the quality inspection order exchange result between the i-th data table to be inspected and the i+1-th data table to be inspected, and the quality inspection completion time W of the first i data tables to be inspected is obtained. i , combined with the update method of i=i+1, it provides a data basis for obtaining the quality inspection time conflict results between the i+1th data table to be quality inspected and the i+2th data table to be quality inspected, realizes the continuous adjustment of the quality inspection order of the data tables to be quality inspected and the continuous quality inspection of the data tables to be quality inspected, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform.

[0073] S5, repeat S2-S4 until i=N is updated, and the N-1th data table to be inspected and the subsequent data table to be inspected in the Nth data table to be inspected are quality inspected through the data quality inspection platform to obtain N problem data lists corresponding to the N data tables to be inspected.

[0074] In the above, by repeatedly executing S2-S4 until i=N is updated, the quality inspection order of N-1 data tables to be quality inspected is adjusted in sequence, and the quality inspection of N-1 data tables to be quality inspected is completed. Further quality inspection is performed on the N-1 data table to be quality inspected and the data table to be quality inspected that is later in the N data table to be quality inspected, and the quality inspection of N data tables to be quality inspected is completed. N problem data lists corresponding to the N data tables to be quality inspected are obtained as the data basis for tasks such as data trend analysis, comparative analysis, classification statistics, traceability analysis, and relationship analysis.

[0075] In the above, the i-th data table to be inspected is uploaded according to the preset upload time T i Input into the preset data quality inspection platform, according to the quality inspection completion time W of the first i-1 data tables to be inspected i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0 , determine whether there is a conflict in quality inspection time between the i-th data table to be inspected and the i+1-th data table to be inspected, and obtain the quality inspection time conflict result, which provides a data basis for adjusting the quality inspection order of the data set to be inspected; according to the difference between the quality levels of historical data, obtain the second quality inspection order replacement degree B 1 i(i+1) , to characterize the degree to which the quality inspection order between the i-th data table to be inspected and the i+1-th data table to be inspected needs to be swapped, as the basis for adjusting the quality inspection order of the data table to be inspected, improving the quality inspection efficiency and real-time quality inspection of the data quality inspection platform; according to the quality inspection order swap result between the i-th data table to be inspected and the i+1-th data table to be inspected, the quality inspection of the data to be inspected is performed, and the quality inspection completion time W of the first i data tables to be inspected is obtained i, combined with the update method of i=i+1, a data basis is provided for obtaining the quality inspection time conflict result between the i+1th data table to be inspected and the i+2th data table to be inspected, and the continuous adjustment of the quality inspection order of the data tables to be inspected and the continuous quality inspection of the data tables to be inspected are realized, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform; by repeatedly executing S2-S4 until i=N is updated, the quality inspection order of N-1 data tables to be inspected is adjusted in sequence, and the quality inspection of N-1 data tables to be inspected is completed, and the quality inspection of the N-1 data tables to be inspected and the data tables to be inspected that are later in the order of the N-1 data tables to be inspected are further quality inspected, and the quality inspection of the N data tables to be inspected is completed, and the quality inspection of the N data tables to be inspected is completed, and the N problem data lists corresponding to the N data tables to be inspected are obtained, thereby improving the quality inspection efficiency and real-time performance of the data quality inspection platform.

[0076] Embodiment 2

[0077] The second embodiment provides a data quality inspection method based on data sampling, and the data quality inspection method based on data sampling includes: Figure 2 As shown:

[0078] S10, obtaining N data tables to be inspected corresponding to N target data sources, the historical data quality level corresponding to each data table to be inspected, and a preset data quality score threshold, wherein N refers to the total number of data tables to be inspected.

[0079] Among them, the specific value of the preset data quality score threshold can be set by the implementer according to actual conditions.

[0080] S20, sampling each data table to be inspected, and obtaining the i-th sampling data table corresponding to the i-th data table to be inspected, where i=1, 2, ..., N.

[0081] In a specific implementation, S20 further includes the following steps:

[0082] S201, according to the historical data quality level D corresponding to the i-th data table to be inspected i and the preset initial sampling ratio L 0 , obtain the target sampling ratio L corresponding to the i-th quality inspection data table i =L 0 ×((2 / (1+e^(-D i )))-1).

[0083] S202, obtaining the data volume of the ith data table to be inspected, and performing a sampling operation based on the data volume of the ith data table to be inspected and the corresponding target sampling ratio L i , obtain the sampling data volume corresponding to the i-th data table to be inspected.

[0084] S203, extracting data from the i-th data table to be inspected according to the amount of sampling data corresponding to the i-th data table to be inspected, to form an i-th sampling data table corresponding to the i-th data table to be inspected.

[0085] Among them, the historical data quality level D corresponding to the i-th data table to be inspected i The larger the value is, the more problematic data there may be in the i-th data table to be inspected. In order to better measure the problem status of the i-th data table to be inspected, more data needs to be extracted for quality inspection. Therefore, the historical data quality level D corresponding to the i-th data table to be inspected is i and the target sampling ratio L i There is a positive correlation.

[0086] The preset initial sampling ratio L 0 The specific value can be set by the implementer according to the actual situation.

[0087] According to the amount of sampling data corresponding to the i-th data table to be inspected, data can be randomly selected from the i-th data table to be inspected to form the i-th sampling data table corresponding to the i-th data table to be inspected.

[0088] In the above, each data table to be inspected is sampled to obtain the i-th sampling data table corresponding to the i-th data table to be inspected, which provides a data basis for measuring the data quality of the i-th data table to be inspected and improves the efficiency of measuring the data quality of the i-th data table to be inspected.

[0089] S30, uploading the i-th sampling data table to the preset data quality inspection platform, performing quality inspection on the i-th sampling data table through the i-th preset quality inspection model in the data quality inspection platform, and obtaining the i-th sampling quality inspection result table corresponding to the i-th sampling data table.

[0090] In a specific implementation, the data quality inspection platform includes N preset quality inspection models, wherein the i-th preset quality inspection model includes M i The i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, and S30 further includes the following steps:

[0091] S301, according to the jth preset quality inspection rule in the i-th preset quality inspection model, perform quality inspection on the kth sampled data in the i-th sampled data table, and obtain a first quality inspection result E of the kth sampled data in the i-th sampled data table relative to the jth preset quality inspection rule in the i-th preset quality inspection model 1 ik j , where k = 1, 2, ..., K(i), K(i) refers to the total amount of data in the i-th sampling data table, j = 1, 2, ..., M i , Mi It refers to the total number of preset quality inspection rules in the i-th preset quality inspection model.

[0092] S302, according to the first quality inspection result E 1 ik j , obtain the first quality inspection label R of the kth sample data in the i-th sample data table relative to the j-th preset quality inspection rule in the i-th preset quality inspection model 1 ik j .

[0093] S303, the first quality inspection label R 1 ik j As the data value of the kth row and jth column in the i-th sampling quality inspection result table.

[0094] S304, traverse k=1, 2..., K(i) and j=1, 2..., M i , forming the i-th sampling quality inspection result table.

[0095] In a specific implementation, S302 further includes the following steps:

[0096] S3021, if the first quality inspection result is E 1 ik j If there is a problem with the data, the corresponding first quality inspection label R 1 ik j is equal to the first preset value.

[0097] S3022, if the first quality inspection result is E 1 ik j If there is no problem with the data, the corresponding first quality inspection label R 1 ik j is equal to the second preset value.

[0098] The specific values ​​of the first preset value and the second preset value can be set by the implementer according to the actual situation. For example, in this embodiment, the first preset value can be set to 1 and the second preset value can be set to 0.

[0099] In the above, the i-th sampling data table is quality inspected through the i-th preset quality inspection model in the data quality inspection platform, and the i-th sampling quality inspection result table corresponding to the i-th sampling data table is obtained, which provides a data basis for measuring the data quality of the i-th data table to be quality inspected, and improves the accuracy of measuring the data quality of the i-th data table to be quality inspected.

[0100] S40, obtaining the i-th data quality score corresponding to the i-th data table to be quality inspected according to the historical data quality level corresponding to the i-th data table to be quality inspected and the i-th sampling quality inspection result table.

[0101] In a specific implementation, S40 further includes the following steps:

[0102] S401: According to the i-th sampling quality inspection result table, obtain the first quantity F corresponding to the first preset value in the i-th sampling quality inspection result table. 1 i The second number F corresponding to the second preset value 2 i .

[0103] S402, according to the second quantity F 2 i and the first quantity F 1 i , obtain the sampling data quality score P corresponding to the i-th data table to be inspected 1 i =F 2 i / (F 1 i +F 2 i ).

[0104] S403, according to the historical data quality level D corresponding to the i-th data table to be inspected i and the sampling data quality score P corresponding to the i-th data table to be inspected 1 i , obtain the i-th data quality score P corresponding to the i-th data table to be inspected 2 i =P 1 i ×((2 / (1+e^(-D i )))-1).

[0105] The second number F corresponding to the second preset value is 2 i The larger the proportion, the smaller the proportion of problematic data in the i-th sampling data table, which in turn means that the smaller the proportion of problematic data in the i-th data table to be inspected, and correspondingly, the higher the sampling data quality score corresponding to the i-th data table to be inspected.

[0106] Above, through historical data quality level D i The data quality of the target data source corresponding to the i-th data table to be inspected is represented by the sample data quality score P 1 iCharacterize the actual data quality of the i-th data table to be inspected, and combine it with the historical data quality level D i and sampling data quality score P 1 i Get the data quality score P corresponding to the i-th data table to be inspected 2 i , improving the data quality score P 2 i accuracy.

[0107] S50: judging whether the i-th data table to be quality inspected meets the preset quality inspection standard according to the i-th data quality score and the data quality score threshold.

[0108] Among them, if the i-th data quality score is greater than the data quality score threshold, then the i-th data table to be inspected is judged to meet the preset quality inspection standard. If the i-th data quality score is less than or equal to the data quality score threshold, then the i-th data table to be inspected is judged to not meet the preset quality inspection standard.

[0109] The specific value of the data quality score threshold can be set by the implementer according to actual conditions.

[0110] S60: If the i-th data table to be inspected does not meet the preset quality inspection standard, the i-th data table to be inspected is returned to the corresponding i-th target data source.

[0111] Among them, the target data source can manage problems on the returned data table to be inspected, and upload the managed data table to the data quality inspection platform again as the data table to be inspected, so as to conduct another sampling quality inspection on the managed data table, obtain the corresponding data quality score, and again judge whether the managed data table meets the preset quality inspection standards, and perform corresponding operations based on the judgment results.

[0112] S70, if the i-th data table to be quality inspected meets the preset quality inspection standard, the i-th data table to be quality inspected is uploaded to the data quality inspection platform, and the i-th data table to be quality inspected is quality inspected by the i-th preset quality inspection model, and the quality inspection result table corresponding to the i-th data table to be quality inspected is obtained.

[0113] In a specific implementation, S70 further includes the following steps:

[0114] S701, according to the jth preset quality inspection rule in the i-th preset quality inspection model, perform quality inspection on the u-th data to be inspected in the i-th data table to be inspected, and obtain the second quality inspection result E of the u-th data to be inspected in the i-th data table to be inspected relative to the j-th preset quality inspection rule in the i-th preset quality inspection model 2 iu j, where u = 1, 2, ..., U(i), U(i) refers to the total amount of data in the i-th data table to be inspected.

[0115] S702, if the second quality inspection result is E 2 iu j If there is a problem with the data, the corresponding second quality inspection label R 2 iu j is equal to the first preset value.

[0116] S703, if the second quality inspection result is E 2 iu j If there is no problem with the data, the corresponding second quality inspection label R 2 iu j is equal to the second preset value.

[0117] S704, the second quality inspection label R 2 iu j As the data value of the u-th row and j-th column in the i-th quality inspection result table.

[0118] S705, traverse u=1, 2, ..., U(i) and j=1, 2, ..., M i , forming the i-th quality inspection result table corresponding to the i-th data table to be inspected.

[0119] In the above, each data table to be inspected is sampled, and the i-th sampling data table corresponding to the i-th data table to be inspected is obtained, which provides a data basis for measuring the data quality of the i-th data table to be inspected, and improves the efficiency of measuring the data quality of the i-th data table to be inspected; the i-th sampling data table is quality inspected by the i-th preset quality inspection model in the data quality inspection platform, and the i-th sampling quality inspection result table corresponding to the i-th sampling data table is obtained, which provides a data basis for measuring the data quality of the i-th data table to be inspected, and improves the accuracy of measuring the data quality of the i-th data table to be inspected; through the historical data quality level D i Characterize the data quality of the target data source corresponding to the i-th data table to be inspected, characterize the data quality of the i-th data table to be inspected actually corresponding to the i-th data table to be inspected through the i-th sampling quality inspection result, and combine it with the historical data quality level D iThe data quality score corresponding to the ith data table to be inspected is obtained from the ith sampling quality inspection result table, thereby improving the accuracy of the data quality score; if the ith data table to be inspected does not meet the preset quality inspection standard, the ith data table to be inspected is returned to the corresponding ith target data source; if the ith data table to be inspected meets the preset quality inspection standard, the ith data table to be inspected is uploaded to the data quality inspection platform, and the ith data table to be inspected is quality inspected by the ith preset quality inspection model, and the quality inspection result table corresponding to the ith data table to be inspected is obtained. The data tables to be inspected are screened by the data quality score, low-quality data are reworked, and high-quality data are quality inspected, thereby reducing the quality inspection pressure of the data quality inspection platform and improving the quality inspection efficiency of the data quality inspection platform.

[0120] Embodiment 3

[0121] The third embodiment provides a rule screening method for data quality inspection, and the rule screening method for data quality inspection includes: Figure 3 As shown:

[0122] S100, obtain N data tables to be quality inspected, N verification data tables, a preset quality score range, a preset data quality inspection platform and Q candidate quality inspection rules to be screened in the data quality inspection platform corresponding to N target data sources, wherein the data quality inspection platform includes N preset quality inspection models, and the i-th preset quality inspection model is used to perform quality inspection on the data corresponding to the i-th target data source, i=1, 2..., N, N refers to the total number of data tables to be quality inspected.

[0123] Among them, the candidate quality inspection rules to be screened are newly added quality inspection rules for data quality inspection. Before adding the candidate quality inspection rules to the data quality inspection platform, it is necessary to measure the quality inspection effects of the candidate quality inspection rules on the data corresponding to each target data source, and determine which target data sources the candidate quality inspection rules can be added to the quality inspection models of the data. This can improve the richness of the quality inspection rules of the data quality inspection platform while improving the quality inspection accuracy of the data quality inspection platform.

[0124] The verification data table may refer to a data table that has been quality inspected by manual methods or other methods. The quality inspection results corresponding to the verification data table may be obtained, and then the corresponding quality score range may be obtained to verify the quality inspection effectiveness of the candidate quality inspection rules for the data corresponding to each target data source.

[0125] S200, perform quality inspection on the vth verification data in the i-th verification data table according to the qth candidate quality inspection rule, and obtain the target quality inspection result W of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule. iv q, where v = 1, 2, ..., V(i), V(i) refers to the total amount of data in the i-th verification data table, and q = 1, 2, ..., Q, Q refers to the total number of preset quality inspection rules.

[0126] S300, according to the target quality inspection result W iv q , get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q .

[0127] In a specific implementation manner, the target quality inspection result includes data having problems and data not having problems, and S300 further includes the following steps:

[0128] S310, if the target quality inspection result is W iv q If there is a problem with the data, the corresponding target quality inspection label R iv q is equal to the first preset value.

[0129] S320, if the target quality inspection result is W iv q If there is no problem with the data, the corresponding target quality inspection label R iv q is equal to the second preset value.

[0130] S400, traverse v=1, 2, ..., V(i), and obtain V(i) target quality inspection labels of the i-th verification data table corresponding to the q-th candidate quality inspection rule.

[0131] S500: Based on V(i) target quality inspection labels, obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule. i q .

[0132] In a specific implementation, S500 further includes the following steps:

[0133] S510, according to V(i) target quality inspection labels, obtain a third quantity F corresponding to the first preset value in V(i) target quality inspection labels 3 i q The fourth quantity F corresponding to the second preset value 4 i q .

[0134] S520, according to the third quantity F 3 i q and the fourth quantity F 4i q , obtain the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule i q =F 4 i q / (F 3 i q +F 4 i q ).

[0135] Among them, the fourth quantity F corresponding to the second preset value 4 i q The larger the proportion of , the smaller the proportion of problematic data in the i-th verification data table relative to the q-th candidate quality inspection rule, which means that the verification data quality score of the i-th verification data table relative to the q-th candidate quality inspection rule is higher.

[0136] As mentioned above, based on V(i) target quality inspection labels, the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule is obtained. i q , which is used to characterize the quality inspection status of the i-th verification data table relative to the q-th candidate quality inspection rule, and serves as the basis for verifying the quality inspection effectiveness of the candidate quality inspection rules for the data corresponding to each target data source.

[0137] S600, if H i q Within the quality score range, the qth candidate quality inspection rule is added to the ith preset quality inspection model, and the ith target quality inspection model is obtained by traversing q=1, 2..., Q.

[0138] In a specific embodiment, the i-th preset quality inspection model includes M i The preset quality inspection rules, S600 also includes the following steps:

[0139] S610, the M in the i-th preset quality inspection model i A preset quality inspection rule is determined as the target quality inspection rule in the i-th target quality inspection model.

[0140] S620, if H i q Within the quality score range, the qth candidate quality inspection rule is determined as the target quality inspection rule in the ith target quality inspection model.

[0141] S630, traverse q=1, 2, ..., Q, and obtain the i-th target quality inspection model, wherein the i-th target quality inspection model includes Φ(i) target quality inspection rules.

[0142] Among them, the quality score range includes the upper limit and lower limit of the quality score. i q Less than or equal to the upper limit of the quality score and greater than or equal to the lower limit of the quality score, it means H i q Within the quality score range, it means that the quality inspection situation of the i-th verification data table relative to the q-th candidate quality inspection rule meets the preset situation, then the q-th candidate quality inspection rule is determined as the target quality inspection rule in the i-th target quality inspection model, which is used to perform quality inspection on the i-th data table to be quality inspected.

[0143] According to H i q By comparing with the quality score range, the effectiveness of the candidate quality inspection rules for the data corresponding to each target data source is judged, and the target quality inspection model is obtained for the quality inspection of the data table to be inspected, which improves the richness of the quality inspection rules of the target quality inspection model and the accuracy of the quality inspection results.

[0144] S700, performing quality inspection on the i-th data table to be inspected by using the i-th target quality inspection model, and obtaining a data quality level corresponding to the i-th data table to be inspected.

[0145] In a specific implementation, S700 further includes the following steps:

[0146] S710, according to the φth target quality inspection rule in the i-th target quality inspection model, perform quality inspection on the u-th data to be inspected in the i-th data table to be inspected, and obtain a third quality inspection result E of the u-th data to be inspected in the i-th data table to be inspected relative to the φth target quality inspection rule in the i-th target quality inspection model. 3 iu φ , where u = 1, 2…, U(i), U(i) refers to the total amount of data in the i-th data table to be quality inspected, and φ = 1, 2…, Ф(i), Ф(i) refers to the total number of target quality inspection rules in the i-th target quality inspection model.

[0147] S720, if the third quality inspection result is E 3 iu φ If there is a problem with the data, the corresponding third quality inspection label R 3 iu φ is equal to the first preset value.

[0148] S730, if the third quality inspection result is E3 iu φ If there is no problem with the data, the corresponding third quality inspection label R 3 iu φ is equal to the second preset value.

[0149] S740, traverse u=1, 2..., U(i) and φ=1, 2..., Φ(i), and obtain the fifth quantity F of the first preset value corresponding to the i-th data table to be inspected 5 i and the corresponding sixth number F of the second preset value 6 i .

[0150] S750, according to F 5 i 、F 6 i and the preset quality level range, and obtain the data quality level corresponding to the i-th data table to be inspected.

[0151] In a specific implementation, S750 further includes the following steps:

[0152] S751, obtain α preset quality level ranges and the quality level corresponding to each preset quality level range, wherein the data quality level corresponding to the βth preset quality level range is β, β=1, 2..., α, α refers to the total number of preset quality level ranges.

[0153] S752, if F 6 i / (F 5 i +F 6 i ) meets the β-th preset quality level range, then the data quality level corresponding to the i-th data table to be inspected is determined to be β.

[0154] The specific value of α and each preset quality level range may be set by the implementer according to actual conditions.

[0155] For example, in this embodiment, α can be set to 4, and the first quality level range is [0, 10), the second quality level range is [10, 60), the first quality level range is [60, 90), and the second quality level range is [90, 100].

[0156] In the above, according to the fifth quantity F of the first preset value corresponding to the i-th to-be-checked data table 5 i , the sixth number F of the corresponding second preset value 6 iand the preset quality level range, the data quality level corresponding to the i-th data table to be inspected is obtained, thereby improving the accuracy of the data quality level.

[0157] In a specific implementation, S700 further includes the following steps:

[0158] S760, traverse u=1, 2, ..., U(i), and obtain the seventh quantity F of the first preset value of the φth target quality inspection rule in the i-th target quality inspection model for the i-th data table to be inspected 7 i φ and the eighth number F of the second preset value 8 i φ .

[0159] S770, according to the seventh quantity F 7 i φ and the eighth quantity F 8 i φ , obtain the first target data quality score of the i-th data table to be inspected relative to the φ-th target quality inspection rule in the i-th target quality inspection model Ж i φ =F 8 i φ / (F 7 i φ +F 8 i φ ).

[0160] S780, traverse φ=1, 2, ..., Φ(i), and obtain all first target data quality scores of the i-th data table to be quality inspected relative to the i-th target quality inspection model.

[0161] S790, obtaining the preset importance corresponding to each target quality inspection rule, and obtaining the second target data quality score σ corresponding to the i-th data table to be quality inspected according to all the first target data quality scores corresponding to the i-th data table to be quality inspected and the preset importance corresponding to each target quality inspection rule. i , where σ i Meet the following conditions:

[0162] σ i =Σ φ Ф(i) (γ i φ × i φ ), where γ i φIt refers to the preset importance level corresponding to the φth target quality inspection rule in the ith target quality inspection model, i φ It refers to the target data quality score of the i-th data table to be quality inspected relative to the φ-th target quality inspection rule in the i-th target quality inspection model.

[0163] S791, according to σ i and the preset quality level range, and obtain the data quality level corresponding to the i-th data table to be inspected.

[0164] Among them, the preset importance corresponding to each target quality inspection rule can be determined according to the quality inspection rule type corresponding to the target quality inspection rule. For example, the quality inspection rule type can include red line problem rules, common problem rules and general problem rules, and the corresponding importance during data quality inspection is from high to low.

[0165] In the above, combined with the first target data quality score of the i-th data table to be inspected relative to all target quality inspection rules in the i-th target quality inspection model, and the preset importance corresponding to each target quality inspection rule, the second target data quality score σ corresponding to the i-th data table to be inspected is obtained. i , which improves the accuracy of the second target data quality score, thereby improving the accuracy of the data quality grade.

[0166] In the above, the vth verification data in the ith verification data table is quality inspected according to the qth candidate quality inspection rule, and the target quality inspection result W of the vth verification data in the ith verification data table relative to the qth candidate quality inspection rule is obtained. iv q , according to the target quality inspection result W iv q , get the target quality inspection label R of the vth verification data in the i-th verification data table relative to the qth candidate quality inspection rule iv q , traverse v = 1, 2 ..., V (i), obtain V (i) target quality inspection labels of the i-th verification data table relative to the q-th candidate quality inspection rule, and the verification data quality score H of the i-th verification data table relative to the q-th candidate quality inspection rule i q , which is used to characterize the quality inspection status of the i-th verification data table relative to the q-th candidate quality inspection rule, and provides a data basis for verifying the quality inspection validity of the candidate quality inspection rules for the data corresponding to each target data source; according to H i qBy comparing with the quality score range, the effectiveness of the candidate quality inspection rules for the data corresponding to each target data source is judged, and the target quality inspection model is obtained for the quality inspection of the data table to be inspected. The i-th data table to be inspected is quality inspected by the i-th target quality inspection model, and the data quality level corresponding to the i-th data table to be inspected is obtained, which improves the richness of the quality inspection rules of the target quality inspection model and the accuracy of the quality inspection results.

[0167] Embodiment 4

[0168] This fourth embodiment provides a data quality inspection device, the data quality inspection device includes: Figure 4 As shown:

[0169] The first data acquisition module 41 is used to acquire N data tables to be inspected, the predicted inspection time of each data table to be inspected, and the historical data quality level corresponding to each data table to be inspected, and upload the i-th data table to be inspected according to the preset upload time T i Input into the preset data quality inspection platform, i=1, 2..., N, N refers to the total number of data tables to be inspected.

[0170] The quality inspection time conflict analysis module 42 is used to analyze the quality inspection completion time W of the first i-1 quality inspection data tables. i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0 , obtain the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, where W0=0, and the quality inspection time conflict result includes quality inspection time conflict and quality inspection time non-conflict.

[0171] The first quality inspection sequence replacement module 43 is used to obtain the quality inspection sequence replacement result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected according to the historical data quality level of the i-th data table to be quality inspected and the historical data quality level of the i+1-th data table to be quality inspected if the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, wherein the quality inspection sequence replacement result includes replacing the quality inspection sequence and not replacing the quality inspection sequence.

[0172] The first data quality inspection module 44 is used to perform quality inspection on the i-th data table to be quality inspected and the data table to be quality inspected that is earlier in the order of the i-th data table to be quality inspected and the i+1-th data table to be quality inspected through the data quality inspection platform according to the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, and obtain the problem data list of the i-th data table to be quality inspected and the data table to be quality inspected that is earlier in the order of the i+1-th data table to be quality inspected, as well as the quality inspection completion time W of the first i data tables to be quality inspected i , update i=i+1.

[0173] The second data quality inspection module 45 is used to repeatedly execute S2-S4 until i=N is updated, and the data quality inspection platform performs quality inspection on the N-1th data table to be quality inspected and the data table to be quality inspected that is subsequent in sequence in the Nth data table to be quality inspected, and obtains N problem data lists corresponding to the N data tables to be quality inspected.

[0174] In a specific implementation, the data quality inspection device further includes:

[0175] The second data acquisition module is used to obtain the data volume of each data table to be quality inspected and the quality inspection speed of the data quality inspection platform.

[0176] The quality inspection time prediction module is used to determine the ratio of the data volume and the quality inspection speed of each data table to be quality inspected as the predicted quality inspection time of the corresponding data table to be quality inspected.

[0177] In a specific implementation, the quality inspection time conflict analysis module 42 further includes:

[0178] The quality inspection completion time prediction submodule is used to predict the quality inspection completion time W of the first i-1 quality inspection data tables. i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , get the predicted quality inspection completion time C of the i-th quality inspection data table i =W i-1 +T i +Y i .

[0179] The time interval acquisition submodule is used to obtain the time T of uploading the i+1th quality inspection data table. i+1 and the predicted quality inspection completion time C of the i-th quality inspection data table i , obtain the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) =T i+1 -C i .

[0180] The first result acquisition submodule is used if G i(i+1) >T0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection time does not conflict.

[0181] The second result acquisition submodule is used if G i(i+1) ≤T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as the quality inspection time conflict.

[0182] In a specific implementation, the data quality inspection device further includes:

[0183] The second quality inspection sequence replacement module, if the quality inspection time conflict result between the i-th data table to be inspected and the i+1-th data table to be inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be inspected, the historical data quality level of the i+1-th data table to be inspected, the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) , preset time interval threshold T 0 The preset first quality inspection sequence exchange degree threshold B 1 , obtain the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected.

[0184] In a specific implementation, the second quality inspection sequence replacement module further includes:

[0185] The quality inspection time overlap degree acquisition submodule is used to obtain the time interval G between the i-th quality inspection data table and the i+1-th quality inspection data table. i(i+1) , and the preset time interval threshold T 0 , obtain the quality inspection time overlap degree A between the i-th quality inspection data table and the i+1-th quality inspection data table i(i+1) =(T 0 -G i(i+1) ) / T 0 .

[0186] The first quality inspection sequence replacement degree acquisition submodule is used to obtain the historical data quality level D of the i-th data table to be inspected according to the quality level D of the historical data of the i-th data table to be inspected. i 、The quality level of historical data in the i+1th data table to be inspected is D i+1 Overlap degree with quality inspection time A i(i+1) , obtain the first quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 1 i(i+1) =A i(i+1) ×((2 / (1+e^(D i -D i+1 )))-1).

[0187] The first quality inspection sequence replacement submodule is used if B 1 i(i+1) >B 1 , then the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as the exchanged quality inspection order.

[0188] The second quality inspection sequence replacement submodule is used if B 1 i(i+1) ≤B 1 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged.

[0189] In a specific implementation, the first quality inspection sequence switching module 43 further includes:

[0190] The second quality inspection sequence replacement degree acquisition submodule is used to obtain the historical data quality level D of the i-th data table to be inspected according to the quality level D of the historical data of the i-th data table to be inspected. i and the quality level D of the historical data of the i+1th data table to be inspected i+1 , obtain the second quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 2 i(i+1) =(2 / (1+e^(D i -D i+1 )))-1.

[0191] The third quality inspection sequence replacement submodule is used if B 2 i(i+1) >B 2 , then the quality inspection order exchange result between the i-th quality inspection data table and the i+1-th quality inspection data table is obtained as the quality inspection order exchange, where B 2 Refers to the preset second quality inspection order replacement degree threshold.

[0192] The fourth quality inspection sequence replacement submodule is used if B 2 i(i+1) ≤B 2 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged.

[0193] In a specific implementation, the first data quality inspection module 44 further includes:

[0194] The first quality inspection order determination submodule is used to determine the i+1th data table to be quality inspected as the data table to be quality inspected that has an earlier order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected if the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a exchange of the quality inspection order.

[0195] The second quality inspection order determination submodule is used to determine the i-th data table to be quality inspected as the data table to be quality inspected that has an earlier order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected if the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is not to exchange the quality inspection order.

[0196] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0197] Embodiment 5

[0198] Embodiment 5 of the present invention provides a non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the steps:

[0199] S1, obtain N data tables to be inspected, the predicted inspection time of each data table to be inspected, and the historical data quality level corresponding to each data table to be inspected, and upload the i-th data table to be inspected according to the preset upload time T i Input into the preset data quality inspection platform, i=1, 2..., N, N refers to the total number of data tables to be inspected.

[0200] S2, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0 , obtain the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, where W0=0, and the quality inspection time conflict result includes quality inspection time conflict and quality inspection time non-conflict.

[0201] S3. If the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be quality inspected and the historical data quality level of the i+1-th data table to be quality inspected, the quality inspection order replacement result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained, wherein the quality inspection order replacement result includes replacing the quality inspection order and not replacing the quality inspection order.

[0202] S4: According to the quality inspection order swap result between the i-th data table to be inspected and the i+1-th data table to be inspected, the data quality inspection platform performs quality inspection on the i-th data table to be inspected and the data table to be inspected that is earlier in the order of the i+1-th data table to be inspected, and obtains the problem data list of the i-th data table to be inspected and the data table to be inspected that is earlier in the order of the i+1-th data table to be inspected, as well as the quality inspection completion time W of the first i data tables to be inspected. i , update i=i+1.

[0203] S5, repeat S2-S4 until i=N is updated, and the N-1th data table to be inspected and the subsequent data table to be inspected in the Nth data table to be inspected are quality inspected through the data quality inspection platform to obtain N problem data lists corresponding to the N data tables to be inspected.

[0204] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0205] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0206] Embodiment 6

[0207] A sixth embodiment of the present invention provides an electronic device, which includes a processor and the non-transitory computer-readable storage medium in the fifth embodiment of the present invention.

[0208] The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A data quality inspection method, characterized in that: The data quality inspection method comprises the following steps: S1, obtain N data tables to be inspected, the predicted inspection time of each data table to be inspected, and the historical data quality level corresponding to each data table to be inspected, and upload the i-th data table to be inspected according to the preset upload time T i Input into the preset data quality inspection platform, i=1, 2..., N, N refers to the total number of data tables to be inspected; S2, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0 , obtaining the quality inspection time conflict result between the i-th quality inspection data table and the i+1-th quality inspection data table, wherein W0=0, the quality inspection time conflict result includes quality inspection time conflict and quality inspection time non-conflict, S2 further includes the following steps: S21, based on the quality inspection completion time W of the first i-1 quality inspection data tables i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , obtain the predicted quality inspection completion time C of the i-th data table to be inspected i =W i-1 +T i +Y i ; S22, according to the upload time T of the i+1th data table to be inspected i+1 and the predicted quality inspection completion time C of the i-th data table to be inspected i , obtain the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) =T i+1 -C i ; S23, if G i(i+1) >T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as follows: the quality inspection time does not conflict; S24, if G i(i+1) ≤T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as a quality inspection time conflict; S3, if the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be quality inspected and the historical data quality level of the i+1-th data table to be quality inspected, the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained, wherein the quality inspection order exchange result includes exchanging the quality inspection order and not exchanging the quality inspection order, and S3 specifically includes the following steps: According to the historical data quality level D of the i-th data table to be inspected i and the quality level D of the historical data of the i+1th data table to be inspected i+1 , obtain the second quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 2 i(i+1) =(2 / (1+e^(D i -D i+1 )))-1; If B 2 i(i+1) >B 2 , then the quality inspection order exchange result between the i-th quality inspection data table and the i+1-th quality inspection data table is obtained as the quality inspection order exchange, where B 2 Refers to the preset second quality inspection sequence replacement degree threshold; If B 2 i(i+1) ≤B 2 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged; Alternatively, if the quality inspection time conflict result between the i-th data table to be inspected and the i+1-th data table to be inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be inspected, the historical data quality level of the i+1-th data table to be inspected, the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) , the preset time interval threshold T 0 The preset first quality inspection sequence exchange degree threshold B 1 , obtaining the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, specifically comprising the following steps: According to the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) , and the preset time interval threshold T 0 , obtain the quality inspection time overlap degree A between the i-th quality inspection data table and the i+1-th quality inspection data table i(i+1) =(T 0 -G i(i+1) ) / T 0 ; According to the historical data quality level D of the i-th data table to be inspected i , the quality level D of the historical data of the i+1th data table to be inspected i+1 Overlap degree with quality inspection time A i(i+1) , obtain the first quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 1 i(i+1) =A i(i+1) ×((2 / (1+e^(D i -D i+1 )))-1); If B 1 i(i+1) >B 1 , then the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as the quality inspection order exchange; If B 1 i(i+1) ≤B 1 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged; S4, according to the quality inspection order exchange result between the i-th data table to be inspected and the i+1-th data table to be inspected, the data quality inspection platform performs quality inspection on the i-th data table to be inspected and the data table to be inspected that is earlier in sequence in the i+1-th data table to be inspected, and obtains a list of problematic data of the i-th data table to be inspected and the data table to be inspected that is earlier in sequence in the i+1-th data table to be inspected, as well as the quality inspection completion time W of the first i data tables to be inspected i , increase the value of i by 1 for the next round of iteration; S5, repeat S2-S4 until i=N is updated, and perform quality inspection on the N-1th data table to be quality inspected and the data table to be quality inspected that comes later in the Nth data table to be quality inspected through the data quality inspection platform, and obtain N problem data lists corresponding to the N data tables to be quality inspected, wherein the problem data lists include the problem data detected by quality inspection, the business table corresponding to the problem data, the business component corresponding to the problem data, and the quality inspection rules corresponding to the problem data.

2. The data quality inspection method according to claim 1, characterized in that: Obtaining the predicted quality inspection time for each data table to be inspected also includes the following steps: S01, obtaining the data volume of each data table to be quality inspected and the quality inspection speed of the data quality inspection platform; S02: Determine the ratio of the data volume of each data table to be quality inspected to the quality inspection speed as the predicted quality inspection time of the corresponding data table to be quality inspected.

3. The data quality inspection method according to claim 1, characterized in that: S4 also includes the following steps: S41, if the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is to exchange the quality inspection order, determine the i+1-th data table to be quality inspected as the data table to be quality inspected that has an earlier order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected; S42, if the result of exchanging the quality inspection order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is not to change the quality inspection order, then determine that the i-th data table to be quality inspected is the data table to be quality inspected that comes earlier in order between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected.

4. A data quality inspection device, characterized in that: The data quality inspection device comprises: The first data acquisition module is used to obtain N data tables to be inspected, the predicted quality inspection time of each data table to be inspected, and the historical data quality level corresponding to each data table to be inspected, and upload the i-th data table to be inspected according to the preset upload time T i Input into the preset data quality inspection platform, i=1, 2..., N, N refers to the total number of data tables to be inspected; The quality inspection time conflict analysis module is used to calculate the quality inspection completion time W of the first i-1 quality inspection data tables. i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , the upload time T of the i+1th quality inspection data table i+1 and the preset time interval threshold T 0 , obtain the quality inspection time conflict result between the i-th quality inspection data table and the i+1-th quality inspection data table, wherein W0=0, the quality inspection time conflict result includes quality inspection time conflict and quality inspection time non-conflict, and the quality inspection time conflict analysis module also includes: The quality inspection completion time prediction submodule is used to predict the quality inspection completion time W of the first i-1 quality inspection data tables. i-1 , the predicted quality inspection time Y of the i-th data table to be inspected i , the upload time T of the i-th data table to be inspected i , obtain the predicted quality inspection completion time C of the i-th data table to be inspected i =W i-1 +T i +Y i ; The time interval acquisition submodule is used to obtain the time interval according to the upload time T of the i+1th quality inspection data table. i+1 and the predicted quality inspection completion time C of the i-th data table to be inspected i , obtain the time interval G between the i-th data table to be inspected and the i+1-th data table to be inspected i(i+1) =T i+1 -C i ; The first result acquisition submodule is used if G i(i+1) >T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as follows: the quality inspection time does not conflict; The second result acquisition submodule is used if G i(i+1) ≤T 0 , then the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as a quality inspection time conflict; A first quality inspection sequence replacement module is used for obtaining a quality inspection sequence replacement result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected according to the historical data quality level of the i-th data table to be quality inspected and the historical data quality level of the i+1-th data table to be quality inspected if the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, wherein the quality inspection sequence replacement result includes replacing the quality inspection sequence and not replacing the quality inspection sequence, and the first quality inspection sequence replacement module specifically includes; The second quality inspection sequence replacement degree acquisition submodule is used to obtain the historical data quality level D of the i-th data table to be inspected according to the quality level D of the historical data of the i-th data table to be inspected. i and the quality level D of the historical data of the i+1th data table to be inspected i+1 , obtain the second quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 2 i(i+1) =(2 / (1+e^(D i -D i+1 )))-1; The third quality inspection sequence replacement submodule is used if B 2 i(i+1) >B 2 , then the quality inspection order exchange result between the i-th quality inspection data table and the i+1-th quality inspection data table is obtained as the quality inspection order exchange, where B 2 Refers to the preset second quality inspection sequence replacement degree threshold; The fourth quality inspection sequence replacement submodule is used if B 2 i(i+1) ≤B 2 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged; Alternatively, a second quality inspection sequence replacement module is used to, if the quality inspection time conflict result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is a quality inspection time conflict, then according to the historical data quality level of the i-th data table to be quality inspected, the historical data quality level of the i+1-th data table to be quality inspected, the time interval G between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected i(i+1) , the preset time interval threshold T 0 The preset first quality inspection sequence exchange degree threshold B 1 , obtaining the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, specifically including: The quality inspection time overlap degree acquisition submodule is used to obtain the time interval G between the i-th quality inspection data table and the i+1-th quality inspection data table according to the time interval G between the i-th quality inspection data table and the i+1-th quality inspection data table. i(i+1) , and the preset time interval threshold T 0 , obtain the quality inspection time overlap degree A between the i-th quality inspection data table and the i+1-th quality inspection data table i(i+1) =(T 0 -G i(i+1) ) / T 0 ; The first quality inspection sequence replacement degree acquisition submodule is used to obtain the historical data quality level D of the i-th data table to be inspected according to the quality level D of the historical data of the i-th data table to be inspected. i , the quality level D of the historical data of the i+1th data table to be inspected i+1 Overlap degree with quality inspection time A i(i+1) , obtain the first quality inspection order exchange degree B between the i-th quality inspection data table and the i+1-th quality inspection data table 1 i(i+1) =A i(i+1) ×((2 / (1+e^(D i -D i+1 )))-1); The first quality inspection sequence replacement submodule is used if B 1 i(i+1) >B 1 , then the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is obtained as the quality inspection order exchange; The second quality inspection sequence replacement submodule is used if B 1 i(i+1) ≤B 1 , then the result of the quality inspection order exchange between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected is that the quality inspection order is not exchanged; The first data quality inspection module is used to perform quality inspection on the i-th data table to be quality inspected and the data table to be quality inspected that is earlier in the order of the i-th data table to be quality inspected and the i+1-th data table to be quality inspected through the data quality inspection platform according to the quality inspection order exchange result between the i-th data table to be quality inspected and the i+1-th data table to be quality inspected, and obtain the problem data list of the i-th data table to be quality inspected and the data table to be quality inspected that is earlier in the order of the i+1-th data table to be quality inspected, as well as the quality inspection completion time W of the first i data tables to be quality inspected i , increase the value of i by 1 for the next round of iteration; The second data quality inspection module is used to repeatedly execute the quality inspection time conflict analysis module, the first quality inspection sequence replacement module and the first data quality inspection module until i=N is updated, and to perform quality inspection on the N-1th data table to be quality inspected and the data table to be quality inspected that is later in sequence in the Nth data table to be quality inspected through the data quality inspection platform, and obtain N problem data lists corresponding to the N data tables to be quality inspected, wherein the problem data lists include the problem data detected by quality inspection, the business table corresponding to the problem data, the business component corresponding to the problem data and the quality inspection rules corresponding to the problem data.

5. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the data quality inspection method as described in any one of claims 1-3.

6. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 5.

Images