Data quality detection method, system, device and storage medium

Abstract

The present invention discloses a data quality detection method, system, device and storage medium, which can be applied to the big data field or the financial field. The method includes: according to the access address identifier in the data detection trigger information, performing data screening on the preset mapping relationship table to obtain the inspection item table, performing field name comparison based on the inspection item table to determine the target field information group corresponding to each target field name, and for each target field name: calling the first type of initial inspection template corresponding to each field attribute information in the target field information group corresponding to the target field name, and performing field replacement on each first type of initial inspection template to obtain multiple first type inspection templates; using the first type of inspection template to perform data quality inspection on the field content corresponding to the target field name, and generating a data quality inspection result corresponding to the field content. The present invention improves the efficiency of data quality detection.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data quality detection method, system, device and storage medium. Background Art

[0002] System data is a critical asset for banks, and its quality directly impacts the efficiency and accuracy of banking operations. The current approach to data quality testing involves verification personnel first developing verification rules based on data standards. Technical personnel then manually develop verification scripts based on these rules. Finally, the developed verification scripts are used to perform data quality testing.

[0003] However, due to the sheer volume and variety of data, manual verification scripts must be repeatedly developed for each business scenario to perform quality checks on each data type. This reduces the efficiency of performing data quality checks on large amounts of data from different business scenarios simultaneously. Therefore, improving data quality testing efficiency while maintaining accuracy has become a pressing issue. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a data quality detection method, system, device, and storage medium to improve detection efficiency while ensuring data quality detection accuracy. The specific technical solution is as follows:

[0005] A method for detecting data quality, comprising:

[0006] When the detection identifier in the received data detection trigger information is a production verification identifier, data screening is performed on the preset mapping relationship table according to the access address identifier in the data detection trigger information to obtain an inspection item table constructed based on the result of the data screening, wherein the preset mapping relationship table stores each field information group in multiple business tables corresponding to each business system, the field information group includes a field name and multiple field attribute information, the field name corresponds to at least one of the field attribute information, and the inspection item table includes multiple field information groups of one business table;

[0007] Performing a field name comparison on the business table to be checked based on the inspection item table, and determining each target field name and its corresponding target field information group based on the result of the field name comparison, wherein the business table to be checked is the business table corresponding to the access address identifier, the target field name is a field name in the business table to be checked, and the target field name is consistent with the field name in the target field information group, and the target field information group is one of the field information groups in the inspection item table;

[0008] For each of the target field names: call the first type of initial verification template corresponding to each of the field attribute information in the target field information group corresponding to the target field name, and perform field replacement on each of the first type of initial verification templates based on the access address identifier and the target field name to obtain multiple first type verification templates; use each of the first type of verification templates to perform data quality detection on the field content corresponding to the target field name, and generate multiple data quality detection results corresponding to the field content.

[0009] Optionally, the method further includes:

[0010] When the detection status identifier is a function test identifier, identifying table structure fields of the initial business table based on the preset mapping relationship table, and determining the field information group corresponding to each table structure field, wherein the initial business table is the to-be-checked business table that has not loaded business data;

[0011] For each of the table structure fields: call the second type of initial verification template corresponding to each of the field attribute information in the field information group corresponding to the table structure field; and perform field replacement on each of the second type of initial verification templates based on the access address identifier and the table structure field to obtain multiple second type verification templates; use each of the second type of verification templates to perform data quality detection on the table structure field, and generate a data quality detection result corresponding to the table structure field.

[0012] Optionally, performing field replacement on each of the first-type initial verification templates based on the access address identifier and the target field name to obtain multiple first-type verification templates includes:

[0013] For each of the first type of initial verification templates:

[0014] Obtaining an access address code corresponding to the access address identifier;

[0015] The access address field in the first type of initial verification template is updated using the access address code, and the access object field in the first type of initial verification template is updated using the target field name to obtain the first type of verification template.

[0016] Optionally, the method further includes:

[0017] Text extraction is performed on each of the data quality detection results to obtain the name of each problem field and the corresponding verification rule description text, and the preset report template is filled with the problem field name and the verification rule description text to obtain a data quality detection report.

[0018] A data quality detection system, comprising:

[0019] a first data screening unit configured to, when the detection identifier in the received data detection trigger information is a production verification identifier, perform data screening on a preset mapping relationship table according to an access address identifier in the data detection trigger information, so as to obtain an inspection item table constructed based on the result of the data screening, wherein the preset mapping relationship table stores field information groups in multiple business tables corresponding to each business system, the field information groups including a field name and multiple field attribute information, the field name corresponding to at least one field attribute information, and the inspection item table including multiple field information groups of one business table;

[0020] a field comparison unit, configured to compare the field names of the to-be-checked business table based on the inspection item table, and determine each target field name and its corresponding target field information group based on the result of the field name comparison, wherein the to-be-checked business table is the business table corresponding to the access address identifier, the target field name is a field name in the to-be-checked business table, and the target field name is consistent with the field name in the target field information group, and the target field information group is one of the field information groups in the inspection item table;

[0021] The first quality detection unit is used to: for each target field name: retrieve the first type of initial verification template corresponding to each of the field attribute information in the target field information group corresponding to the target field name, and perform field replacement on each of the first type of initial verification templates based on the access address identifier and the target field name to obtain multiple first type verification templates; use each of the first type of verification templates to perform data quality detection on the field content corresponding to the target field name, and generate multiple data quality detection results corresponding to the field content.

[0022] Optionally, the system further includes:

[0023] a second data screening unit, configured to, when the detection status identifier is a function test identifier, identify table structure fields of an initial business table based on the preset mapping relationship table, and determine the field information group corresponding to each table structure field, wherein the initial business table is the to-be-checked business table that has not been loaded with business data;

[0024] The second quality detection unit is used to: retrieve the second type of initial verification template corresponding to each of the field attribute information in the field information group corresponding to the table structure field for each of the table structure fields; and perform field replacement on each of the second type of initial verification templates based on the access address identifier and the table structure field to obtain multiple second type verification templates; use each of the second type of verification templates to perform data quality detection on the table structure field, and generate a data quality detection result corresponding to the table structure field.

[0025] Optionally, when the field replacement is performed on each of the first-type initial verification templates based on the access address identifier and the target field name to obtain multiple first-type verification templates, the first quality detection unit is configured to:

[0026] For each of the first type of initial verification templates:

[0027] Obtaining an access address code corresponding to the access address identifier;

[0028] The access address field in the first type of initial verification template is updated using the access address code, and the access object field in the first type of initial verification template is updated using the target field name to obtain the first type of verification template.

[0029] Optionally, the system further includes:

[0030] The report generation unit is used to perform text extraction on each of the data quality detection results, obtain the name of each problem field and the corresponding verification rule description text, and use the problem field name and the verification rule description text to fill in the preset report template to obtain a data quality detection report.

[0031] A data quality detection device, comprising:

[0032] processor;

[0033] a memory for storing instructions executable by the processor;

[0034] The processor is configured to execute the instructions to implement any of the above-mentioned methods for detecting data quality.

[0035] A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of a data quality detection device, enables the detection device to perform any of the data quality detection methods described above.

[0036] The embodiment of the present invention provides a data quality detection method, system, device and storage medium. It can store each field information group in multiple business tables corresponding to each business system by configuring a preset mapping relationship table, so that when performing data quality detection on multiple business data under different business scenarios, there is no need for developers to temporarily query the verification standards corresponding to each business data, but directly search in the preset mapping relationship table by field name, thereby improving the query efficiency of the data verification standard. In addition, by configuring the preset mapping relationship table to record the mapping relationship between different fields and their field attribute information, and pre-configuring the corresponding relationship between each field attribute information and the initial verification template, it is possible to achieve that when data quality detection is required for different business tables, or when data quality detection is required for business tables containing multiple fields, there is no need for developers to develop corresponding detection scripts for different fields under different business scenarios, but instead the corresponding verification template is called based on the pre-built mapping relationship to construct the detection script. This saves the detection script development time and improves the efficiency of data quality detection. It can be seen that the present invention improves the efficiency of data quality detection.

[0037] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A flow chart of a data quality detection method provided by an embodiment of the present invention;

[0040] Figure 2 A block diagram of a data quality detection system provided by an embodiment of the present invention;

[0041] Figure 3 A block diagram of a data quality detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The method, system, device, and storage medium for determining an interest rate risk indicator provided by this application can be used in the big data field or the financial field. The above is only an example and does not limit the application field of the method, system, device, and storage medium for determining an interest rate risk indicator provided by this application.

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] The embodiment of the present invention provides a method for detecting data quality, such as Figure 1 As shown, the data quality detection method includes:

[0045] S101. When the detection identifier in the received data detection trigger information is a production verification identifier, data is screened on the preset mapping relationship table according to the access address identifier in the data detection trigger information to obtain an inspection item table constructed based on the results of the data screening, wherein the preset mapping relationship table stores various field information groups in multiple business tables corresponding to each business system, the field information group includes a field name and multiple field attribute information, the field name corresponds to at least one field attribute information, and the inspection item table includes multiple field information groups of a business table.

[0046] Optionally, in an optional embodiment of the present invention, the field names in the above field information group may be not only data fields or table structure fields in a data table, but also metadata items in a business system.

[0047] Optionally, in an optional embodiment of the present invention, the access address identifier is used to determine the location of the field name and the identifier of the business table to be checked, such as a system identifier and a business table identifier.

[0048] It should be noted that in actual application scenarios, the data detection trigger information can be a trigger information sent by an upstream system through a workflow to trigger the data detection operation. By configuring the data detection trigger information as a signal to trigger the corresponding data quality detection, the present invention realizes the automation of the data quality detection process and improves detection efficiency.

[0049] It should be noted that, in actual application scenarios, the above-mentioned commissioning verification identifier is an identifier used to characterize that the current detection status is in the commissioning verification stage. Since the data quality detection is different in different test stages. For example, in the commissioning verification stage, the business table has completed the construction of the table structure. In this process, only the business data generated in the business table needs to be tested for data quality. Therefore, the present invention improves the detection accuracy and detection flexibility by configuring the above-mentioned commissioning verification identifier to distinguish the detection object and to determine the type of verification template for data quality detection of the detection object.

[0050] It should be noted that in actual application scenarios, the above-mentioned preset mapping relationship table can be constructed by developers based on business needs, and is used to record the mapping relationship between each table field and its corresponding data verification standard. Based on different business needs, the specific structure of the above-mentioned preset mapping relationship table can be various, and an exemplary one is provided here:

[0051] Table 1 shows a preset mapping relationship table for a banking integrated system:

[0052]

[0053] Table 1

[0054] Each row of data in Table 1 represents a field information group. The Chinese field name, English field name, dictionary Chinese name, and dictionary English name in Table 1 can all be field names. The enterprise-level data dictionary ID, data category, data type, data format, and code value range represent field attribute information. In addition to the field attribute information recorded in Table 1, this field attribute information can also include descriptive text for each verification rule.

[0055] It should be noted that the numbers in parentheses in Table 1 above are identifiers of the data verification standards corresponding to the field attribute information. These identifiers represent the correspondence between the field attribute information and its corresponding initial verification template. In actual application scenarios, the initial verification template corresponding to the field attribute information can be retrieved based on the identifier of the data verification standard in the field attribute information.

[0056] It should be noted that the present invention configures the above-mentioned preset mapping relationship table to store the field information groups in multiple business tables corresponding to each business system, so that when performing data quality testing on various business data in different business scenarios, there is no need for developers to temporarily query the verification standards corresponding to each business data, but directly search in the above-mentioned preset mapping relationship table through the field name, thereby improving the query efficiency of the data verification standards, and further improving the efficiency of subsequent data quality testing of the data.

[0057] It should be noted that the present invention obtains an inspection item list constructed based on the result of data screening through data screening, which can remove interference data and thus improve the efficiency of subsequent data quality detection.

[0058] S102. Perform a field name comparison on the business table to be checked based on the inspection item table, and determine each target field name and its corresponding target field information group based on the result of the field name comparison, wherein the business table to be checked is a business table corresponding to the access address identifier, the target field name is a field name in the business table to be checked, and the target field name is consistent with the field name in the target field information group, and the target field information group is a field information group in the inspection item table.

[0059] It should be noted that in actual application scenarios, the above Figure 1 There are many implementations of step S102, and an exemplary implementation is provided here:

[0060] Assume that the business table to be checked contains only two field names and their corresponding field contents, where the first field name is "Customer Gender" and its field content is "0", and the second field name is "Date" and its field content is "20011221".

[0061] After comparing the field names, the target field information groups corresponding to the two target field names are shown in Table 2 below:

[0062]

[0063] Table 2

[0064] It should be noted that the above Tables 1 and 2 are merely optional embodiments of the present invention, and the present invention does not impose excessive limitations or redundant descriptions on the specific contents of Tables 1 and 2.

[0065] S103. For each target field name: retrieve the first type of initial verification template corresponding to each field attribute information in the target field information group corresponding to the target field name, and perform field replacement on each first type of initial verification template based on the access address identifier and the target field name to obtain multiple first type verification templates; use each first type of verification template to perform data quality detection on the field content corresponding to the target field name, and generate multiple data quality detection results corresponding to the field content.

[0066] Optionally, in an optional embodiment of the present invention, the first type of initial verification template is a template corresponding to the production verification mark. The initial verification template can be a query statement edited by a developer using Structured Query Language (SQL) based on business needs.

[0067] It should be noted that in actual application scenarios, since there will be multiple fields in the same business table at the same time, and there will also be differences in the fields in different business tables. Therefore, the present invention records the mapping relationship between different fields and their field attribute information through a preset mapping relationship table, and pre-configures the corresponding relationship between each field attribute information and the initial verification template, so that when it is necessary to perform data quality detection on different business tables, or to perform data quality detection on a business table containing multiple fields, there is no need for developers to develop corresponding detection scripts for different fields. Instead, the corresponding verification template is called based on the pre-built mapping relationship to build a detection script. This saves the time for developing detection scripts and improves the efficiency of data quality detection.

[0068] It should be noted that in actual application scenarios, in order to improve the efficiency of data quality testing, when multiple verification templates are obtained, the verification templates corresponding to multiple target field names under the same business table can be spliced ​​and compiled into a complete script file, so that the script file can be used to perform data quality testing on the business table.

[0069] The present invention stores each field information group in a plurality of business tables corresponding to each business system by configuring a preset mapping relationship table, so that when performing data quality detection on a variety of business data under different business scenarios, it is not necessary for the developer to temporarily query the verification standard corresponding to each business data, but to directly search in the preset mapping relationship table by the field name, thereby improving the query efficiency of the data verification standard. In addition, by configuring the preset mapping relationship table to record the mapping relationship between different fields and their field attribute information, and pre-configuring the corresponding relationship between each field attribute information and the initial verification template, it is possible to achieve that when data quality detection is required for different business tables, or when data quality detection is required for a business table containing multiple fields, it is not necessary for the developer to develop corresponding detection scripts for different fields under different business scenarios, but to call the corresponding verification template based on the pre-built mapping relationship to construct the detection script. This saves the detection script development time and improves the efficiency of data quality detection. It can be seen that the present invention improves the efficiency of data quality detection.

[0070] Optional, as above Figure 1 The data quality detection method shown also includes:

[0071] When the detection status is identified as a functional test identifier, the table structure fields of the initial business table are identified based on the preset mapping relationship table, and the field information group corresponding to each table structure field is determined, wherein the initial business table is a business table to be checked that has not loaded business data.

[0072] For each table structure field: call the second type of initial verification template corresponding to each field attribute information in the field information group corresponding to the table structure field; and replace the field of each second type of initial verification template based on the access address identifier and the table structure field to obtain multiple second type verification templates; use each second type of verification template to perform data quality detection on the table structure field, and generate a data quality detection result corresponding to the table structure field.

[0073] It should be noted that, in actual application scenarios, the above-mentioned functional test identifier is an identifier used to characterize that the current detection status is in the functional test stage. In the functional test stage, the business table has completed the development and construction of the table structure, but the corresponding business data is not loaded in the business table at this time. Since the table structure of the business table directly affects the quality of the business data finally generated, for example, the developer does not set the table structure field format according to the business requirements, but according to his own habits, resulting in the final generated business data format not meeting the business requirements. This means that when erroneous data is found in the production verification stage due to the table structure fields not being developed according to business requirements, the table structure needs to be re-checked, thereby reducing the efficiency of data quality detection. The present invention, by configuring the above-mentioned functional test identifier to distinguish the detection objects, avoids the risk of erroneous data generated due to non-standard development of the table structure, resulting in the need to re-check the table structure, thereby improving the efficiency of data quality detection.

[0074] It should be noted that the second type of initial verification template is an initial verification template corresponding to the functional test identifier.

[0075] It should be noted that, in actual application scenarios, the specific information in the field information group corresponding to the above table structure fields may be as shown in Table 3 below:

[0076]

[0077] Table 3

[0078] It should be noted that, in actual application scenarios, the second type of initial verification template can be a query statement compiled by developers using Structured Query Language (SQL). The present invention does not make too many restrictions or elaborate on the specific construction process of the second type of initial verification template.

[0079] It should be noted that, in actual application scenarios, the order in which the functional test identifier and the production verification identifier are issued can be customized based on the actual application scenario requirements. For example, during normal project development and testing, the test identifier can be issued first, followed by the production verification identifier. When summarizing and analyzing results, the production verification identifier and the functional test identifier can be issued simultaneously. This disclosure does not impose any further limitations or elaboration on this matter.

[0080] Optionally, field replacement is performed on each of the first type initial verification templates based on the access address identifier and the target field name to obtain multiple first type verification templates, including:

[0081] For each first type of initial verification template:

[0082] Get the access address code corresponding to the access address identifier;

[0083] The access address field in the first type of initial verification template is updated using the access address code, and the access object field in the first type of initial verification template is updated using the target field name to obtain the first type of verification template.

[0084] It should be noted that, in actual application scenarios, there are many implementation methods for obtaining the first type of verification template. Here, an exemplary implementation method is provided:

[0085] Assume the current target field name is "Customer Gender (chk_col)" and the access address identifier for this target field name corresponds to the access address code "chk_db." The field attribute information corresponding to this target field name indicates that its verification rule category is "Code Value Validity." The field attribute information corresponding to this target field name indicates that its corresponding business system data item is ('0', '1'). After field replacement, the resulting first type of verification template can be: Select display_colfromchk_db.chk_tab where chk_col notin('0', '1').

[0086] It should be noted that, in actual application scenarios, the specific format and content of the above-mentioned verification template can be set according to the actual application scenario, and the present invention does not make too many restrictions or elaborate on this.

[0087] Optional, as above Figure 1 The data quality detection method shown also includes:

[0088] Text extraction is performed on each data quality test result to obtain the name of each problem field and the corresponding verification rule description text, and the problem field name and verification rule description text are used to fill in the preset report template to obtain the data quality test report.

[0089] It should be noted that in actual application scenarios, the above problematic field names can be configured by configuring the above first type of verification template to add a label to the target field name if the field content fails the data quality test. The label content is then added to the data quality test results.

[0090] It should be noted that in actual application scenarios, there are many ways to perform text extraction, such as the Naive Bayes algorithm, the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm, and the Support Vector Machine (SVM).

[0091] Corresponding to the above method embodiment, the present invention also provides a data quality detection system, such as Figure 2 As shown, the data quality detection system includes:

[0092] The first data screening unit 201 is used to screen data on the preset mapping relationship table according to the access address identifier in the data detection trigger information when the detection identifier in the received data detection trigger information is a production verification identifier, so as to obtain an inspection item table constructed based on the result of the data screening, wherein the preset mapping relationship table stores various field information groups in multiple business tables corresponding to each business system, the field information group includes a field name and multiple field attribute information, the field name corresponds to at least one field attribute information, and the inspection item table includes multiple field information groups of a business table.

[0093] The field comparison unit 202 is used to compare the field names of the business table to be checked based on the inspection item table, and determine each target field name and its corresponding target field information group based on the result of the field name comparison, wherein the business table to be checked is a business table corresponding to the access address identifier, the target field name is a field name in the business table to be checked, and the target field name is consistent with the field name in the target field information group, and the target field information group is a field information group in the inspection item table.

[0094] The first quality detection unit 203 is used to: for each target field name: retrieve the first type of initial verification template corresponding to each field attribute information in the target field information group corresponding to the target field name, and perform field replacement on each first type of initial verification template based on the access address identifier and the target field name to obtain multiple first type verification templates; use each first type of verification template to perform data quality detection on the field content corresponding to the target field name, and generate multiple data quality detection results corresponding to the field content.

[0095] Optional, as above Figure 2 The data quality detection system shown also includes:

[0096] a second data screening unit configured to, when the detection status identifier is a function test identifier, identify table structure fields of the initial business table based on a preset mapping relationship table and determine a field information group corresponding to each table structure field, wherein the initial business table is a business table to be checked that has not been loaded with business data;

[0097] The second quality detection unit is used to: for each table structure field: retrieve the second type of initial verification template corresponding to each field attribute information in the field information group corresponding to the table structure field; and replace the field of each second type of initial verification template based on the access address identifier and the table structure field to obtain multiple second type verification templates; use each second type of verification template to perform data quality detection on the table structure field, and generate a data quality detection result corresponding to the table structure field.

[0098] Optionally, when a plurality of first-type verification templates are obtained by performing field replacement on each first-type initial verification template based on the system identifier, the business table identifier, and the target field name, the first quality detection unit 203 is configured to:

[0099] For each first type of initial verification template:

[0100] Obtain the address codes corresponding to the system identifier and the business table identifier respectively, and concatenate the address codes to obtain the access address code;

[0101] The access address field in the first type of initial verification template is updated using the access address code, and the access object field in the first type of initial verification template is updated using the target field name to obtain the first type of verification template.

[0102] Optional, as above Figure 2 The data quality detection system shown also includes:

[0103] The report generation unit is used to extract text from each data quality test result, obtain the name of each problem field and the corresponding verification rule description text, and use the problem field name and verification rule description text to fill in the preset report template to obtain a data quality test report.

[0104] The embodiment of the present invention also provides a data quality detection device, such as Figure 3 As shown, the data quality detection equipment includes:

[0105] Processor 301;

[0106] a memory 302 for storing instructions executable by the processor 301;

[0107] The processor 301 is configured to execute instructions to implement any of the above-mentioned data quality detection methods.

[0108] An embodiment of the present invention further provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of a data quality detection device, the detection device can perform any of the above-mentioned data quality detection methods.

[0109] Memory may include non-permanent memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory includes at least one memory chip. Memory is an example of a computer-readable medium.

[0110] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. It should also be noted that the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, commodity, or device comprising the element.

[0113] Each embodiment in this specification is described in a related manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiment is generally similar to the method embodiment, so the description is relatively simple. For related parts, refer to the description of the method embodiment.

[0114] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for detecting data quality, characterized in that: The method comprises: When the detection identifier in the received data detection trigger information is a production verification identifier, data screening is performed on the preset mapping relationship table according to the access address identifier in the data detection trigger information to obtain an inspection item table constructed based on the result of the data screening, wherein the preset mapping relationship table stores each field information group in multiple business tables corresponding to each business system, the field information group includes a field name and multiple field attribute information, the field name corresponds to at least one of the field attribute information, and the inspection item table includes multiple field information groups of one business table; Performing a field name comparison on the business table to be checked based on the inspection item table, and determining each target field name and its corresponding target field information group based on the result of the field name comparison, wherein the business table to be checked is the business table corresponding to the access address identifier, the target field name is a field name in the business table to be checked, and the target field name is consistent with the field name in the target field information group, and the target field information group is one of the field information groups in the inspection item table; For each of the target field names: call the first type of initial verification template corresponding to each of the field attribute information in the target field information group corresponding to the target field name, and perform field replacement on each of the first type of initial verification templates based on the access address identifier and the target field name to obtain multiple first type verification templates; use each of the first type of verification templates to perform data quality detection on the field content corresponding to the target field name, and generate multiple data quality detection results corresponding to the field content.

2. The method according to claim 1, characterized in that The method further comprises: When the detection status identifier is a function test identifier, identifying table structure fields of the initial business table based on the preset mapping relationship table, and determining the field information group corresponding to each table structure field, wherein the initial business table is the to-be-checked business table that has not loaded business data; For each of the table structure fields: call the second type of initial verification template corresponding to each of the field attribute information in the field information group corresponding to the table structure field; and perform field replacement on each of the second type of initial verification templates based on the access address identifier and the table structure field to obtain multiple second type verification templates; use each of the second type of verification templates to perform data quality detection on the table structure field, and generate a data quality detection result corresponding to the table structure field.

3. The method according to claim 1, characterized in that The performing field replacement on each of the first-type initial verification templates based on the access address identifier and the target field name to obtain multiple first-type verification templates includes: For each of the first type of initial verification templates: Obtaining an access address code corresponding to the access address identifier; The access address field in the first type of initial verification template is updated using the access address code, and the access object field in the first type of initial verification template is updated using the target field name to obtain the first type of verification template.

4. The method according to claim 1, wherein The method further comprises: Text extraction is performed on each of the data quality detection results to obtain the name of each problem field and the corresponding verification rule description text, and the preset report template is filled with the problem field name and the verification rule description text to obtain a data quality detection report.

5. A data quality detection system, characterized in that: The system comprises: a first data screening unit configured to, when the detection identifier in the received data detection trigger information is a production verification identifier, perform data screening on a preset mapping relationship table according to an access address identifier in the data detection trigger information, so as to obtain an inspection item table constructed based on the result of the data screening, wherein the preset mapping relationship table stores field information groups in multiple business tables corresponding to each business system, the field information groups including a field name and multiple field attribute information, the field name corresponding to at least one field attribute information, and the inspection item table including multiple field information groups of one business table; a field comparison unit, configured to compare the field names of the to-be-checked business table based on the inspection item table, and determine each target field name and its corresponding target field information group based on the result of the field name comparison, wherein the to-be-checked business table is the business table corresponding to the access address identifier, the target field name is a field name in the to-be-checked business table, and the target field name is consistent with the field name in the target field information group, and the target field information group is one of the field information groups in the inspection item table; The first quality detection unit is used to: for each target field name: retrieve the first type of initial verification template corresponding to each of the field attribute information in the target field information group corresponding to the target field name, and perform field replacement on each of the first type of initial verification templates based on the access address identifier and the target field name to obtain multiple first type verification templates; use each of the first type of verification templates to perform data quality detection on the field content corresponding to the target field name, and generate multiple data quality detection results corresponding to the field content.

6. The system according to claim 5, characterized in that The system further comprises: a second data screening unit, configured to, when the detection status identifier is a function test identifier, identify table structure fields of an initial business table based on the preset mapping relationship table, and determine the field information group corresponding to each table structure field, wherein the initial business table is the to-be-checked business table that has not been loaded with business data; The second quality detection unit is used to: retrieve the second type of initial verification template corresponding to each of the field attribute information in the field information group corresponding to the table structure field for each of the table structure fields; and perform field replacement on each of the second type of initial verification templates based on the access address identifier and the table structure field to obtain multiple second type verification templates; use each of the second type of verification templates to perform data quality detection on the table structure field, and generate a data quality detection result corresponding to the table structure field.

7. The system according to claim 5, characterized in that In the case where the fields of the initial verification templates of the first type are replaced based on the access address identifier and the target field name to obtain multiple verification templates of the first type, the first quality detection unit is configured to: For each of the first type of initial verification templates: Obtaining an access address code corresponding to the access address identifier; The access address field in the first type of initial verification template is updated using the access address code, and the access object field in the first type of initial verification template is updated using the target field name to obtain the first type of verification template.

8. The system according to claim 5, wherein: The system further comprises: The report generation unit is used to perform text extraction on each of the data quality detection results, obtain the name of each problem field and the corresponding verification rule description text, and use the problem field name and the verification rule description text to fill in the preset report template to obtain a data quality detection report.

9. A data quality detection device, characterized in that: The detection equipment includes: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data quality detection method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of a data quality detection device, the detection device is enabled to perform the data quality detection method according to any one of claims 1 to 4.

Images