Data quality inspection method and device, equipment and storage medium
By introducing the decoupling design of the audit engine and the audit SDK in the data quality audit, the problem of poor flexibility in the existing technology is solved, and higher data quality audit flexibility and adaptability are achieved.
Patent Information
- Application Number
- CN202411905524.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-05-09
AI Technical Summary
The existing data quality audit methods are poorly flexible and cannot effectively respond to changes in audit rules and data sources. The overall code needs to be remodeled and deployed.
By introducing the decoupled design of the audit engine and the audit SDK in the data quality audit, the modularization of the audit engine and the audit SDK is realized, allowing separate modifications or upgrades to the audit SDK, improving the flexibility of the modification and upgrading of the data quality audit.
It improves the overall flexibility of data quality auditing, reduces the need for code modification and deployment, and enhances the adaptability and scalability of the system.
Smart Images
Figure CN119961255A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a data quality audit method, device, equipment and storage medium. Background Art
[0002] With the rapid development of social digitization, more and more data needs to be accessed, cleaned, managed and stored. In order to improve the quality of data and reduce the processing and storage of useless data, it is necessary to introduce data quality auditing.
[0003] Data quality audit is to audit the data through corresponding audit rules, and then evaluate the data from six dimensions: accuracy, completeness, consistency, validity, uniqueness and timeliness.
[0004] The existing data quality auditing methods usually encode the auditing rules in the code. As the business develops and the data grows, the corresponding auditing rules and data sources may change. When the rules change, the code needs to be modified and redeployed, which is less flexible. Summary of the invention
[0005] The embodiments of the present application provide a data quality audit method, apparatus, device and storage medium, which can solve the technical problem of poor flexibility of data quality audit and improve the flexibility of data quality audit.
[0006] In a first aspect, an embodiment of the present application provides a data quality audit method, comprising:
[0007] receiving an audit start instruction, and displaying an information menu in an interactive interface in response to the audit start instruction;
[0008] Receive basic information and start instructions based on information menu input, the basic information includes data source information and audit rule information;
[0009] In response to the start instruction, the audit engine is started, and the basic information is parsed and processed by the audit engine to obtain the audit task configuration information, which includes the data source link and the audit rule number;
[0010] The audit engine calls the audit SDK to perform audit processing according to the audit task configuration information to obtain the initial audit results, wherein the audit engine and the audit SDK are decoupled.
[0011] Furthermore, the audit engine calls the audit SDK to perform audit processing according to the audit task configuration information to obtain the initial audit results, including:
[0012] The audit engine obtains the data to be audited from the target database one by one according to the data source link;
[0013] The audit engine determines the rule parameters according to the audit rule number and the preset mapping relationship;
[0014] The audit engine calls the audit SDK according to the rule parameters and transmits the data to be audited to the audit SDK one by one;
[0015] Perform data quality audit on the received data to be audited through the audit SDK, obtain the initial audit results, and feed back the initial audit results to the audit engine;
[0016] The audit engine receives the initial audit results fed back by the audit SDK.
[0017] Furthermore, the audit engine obtains the data to be audited from the target database one by one according to the data source link, including:
[0018] The audit engine determines the target database based on the data source link. The target databases include MySQL, ORACLE, Hive, Clickhouse and Kafka.
[0019] The audit engine traverses the data in the target database according to the data source link, and obtains the data to be audited one by one.
[0020] Furthermore, the audit engine traverses the data in the target database according to the data source link, and obtains the data to be audited one by one, including:
[0021] When the target database is a Kafka database, create the first consumer program through the audit engine;
[0022] The first consumer program monitors the data in the Kafka database based on the data source link to obtain the first audit data one by one;
[0023] The first audit data is deserialized through the first consumer program to obtain the data to be audited.
[0024] Furthermore, after the audit engine receives the initial audit results from the audit SDK, it includes:
[0025] Create a producer program through the audit engine, and serialize the initial audit results through the producer program to obtain the target audit results;
[0026] Write the target audit results into the target Kafka database through the producer program and generate query interface information;
[0027] Based on the query interface information, an audit result query control is generated and displayed in the interactive interface.
[0028] Further, according to the query interface information, after the audit result query control is generated and displayed on the interactive interface, it includes:
[0029] Receive a click operation based on the audit result query control, and retrieve the initial audit result based on the query interface information according to the click operation;
[0030] The initial audit results are displayed in the interactive interface.
[0031] Further, receiving a click operation based on the audit result query control, and retrieving the initial audit result based on the query interface information according to the click operation, including:
[0032] Receive a click operation based on the audit result query control, and create a second consumer program through the audit engine;
[0033] The second consumer program obtains the corresponding target audit results from the target Kafka database according to the query interface information corresponding to the click operation;
[0034] The target audit result is deserialized through the second consumer program to obtain the initial audit result.
[0035] In a second aspect, an embodiment of the present application provides a data quality audit device, including:
[0036] A first instruction receiving module is used to receive an audit start instruction, and in response to the audit start instruction, display an information menu in the interactive interface;
[0037] A second instruction receiving module is used to receive basic information and start instructions based on the information menu input, the basic information includes data source information and audit rule information;
[0038] An information analysis module is used to start the audit engine in response to the start instruction, and to parse and process the basic information through the audit engine to obtain the audit task configuration information, which includes the data source link and the audit rule number;
[0039] The SDK calling module is used to call the audit SDK through the audit engine according to the audit task configuration information to perform audit processing and obtain initial audit results, wherein the audit engine and the audit SDK are decoupled.
[0040] In a third aspect, an embodiment of the present application provides a data quality audit device, including:
[0041] memory and one or more processors;
[0042] A memory for storing one or more programs;
[0043] When one or more programs are executed by one or more processors, the one or more processors implement the data quality audit method of the first aspect.
[0044] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute the data quality audit method of the first aspect.
[0045] In the embodiment of the present application, during a data quality audit, an information menu is displayed in an interactive interface according to a received audit start instruction, and basic information and a start instruction based on the information menu input are received. In response to the start instruction, an audit engine is started, and the basic information is parsed and processed by the audit engine to obtain audit task configuration information; the audit engine calls the audit SDK to perform audit processing according to the audit task configuration information to obtain an initial audit result; wherein the audit engine and the audit SDK are decoupled. By adopting the above-mentioned technical means, the audit engine and the audit SDK can be decoupled to achieve modularization of the audit engine and the audit SDK. When the audit rules or data sources change, the audit SDK can be modified or upgraded separately. Compared with the existing method that requires the entire code to be re-modified and deployed, the flexibility of modification and upgrade of data quality audits is improved, thereby improving the overall flexibility of data quality audits. In addition, various types of audit rules can be configured through the audit SDK, and the corresponding audit rules can be selected according to the information menu. Subsequently, the audit SDK can be called according to the audit rules selected in the information menu to perform data quality audits based on the corresponding audit rules, thereby improving the flexibility of data quality audits.
[0046] The beneficial effects of the data quality auditing device, data quality auditing equipment and storage medium provided above can refer to the beneficial effects of the data quality auditing method. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flow chart of a data quality audit method provided by an embodiment of the present application;
[0048] Figure 2 This is a schematic diagram of an information menu of an interactive interface provided by an embodiment of the present application;
[0049] Figure 3 This is a result query diagram of an interactive interface provided in an embodiment of the present application;
[0050] Figure 4 is a flow chart of another data quality audit method provided in an embodiment of the present application;
[0051] Figure 5 It is a structural schematic diagram of a data quality audit device provided in an embodiment of the present application;
[0052] Figure 6 It is a structural diagram of a data quality audit device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
[0054] Data quality audit is to audit data through corresponding audit rules, and then use the audit rules in combination to evaluate the data from six dimensions: accuracy, completeness, consistency, validity, uniqueness and timeliness. In the existing data quality audit methods, some audit rules are relatively simple and cannot meet the needs of complex businesses; some audit rules are encoded in the code. With the development of business and the growth of data, the corresponding audit rules and data sources may change. When the rules change, the code needs to be modified and redeployed, which is less flexible.
[0055] Based on this, a data quality audit method, apparatus, device and storage medium are provided in an embodiment of the present application, which aims to display an information menu in an interactive interface according to a received audit start instruction during data quality audit, and receive basic information and startup instructions input based on the information menu, and start the audit engine in response to the startup instruction, and parse and process the basic information through the audit engine to obtain audit task configuration information; the audit engine calls the audit SDK to perform audit processing according to the audit task configuration information to obtain an initial audit result; wherein the audit engine and the audit SDK are decoupled. By adopting the above-mentioned technical means, the audit engine and the audit SDK can be decoupled to achieve modularization of the audit engine and the audit SDK. When the audit rules or data sources change, the audit SDK can be modified or upgraded separately. Compared with the existing method that requires the entire code to be re-modified and deployed, the flexibility of modification and upgrade of data quality audits is improved, thereby improving the overall flexibility of data quality audits. In addition, various types of audit rules can be configured through the audit SDK, and the corresponding audit rules can be selected according to the information menu. Subsequently, the audit SDK can be called according to the audit rules selected in the information menu to perform data quality audits based on the corresponding audit rules, thereby improving the flexibility of data quality audits.
[0056] Figure 1 A flow chart of a data quality audit method provided in an embodiment of the present application is given. The data quality audit method provided in this embodiment can be executed by a data quality audit device, which can be implemented by software and / or hardware. The data quality audit device can be composed of two or more physical entities, or can be composed of one physical entity. Generally speaking, the data quality audit device can be a computer device.
[0057] The following description takes computer equipment as the subject of the data quality audit method as an example. Figure 1 The data quality audit method specifically includes:
[0058] S101. Receive an audit start instruction, and in response to the audit start instruction, display an information menu in an interactive interface.
[0059] A corresponding data quality audit application is installed in the data quality audit device. Based on the startup of the data quality audit application, the interactive interface of the data quality audit application is entered. The user can enter the audit start instruction in the interactive interface of the data quality audit application. Exemplarily, the audit start instruction can be entered by clicking on the corresponding control (e.g., the "start" control) in the interactive interface.
[0060] The data quality audit device receives an audit start instruction, and in response to the audit start instruction, displays an information menu in an interactive interface, where the information menu is used to provide selection of data source information and selection of audit rules.
[0061] Figure 2 is a schematic diagram of an information menu of an interactive interface provided in an embodiment of the present application, referring to Figure 2 In the interactive interface of the data quality audit application, an information menu is displayed, and the information menu includes a first control 11 and a second control 12. When the first control 11 is clicked, a first submenu 13 is displayed, and the first submenu 13 is used to display the data source control to be selected. For example, the data source control of the first submenu 13 includes data source 1, data source 2, data source 3, data source 4 and data source 5. The user can click on the data source control to determine which data source to perform a data quality audit on. When the second control 12 is clicked, a second submenu 14 is displayed, and the second submenu 14 is used to display the audit rule control to be selected. For example, the audit rule control of the second submenu 14 includes audit rule 1, audit rule 2, audit rule 3 and audit rule 4. The user can click on the audit rule control to determine which audit rule to use for data quality audit.
[0062] It should be noted that the audit rules corresponding to each audit rule control are preset in the audit SDK, and the preset audit rules can be saved in the audit SDK in the form of code, configuration file, database and / or cloud storage.
[0063] Audit SDK (Software Development Kit) refers to a software development kit used for audit or monitoring functions. Audit SDK contains a series of software components, development libraries, tools, interfaces and documents, which can realize the rapid construction, integration and use of specific software or services. The audit SDK (tool) has built-in preset audit rules, such as audit rules 1-4. Each audit rule corresponds to an audit rule number. The audit rule number can be selected in the interactive interface to determine which audit rule to use for the audit. By presetting multiple types of audit rules in the audit SDK, most data quality audit business scenarios can be covered.
[0064] Exemplarily, install the audit SDK (tool), create an audit plan in the data quality audit system, and add the path of the audit SDK to the build path of the audit plan. Later, during the data quality audit, you can access the API and library provided by the audit SDK.
[0065] S102, receiving basic information and start instructions based on information menu input, the basic information including data source information and audit rule information.
[0066] After the menu information is displayed on the interactive interface, basic information and start instructions based on the menu information input are received, wherein the basic information includes data source information and audit rule information. The data source information can be obtained by triggering the aforementioned data source control, and the data source information includes information such as the audit target (i.e., data source number, such as data source 1), range, data type, and time requirement. The audit rule information can be obtained by triggering the aforementioned audit rule control, and the audit rule information can be the audit rule number, such as audit rule 1. The start instruction is triggered based on the start instruction control in the interactive interface, such as Figure 2 The "Start Audit" control in .
[0067] Exemplarily, data sources include relational databases and non-relational databases. Among them, relational databases are structured databases that use relational modules to organize data. Therefore, when performing data quality audits on data in relational databases, the data can be directly read and audited. Non-relational databases are generally used to store data that is not fixed in type or has no fixed rules. Non-relational databases are a collection of structured data storage methods, which can be documents or key-value pairs. Non-relational data is not suitable for storage in rows and columns of data tables. Non-relational data is usually stored in data sets, such as documents, key-value pairs, or graph structures. Therefore, when performing data quality audits on data in non-relational databases, the read data cannot be directly audited. It is necessary to perform corresponding data format processing before it can be audited.
[0068] Exemplarily, the data sources include MySOL database, ORACLE database, Hive database, Clickhouse database and Kafka database, wherein MySOL database, ORACLE database, Hive database and Clickhouse database are relational databases, and Kafka database is a non-relational database.
[0069] S103. In response to the start instruction, the audit engine is started, and the basic information is parsed and processed by the audit engine to obtain the audit task configuration information, which includes the data source link and the audit rule number.
[0070] After receiving the start command, the audit engine is started in response to the start command. The audit engine receives the basic information obtained based on the information menu, and parses the basic information to obtain the audit task configuration information, which is used to guide subsequent audit work. The audit task configuration information includes data source links and audit rule numbers, etc. The basic information is the information entered by the user through the information menu of the interactive interface, which corresponds to a table or a form in the background. Therefore, the audit engine needs to parse the basic information to parse out the corresponding data source link (i.e. determine the target database), audit scope, data type, time requirements and audit rule number and other information. Subsequently, the target database can be determined based on the data source link, and which audit rule to use can be determined based on the audit rule number, and the corresponding field name and rule list can be determined.
[0071] S104. The audit engine calls the audit SDK according to the audit task configuration information to perform audit processing and obtain initial audit results, wherein the audit engine and the audit SDK are decoupled.
[0072] The audit engine determines the target database based on the data source link, and obtains the data to be audited from the target database one by one according to the audit scope. The audit engine determines the rule parameters according to the audit rule number and the preset mapping relationship, where the rule parameters can be parameters such as field names and rule lists. The audit engine calls the audit SDK according to the rule parameters, and transmits the data to be audited to the audit SDK one by one. The audit SDK performs data quality audit on the received data to be audited, obtains the initial audit results, and feeds back the initial audit results to the audit engine. The audit engine receives the initial audit results fed back by the audit SDK.
[0073] It should be noted that the audit engine and the audit SDK are decoupled and modularized. When the audit rules or data sources change, the audit engine and the audit SDK can be modified or upgraded separately. Compared with the existing method that requires the entire code to be modified and redeployed, this improves the modification flexibility of data quality audits, thereby improving the overall flexibility of data quality audits.
[0074] Since the target database may be a relational database or a non-relational database, different data quality audit processes need to be performed on relational databases and non-relational databases.
[0075] The audit engine determines the target database based on the data source link. When the target database is a relational database, that is, the target database is a MySOL database, an ORACLE database, a Hive database, or a Clickhouse database, the read data can be directly transmitted to the audit SDK for audit processing. Therefore, the audit engine can traverse the corresponding data range in the target database according to the data source link and read the corresponding data to be audited one by one. The audit engine calls the audit SDK according to the rule parameters and transmits the data to be audited to the audit SDK one by one. The audit SDK performs data quality audit processing on the received data to be audited based on the corresponding audit rules, obtains the initial audit results, and feeds back the initial audit results to the audit engine. The audit engine receives the initial audit results fed back by the audit SDK.
[0076] The target database is determined by the audit engine according to the data source link. When the target database is a non-relational database, that is, when the target database is a kafka database, the non-relational database is not in the form of a structured data table, but may be in the form of a document, a key-value pair or a graph structure. If the read data is directly transmitted to the audit SDK, the audit SDK cannot recognize the corresponding data. Therefore, when the target database is a kafka database, a first consumer program can be created through the audit engine, and the first consumer program monitors the corresponding data range in the kafka database based on the data source link to read the corresponding first audit data one by one. The first consumer program deserializes the first audit data to obtain the data to be audited. In the above manner, the data to be audited can be obtained one by one. The audit engine calls the audit SDK according to the rule parameters, and transmits the data to be audited to the audit SDK one by one. The audit SDK performs data quality audit processing on the received data to be audited based on the corresponding audit rules, obtains the initial audit result, and feeds back the initial audit result to the audit engine. The initial audit result fed back by the audit SDK is received by the audit engine.
[0077] As described above, through the data quality audit method provided in this embodiment, data quality audit can be performed on relational databases and non-relational databases, which expands the business scenarios of data quality audit and thus improves the user experience.
[0078] To improve the persistence and reliability of audit result data, the audit result data can be pushed to the Kafka database for storage. The audit SDK performs data quality audit processing to obtain the initial audit result, and feeds back the initial audit result to the audit engine. The audit engine receives the initial audit result and creates a producer program. The producer program serializes the initial audit result to obtain the target audit result. The producer program writes the target audit result to the target Kafka database and generates query interface information, which is used to retrieve the initial audit result. Based on the query interface information, the audit result query control is generated and displayed in the interactive interface.
[0079] When the user needs to query the corresponding audit result, he can click the corresponding audit result query control in the interactive interface. The data quality audit device receives the click operation based on the audit result query control, creates a second consumer program through the audit engine; determines the corresponding query interface information according to the click operation through the second consumer program, and obtains the corresponding target audit result from the target kafka database according to the query interface information; deserializes the target audit result through the second consumer program to obtain the corresponding initial audit result, and displays the initial audit result in the interactive interface.
[0080] Figure 3 This is a result query diagram of an interactive interface provided in an embodiment of the present application, referring to Figure 3 , the target audit result is written into the target kafka database through the audit engine, and the query interface information is generated. According to the query interface information, the audit result query control 15 is generated and displayed in the interactive interface. When the user needs to query the corresponding audit result, the corresponding audit result query control 15 can be clicked in the interactive interface. For example, suppose the user clicks the audit result query control 15 corresponding to "XX1's audit result", that is, the "query" control in the figure. The data quality audit system receives the click operation based on the audit result query control 15, and creates a second consumer program through the audit engine; the corresponding query interface information is determined according to the click operation through the second consumer program, and the target audit result corresponding to "XX1" is obtained from the target kafka database according to the query interface information; the target audit result is deserialized through the second consumer program to obtain the initial audit result corresponding to "XX1", and the initial audit result is displayed in the interactive interface.
[0081] Based on the above implementation, Figure 4 is a flowchart of another data quality audit method provided in an embodiment of the present application, referring to Figure 4 The data quality audit method specifically includes:
[0082] S201. Audit begins.
[0083] The audit start instruction may be triggered based on the interactive interface of the corresponding data quality audit application or data quality audit system. The data quality audit application or data quality audit system executes subsequent data quality audit steps based on the audit start instruction.
[0084] S202. Create an audit task and select a data source.
[0085] Audit tasks can be created based on the interactive interface of the data quality audit application or data quality audit system, and the data source to be audited can be selected based on the user's interactive operation. The data sources include MySQL database, ORACLE database, Hive database, Clickhouse database and Kafka database. Creating an audit task includes selecting the data source to be audited (data table or database) and the corresponding audit rules to obtain basic basic information.
[0086] It should be noted that if the user does not select an audit rule, the default audit rule will be used for subsequent data quality audit processing.
[0087] S203. Configure rules for the fields according to business requirements.
[0088] According to the basic information when creating the audit task mentioned above, determine the business requirements, where the business requirements include the fields that need to be audited, the data type of the field, the expected data format and range, and the specific content of the audit rules. Configure audit rules for the fields according to business requirements. Exemplarily, the audit rules include: field validation, data range validation, format validation, and business logic validation. Among them, field validation is used to check whether the field exists, whether it is empty, or whether it conforms to the expected data type. Data range validation is used to check whether the value of the field is within the expected range, such as whether the age is between 0 and 150, whether the date is within the valid range, etc. Format validation is used to check whether the format of the field is as expected, such as whether the email address conforms to the format specification of the email, whether the phone number conforms to the format specification of the phone number, etc. Business logic validation is used to check whether the value of the field meets specific conditions according to the business logic, such as checking whether the order status is "paid" before it can be shipped. Configure rules for the field according to business needs and determine basic information.
[0089] S204: Start the audit task and pass the basic information to the audit engine.
[0090] Start the audit task, start the audit engine, and pass the basic information to the audit engine.
[0091] S205: The audit engine starts processing.
[0092] The audit engine parses and processes the basic information to obtain the audit task configuration information, which includes the data source link and the audit rule number.
[0093] S206: Determine whether the data source is a relational database.
[0094] The audit engine determines the target database based on the data source link and determines whether the target database is a relational database. If so, execute S207; if not, execute S208.
[0095] S207, query data and call the audit method of the audit SDK in a loop.
[0096] The audit engine determines the target database based on the data source link and determines whether the target database is a relational database. If the target database is a relational database, that is, the target database is a MySOL database, ORACLE database, Hive database or Clickhouse database, the audit engine traverses the corresponding data range in the target database and obtains the data to be audited one by one. The audit engine calls the audit SDK according to the rule parameters and transmits the data to be audited to the audit SDK one by one. The audit SDK performs data quality audit on the received data to be audited, obtains the initial audit results, and feeds back the initial audit results to the audit engine. The audit engine receives the initial audit results fed back by the audit SDK.
[0097] S208. Create a consumer program to monitor data in real time. When receiving data, call the audit method of the audit SDK.
[0098] The audit engine determines the target database based on the data source link, and determines whether the target database is a relational database. If the target database is a non-relational database, that is, the target database is a Kafka database, the audit engine creates a first consumer program, and the first consumer program monitors the corresponding data range in the Kafka database based on the data source link to obtain the first audit data one by one. The first consumer program deserializes the first audit data to obtain the data to be audited. In the above manner, the data to be audited can be obtained one by one. The audit engine calls the audit SDK according to the rule parameters, and transmits the data to be audited to the audit SDK one by one. The audit SDK performs data quality audit on the received data to be audited, obtains the initial audit result, and feeds back the initial audit result to the audit engine. The audit engine receives the initial audit result fed back by the audit SDK.
[0099] S209. Push the audit results to the Kafka database.
[0100] In order to improve the persistence and reliability of the audit result data, the audit result data can be pushed to the kafka database for storage. The audit engine receives the initial audit results, and then creates a producer program through the audit engine. The initial audit results are serialized through the producer program to obtain the target audit results. The target audit results are written to the target kafka database through the producer program, and query interface information is generated. The query interface information is used to retrieve the corresponding initial audit results when the audit results need to be viewed later. As mentioned above, by writing the audit results into the target kafka database, the audit results can be persisted, thereby improving the reliability of the audit result storage.
[0101] S210. The audit is completed.
[0102] As mentioned above, by setting up the audit SDK (tool), the preset audit rules are built into the audit SDK, which can cover most of the business scenarios of data quality audit. When calling the audit SDK, it is necessary to pass in the data, field name, rule list or audit rule number applied to the field. In this embodiment, the audit engine passes the corresponding data to be audited and the audit rule number to the audit SDK. After the audit SDK performs data quality audit processing on the received data to be audited, it returns the initial audit result to the audit engine. Among them, the initial audit result includes whether it passes and the reason for failure, etc.
[0103] In the above, the audit engine receives the basic information corresponding to the audit task, parses the basic information, and obtains the audit task configuration information. The audit engine calls the audit SDK according to the audit task configuration information, so that the audit SDK performs data quality audit processing on the audit data and feedbacks the initial audit result. After the audit engine performs corresponding processing on the received initial audit result, it obtains the target audit result and pushes the target audit result to the target Kafka database for persistent storage.
[0104] As mentioned above, an interactive interface is provided to the user for interactive operation through a data quality audit application or a data quality audit system. The data quality audit application or the data quality audit system supports the selection of a data source based on an interactive interface, and supports the configuration of audit rules for the corresponding fields under the data source to form an audit plan. When the audit task is started, the audit plan is first converted into task information that the audit engine can process (i.e., the aforementioned audit task configuration information), and then the audit engine performs a data quality audit based on the task information. The data quality audit application or the data quality audit system provided in this embodiment supports the creation of a new audit plan, and the content of the new audit plan includes the data source (data table) to be audited and the audit rules for configuring each field. The data quality audit application or the data quality audit system provided in this embodiment supports the operation of the audit plan, and the data source information, fields, and the audit rules configured by the fields can be transmitted to the audit engine for data quality audit. The data quality audit application or data quality audit system provided in this embodiment can support the audit engine to output Kafka-type audit results, write the target audit results into the target Kafka database, and provide query interface information for subsequent query of the audit results.
[0105] As mentioned above, through the preset audit rules built into the audit SDK, the audit engine implements data quality audit by calling the audit SDK, and uses kafka as the message notification queue to transmit the audit results. The coupling between the modules is low, and it is easy to implement and expand. The overall architecture is divided into three major modules: audit SDK, audit engine, and data quality audit system (interactive platform), and the coupling between the modules is low. When the audit SDK adds new audit rules, the data quality audit system only needs to initialize the new audit rules through SQL scripts, and the new audit rules can be referenced in the audit plan. In this process, the interaction between the audit engine and the audit SDK will not change, and the newly added audit rules are transparent to the audit engine, thereby greatly reducing the coupling between the audit engine and the audit SDK. In addition, this embodiment uses kafka as the message middleware for the audit results to avoid the audit results being directly stored in the warehouse and affecting the audit performance, thereby improving the audit efficiency.
[0106] In the above, according to the received audit start instruction, the information menu is displayed in the interactive interface, and the basic information and the start instruction based on the information menu input are received. In response to the start instruction, the audit engine is started, and the basic information is parsed and processed by the audit engine to obtain the audit task configuration information; the audit engine calls the audit SDK according to the audit task configuration information to perform the audit process and obtain the initial audit result; wherein, the audit engine and the audit SDK are decoupled. By adopting the above technical means, the audit engine and the audit SDK can be decoupled to realize the modularization of the audit engine and the audit SDK. When the audit rules or data sources change, the audit SDK can be modified or upgraded separately. Compared with the existing method that requires the whole code to be re-modified and deployed, the flexibility of the modification and upgrade of the data quality audit is improved, thereby improving the overall flexibility of the data quality audit; in addition, various types of audit rules can be configured through the audit SDK, and the corresponding audit rules can be selected according to the information menu. Subsequently, the audit SDK can be called according to the audit rules selected in the information menu to perform data quality audit processing based on the corresponding audit rules, thereby improving the flexibility of the data quality audit.
[0107] Based on the above embodiments, Figure 5 A schematic diagram of the structure of a data quality audit device provided in an embodiment of the present application. Figure 5 The data quality audit device provided in this embodiment specifically includes: a first instruction receiving module 21, a second instruction receiving module 22, an information parsing module 23 and an SDK calling module 24.
[0108] The first instruction receiving module 21 is used to receive an audit start instruction, and in response to the audit start instruction, display an information menu in the interactive interface;
[0109] The second instruction receiving module 22 is used to receive basic information and start instructions based on the information menu input, the basic information includes data source information and audit rule information;
[0110] The information parsing module 23 is used to start the audit engine in response to the start instruction, and parse the basic information through the audit engine to obtain the audit task configuration information, which includes the data source link and the audit rule number;
[0111] The SDK calling module 24 is used to call the audit SDK to perform audit processing according to the audit task configuration information through the audit engine to obtain the initial audit result, wherein the audit engine and the audit SDK are decoupled.
[0112] In one embodiment, the SDK calling module 24 includes: a data reading submodule, a rule parameter determination submodule, a calling submodule, an audit processing submodule and a result feedback submodule;
[0113] The data reading submodule is used to obtain the data to be audited from the target database one by one through the audit engine according to the data source link;
[0114] The rule parameter determination submodule is used to determine the rule parameters according to the audit rule number and the preset mapping relationship through the audit engine;
[0115] The calling submodule is used to call the audit SDK according to the rule parameters through the audit engine, and transmit the data to be audited to the audit SDK one by one;
[0116] The audit processing submodule is used to perform data quality audit on the received data to be audited through the audit SDK, obtain the initial audit results, and feed back the initial audit results to the audit engine;
[0117] The result feedback submodule is used to receive the initial audit results fed back by the audit SDK through the audit engine.
[0118] In one embodiment, the data reading submodule includes: a database type determination unit and a first data reading unit;
[0119] The database type determination unit is used to determine the target database according to the data source link through the audit engine. The target databases include MySQL database, ORACLE database, Hive database, Clickhouse database and Kafka database.
[0120] The first data reading unit is used to traverse the data in the target database according to the data source link through the audit engine, and obtain the data to be audited one by one.
[0121] In one embodiment, the data reading submodule includes: a program creation unit, a second data reading unit and a first data processing unit;
[0122] A program creation unit, used to create a first consumer program through an audit engine when the target database is a Kafka database;
[0123] A second data reading unit is used to monitor the data in the Kafka database based on the data source link through the first consumer program to obtain the first audit data one by one;
[0124] The first data processing unit is used to deserialize the first audit data through the first consumer program to obtain the data to be audited.
[0125] In one embodiment, the data quality audit device further includes: a result preprocessing module, a result writing module and a control generation module;
[0126] The result preprocessing module is used to create a producer program through the audit engine, and serialize the initial audit results through the producer program to obtain the target audit results;
[0127] The result writing module is used to write the target audit results into the target Kafka database through the producer program and generate query interface information;
[0128] The control generation module is used to generate and display the audit result query control in the interactive interface according to the query interface information.
[0129] In one embodiment, the data quality audit device further includes: a query module and a result display module;
[0130] A query module is used to receive a click operation based on the audit result query control, and retrieve the initial audit result based on the query interface information according to the click operation;
[0131] The result display module is used to display the initial audit results in the interactive interface.
[0132] In one embodiment, the query module includes: a query control triggering submodule, a query result obtaining submodule, and a query result processing submodule;
[0133] The query control trigger submodule is used to receive a click operation based on the audit result query control and create a second consumer program through the audit engine;
[0134] The query result acquisition submodule is used to obtain the corresponding target audit result from the target Kafka database according to the query interface information corresponding to the click operation through the second consumer program;
[0135] The query result processing submodule is used to deserialize the target audit result through the second consumer program to obtain the initial audit result.
[0136] The data quality auditing device provided in the embodiment of the present application can be used to execute the data quality auditing method provided in the above embodiment, and has corresponding functions and beneficial effects.
[0137] The present application embodiment provides a data quality audit device, referring to Figure 6 The data quality audit device includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The number of processors in the data quality audit device can be one or more, and the number of memories in the data quality audit device can be one or more. The processor, memory, communication module, input device and output device of the data quality audit device can be connected through a bus or other methods.
[0138] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the data quality audit method of any embodiment of the present application (for example, the first instruction receiving module, the second instruction receiving module, the information parsing module and the SDK calling module in the data quality audit device). The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.
[0139] The communication module 33 is used for data transmission.
[0140] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory, that is, implements the above-mentioned data quality audit method.
[0141] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.
[0142] The data quality auditing device provided above can be used to execute the data quality auditing method provided in the above embodiment, and has corresponding functions and beneficial effects.
[0143] An embodiment of the present application also provides a storage medium storing computer executable instructions, which are used to execute a data quality audit method when executed by a computer processor. The data quality audit method includes: receiving an audit start instruction, and displaying an information menu in an interactive interface in response to the audit start instruction; receiving basic information and a startup instruction based on the information menu input, the basic information including data source information and audit rule information; starting an audit engine in response to the startup instruction, parsing the basic information through the audit engine to obtain audit task configuration information, the audit task configuration information including a data source link and an audit rule number; calling the audit SDK through the audit engine to perform audit processing according to the audit task configuration information to obtain an initial audit result, wherein the audit engine and the audit SDK are decoupled.
[0144] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.
[0145] Of course, the computer executable instructions of a storage medium storing computer executable instructions provided in an embodiment of the present application are not limited to the data quality audit method described above, and can also execute related operations in the data quality audit method provided in any embodiment of the present application.
[0146] The data quality auditing device, storage medium and data quality auditing equipment provided in the above embodiments can execute the data quality auditing method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the data quality auditing method provided in any embodiment of the present application.
[0147] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.
Claims
1. A data quality audit method, characterized in that: include: Receiving an audit start instruction, and displaying an information menu in an interactive interface in response to the audit start instruction; Receiving basic information and a start instruction input based on the information menu, wherein the basic information includes data source information and audit rule information; In response to the start instruction, the audit engine is started, and the basic information is parsed and processed by the audit engine to obtain audit task configuration information, wherein the audit task configuration information includes a data source link and an audit rule number; The audit engine calls the audit SDK to perform audit processing according to the audit task configuration information to obtain an initial audit result, wherein the audit engine and the audit SDK are decoupled.
2. The method according to claim 1, characterized in that The audit engine calls the audit SDK to perform audit processing according to the audit task configuration information to obtain the initial audit result, including: The audit engine obtains the data to be audited from the target database one by one according to the data source link; Determine the rule parameters according to the audit rule number and the preset mapping relationship by the audit engine; The audit engine calls the audit SDK according to the rule parameters, and transmits the data to be audited to the audit SDK one by one; Performing data quality audit on the received data to be audited through the audit SDK to obtain an initial audit result, and feeding back the initial audit result to the audit engine; The initial audit results fed back by the audit SDK are received through the audit engine.
3. The method according to claim 2, characterized in that The step of obtaining the data to be audited from the target database one by one according to the data source link by the audit engine includes: Determine the target database according to the data source link through the audit engine, and the target database includes MySOL database, ORACLE database, Hive database, Clickhouse database and Kafka database; The audit engine traverses the data in the target database according to the data source link to obtain the data to be audited one by one.
4. The method according to claim 3, characterized in that The audit engine traverses the data in the target database according to the data source link to obtain the data to be audited one by one, including: When the target database is a Kafka database, creating a first consumer program through the audit engine; The first consumer program monitors the data in the Kafka database based on the data source link to obtain the first audit data one by one; The first audit data is deserialized through the first consumer program to obtain the data to be audited.
5. The method according to claim 2, characterized in that: After the audit engine receives the initial audit result fed back by the audit SDK, the method includes: Creating a producer program through the audit engine, and serializing the initial audit result through the producer program to obtain a target audit result; The target audit result is written into the target Kafka database through the producer program, and query interface information is generated; Based on the query interface information, an audit result query control is generated and displayed on the interactive interface.
6. The method according to claim 5, characterized in that After the audit result query control is generated and displayed on the interactive interface according to the query interface information, the method includes: Receiving a click operation based on the audit result query control, and retrieving the initial audit result based on the query interface information according to the click operation; The initial audit result is displayed in the interactive interface.
7. The method according to claim 6, characterized in that The receiving a click operation based on the audit result query control, and retrieving the initial audit result based on the query interface information according to the click operation, comprises: Receiving a click operation based on the audit result query control, and creating a second consumer program through the audit engine; Obtaining the corresponding target audit result from the target Kafka database through the second consumer program according to the query interface information corresponding to the click operation; The target audit result is deserialized by the second consumer program to obtain the initial audit result.
8. A data quality audit device, characterized in that: include: A first instruction receiving module is used to receive an audit start instruction, and in response to the audit start instruction, display an information menu in the interactive interface; A second instruction receiving module, used for receiving basic information and a start instruction input based on the information menu, wherein the basic information includes data source information and audit rule information; An information parsing module, used to start the audit engine in response to the startup instruction, and parse the basic information through the audit engine to obtain audit task configuration information, wherein the audit task configuration information includes a data source link and an audit rule number; The SDK calling module is used to call the audit SDK through the audit engine according to the audit task configuration information to perform audit processing and obtain initial audit results, wherein the audit engine and the audit SDK are decoupled.
9. A data quality audit device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
10. A storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to perform the method according to any one of claims 1 to 7 when executed by a processor.