A data auditing method, device, system and electronic device

Abstract

The present disclosure relates to the technical field of data processing, in particular to a data auditing method, device, system and electronic equipment. The method comprises: obtaining a big data computing engine type, determining a preset scheduling template based on the engine type, wherein the scheduling template comprises a data auditing task script template based on the engine; obtaining a data source type and an auditing rule, generating a task script for data auditing of the data source based on the auditing rule based on the data source type, the auditing rule and the scheduling template, wherein the task script is based on the big data computing engine; obtaining a scheduling requirement, executing the task script based on the scheduling requirement, generating an auditing report and a running record.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to a data auditing method, apparatus, system, and electronic device. Background Technology

[0002] Data quality issues such as redundancy, distortion, missing data, and inconsistency severely hinder data from realizing its maximum value. Several software programs, such as Apache Griffin, Qualitis, and DataWorks, have emerged to integrate and manage dataset quality issues. However, these three mainstream data quality management software programs only support auditing limited data source types based on limited data quality audit rules. Adding support for new data source types and new data quality audit rules is extremely complex, making it impossible to respond promptly to customer-specified new data source types or audits based on customer-specified new audit rules. Furthermore, they either support the Spark or Flink computing engines, failing to meet the needs of different customers using different preferred computing engines for data auditing. Summary of the Invention

[0003] This disclosure presents a data auditing method, a data auditing device, a data auditing system, and an electronic device to address the problem that existing data quality management software cannot respond promptly to diverse customer needs due to its architecture by creating data quality auditing tasks based on templates.

[0004] A first aspect of this disclosure provides a data auditing method, including:

[0005] Obtain the big data computing engine type, and determine a preset scheduling template based on the engine type, wherein the scheduling template includes a data audit task script template based on the engine;

[0006] Obtain the data source type and audit rules, and generate a task script to perform data audit on the data source based on the audit rules and the scheduling template, wherein the task script is based on the big data computing engine;

[0007] Obtain scheduling requirements, execute the task script based on the scheduling requirements, and generate audit reports and operation records.

[0008] In some embodiments, the data auditing method further includes:

[0009] If the audit report contains abnormal data, an alarm will be triggered for the abnormal data according to the audit rules.

[0010] In some embodiments, generating a task script to perform data audit on the data source based on the audit rules and the scheduling template includes:

[0011] Generate the table creation statement in the task script based on the data source type;

[0012] The selection statements in the task script are generated based on the audit rules;

[0013] The task script is generated by replacing the corresponding parts in the template with the table creation statement and the selection statement, wherein the task script is based on the big data computing engine.

[0014] In some embodiments, executing the task script based on the scheduling requirements includes:

[0015] Update the scheduling template based on the data source type and the audit rules;

[0016] Data quality scheduling parameters are generated based on the updated scheduling template, and the parameters are transmitted to the scheduling system, wherein the parameters include the task script;

[0017] The scheduling system generates the data quality audit task schedule based on the parameters and the scheduling requirements, and the schedule includes the task script.

[0018] In some embodiments, determining the preset scheduling template based on the engine type includes:

[0019] Retrieve a preset scheduling template from the template library based on the big data computing engine type;

[0020] If the scheduling template is not found, the search continues in the data dictionary.

[0021] In some embodiments, the big data computing engine type is Spark.

[0022] A second aspect of this disclosure provides a data auditing apparatus, comprising:

[0023] The determination module is used to obtain the big data computing engine type and determine a preset scheduling template based on the engine type, wherein the scheduling template includes a data audit task script template based on the engine;

[0024] A generation module is used to obtain the data source type and audit rules, and generate a task script to perform data audit on the data source based on the audit rules and the scheduling template, wherein the task script is based on the big data computing engine;

[0025] The audit module is used to obtain scheduling requirements and execute the task script based on the scheduling requirements.

[0026] A third aspect of this disclosure provides a data auditing system, including a database, and further comprising:

[0027] The scheduling unit is used to obtain the data source type and audit rules to be audited, generate a Spark execution script with specified audit rules based on the data source type and audit rules, obtain scheduling requirements, generate and execute data audit tasks based on the scheduling requirements and the Spark execution script, obtain audit report data, and write the data to the database.

[0028] In some embodiments of this disclosure, the data auditing system further includes:

[0029] The audit management unit is used to obtain alarm rules for audit anomaly data, retrieve the audit report data from the database, and issue alarms for the abnormal data in the audit report data based on the alarm rules.

[0030] A fourth aspect of this disclosure provides an electronic device, including a memory and a processor.

[0031] The memory is used to store computer programs;

[0032] The processor is configured to implement the method according to the first aspect of this disclosure when executing the computer program.

[0033] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0034] A sixth aspect of this disclosure provides a computer program product, including a computer program and instructions, which, when executed by a processor, implement the method described in the first aspect of this disclosure.

[0035] This disclosure embodiment allows customers to quickly generate data quality audit tasks by creating data quality audit tasks based on templates, and execute them at the customer's specified time. This flexibly meets the diverse needs of different customers. Attached Figure Description

[0036] The features and advantages of this disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the scope of this disclosure in any way.

[0037] Figure 1 This is a schematic diagram of a computer system to which this disclosure applies;

[0038] Figure 2 This is a flowchart illustrating a data auditing method according to some embodiments of the present disclosure;

[0039] Figure 3 This is a detailed flowchart of a data auditing method according to some other embodiments of the present disclosure;

[0040] Figure 4 This is a schematic diagram of a data auditing apparatus according to some embodiments of the present disclosure;

[0041] Figure 5 This is an architecture diagram of a data auditing system according to some embodiments of the present disclosure;

[0042] Figure 6 This is a schematic diagram of an electronic device shown in some embodiments of this disclosure. Detailed Implementation

[0043] In the following detailed description, numerous specific details of this disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this disclosure are a method of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if they can achieve the same purpose.

[0044] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to, or communicate with, other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used in this disclosure includes any one and all combinations of one or more of the associated listed items.

[0045] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As shown in this specification and claims, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified features, integrals, steps, operations, elements, and / or components, and such expressions do not constitute an exclusive list, in which other features, integrals, steps, operations, elements, and / or components may also be included.

[0046] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this disclosure can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this disclosure. It is understood that the drawings are not drawn to scale.

[0047] Various structural diagrams are used in this disclosure to illustrate various variations of embodiments according to this disclosure. It should be understood that the preceding or following structures are not intended to limit this disclosure. The scope of protection of this disclosure is defined by the claims.

[0048] Figure 1 This is a schematic diagram of a computer system to which this disclosure applies. For example... Figure 1 The system shown has an audit server connected to one or more databases (database 1, database 2, database 3, ..., database N, where N is a natural number), an interactive client, and an audit data storage device via a network. The audit server obtains audit requirements (including audit rules, database type to be audited, big data computing engine, audit time, etc.) from the interactive client, generates a data audit task schedule based on the audit requirements, generates a data audit task schedule instance at the audit time, performs data audit on the database to be audited according to the audit rules, and saves the generated audit report and operation record in the audit data storage device.

[0049] in:

[0050] The audit server integrates a big data computing engine and can execute audit tasks via scripts. The audit server can be a single machine, a cluster, or a distributed server.

[0051] The interactive client can be a mobile device based on a mobile operating system, such as Android or Apple, or a personal computer, workstation, or server based on a Windows operating system. The interactive client can obtain the audit request input by the user and send the audit request to the audit server.

[0052] In some embodiments of this disclosure, the interactive client is deployed as software on the audit server. The client can select the database type to be audited, audit rules, audit time, and big data computing engine in the interactive client. If the user does not select a big data computing engine, in some embodiments of this disclosure, the system defaults to the Spark computing engine.

[0053] Audit report data generated during the execution of data audit tasks is written to the audit data storage device. The audit data storage device can be a disk deployed on the audit server or a separate storage server, which can be any of a single machine, a cluster, or a distributed server.

[0054] The database to be audited can be of various types, such as MySQL, Hive, and Elasticsearch. In particular, the computer system supports dynamically adding data audits for new types of databases.

[0055] Figure 2 This is a flowchart illustrating a data auditing method according to some embodiments of the present disclosure. In some embodiments, the data auditing method may be provided by... Figure 1 The audit server shown is executing.

[0056] like Figure 2 As shown, the data auditing method includes the following steps:

[0057] S201, Obtain the big data computing engine type, and determine a preset scheduling template based on the engine type, wherein the scheduling template includes a data audit task script template based on the engine.

[0058] Specifically, in some embodiments of this disclosure, the template library of the audit server has preset scheduling templates corresponding to various big data computing engine types, such as shell scheduling templates, Spark scheduling templates, Flink scheduling templates, etc., and the scheduling templates include data audit task script templates based on the engine.

[0059] In some embodiments of this disclosure, a user-specified big data engine is obtained from the interactive client. If the client does not specify one, the Spark engine is used by default. A preset scheduling template is retrieved from the template library based on the big data computing engine type.

[0060] If the scheduling template is not found, the search continues in the data dictionary.

[0061] For newly emerging big data computing engines, we will pre-create corresponding scheduling templates and upload them to the data dictionary. When the scheduling template is used for the first time, it will be saved in the template library.

[0062] S202, Obtain the data source type and audit rules, and generate a task script to perform data audit on the data source based on the audit rules and the scheduling template, wherein the task script is based on the big data computing engine;

[0063] Specifically, data audits that comply with international standards should at least include the following audit rules:

[0064] Aggregate calculation, completeness, uniqueness, and accuracy.

[0065] in:

[0066] Aggregation calculation refers to aggregating data from certain dimensions and configuring alarm rules with custom metrics, such as maximum value, minimum value, average value, sum, null value, etc.

[0067] Completeness refers to the process of identifying whether data is complete and whether there are missing values. It involves validating one or more fields to check for NULL, null, or empty values.

[0068] As the data industry becomes more deeply integrated with business processes, more audit rules may emerge.

[0069] Users can select the database type to be audited and the applicable audit rules in the interactive client. In some embodiments of this disclosure,

[0070] Generate table creation statements based on the data source type and the big data computing engine;

[0071] Based on the audit rules, a selection statement based on the big data computing engine is generated;

[0072] By replacing the corresponding parts of the data audit task script template in the scheduling template with the table creation statement and the selection statement, a task script based on the big data engine can be generated.

[0073] In some embodiments of this disclosure, the big data computing engine is Spark.

[0074] Generate Spark table creation statements based on data source type;

[0075] Generate Spark selection statements based on audit rules;

[0076] By replacing the corresponding content in the Spark scheduling template with Spark table creation statements and Spark select statements, scheduling parameters can be generated, which contain Spark task scripts.

[0077] S203, Obtain scheduling requirements, execute the task script based on the scheduling requirements, and generate an audit report and operation record.

[0078] Specifically, the parameters are sent to the scheduling system, which can then generate a data quality audit task schedule based on the scheduling requirements obtained from the interactive client, at the client's desired audit time. The schedule integrates Spark task scripts. Executing the schedule creates a Spark task. The Spark task, i.e., the audit task instance, is executed, the audit results are written back, and an audit report and execution log are generated. Notably, the scheduling can be performed periodically.

[0079] S204, If the audit report contains abnormal data, trigger an alarm for the abnormal data according to the audit rules.

[0080] Figure 3 This is a detailed flowchart illustrating a data auditing method according to some embodiments of the present disclosure.

[0081] Figure 3 The data auditing method shown uses the Spark computing engine and includes the following steps:

[0082] To create an audit rule, first select the required data source type, such as MySQL, Hive, or Elasticsearch. After selecting the data type, submit it to the system administrator for task assembly. During task assembly, the system will retrieve the corresponding template from the template engine library based on the data source type. Once a match is found, the task will proceed to assembly.

[0083] Audit rule definition, also known as selecting audit rules, includes criteria such as aggregation calculation and integrity. After selecting a rule, it is incorporated into the task assembly process and assembled with the selected template. This process replaces the content variables in the template and generates a Spark execution script that conforms to the specified audit rules.

[0084] Once the task is assembled, it is placed in the scheduler for execution. First, a pre-made data quality scheduling template is matched against the scheduling template library. After transformation, it is converted into data quality scheduling parameters containing Spark task scripts. These parameters are then passed to the scheduling system to create a schedule, which integrates the Spark engine. The schedule is then executed, and the schedule instance creates a Spark task.

[0085] The executed Spark task, which is also the audit task instance, writes back the audit results: generating an audit report and execution log. If abnormal data is detected, an alert is triggered according to the audit rules.

[0086] Figure 4 This is a schematic diagram of a data auditing device according to some embodiments of the present disclosure. Figure 4 As shown, the data auditing device 400 includes a determining module 410, a generating module 420, and an auditing module 430. In some embodiments, the data auditing function can be performed by... Figure 1 The audit server in the system executes the process. Among them:

[0087] The determination module 410 is used to obtain the big data computing engine type and determine a preset scheduling template based on the engine type, wherein the scheduling template includes a data audit task script template based on the engine.

[0088] The generation module 420 is used to obtain the data source type and audit rules, and generate a task script to perform data audit on the data source based on the audit rules and the scheduling template, wherein the task script is based on the big data computing engine.

[0089] Audit module 430 is used to obtain scheduling requirements and execute the task script based on the scheduling requirements.

[0090] Some embodiments of this disclosure provide a data auditing system, including a database, and further including:

[0091] The scheduling unit is used to obtain the data source type and audit rules to be audited, generate a Spark execution script with specified audit rules based on the data source type and audit rules, obtain scheduling requirements, generate and execute data audit tasks based on the scheduling requirements and the Spark execution script, obtain audit report data, and write the data to the database.

[0092] In other embodiments of this disclosure, the data auditing system further includes:

[0093] The audit management unit is used to obtain alarm rules for audit anomaly data, retrieve the audit report data from the database, and issue alarms for the abnormal data in the audit report data based on the alarm rules.

[0094] Figure 5 This is an architecture diagram of a data auditing system according to some embodiments of this disclosure. Figure 5 In the system shown:

[0095] In system management, configure tags, alarm templates, and data source management, and configure data quality audit rules.

[0096] The scheduling process integrates the Spark computing engine and configures a Spark engine template. Data quality audit rules are created by copying the Spark engine template, replacing the content variables in the template, and generating Spark execution scripts for the specified audit rules.

[0097] The scheduler integrates the Spark engine and can execute Spark tasks through generated scripts. The scheduler can also be set to execute tasks on a scheduled basis.

[0098] The executed Spark task (audit task) will write the processed data (audit report data) to the database and trigger alerts based on the audit rules.

[0099] One embodiment of this disclosure provides an electronic device 600, which includes a memory 620 and a processor 610. The memory 620 is used to store a computer program; the processor 610 is used to implement, when executing the computer program, [the following is an example of implementing a function]. Figure 2 The data auditing methods described in S201-S204.

[0100] One embodiment of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements... Figure 2 The data auditing methods described in S201-S204.

[0101] One embodiment of this disclosure provides a computer program product, including a computer program and instructions, which, when executed by a processor, implement... Figure 2 The data auditing methods described in S201-S204.

[0102] In summary, the data auditing methods, data auditing devices, data auditing systems, electronic devices, computer-readable storage media, and program products provided in the embodiments of this disclosure, by creating data quality auditing tasks based on templates, can support customers in quickly generating data quality auditing tasks by selecting the database to be audited, auditing rules, and big data computing engine, and executing them at the time specified by the customer, thereby flexibly meeting the diverse needs of different customers.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0104] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0105] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0106] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this disclosure and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this disclosure should be included within the protection scope of this disclosure. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A data auditing method, characterized in that, include: Obtain the big data computing engine type, and determine a preset scheduling template based on the engine type, wherein the scheduling template includes a data audit task script template based on the engine; Obtain the data source type and audit rules, and generate a task script to perform data audit on the data source based on the audit rules and the scheduling template, wherein the task script is based on the big data computing engine; Obtain scheduling requirements, execute the task script based on the scheduling requirements, and generate an audit report and operation log; The step of generating a task script to perform data audit on the data source based on the data source type, the audit rules, and the scheduling template includes: generating table creation statements in the task script based on the data source type; generating selection statements in the task script based on the audit rules; and replacing the corresponding parts in the template with the table creation statements and selection statements to generate the task script. The step of determining the preset scheduling template based on the engine type includes: searching for a preset scheduling template in the template library according to the big data computing engine type; if the scheduling template is not found, then continuing to search in the data dictionary.

2. The method according to claim 1, characterized in that, Also includes: If the audit report contains abnormal data, an alarm will be triggered for the abnormal data according to the audit rules.

3. The method according to claim 1, characterized in that, The execution of the task script based on the scheduling requirements includes: The scheduling template is updated based on the data source type and the audit rules; Data quality scheduling parameters are generated based on the updated scheduling template, and the parameters are transmitted to the scheduling system, wherein the parameters include the task script; The scheduling system generates the data quality audit task schedule based on the parameters and the scheduling requirements, and the schedule includes the task script.

4. The method according to any one of claims 1-3, characterized in that: The big data computing engine type is Spark.

5. A data auditing device, characterized in that, include: The determination module is used to obtain the big data computing engine type and determine a preset scheduling template based on the engine type. Specifically, it retrieves a preset scheduling template from the template library according to the big data computing engine type. If the scheduling template is not found, the search continues in the data dictionary; wherein the scheduling template includes a data audit task script template based on the engine; A generation module is used to obtain the data source type and audit rules, and generate a task script to perform data audit on the data source based on the audit rules and the scheduling template, wherein the task script is based on the big data computing engine; The audit module is used to obtain scheduling requirements and execute the task script based on the scheduling requirements. The generation module is further configured to: generate table creation statements in the task script based on the data source type; generate selection statements in the task script based on the audit rules; and replace the corresponding parts in the template with the table creation statements and the selection statements to generate the task script.

6. A data auditing system, comprising a database, characterized in that, Also includes: The scheduling unit is used to obtain the data source type and audit rules to be audited, generate a Spark execution script with specified audit rules based on the data source type and audit rules, obtain scheduling requirements, generate and execute data audit tasks based on the scheduling requirements and the Spark execution script, obtain audit report data and write the data to the database; and retrieve preset scheduling templates in the template library according to the big data computing engine type. If the scheduling template is not found, continue searching in the data dictionary; The scheduling unit is further configured to: generate table creation statements in the Spark execution script based on the data source type; generate selection statements in the Spark execution script based on the audit rules; and replace the corresponding parts in the template with the table creation statements and the selection statements to generate the Spark execution script.

7. The data auditing system according to claim 6, characterized in that, Also includes: The audit management unit is used to obtain alarm rules for audit anomaly data, retrieve the audit report data from the database, and issue alarms for the abnormal data in the audit report data based on the alarm rules.

8. An electronic device, characterized in that: Including memory and processor, The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the method according to any one of claims 1-4.

Images