Data auditing method and device based on auditing model, equipment and storage medium

Abstract

The invention provides a data auditing method and device based on an auditing model, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the steps that a target auditing task is selected from multiple auditing tasks of a first auditing task list, each auditing task comprises multiple auditing fields, and the target auditing field is any one of multiple auditing fields of the target auditing task; according to the target auditing task and the target auditing field, selecting a target auditing model from a plurality of preset auditing models to generate a target auditing script, the plurality of auditing models including a uniqueness auditing model, a conformity auditing model, an existence auditing model, a relevance auditing model, a timeliness auditing model and a volatility auditing model; and auditing the target auditing field according to the target auditing script to obtain an auditing report of the target auditing field. According to the method, the problem of how to effectively control the data auditing task is solved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data auditing method, apparatus, device, and storage medium based on an auditing model. Background Art

[0002] A complete data auditing task includes auditing the key data after collection and aggregation according to a preset data quality evaluation standard, and outputting an audit report. That is, according to various pre-established data auditing rules, these auditing rules are implemented through scripts or tools to complete the auditing task, and then an audit report is output, where the audit report includes audited abnormal data.

[0003] However, due to the numerous data auditing rules, the data tables or data fields they target are different, and the judgment rules are also different. Usually, it is necessary to independently construct an auditing task for each data auditing rule, resulting in a huge workload for data auditing. Moreover, as the number of data auditing tasks gradually increases, it is no longer possible to effectively manage the data auditing tasks. Summary of the Invention

[0004] This application provides a data auditing method, apparatus, device, and storage medium based on an auditing model to solve the problem of how to effectively manage data auditing tasks.

[0005] In a first aspect, this application provides a data auditing method based on an auditing model, and the method includes:

[0006] Select a target auditing task from multiple auditing tasks in a first auditing task list, where each auditing task includes multiple auditing fields, and the target auditing field is any one of the multiple auditing fields of the target auditing task;

[0007] According to the target auditing task and the target auditing field, select a target auditing model from a preset multiple auditing models to generate a target auditing script, where the multiple auditing models include: uniqueness auditing model, compliance auditing model, existence auditing model, relevance auditing model, timeliness auditing model, and volatility auditing model;

[0008] Audit the target auditing field according to the target auditing script to obtain an audit report of the target auditing field, where the audit report includes audited abnormal data.

[0009] In a possible design, each auditing task includes a task attribute, and the target task attribute is the task attribute of the target auditing task;

[0010] Selecting a target auditing model from a preset multiple auditing models according to the target auditing task and the target auditing field to generate a target auditing script includes:

[0011] Select a target audit model from multiple preset audit models according to the target audit task and target audit fields;

[0012] Generate a target audit script according to the target task attribute and the target audit model.

[0013] In a possible design, multiple audit scripts are pre-stored in each audit model;

[0014] Generate a target audit script according to the target task attribute and the target audit model, including:

[0015] Select a target audit script from multiple audit scripts pre-stored in the target audit model according to the target task attribute and the target audit model.

[0016] In a possible design, the uniqueness audit model is used to perform uniqueness audit on the audit fields;

[0017] The compliance audit model is used to perform compliance audit on the audit fields;

[0018] The existence audit model is used to perform existence audit on the audit fields;

[0019] The relevance audit model is used to perform relevance audit on the audit fields;

[0020] The timeliness audit model is used to perform timeliness audit on the audit fields;

[0021] The volatility audit model is used to perform volatility audit on the audit fields.

[0022] In a possible design, the compliance audit model includes: a data non-empty audit sub-model, a field length audit sub-model, a numerical unidirectional audit sub-model, a numerical range audit sub-model, a value range audit sub-model, an in-table comparison audit sub-model, a decision function audit sub-model, and a custom function audit sub-model;

[0023] The data non-empty audit sub-model is used to perform non-empty audit on the audit fields;

[0024] The field length audit sub-model is used to perform length audit on the audit fields;

[0025] The numerical unidirectional audit sub-model is used to perform numerical size audit on the audit fields;

[0026] The numerical range audit sub-model is used to perform numerical range audit on the audit fields;

[0027] The value range audit sub-model is used to perform value range audit on the audit fields;

[0028] The in - table comparison auditing sub - model is used to perform in - table comparison auditing on auditing fields;

[0029] The decision - function auditing sub - model is used to perform decision - function auditing on auditing fields;

[0030] The custom - function auditing sub - model is used to perform custom - function auditing on auditing fields.

[0031] In a possible design, each auditing task includes first indication information, and the first indication information is used to indicate whether the corresponding auditing task meets a preset condition;

[0032] Selecting a target auditing task from multiple auditing tasks in the first auditing task list includes:

[0033] When the first indication information of the target auditing task indicates that the target auditing task meets the preset condition, selecting the target auditing task from multiple auditing tasks in the first auditing task list.

[0034] In a possible design, the auditing report further includes second indication information, and the second indication information is used to indicate whether auditing abnormal data is detected in the auditing field;

[0035] After obtaining the auditing report of the target auditing field by auditing the target auditing field according to the target auditing script, the method further includes:

[0036] When the second indication information indicates that auditing abnormal data is detected, transferring the target auditing task from the first auditing task list to the second auditing task list;

[0037] When the second indication information indicates that no auditing abnormal data is detected, transferring the target auditing task from the first auditing task list to the third auditing task list.

[0038] In a second aspect, the present application provides a data auditing device based on an auditing model, and the device includes:

[0039] A task selection module, configured to select a target auditing task from multiple auditing tasks in the first auditing task list, where each auditing task includes multiple auditing fields, and the target auditing field is any one of the multiple auditing fields of the target auditing task;

[0040] A model selection module, configured to select a target auditing model from multiple preset auditing models according to the target auditing task and the target auditing field to generate a target auditing script, where the multiple auditing models include: uniqueness auditing model, compliance auditing model, existence auditing model, relevance auditing model, timeliness auditing model, and volatility auditing model;

[0041] A report generation module, configured to audit target audit fields according to a target audit script, and obtain an audit report of the target audit fields, where the audit report includes audit exception data.

[0042] In a third aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0043] The memory stores computer-executable instructions;

[0044] The processor executes the computer-executable instructions stored in the memory to implement a data auditing method based on an audit model according to the invention content of the first aspect.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement a data auditing method based on an audit model according to the invention content of the first aspect.

[0046] In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it is used to implement a data auditing method based on an audit model according to the invention content of the first aspect.

[0047] A data auditing method, device, equipment and storage medium based on an audit model provided by the present application select a target audit task from multiple audit tasks in a first audit task list; and select a target audit model from a preset multiple audit models according to the target audit task and target audit fields to generate a target audit script; thereby auditing the target audit fields according to the target audit script to obtain an audit report of the target audit fields. The following technical effects are achieved: By storing all multiple audit tasks of multiple data sources in the first audit task list, and selecting a target audit task from multiple audit tasks in the first audit task list for data auditing according to the priority order of the audit tasks, or according to the preset start and completion times of the audit tasks, or according to preset conditions such as whether the audit tasks have been transmitted to the target computing engine, etc., the problem of how to uniformly manage all data audit tasks with a set of unified data audit task tables is solved; By enabling six audit models to respectively cover different evaluation types of data auditing, a comprehensive coverage of all evaluation types of data auditing by the six audit models is realized; By auditing the target audit fields according to the generated audit script and generating an audit report of the target audit fields, the specific implementation problem of the audit task is solved. According to the audit report, the execution result of the audit task can also be intuitively viewed. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0049] The accompanying drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0050] Figure 1 Schematic diagram of the system architecture of a data auditing method based on an auditing model provided by an embodiment of the present application;

[0051] Figure 2 Application scenario framework diagram of a data auditing method based on an auditing model provided by an embodiment of the present application;

[0052] Figure 3 Flow diagram of a data auditing method based on an auditing model provided by an embodiment of the present application Figure 1 ;

[0053] Figure 4 Flow diagram of a data auditing method based on an auditing model provided by an embodiment of the present application Figure 2 ;

[0054] Figure 5 Flow diagram of a data auditing method based on an auditing model provided by an embodiment of the present application Figure 3 ;

[0055] Figure 6 Schematic diagram of the structure of a data auditing device based on an auditing model provided by an embodiment of the present application;

[0056] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application.

[0057] Reference numerals:

[0058] 110 - Auditing server; 111 - Auditing model storage module; 112 - Auditing task configuration module; 113 - Auditing task processing module;

[0059] 210 - Data source;

[0060] 300 - Data auditing device based on an auditing model; 310 - Task selection module; 320 - Model selection module; 330 - Report generation module;

[0061] 400 - Electronic device; 410 - Processor; 420 - Memory; 430 - Communication component; 440 - Bus. Detailed implementation

[0062] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0063] In the embodiments of the present application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and roles. Those skilled in the art can understand that the terms "first" and "second" do not limit the quantity and execution order, and the terms "first" and "second" do not necessarily mean different. It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly, using words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner. In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more.

[0064] It should be noted that "when... " in the embodiments of the present application can be at the instant when a certain situation occurs, or within a period of time after a certain situation occurs. The embodiments of the present application do not make specific limitations on this. In addition, a data auditing method based on an auditing model provided in the embodiments of the present application is only an example, and a data auditing method based on an auditing model may also include more or less content.

[0065] To facilitate a clear description of the technical solutions in the embodiments of the present application, the following briefly introduces some terms and technologies involved in the embodiments of the present application:

[0066] First auditing task list: A list of all data auditing tasks to be audited.

[0067] Second auditing task list: A list of tasks that failed the audit, saving all auditing tasks that did not pass the audit.

[0068] Third auditing task list: A list of tasks that passed the audit, saving all auditing tasks that passed the audit.

[0069] Data plays a very important role in the business development of each enterprise. Whether the data value can be fully reflected is closely related to the quality of the data. In order to improve the quality of the data and ensure the accuracy and effectiveness of the data, it is necessary for enterprises to carry out data governance work. At present, with the gradual embedding of data governance work in the business operation activities of enterprises, the requirements for data quality in each business activity are also gradually increasing. As an important means of data quality evaluation, the workload of data auditing is also increasing sharply, and the scope of data auditing is becoming wider and wider, resulting in more difficult management of data auditing work.

[0070] Based on this, the embodiments of the present application provide a data auditing method, device, equipment and storage medium based on an auditing model, which can be used in the field of data processing technology, aiming to solve the above technical problems in the prior art.

[0071] The following uses specific embodiments to detail the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0072] Figure 1 It is a schematic diagram of the system architecture of a data auditing method based on an auditing model provided by an embodiment of the present application. It should be noted that Figure 1 The figure shown is only an example of the system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0073] As Figure 1 shown, the system architecture where the method is located is an auditing server 110, and the auditing server 110 is used to perform data auditing on multiple data auditing tasks of each data source and output an auditing report based on the results of the data auditing. The auditing server 110 includes: an auditing model storage module 111, an auditing task configuration module 112, and an auditing task processing module 113. Among them:

[0074] The auditing model storage module 111 stores multiple auditing models for auditing the data auditing tasks of each data source. Each auditing model includes multiple auditing scripts.

[0075] The auditing task configuration module 112 is used to configure the corresponding auditing model for multiple auditing tasks of each data source.

[0076] The auditing task processing module 113 is used to process each data auditing task and generate a corresponding auditing report.

[0077] Figure 2This is an application scenario framework diagram of a data auditing method based on an auditing model provided by an embodiment of the present application.

[0078] As Figure 2 shown, the application scenario framework where this method is located includes: an auditing server 110 and a data source 210. The data source 210 includes multiple auditing tasks to be audited. The auditing server 110 performs data auditing on each auditing task of the data source 210 and outputs an audit report.

[0079] Figure 3 This is a flowchart of a data auditing method based on an auditing model provided by an embodiment of the present application Figure 1 . In the present application, the auditing server can sequentially perform data auditing on multiple auditing tasks of multiple data sources. Among them, the processing methods for each auditing task are the same. In this embodiment, only the method for performing data auditing on one auditing field of any one of the auditing tasks will be described in detail. As Figure 3 shown, this method includes:

[0080] S101. Select a target auditing task from multiple auditing tasks in the first auditing task list;

[0081] Specifically, the auditing server stores all the multiple auditing tasks of multiple data sources in the first auditing task list, that is, in the list of all data auditing tasks to be audited. And according to the priority order of the auditing tasks, or according to the preset start and completion times of the auditing tasks, or according to preset conditions such as whether the auditing tasks have been transmitted to the target computing engine, etc., a target auditing task is selected from the multiple auditing tasks in the first auditing task list for data auditing. Since each of these auditing tasks may include multiple fields to be audited, it is also necessary to determine any one of the auditing fields to be audited in the multiple auditing fields of the target auditing task as the target auditing field. The auditing task can be represented as an auditing table, and the auditing field is the data field in the auditing table.

[0082] S102. Select a target auditing model from multiple preset auditing models according to the target auditing task and the target auditing field to generate a target auditing script;

[0083] Specifically, the auditing server selects the auditing model corresponding to the target auditing task and the target auditing field from six preset auditing models to further generate an auditing script, so as to perform data auditing on the target auditing field. The six auditing models here are: uniqueness auditing model, compliance auditing model, existence auditing model, relevance auditing model, timeliness auditing model, and volatility auditing model. These six auditing models cover all evaluation types of data auditing. Generally, the evaluation types of data auditing can be divided into six types, namely,

[0084] Uniqueness audit: Generally, it is for single-table audit, requiring that the data in the audit fields within the table is non-null and uniquely determined;

[0085] Integrity audit: Requires that the data in the audit fields within the table meet the preset requirements, such as not being null, etc.;

[0086] Accuracy audit: Requires that the data in the audit fields within the table can truly reflect the actual situation, such as the validity degree of the value range, the compliance degree of the data format, etc. Generally, it can be specifically divided into three categories. For example, single-field accuracy (i.e., a single field conforms to the preset format), value-range accuracy (i.e., the data value is within the preset range), and value-dictionary accuracy (i.e., the data value is within the data dictionary);

[0087] Validity audit: Requires that the data value conforms to the business logic rules. For example, in-table association validity (i.e., the value of a field within the same table must be calculated from the values of other fields), cross-table association validity (i.e., the data of one table is obtained by processing, counting, or summarizing one or more fields of another table), and data fluctuation validity (i.e., by comparing the data table month-on-month, the amplitude change of its sum or summary value is within a certain range);

[0088] Consistency audit: Requires that the values of the same business object be consistent in the context, generally referring to foreign key compliance. That is, the foreign key of the child table must be included in the primary key of the parent table;

[0089] Timeliness audit: Requires that the time point when the data is provided meets the preset requirements. Generally, a log table corresponding to the data-providing time point is required.

[0090] The six audit models respectively cover different evaluation types of data audit, specifically:

[0091] Uniqueness audit model: Corresponding to the uniqueness audit in the evaluation type, it is used to perform the audit of the unique non-null condition on the audit fields of a single table;

[0092] Compliance audit model: Corresponding to the integrity audit, accuracy audit (single-field accuracy and value-range accuracy), and validity audit (in-table association validity) in the evaluation type, it is used to perform various compliance rule judgments on the audit fields of a single table, such as multiple audit rules including field non-null, field length limit, value range, finite value, regular function judgment, and custom function judgment;

[0093] Existence Audit Model: Corresponding to the accuracy audit (value dictionary accuracy), validity audit (cross-table association validity of single fields), and consistency audit in the evaluation type, it is used to determine the direct association relationship between the parent table and the child table, and requires that the data of the audit fields in the child table must exist in the values of the audit fields in the parent table;

[0094] Relevance Audit Model: Corresponding to the validity audit (cross-table association validity of multi-field aggregation) in the evaluation type, it is used to perform an association judgment on the aggregation operation of the child table, and requires that the values of the audit fields in the parent table be consistent with the data summary value of the child table;

[0095] Timeliness Audit Model: Corresponding to the timeliness audit in the evaluation type, it is used to determine the production completion time of the data, and requires that each piece of data has a complete production completion log record and a corresponding log record table;

[0096] Volatility Audit Model: Corresponding to the validity audit (data volatility validity) in the evaluation type, it is used to compare the abnormal fluctuations of the data or the abnormal fluctuations of the number of records by performing a month-on-month comparison on the data table.

[0097] S103. Audit the target audit fields according to the target audit script to obtain an audit report on the target audit fields;

[0098] Specifically, the audit server performs data auditing on the target audit fields according to the generated audit script and generates an audit report on the target audit fields. The audit report indicates the execution result of the audit task, and the audit report includes audit exception data.

[0099] A data auditing method based on an auditing model provided in this embodiment selects a target auditing task from multiple auditing tasks in a first auditing task list; and selects a target auditing model from multiple preset auditing models according to the target auditing task and target auditing fields to generate a target auditing script; thereby auditing the target auditing fields according to the target auditing script to obtain an auditing report for the target auditing fields. The following technical effects are achieved: By storing all the auditing tasks of multiple data sources in the first auditing task list, and selecting a target auditing task from the multiple auditing tasks in the first auditing task list according to the priority order of the auditing tasks, or according to the preset start and completion times of the auditing tasks, or according to preset conditions such as whether the auditing task has been transmitted to the target computing engine, etc., the problem of how to uniformly manage all data auditing tasks with a set of unified data auditing task tables is solved; by enabling six auditing models to respectively cover different evaluation types of data auditing, a comprehensive coverage of all evaluation types of data auditing by the six auditing models is achieved; by auditing the target auditing fields according to the generated auditing script and generating an auditing report for the target auditing fields, the specific implementation problem of the auditing task is solved, and according to the auditing report, the execution result of the auditing task can also be intuitively viewed.

[0100] Figure 4 Schematic flow of a data auditing method based on an auditing model provided in an embodiment of the present application Figure 2 . In a possible example, as Figure 4 shown, on the basis of the Figure 3 embodiment, this embodiment will elaborate on how to generate a target auditing script according to the target auditing task and target auditing fields. S102. Selecting a target auditing model from multiple preset auditing models according to the target auditing task and target auditing fields to generate a target auditing script includes:

[0101] S201. Select a target auditing model from multiple preset auditing models according to the target auditing task and target auditing fields;

[0102] Specifically, each auditing task includes task attributes. The target task attributes are the task attributes of the target auditing task. After selecting a target auditing model from multiple preset auditing models according to the target auditing task and target auditing fields, it further includes determining the target task attributes according to the target auditing task. The target task attributes can be one or more of the attributes such as auditing table name, auditing field name, filtering conditions for data fields, parent table reference field name, parent table filtering conditions, grouping field list, associated table name, calculation formula, aggregation method, grouping field list, time required to complete the task, log generation period, log table name, and log table filtering conditions, etc.

[0103] S202. Select a target audit script from multiple pre-stored audit scripts of the target audit model according to the target task attribute and the target audit model;

[0104] Specifically, each audit model pre-stores multiple audit scripts. The target audit script is generated according to the target task attribute and the target audit model. That is, the target audit script is selected from multiple pre-stored audit scripts of the target audit model according to the target task attribute and the target audit model.

[0105] Figure 5 Flow schematic of a data audit method based on an audit model provided by an embodiment of the present application Figure 3 In a possible example, as Figure 5 shown, based on the Figure 3 embodiment, this embodiment details how to select a target audit task from multiple audit tasks in the first audit task list and the further operation process after obtaining the audit report of the target audit field. As Figure 5 shown, the method includes:

[0106] S301. When the first indication information of the target audit task indicates that the target audit task meets a preset condition, select the target audit task from multiple audit tasks in the first audit task list;

[0107] Specifically, all multiple audit tasks from multiple data sources are stored in the first audit task list in its entirety, and the audit server executes these audit tasks in a certain order in sequence. Each audit task contains first indication information, which is used to indicate whether the corresponding audit task meets the preset condition. When the first indication information indicates that the audit task meets the preset condition, the server starts the execution mode for the audit task. Among them, the first indication information may refer to any one or more of the preset conditions such as the priority order of the audit task, the preset start and completion times of the audit task, or whether the audit task has been transmitted to the target computing engine.

[0108] S302. Select a target audit model from multiple preset audit models according to the target audit task and the target audit field to generate a target audit script;

[0109] S303. Audit the target audit field according to the target audit script to obtain an audit report of the target audit field;

[0110] S302 - S303 is similar to S102 - S103, and this embodiment will not elaborate further.

[0111] S304. When the second indication information indicates that audit exception data is detected, transfer the target audit task from the first audit task list to the second audit task list;

[0112] Specifically, the audit report further includes second indication information, which is used to indicate whether audit exception data is detected in the audit field. After the audit server performs data audit on the target audit field according to the generated audit script and generates the audit report of the target audit field, when the second indication information indicates that audit exception data is detected in the audit field, the target audit task is transferred to the second audit task list, that is, the task list of audit failure.

[0113] S305. When the second indication information indicates that no audit exception data is detected, transfer the target audit task from the first audit task list to the third audit task list;

[0114] Specifically, when the second indication information indicates that no audit exception data is detected in the audit field, the target audit task is transferred to the third audit task list, that is, the task list of audit success.

[0115] In a possible design, the uniqueness audit model is used to perform uniqueness audit on the audit field. It only performs independent audit judgment on a single field of a single table. The task attributes are: audit table name, audit field name, and filtering condition. The judgment rule is: if the audit table meets the filtering condition, the audit field remains non-null and unique. This audit model is suitable for auditing the primary key and key identifier of the audit table, such as the personnel identifier and ID number of the personnel table. In this embodiment, the format of the restriction condition is filtering condition (empty). Specific examples are as follows:

[0116] Case 1. If it is required that the ID number in the user table must remain non-null and unique, the uniqueness audit model is applicable. The configured audit model is: audit table name (user table), audit field name (ID no), audit evaluation type is uniqueness audit, filtering condition (empty). When the filtering condition is met and the audit field is empty, it does not conform;

[0117] The compliance audit model is used to perform compliance audit on the audit field. It performs audit judgment on the format and value range of a single field of a single table. The task attributes are: audit table name, audit field name, and filtering condition. The judgment rule is: if the audit table meets the filtering condition, the audit field is data that passes the compliance audit.

[0118] The existence verification model is used to perform existence verification on verification fields. It performs associated matching on the verification fields of two verification tables, requiring that the data of the verification fields in the verification table must exist in the associated table (parent table). The task attributes include: verification table name, verification field name, filtering condition (if any), parent table name, parent table reference field name, parent table filtering condition (if any). This model is mainly applicable to foreign key verification, dictionary table verification, and single-record association verification. For example, the educational background of a person must exist in the educational background table, and the order number must exist in the order table. Specific examples are as follows:

[0119] Case 2: If it is required that the product identifier of mobile network users (net type code = '10') in the user table must be in the product identifier field (product id) of mobile network products (net type code = 'A') in the product table, then the existence verification model is applicable. The configured verification model is: verification table name (usertable), verification field name (product id), verification evaluation type is existence verification, filtering condition (net type code = '10'), parent table name (product), parent table reference field name (product id), parent table filtering condition (net type code = 'A'). When the filtering condition is met, it is compliant;

[0120] The relevance verification model is used to perform relevance verification on verification fields. It makes a determination by comparing and referencing with the associated table for the verification data of a single table, focusing on the verification comparison between the data in the parent table and the results summarized or counted from the child table. The task attributes include: verification table name, verification field name, grouping field list, filtering condition (if any), associated table name, summary field name (or calculation formula), aggregation method (counting or summing), grouping field list (the quantity and order should be the same as those in the verification table), filtering condition (if any). The application scenarios of this model are: verification of the summary table result and the detail table data, such as the verification comparison between the data in the monthly data table summarized from the daily data table and the data in the original daily data table. Specific examples are as follows:

[0121] Case 3. If it is required that the total income of the user (user ID) in the user month income table be consistent with the total sum of the income of the user (user ID) in the user day income table, the relevance audit model applies. The configured audit model: audit table name (user month income), audit field name (income), audit evaluation type is relevance audit, grouping field list (user ID), filter condition (none), associated table name (user day income), summary field name (income), aggregation method (sum), grouping field list (user ID), filter condition (none). When the filter condition is met, it is compliant;

[0122] The timeliness audit model is used to conduct timeliness audits on audit fields. It only audits the specific production situation of a single table and is limited to data tables with complete data production logs. The task attributes include: data table content, time required to complete the task, log generation cycle, log table name, log table data field name, log table accounting period field name, log table status field name, log table completion time field name, log table completion status flag, and log table filter condition (if any). Since a large amount of data generation processes may share one or more log tables, fields directly associated with the log table can also be made into a log template table, which can greatly simplify the configuration work of data table log data. This model mainly checks whether important data can be completed within the preset time, such as income day data needs to be completed before 8:00 every morning, and financial report data needs to be completed at 12:00 on the 5th of each month, etc. Specific examples are as follows:

[0123] Case 4. If it is required that the user day income table be generated at 7:00 every day, the timeliness audit model applies. The configured audit model: audit table name (user day income), audit field name (blank), audit evaluation type is timeliness audit, cycle identifier (every day), expected completion time (7:00), log table name (log table), key field name (table name), accounting period field name (caledate), status field name (status), completion time field name (finish time), and the status is updated upon successful audit;

[0124] The volatility auditing model is used to conduct volatility auditing on auditing fields. It compares by month-on-month for the same auditing form to check for abnormal fluctuations. The task attributes include: auditing form name, auditing field name (if it is the number of records, it is 0), accounting period field name, benchmark ratio, passing method (greater than, less than, not less than, not greater than, equal to), filtering conditions (if any). Such as auditing the abnormal change of the data table record number, auditing the sharp increase or decrease of income data, etc. Specific examples are as follows:

[0125] Case Five, if it is required that for the user daily income table (user day income), the daily cumulative income fluctuation does not exceed 10%, then the volatility auditing model is applicable. The configured auditing model: auditing form name (user day income), auditing field name (curincome), auditing evaluation type is volatility auditing, accounting period field name (caledate), accounting period type is daily accounting period, auditing ratio 10%, passing direction (not greater than), no filtering conditions.

[0126] In a possible design, the compliance auditing model includes: eight sub-models, namely the non-empty data auditing sub-model, the field length auditing sub-model, the numerical unidirectional auditing sub-model, the numerical range auditing sub-model, the value range auditing sub-model, the in-table comparison auditing sub-model, the decision function auditing sub-model, and the custom function auditing sub-model. In this embodiment, the format of the restriction condition is the filtering condition (less than 0).

[0127] The non-empty data auditing sub-model is used to conduct non-empty auditing on auditing fields, that is, it is specified that the auditing fields cannot be empty;

[0128] The field length auditing sub-model is used to conduct length auditing on auditing fields, that is, to make a compliance judgment on the length of the auditing fields. Multiple evaluation types can be specified, such as not less than the preset number of characters, not greater than the preset number of characters, or must be the preset number of characters, etc. Specifically, for example, the ID number must be 15 / 18 digits, and the mobile phone number can only be 11 digits, etc.;

[0129] The numerical unidirectional auditing sub-model is used to conduct numerical size auditing on auditing fields, that is, to judge the numerical size of the auditing fields, and compare the size with the preset value. It can be divided into five judgment modes: greater than, less than, not less than, not greater than, equal to. For example, the working years cannot be higher than 100 years. Specific examples are as follows:

[0130] Case 6: If it is required that the work life in the user table cannot be less than 0, then the numerical one-way audit sub-model in the compliance audit model is applicable. The configured audit model is: audit table name (user table), audit field name (work life), audit evaluation type is compliance audit, the judgment direction is less than, the judgment value is 0, and when the filtering condition (less than 0) is met, it does not conform;

[0131] The numerical range audit sub-model is used to conduct a numerical range audit on the audit field, that is, to judge the value of the audit field, and it is required that the audit data is within the preset interval range. There are four types of interval rules: not included, left included, right included, and fully included. For example, a person's age must be between 0 and 150 years old;

[0132] The value range audit sub-model is used to conduct a value range audit on the audit field, that is, to judge the value of the audit field, and it is required that the audit data is in the preset data list, and a finite data value list is needed (generally, the number of data values is small, not exceeding 10); such as the valid status of an order can only be new, in transit, completed, or expired;

[0133] The in-table comparison audit sub-model is used to conduct an in-table comparison audit on the audit field, that is, to compare the sizes of two audit fields in the audit table, and there are five judgment modes: greater than, less than, equal to, not less than, and not greater than. For example, the cumulative income should be greater than or equal to the current income.

[0134] The decision function audit sub-model is used to conduct a decision function audit on the audit field, that is, to compare and judge the audit field with the preset regular function. For example, a phone number can only be numbers, dashes, and parentheses, and dashes and parentheses can only appear once, etc. Specific examples are as follows:

[0135] Case 7: If it is required that the mobile no in the user table must conform to the domestic mobile phone rules, then the decision function audit sub-model in the compliance audit model is applicable. The configured audit model is: audit table name (user table), audit field name (mobile no). Judgment function (^1[3-9][0-9]{9}$), which means that the audit field has a total of 11 digits, "1" means the first digit is fixed as 1, "[3-9]" means the second digit is any number from 3 to 9, and "[0-9]{9}" means the remaining 9 digits are any number from 0 to 9 respectively. When the filtering condition is met, it conforms;

[0136] The custom function audit sub-model is used to conduct a custom function audit on the audit field, that is, to compare and judge the audit field with the custom regular function.

[0137] A data auditing method based on an auditing model provided in this embodiment selects a target auditing task from multiple auditing tasks in a first auditing task list; and selects a target auditing model from multiple preset auditing models according to the target auditing task and target auditing fields to generate a target auditing script; thereby auditing the target auditing fields according to the target auditing script to obtain an auditing report of the target auditing fields. The following technical effects are achieved: By storing all the auditing tasks of multiple data sources in the first auditing task list, and selecting the target auditing task from the multiple auditing tasks in the first auditing task list according to the priority order of the auditing tasks, or according to the preset start and completion times of the auditing tasks, or according to whether the auditing tasks have been transmitted to the target computing engine and other preset conditions for data auditing, the problem of how to uniformly manage all data auditing tasks with a unified data auditing task table is solved; by enabling six auditing models to respectively cover different evaluation types of data auditing, the comprehensive coverage of all evaluation types of data auditing by the six auditing models is realized; by auditing the target auditing fields according to the generated auditing script and generating an auditing report of the target auditing fields, the problem of the specific implementation of the auditing task is solved, and according to the auditing report, the execution result of the auditing task can also be intuitively viewed; selecting the target auditing script from multiple preset auditing scripts of the target auditing model according to the target task attribute and target auditing model solves the problem of how to obtain the auditing script; by setting eight sub-models in the compliance auditing model, the compliance auditing model more specifically covers a variety of different data auditing types.

[0138] Embodiments of the present invention can divide the functional modules of an electronic device or a main control device according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present invention is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0139] Figure 6 It is a structural schematic diagram of a data auditing device based on an auditing model provided in an embodiment of the present application. As Figure 6 shown, the data auditing device 300 based on the auditing model includes:

[0140] A task selection module 310, a model selection module 320, and a report generation module 330;

[0141] A task selection module 310 is configured to select a target audit task from multiple audit tasks in a first audit task list. Each audit task includes multiple audit fields, and a target audit field is any one of the multiple audit fields of the target audit task.

[0142] A model selection module 320 is configured to select a target audit model from a preset multiple audit models according to the target audit task and the target audit field, so as to generate a target audit script. The multiple audit models include: a uniqueness audit model, a compliance audit model, an existence audit model, a relevance audit model, a timeliness audit model, and a volatility audit model.

[0143] A report generation module 330 audits the target audit field according to the target audit script, and obtains an audit report of the target audit field, where the audit report includes audit exception data.

[0144] In a possible design, each audit task includes a task attribute, and the target task attribute is the task attribute of the target audit task.

[0145] The model selection module 320 includes: a target audit model selection module and a script generation module.

[0146] The target audit model selection module is configured to select a target audit model from a preset multiple audit models according to the target audit task and the target audit field.

[0147] The script generation module is configured to generate a target audit script according to the target task attribute and the target audit model.

[0148] In a possible design, each audit model pre-stores multiple audit scripts.

[0149] The script generation module is further configured to select a target audit script from the multiple audit scripts pre-stored in the target audit model according to the target task attribute and the target audit model.

[0150] In a possible design, the uniqueness audit model is used to perform a uniqueness audit on the audit field.

[0151] The compliance audit model is used to perform a compliance audit on the audit field.

[0152] The existence audit model is used to perform an existence audit on the audit field.

[0153] The relevance audit model is used to perform a relevance audit on the audit field.

[0154] The timeliness audit model is used to perform a timeliness audit on the audit field.

[0155] The volatility audit model is used to perform a volatility audit on the audit field.

[0156] In a possible design, the compliance audit model includes: a data non - null audit sub - model, a field length audit sub - model, a numerical unidirectional audit sub - model, a numerical range audit sub - model, a value range audit sub - model, an in - table comparison audit sub - model, a decision function audit sub - model, and a custom function audit sub - model;

[0157] The data non - null audit sub - model is used to perform a non - null audit on the audit fields;

[0158] The field length audit sub - model is used to perform a length audit on the audit fields;

[0159] The numerical unidirectional audit sub - model is used to perform a numerical magnitude audit on the audit fields;

[0160] The numerical range audit sub - model is used to perform a numerical range audit on the audit fields;

[0161] The value range audit sub - model is used to perform a value range audit on the audit fields;

[0162] The in - table comparison audit sub - model is used to perform an in - table comparison audit on the audit fields;

[0163] The decision function audit sub - model is used to perform a decision function audit on the audit fields;

[0164] The custom function audit sub - model is used to perform a custom function audit on the audit fields.

[0165] In a possible design, each audit task includes first indication information, and the first indication information is used to indicate whether the corresponding audit task meets a preset condition;

[0166] The task selection module 310 is further configured to select a target audit task from multiple audit tasks in the first audit task list when the first indication information of the target audit task indicates that the target audit task meets the preset condition.

[0167] In a possible design, the audit report further includes second indication information, and the second indication information is used to indicate whether audit exception data is detected for the audit fields;

[0168] The report generation module 330 includes a first transfer module and a second transfer module;

[0169] The first transfer module is configured to transfer the target audit task from the first audit task list to the second audit task list when the second indication information indicates that audit exception data is detected;

[0170] The second transfer module; when the second indication information indicates that no audit exception data is detected, transfers the target audit task from the first audit task list to the third audit task list.

[0171] A data auditing device based on an auditing model provided in this embodiment can execute the data auditing method based on an auditing model in the above embodiment. The implementation principle and technical effect are similar, and will not be elaborated here in this embodiment.

[0172] In the specific implementation of the above-mentioned data auditing device based on an auditing model, each module can be implemented as a processor. The processor can execute computer-executable instructions stored in the memory, so that the processor executes the above-mentioned data auditing method based on an auditing model.

[0173] Figure 7 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As Figure 7 shown, the electronic device 400 includes: at least one processor 410 and a memory 420. The electronic device 400 also includes a communication component 430. Among them, the processor 410, the memory 420, and the communication component 430 are connected through a bus 440.

[0174] In the specific implementation process, at least one processor 410 executes computer-executable instructions stored in the memory 420, so that at least one processor 410 executes a data auditing method based on an auditing model executed on the electronic device side as above.

[0175] For the specific implementation process of the processor 410, reference can be made to the above method embodiment. The implementation principle and technical effect are similar, and will not be elaborated here in this embodiment.

[0176] In the above embodiment, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, abbreviated: CPU), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated: DSP), application specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated: ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0177] The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory.

[0178] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0179] The functions implemented for the electronic device and the master control device are introduced for the solution provided by the embodiments of the present invention. It can be understood that, in order to implement the above functions, the electronic device or the master control device includes the corresponding hardware structure and / or software module for executing each function. Combining the units and algorithm steps of each example described in the embodiments disclosed in the embodiments of the present invention, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiments of the present invention.

[0180] This application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, it is used to implement the above data auditing method based on the auditing model.

[0181] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0182] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the electronic device or the master control device.

[0183] The present application also provides a computer program product, which includes: a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and the execution of the computer program by the at least one processor causes the electronic device to execute the solution provided in any of the above embodiments.

[0184] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0185] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations. In the hardware implementation, the division of the functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component can have multiple functions, or a function or step can be executed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or can be implemented as hardware, or can be implemented as an integrated circuit, such as an application-specific integrated circuit.

[0186] So far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data auditing method based on an auditing model, characterized in that, The method includes: Selecting a target audit task from multiple audit tasks in a first audit task list, where each audit task includes multiple audit fields, and the target audit field is any one of the multiple audit fields of the target audit task; Selecting a target audit model from a preset multiple audit models according to the target audit task and the target audit field to generate a target audit script, where the multiple audit models include: a uniqueness audit model, a compliance audit model, an existence audit model, a relevance audit model, a timeliness audit model, and a volatility audit model; Auditing the target audit field according to the target audit script to obtain an audit report of the target audit field, where the audit report includes audit exception data.

2. The method according to claim 1, wherein Each audit task includes a task attribute, and the target task attribute is the task attribute of the target audit task; The step of selecting a target audit model from a preset multiple audit models according to the target audit task and the target audit field to generate a target audit script includes: Selecting the target audit model from the preset multiple audit models according to the target audit task and the target audit field; Generating the target audit script according to the target task attribute and the target audit model.

3. The method according to claim 2, wherein Each audit model pre-stores multiple audit scripts; The step of generating the target audit script according to the target task attribute and the target audit model includes: Selecting the target audit script from the multiple audit scripts pre-stored in the target audit model according to the target task attribute and the target audit model.

4. The method according to claim 3, characterized in that, The uniqueness audit model is used to perform a uniqueness audit on the audit field; The compliance audit model is used to perform a compliance audit on the audit field; The existence audit model is used to perform an existence audit on the audit field; The relevance audit model is used to perform a relevance audit on the audit field; The timeliness audit model is used to perform a timeliness audit on the audit field; The volatility audit model is used to perform a volatility audit on the audit field.

5. The method according to claim 4, wherein The compliance audit model includes: a data non-empty audit sub-model, a field length audit sub-model, a numerical unidirectional audit sub-model, a numerical range audit sub-model, a value range audit sub-model, an in-table comparison audit sub-model, a decision function audit sub-model, and a custom function audit sub-model; The data non-empty audit sub-model is used to perform a non-empty audit on the audit field; The field length audit sub-model is used to perform a length audit on the audit field; The numerical unidirectional audit sub-model is used to perform a numerical size audit on the audit field; The numerical range audit sub-model is used to perform a numerical range audit on the audit field; The value range audit sub-model is used to perform a value range audit on the audit field; The in-table comparison audit sub-model is used to perform an in-table comparison audit on the audit field; The decision function audit sub-model is used to perform a decision function audit on the audit field; The custom function audit sub-model is used to perform a custom function audit on the audit field.

6. The method according to any one of claims 1-5, characterized in that, Each of the audit tasks includes first indication information for indicating whether the corresponding audit task meets a preset condition; Selecting a target audit task from multiple audit tasks in the first audit task list includes: When the first indication information of the target audit task indicates that the target audit task meets the preset condition, selecting the target audit task from multiple audit tasks in the first audit task list.

7. The method according to claim 6, wherein The audit report further includes second indication information for indicating whether audit abnormal data is detected in the audit field; After obtaining the audit report of the target audit field by auditing the target audit field according to the target audit script, the method further includes: When the second indication information indicates that the audit abnormal data is detected, transferring the target audit task from the first audit task list to a second audit task list; When the second indication information indicates that the audit abnormal data is not detected, transferring the target audit task from the first audit task list to a third audit task list.

8. A data auditing device based on an auditing model, characterized in that The device includes: A task selection module, configured to select a target audit task from multiple audit tasks in the first audit task list, where each audit task includes multiple audit fields, and the target audit field is any one of the multiple audit fields of the target audit task; A model selection module, configured to select a target audit model from a preset multiple audit models according to the target audit task and the target audit field to generate a target audit script, where the multiple audit models include: a uniqueness audit model, a compliance audit model, an existence audit model, a relevance audit model, a timeliness audit model, and a volatility audit model; A report generation module, configured to audit the target audit field according to the target audit script to obtain an audit report of the target audit field, where the audit report includes audit abnormal data.

9. An electronic device, characterized in that, including: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement a data audit method based on an audit model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement a data audit method based on an audit model according to any one of claims 1 to 7.

Images