Method and device for generating data quality control task and related product

Abstract

The invention discloses a data quality management and control task generation method and device and a related product, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and the like. In the method, for a to-be-controlled field of a target data table, task information of a to-be-generated control task is acquired; the task information comprises management and control indexes; the management and control index is used for indicating management and control content related to the data quality of the field to be managed and controlled; checking the task information based on the uniqueness checking rule to obtain a uniqueness checking result of the to-be-generated management and control task; if the uniqueness auditing result is that the uniqueness auditing of the to-be-generated management and control task is passed, configuring an alarm rule for the to-be-generated management and control task based on the task information; and based on the task information and the alarm rule, generating a target management and control task for managing and controlling the data quality of the field to be managed and controlled. Thus, the target management and control task is generated under the condition that the to-be-generated management and control task has uniqueness for the to-be-managed and control field, resource waste can be avoided, and the execution performance of the task is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and related products for generating data quality control tasks. Background Art

[0002] With the rapid development of information technology, all kinds of information in people's lives and work have become data. In various business scenarios, data is generated, collected, and stored in various data tables. With the rapid growth of data volume, data quality issues have gradually emerged. Fields are the basic components of data tables. The accuracy and consistency of their values ​​are directly related to the overall quality of the data table. Therefore, quality control of field values ​​in data tables is necessary to ensure data reliability and accuracy.

[0003] In related technologies, users can configure control tasks through data quality control platforms to manage the data quality of data tables. However, current data quality control platforms, designed for different users, often generate multiple identical control tasks for the same field in a data table, resulting in redundant control tasks. Redundant control tasks can waste both storage and computing resources during execution, impacting the performance of control tasks. Summary of the Invention

[0004] The embodiments of the present application provide a method, device and related products for generating data quality control tasks, the purpose of which is to ensure the uniqueness of control tasks for the same field in a data table, reduce the waste of storage resources and computing resources, and thereby improve the execution performance of control tasks.

[0005] In a first aspect, the present application provides a method for generating a data quality control task, the method comprising:

[0006] For the field to be controlled in the target data table, task information of a control task to be generated is obtained; the task information includes a control indicator; the control indicator is used to indicate the control content related to the data quality of the field to be controlled;

[0007] Review the task information based on the uniqueness review rule to obtain a uniqueness review result of the control task to be generated;

[0008] If the uniqueness review result is that the uniqueness review of the to-be-generated control task is passed, configuring an alarm rule for the to-be-generated control task based on the task information;

[0009] Based on the task information and the alarm rule, a target control task for controlling the data quality of the field to be controlled is generated.

[0010] A second aspect of the present application provides a device for generating a data quality control task, the device comprising:

[0011] An information acquisition module is used to acquire task information of a control task to be generated for a field to be controlled in a target data table; the task information includes a control indicator; the control indicator is used to indicate control content related to the data quality of the field to be controlled;

[0012] A uniqueness review module, configured to review the task information based on uniqueness review rules and obtain a uniqueness review result of the control task to be generated;

[0013] a rule configuration module, configured to configure an alarm rule for the to-be-generated control task based on the task information if the uniqueness audit result is that the uniqueness audit of the to-be-generated control task is passed;

[0014] A task generation module is used to generate a target control task for controlling the data quality of the field to be controlled based on the task information and the alarm rule.

[0015] A third aspect of the present application provides a device for generating a data quality control task, the device comprising a processor and a memory:

[0016] The memory is used to store a computer program and transmit the computer program to the processor;

[0017] The processor is configured to execute the steps of the method for generating a data quality control task provided in the first aspect according to the instructions in the computer program.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program. When the computer program is executed by a device for generating a data quality control task, the computer program implements the steps of the method for generating a data quality control task provided in the first aspect.

[0019] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a data quality control task generation device, implements the steps of the data quality control task generation method provided in the first aspect.

[0020] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0021] In this method, for the fields to be controlled in the target data table, the task information of the control task to be generated is first obtained, wherein the task information includes control indicators, which are used to indicate the control content related to the data quality of the fields to be controlled; then, the task information is reviewed based on the uniqueness review rules to obtain the uniqueness review result of the control task to be generated; if the uniqueness review result is that the uniqueness review of the control task to be generated is passed, an alarm rule is configured for the control task to be generated based on the task information; then, based on the task information and the alarm rule, a target control task for controlling the data quality of the fields to be controlled is generated.

[0022] Because the task information of the pending control task reflects its characteristics and forms the basis for determining its uniqueness, the uniqueness audit result for the pending control task indicates whether the same control task already exists for the pending control field in the target data table. If the uniqueness audit result indicates that the pending control task has passed, this indicates that the generated target control task is unique for the pending control field. This prevents the generation of duplicate control tasks, thereby avoiding the waste of storage and computing resources and improving the execution performance of control tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A scenario architecture diagram of a method for generating data quality control tasks provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a method for generating a data quality control task provided in an embodiment of the present application;

[0025] Figure 3 A code diagram of a deployment control logic provided in an embodiment of the present application;

[0026] Figure 4 A code diagram of a deployment control logic and alarm logic provided in an embodiment of the present application;

[0027] Figure 5 A code diagram of another deployment control logic provided in an embodiment of the present application;

[0028] Figure 6a A flowchart of a specific task uniqueness review method provided in an embodiment of the present application;

[0029] Figure 6b A flowchart of a specific task generation and alarm information push method provided in an embodiment of the present application;

[0030] Figure 7A schematic diagram of the structure of a device for generating data quality control tasks provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of the structure of the server in the embodiment of the present application;

[0032] Figure 9 This is a structural diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In the related art, users can operate through the data quality control platform to configure and obtain control tasks. However, the data quality control platform usually faces multiple users, and the needs of multiple users may be the same or different. It is easy to generate multiple identical control tasks for the same field in the data table, resulting in redundancy of control tasks. For example, a data table involves two business areas, and the business personnel of the two business areas need to set the same control tasks for the same field in the data table, but the information of the two business personnel is not interoperable, so two identical control tasks will be generated, which will cause redundancy of control tasks. The redundancy of control tasks will cause waste of resources. For example, storing redundant control tasks will cause waste of storage resources, and executing the same control tasks will cause waste of computing resources. These situations will affect the execution performance of the data quality control platform in executing control tasks.

[0034] In view of the above problems, a method, device and related products for generating data quality control tasks are provided in the present application, with the purpose of ensuring the uniqueness of the control tasks for the same field in the data table, reducing the waste of storage resources and computing resources, and thus improving the execution performance of the control tasks. In the technical solution provided in the present application, for the field to be controlled in the target data table, the task information of the control task to be generated is obtained; the task information includes control indicators; the control indicators are used to indicate the control content related to the data quality of the field to be controlled; the task information is audited based on the uniqueness audit rules to obtain the uniqueness audit result of the control task to be generated; if the uniqueness audit result is that the uniqueness audit of the control task to be generated is passed, then the alarm rules are configured for the control task to be generated based on the task information; based on the task information and the alarm rules, a target control task for controlling the data quality of the field to be controlled is generated.

[0035] In this way, based on the task information of the control task to be generated, it is determined whether the same control task already exists. When the uniqueness of the control task to be generated is ensured and passed, that is, when the control task to be generated is unique to the field to be controlled, there will be no control task that is the same as the generated target control task. Therefore, it can avoid wasting storage resources and computing resources, thereby improving the execution performance of the task.

[0036] Next, the execution subject of the method for generating data quality control tasks provided in the embodiment of the present application is introduced.

[0037] The execution subject of the method for generating data quality control tasks provided in the embodiment of the present application may be a terminal device. The terminal device may execute the method for generating data quality control tasks, and generate target control tasks for the fields to be controlled in the target data table. For example, for the fields to be controlled in the target data table, the terminal device obtains the task information of the control tasks to be generated. As an example, the terminal device may specifically include but is not limited to mobile phones, desktop computers, tablet computers, laptop computers, PDAs, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The execution subject of the method for generating data quality control tasks provided in the embodiment of the present application may also be a server. For example, for the fields to be controlled in the target data table, the server obtains the task information of the control tasks to be generated.

[0038] In addition, the method for generating data quality control tasks provided in the embodiments of the present application can also be performed collaboratively by a terminal device and a server. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. Therefore, the embodiments of this application do not limit the implementation subject of the technical solution of this application.

[0039] Figure 1 This is a scenario architecture diagram of a method for generating data quality control tasks provided in an embodiment of the present application. Figure 1 It includes servers and various forms of terminal devices. Figure 1 The server shown can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. Furthermore, the server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0040] The method for generating data quality control tasks provided in this application relates to the field of artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0042] Next, from the perspective of the terminal device, the method for generating the data quality control task provided in the embodiment of the present application is specifically introduced.

[0043] See also Figure 2 , which is a flowchart of a method for generating a data quality control task provided in an embodiment of the present application.

[0044] Combine Figure 2 As shown, the method for generating the data quality control task may include the following steps:

[0045] S201: For the field to be controlled in the target data table, obtain task information of the control task to be generated.

[0046] In the embodiments of the present application, the target data table refers to a table used to store data, consisting of vertical columns and horizontal rows. The columns represent different fields of the data, which are used to represent the attributes of the data, while the rows represent the values ​​under the fields, which are used to represent the specific information of the data. For example, data table 1 includes field 1 "name" and field 2 "age". The values ​​under field 1 can include "Zhou", "Wu", and "Zhang", etc., and the values ​​under field 2 can include "32", "38", and "25", etc.

[0047] The field to be controlled refers to any field that needs to control data quality. For example, the field to be controlled can be the "name" mentioned above, etc., and this application does not limit this.

[0048] A pending control task is a control task that needs to be generated for the field to be controlled. It is used to control the value of the field to be controlled in the target data table to ensure the accuracy, completeness, consistency, and reliability of the data.

[0049] Task information refers to information related to the control task to be generated. In some embodiments, task information may include control indicators, which are used to indicate control content related to the quality of the field data to be controlled. For example, control indicators may include the number of null values ​​(also called the number of empty values), the number of zero values, the number of integer data anomalies, etc.

[0050] In one possible implementation, a data quality control pool can be established, which can store multiple control tasks. These can include control tasks pre-configured by developers (also known as default control tasks) and control tasks configured by users of the data quality control pool. This application does not limit this. It should be emphasized that the control tasks stored in the data quality control pool are unique, that is, there will be no duplicate control tasks for the same field in the same data table.

[0051] Exemplarily, the terminal device may provide the user with a configuration page of the data quality control pool, and the user may submit task information through the configuration page operation. In response to the user's operation, the terminal device may obtain task information of the control task to be generated.

[0052] Specifically, the configuration page can display the task information of some management and control tasks, and the user can directly check the box to enable the terminal device to obtain the task information, that is, there is no need for the user to manually enter the task information. The management and control tasks generated based on the user's check operation can be called basic management and control tasks; in addition, the management and control tasks generated by the task information manually entered by the user can be called custom management and control tasks, which is not limited in this application.

[0053] In a possible implementation of the present application, the task information may also include data table information of the target data table and field information of the fields to be controlled. The data table information refers to relevant information describing the attributes of the data table. For example, the data table information may include: the cluster name of the cluster to which the data table belongs, the library name of the library to which it belongs, the table name, the creation time, the task identifier, etc. The field information refers to relevant information describing the attributes of the field. For example, the field information may include the field name, field annotation, whether it is a core field, etc.

[0054] Accordingly, the method for generating the data quality control task may further include the following steps:

[0055] A1: If it is determined that the data table information of the target data table and the field information of the field to be controlled exist in the data quality control pool, the steps of reviewing the task information based on the uniqueness review rule are performed to obtain the uniqueness review result of the control task to be generated.

[0056] If it is determined that the data table information of the target data table and the field information of the field to be controlled exist in the data quality control pool, indicating that a control task for the field to be controlled has been generated through the data quality control pool, S202 can be continued.

[0057] For example, the table name and other information of the target data table can be matched with the information of the data table stored in the data quality control pool. If the same information is matched, it indicates that the data quality control pool has stored the data table information of the target data table. If the same information is not matched, it indicates that the data quality control pool does not store the data table information of the target data table.

[0058] For example, similarly, the field name and other information of the field to be controlled can be matched with the information of the field in the target data table stored in the data quality control pool. If the same information is matched, it means that the data quality control pool has stored the field information of the field to be controlled. If the same information is not matched, it means that the data quality control pool does not store the field information of the field to be controlled.

[0059] A2: If it is determined that the data table information of the target data table and the field information of the field to be controlled do not exist in the data quality control pool, the data table information of the target data table and the field information of the field to be controlled are stored in the data quality control pool, and the task information is audited based on the uniqueness audit rule to obtain the uniqueness audit result of the control task to be generated.

[0060] If the data table information of the target data table and the field information of the field to be controlled do not exist in the data quality control pool, it indicates that this is the first time that the control task of the target data table is generated through the data quality control pool. Therefore, it is necessary to first store the data table information of the target data table and the field information of the field to be controlled, and then continue to execute S202.

[0061] Exemplarily, when storing the data table information of the target data table, assuming that the target data table is stored in a database, the terminal device can obtain and store the data table information by pulling the task cycle information, metadata information (including the partition name and partition type of the data table), the creation time of the data table, etc. of the target data table from the database.

[0062] It should be noted that in the embodiment of the present application, the data table can be simply referred to as a table.

[0063] As another example, data table information can be obtained and stored through the content such as the follower and table level entered by the user on the configuration page of the data quality control pool. This application does not limit this.

[0064] As an example, see Table 1, which exemplarily shows the table structure of a data table storing data table information, including fields storing the data table information, field types, and field comments.

[0065] Table 1: The table structure of the data table that stores data table information

[0066] Field (column name) Field Type Field Comments system_id varchar(128) Cluster name db_name varchar(128) Library Name tbl_name varchar(128) Table name fis_part tinyint(1) Whether to partition part_name varchar(128) Partition name part_type varchar(128) Partition Type cycle_type varchar(128) Task cycle table_level varchar(128) Table level task_id varchar(128) Task ID task_type varchar(128) Task Type script_name varchar(1024) Script Name mnt_user varchar(128) Follow people mnt_date varchar(128) Setup time is_monitor tinyint(1) Has it been controlled? is_core_tbl tinyint(1) Is it a core table?

[0067] A3: Based on the task information, if it is determined that the data table information of the target data table exists in the data quality control pool but the field information of the field to be controlled does not exist, the field information of the field to be controlled is stored in the data quality control pool, and the task information is audited based on the uniqueness audit rule to obtain the uniqueness audit result of the control task to be generated.

[0068] If the data table information of the target data table exists in the data quality control pool, but the field information of the field to be controlled does not exist, it indicates that the control tasks of other fields in the target data table have been generated through the data quality control pool, and this is the first time that the control task of the field to be controlled is generated through the data quality control pool. Therefore, it is necessary to store the field information of the field to be controlled first, and then continue to execute S202.

[0069] For example, field information can be obtained and stored through the field name, field type, field comment, whether it is a core field, whether it is an enumeration field, whether it is a primary key, etc. entered by the user on the configuration page of the data quality control pool. This application does not limit this.

[0070] As another example, assuming that the target data table is stored in a database, the terminal device can obtain and store field information by pulling the field name, field type, etc. of the field to be controlled from the database, and this application does not limit this.

[0071] As an example, see Table 2, which exemplarily shows the table structure of a data table storing field information, including the fields storing the field information, field types, and field comments.

[0072] Table 2: The table structure of the data table that stores field information

[0073] Field (column name) Field Type Field Comments field_name varchar(128) Field Name field_type varchar(128) Field Type field_comment varchar(128) Field Comments is_core_field tinyint(1) Is it a core field? is_enum tinyint(1) Is it an enumeration field? is_key tinyint(1) Is it a primary key?

[0074] In addition, in some embodiments, before storing the field information of the field to be controlled into the data quality control pool, it is determined whether the field to be controlled has a field name. If it does not have a field name, it indicates that the field to be controlled is a user-defined field, and a prompt message can be displayed to prompt the user to enter the field name of the field to be controlled. Subsequently, through the field name entered by the user, the terminal device can obtain the field information of the field name of the field to be controlled.

[0075] In some embodiments, the task information may also include the control type, control indicator name, control rule description, control cycle, establishment time, and other task information used to describe the attributes of the control task to be generated.

[0076] For example, the above task information can be obtained through the content entered by the user on the configuration page of the data quality control pool, or automatically pulled from the database, etc., and this application does not limit this.

[0077] As an example, see Table 3, which exemplarily shows the table structure of a data table storing task information, including fields storing field information, field types, and field comments.

[0078] Table 3: The table structure of the data table storing task information

[0079]

[0080] As an example, see Table 4, which exemplarily shows the data table information, field information, and task information of the target data table to which the field to be controlled belongs, including field comments and field values.

[0081] Table 4 Field information table of fields to be controlled

[0082]

[0083]

[0084] S202: Review the task information based on the uniqueness review rule to obtain the uniqueness review result of the control task to be generated.

[0085] In this embodiment of the present application, the uniqueness audit rule is used to audit whether the control task to be generated is unique for the field to be controlled, that is, whether the same control task already exists. A pass in the audit indicates that the control task to be generated is unique; a failure in the audit indicates that the control task to be generated is not unique and the same control task already exists.

[0086] Exemplarily, based on the above example, the data quality control pool includes multiple control tasks. The task information is audited based on the uniqueness audit rule to obtain a uniqueness audit result of whether the control task to be generated is unique in the data quality control pool, that is, to determine whether there is the same control task for the field to be controlled in the data quality control pool.

[0087] In a possible implementation of the present application, the uniqueness review rule may include a first review rule and a second review rule.

[0088] Among them, the first review rule is to determine whether there is a logical expression that is the same as the target logical expression of the control indicator in the data quality control pool for the field to be controlled.

[0089] In an embodiment of the present application, a logical expression is used to indicate the control logic of a control indicator. In some embodiments, the logical expression includes a logical condition. The logical condition is used to determine whether the value of the field to be controlled meets the requirements. For example, the logical condition can be that the value of the field to be controlled is a null value, indicating that data with a null value is determined under the field to be controlled; or that the value of the field to be controlled is 0, indicating that data with a value of 0 is determined under the field to be controlled.

[0090] In addition, in some embodiments, a logical expression may include logical conditions and logical rules. Logical rules are rules used to combine logical conditions to generate logical expressions with more complex logic, such as using operators such as logical AND, logical OR, and logical NOT to connect logical conditions. For example, logical condition 1 is that the value of the field to be controlled is empty, and logical condition 2 is that the value of the field to be controlled is zero. Using logical OR to connect logical condition 1 and logical condition 2 indicates that data with a value of 0 or a null value is determined under the field to be controlled.

[0091] In some embodiments, the data quality control pool includes control tasks, each of which has a logical expression for a control indicator. Therefore, for the field to be controlled, a match can be performed against the logical expression for the control indicator in the data quality control pool based on the target logical expression for the control indicator. A successful match indicates that the data quality control pool has a control task identical to the control task to be generated; a failed match indicates that the data quality control pool does not have a control task identical to the control task to be generated.

[0092] The above-mentioned second review rule is for the field to be controlled, based on the mapping relationship between the target indicator name and the similar indicator name of the control indicator, to determine whether there is a similar indicator name with a mapping relationship with the target indicator name in the data quality control pool.

[0093] In some embodiments, the mapping relationship between the target indicator name and the similar indicator name of the control indicator refers to pre-stored similar indicator names that are similar to the target indicator name. For example, the target indicator name is "number of null values" and the similar indicator names are "number of null values ​​ratio" and "number of null values ​​situation".

[0094] In some embodiments, the data quality control pool includes a control task, and the control indicator of the control task has an indicator name. Therefore, based on the mapping relationship between the target indicator name and the similar indicator name of the control indicator, it can be matched with the indicator name stored in the data quality control pool. If there is a similar indicator name, it indicates that the data quality control pool has a control task that is the same as the control task to be generated; if there is no similar indicator name, the matching failure indicates that the data quality control pool does not have a control task that is the same as the control task to be generated.

[0095] It should be emphasized that in the embodiments of the present application, similar control tasks are also determined to be the same control task. For example, the number of null values ​​and the percentage of vacant values ​​are for the same attribute of the field to be controlled, so they also indicate the same control task.

[0096] Accordingly, S202 may specifically be: reviewing the task information based on the first review rule and the second review rule to obtain a unique review result of the control task to be generated.

[0097] In some embodiments, if it is determined based on the first audit rule that there is no logical expression in the data quality control pool that is identical to the target logical expression of the control indicator, and based on the second audit rule that there is no similar indicator name in the data quality control pool that has a mapping relationship with the target indicator name, then the uniqueness audit result is that the uniqueness audit of the control task to be generated has passed. Otherwise, that is, if it is determined based on the first audit rule that there is a logical expression in the data quality control pool that is identical to the target logical expression of the control indicator, or if it is determined based on the second audit rule that there is a similar indicator name in the data quality control pool that has a mapping relationship with the target indicator name, then the uniqueness audit result is that the uniqueness audit of the control task to be generated has failed.

[0098] In this way, by reviewing the uniqueness of the control tasks to be generated through the first review rule and the second review rule, it is possible to avoid the existence of identical control tasks for the same field in the data quality control pool, saving storage resources and computing resources, and helping to improve the execution performance of control tasks.

[0099] In a possible implementation of the present application, the above-mentioned uniqueness review rule may also include a third review rule, which is to determine whether there is a target data sub-table storing the same data as the target data table in the data quality control pool, and based on the first review rule and the second review rule, review the target data sub-table to determine whether the field to be controlled has the same control task as the control task to be generated.

[0100] Among them, the target data sub-table is a data table that stores the same content as the target data table, indicating that the target data sub-table also includes the field to be controlled. In the data quality control pool, the field to be controlled in the target data sub-table can be reviewed based on the above-mentioned first review rule and second review rule.

[0101] Accordingly, S202 may specifically be: reviewing the task information based on the first review rule, the second review rule, and the third review rule to obtain a unique review result of the control task to be generated.

[0102] In some embodiments, if the first audit rule determines that the data quality control pool does not contain a logical expression identical to the target logical expression of the control indicator, and the second audit rule determines that the data quality control pool does not contain a similar indicator name that has a mapping relationship with the target indicator name, and the third audit rule determines that the target data sub-table exists, and the first and second audit rules pass the uniqueness audit of the to-be-controlled field in the target data sub-table, then the uniqueness audit result is that the uniqueness audit of the to-be-generated control task has passed. Otherwise, the uniqueness audit result is that the uniqueness audit of the to-be-generated control task has failed.

[0103] In this way, by considering the target data sub-table that stores the same data as the target data table and auditing it, the existence of the same control tasks for the same field in the data quality control pool can be further avoided, which is conducive to saving storage resources and computing resources and improving the execution performance of control tasks.

[0104] S203: If the uniqueness review result is that the uniqueness review of the to-be-generated control task is passed, an alarm rule is configured for the to-be-generated control task based on the task information.

[0105] In the embodiments of the present application, an alarm rule refers to a rule set to trigger a management and control task alarm. In some embodiments, an alarm rule (also referred to as an alarm indicator) may include an alarm threshold, which is used to determine when an alarm is triggered. For example, if the alarm rule specifies that the value of a management and control indicator exceeds the alarm threshold, then the alarm rule is met, an alarm is triggered, and an alarm message is generated to prompt the user.

[0106] If the uniqueness review result indicates that the uniqueness review of the control task to be generated has passed, an alarm rule can be configured for the control task to be generated.

[0107] For example, the user may input alarm rules on the configuration page of the data quality control pool, and the terminal device may first obtain the alarm rules and then configure the generated control tasks based on the alarm rules. This application does not limit this.

[0108] In a possible implementation of the present application, the method for generating the data quality control task may further include:

[0109] B1: If the uniqueness review result indicates that the to-be-generated control task has failed the uniqueness review, a pre-generated control task that matches the task information is determined from the control tasks included in the data quality control pool.

[0110] If the uniqueness review of the control task to be generated fails, indicating that a control task identical to the control task to be generated already exists, a control task that matches the task information, that is, the same as the control task to be generated, can be determined as the pre-generated control task.

[0111] For example, if a uniqueness review is performed in a data quality control pool, a pre-generated control task can be found in the data quality control pool.

[0112] B3: If it is determined that the pre-generation control task is not configured with an alarm rule, an alarm rule is configured for the pre-generation control task based on the task information of the pre-generation control task.

[0113] Exemplarily, based on the above example, if the pre-generated control tasks stored in the data quality control pool are not configured with alarm rules, they can be configured using the alarm rules entered by the user on the configuration page of the data quality control pool.

[0114] B3: If it is determined that the pre-generated control task has been configured with an alarm rule, then in response to the modification operation on the alarm threshold of the alarm rule, the alarm threshold is modified.

[0115] Exemplarily, based on the above example, if the pre-generated control tasks stored in the data quality control pool are configured with alarm rules, the user can modify the alarm threshold in the alarm rule by performing an operation on the configuration page of the data quality control pool to obtain an alarm rule that meets user needs.

[0116] In addition, in some embodiments, if it is determined that the pre-generated management and control task has been configured with an alarm rule and the user has not performed any operation on the alarm threshold, it can be determined that the alarm rule has met the user's needs and does not need to be modified.

[0117] It should be noted that, in the embodiment of the present application, the pre-generated control task refers to a control task that has been generated, but it is necessary to further determine whether the alarm rules have been configured.

[0118] It should be noted that in order to meet the needs of different users, multiple alarm rules can be configured for the control task to be generated. For example, for the same alarm rule, the alarm thresholds are different. This application does not limit the number of alarm rules. For example, alarm rule 1 is: the number of null values ​​in the field to be controlled exceeds 20; alarm rule 2 is: the number of null values ​​in the field to be controlled exceeds 30.

[0119] In this way, when there is a pre-generated control task that is the same as the control task to be generated, it is possible to further determine whether the alarm rule meets the user needs, which is conducive to improving the user experience.

[0120] Furthermore, in one possible implementation, users can review alert rules. For example, for the same alert rule, why different alert thresholds are required, and whether different thresholds are necessary, can be determined. Furthermore, for example, existing alert rules can be reused, though this application does not limit this. This further avoids wasted storage resources and improves alert execution performance.

[0121] In addition, in some embodiments, the alarm rules may also include relevant data table information such as the database name of the target data table, field information such as the field name of the field to be controlled, task information such as the control indicator name of the dependent control task, and alarm information such as the alarm period, alarm threshold source, and alarm threshold type. The above information can be collectively referred to as alarm configuration information.

[0122] As an example, see Table 5, which exemplarily shows the table structure of a data table storing alarm configuration information, including field names, field types, and field comments.

[0123] Table 5: Table structure of the data table storing alarm configuration information

[0124]

[0125]

[0126] As an example, see Table 6, which exemplarily shows alarm configuration information, including field annotations and field values.

[0127] Table 6 Alarm configuration information table

[0128]

[0129] S204: Based on the task information and the alarm rules, a target control task is generated for controlling the data quality of the field to be controlled. Based on the task information of the control task to be generated and the alarm rules configured therefor, a target control task can be generated, which can control the data quality of the field to be controlled.

[0130] In a possible implementation of the present application, S204 may specifically include: performing deployment processing based on the control logic of the control indicators in the task information, and performing deployment processing based on the alarm logic of the alarm rules to generate a target control task.

[0131] In some embodiments, the control logic refers to the calculation logic of the control indicators, and the alarm logic refers to the judgment logic of the alarm rules.

[0132] Exemplarily, the above-mentioned control indicator is the proportion of null values ​​in multiple values ​​under the field to be controlled; the alarm rule is that the proportion of null values ​​exceeds the ratio threshold, then the deployment control logic can be: first, obtain multiple values ​​under the field to be controlled in the target data table; then determine the number of null values ​​from the multiple values, and take the ratio of the number of null values ​​to the number of multiple values ​​as the proportion of null values; the deployment control logic can be: if the proportion of null values ​​exceeds the ratio threshold, then generate an alarm message.

[0133] As an example, see Figure 3 , which is a code diagram of a deployment control logic provided by an embodiment of the present application. Figure 3 As shown, the target data table includes three fields to be controlled, and the control indicators are all the results of the proportion of null values ​​in multiple values ​​under the fields to be controlled. You can access the target data table once, splice the control identifiers of each field to be controlled, that is, splice the partition time, order type, order number, basic indicator, proportion indicator and null quantity proportion, and then splice the corresponding control logic. In this way, the control logic of these three fields to be controlled can be deployed. Then execute Figure 3 As shown in the code, you can output three lines of control results by converting one line into multiple lines using the code "later view explode".

[0134] As an example, see Figure 4 , which is a code diagram of a deployment control logic and alarm logic provided by an embodiment of the present application. Figure 4 As shown, the target data table includes three fields to be controlled, and the control indicators are all the results of the proportion of null values ​​in multiple values ​​under the fields to be controlled. You can determine the control indicators of each field to be controlled by accessing the target data table once, and deploy the corresponding alarm logic respectively.

[0135] In addition, in some embodiments, a blocking rule can be configured for the control task to be generated, and a blocking task related to the target control task can be generated based on the blocking rule. When executing the target control task, if the blocking rule is met, the execution of the target control task will be suspended.

[0136] As another example, the above-mentioned control indicator is the proportion of enumerated outlier values ​​in multiple values ​​under the field to be controlled; the alarm rule is that the proportion of enumerated outlier values ​​exceeds the ratio threshold, then the deployment control logic can be: in the target data table, obtain multiple values ​​under the field to be controlled; then based on one or more preset enumeration values, determine from the multiple values ​​a value different from the preset enumeration value as the enumeration outlier value, and take the ratio of the number of enumeration outlier values ​​to the number of multiple values ​​as the proportion of enumeration outlier values; the deployment control logic can be: if the proportion of enumeration outlier values ​​exceeds the ratio threshold, then generate an alarm message.

[0137] As an example, see Figure 5 , which is a code diagram of another deployment control logic provided by the embodiment of this application. Figure 5 As shown, the target data table includes two fields to be controlled. The control indicators are the periodic data volume and the proportion of enumerated abnormal values ​​in the multiple values ​​of the controlled field. You can deploy the control logic for these two fields by accessing the target data table once, combining the control indicators for each field to be controlled—that is, combining the basic indicators (quantity indicator, periodic data volume, fixed format indicator, proportion indicator, and proportion of enumerated abnormal values) and then combining the corresponding control logic.

[0138] It should be noted that the above is only an example. You can also deploy control logic for other control indicators such as minimum values, and this application does not limit this.

[0139] In this way, by deploying control logic and alarm logic, target control tasks can be generated to control the fields to be controlled. When the alarm rules are met, alarm messages are generated to prompt users, which is conducive to maintaining data quality.

[0140] In addition, in some embodiments, after generating a target control task, the target control task can be executed for the field to be controlled; if the value of the field to be controlled meets the alarm rule, an alarm message is generated; and then the alarm message is sent to the subscribed user.

[0141] As an example, see Table 7, which exemplifies the control capabilities of multiple control tasks in the data quality control pool. Specifically, it can be reflected from the four dimensions of data quality, including completeness, accuracy, consistency and timeliness.

[0142] Table 7 Data quality control statistics

[0143]

[0144]

[0145]

[0146]

[0147] Among them, the ISP indicator refers to the population stability index (PSI) indicator, which is used to control the stability of the data. As shown in Table 7, the control object of the control task, the control indicator type of the control task, the control indicator name, and whether it is default control are displayed. Default control means that the control task is the default control task, which can be a control task pre-generated by the terminal device for all data tables in response to the operation of technical personnel such as development, that is, all data tables need to be controlled. Non-default control means that the control task can be a control task generated by the terminal device in response to the user's operation. The default control prerequisite means that the corresponding control task is the prerequisite for the default control task.

[0148] In addition, it should be noted that this application does not limit the content of Table 7, and notes can also be included for each row. For example, the notes for multiple rows corresponding to consistency can be: 1. Generally, the consistency comparison of downstream data of the same source but different branches 2. The consistency of upstream and downstream data of the same data link 3. The consistency of data in different cycles of the same data (generally, the data consistency of two adjacent data cycles). This application does not limit this.

[0149] In a possible implementation of the present application, the method for generating data quality control tasks may also include: storing the generated target control tasks in a data quality control pool; then determining multiple control tasks for controlling the data quality of the target data table from the data quality control pool; and then executing multiple control tasks simultaneously.

[0150] In some embodiments, the target data table may include multiple fields, and one field has one or more management and control tasks. Then, multiple management and control tasks related to the target data table are determined from the data quality control pool and executed.

[0151] This solves the problem that management and alarm cannot be performed simultaneously for the same target data table, making it easier for users to fully understand the data quality of the entire target data table.

[0152] In a possible implementation of the present application, the target data table is stored in a database. Accordingly, the method for generating the data quality control task may further include: storing the target control task in a data quality control pool; then regularly updating the database according to a preset period; and then deleting the target control task from the data quality control pool if the target data table does not exist in the updated database, or if the target data table exists in the updated database and there is no field to be controlled.

[0153] In some embodiments, the database can store multiple data tables, including a target data table. The database is updated regularly according to a preset period. For example, if the preset period is 7 days, the database can be updated once every 7 days to delete unnecessary data. If the target data table does not exist in the updated database, or the target data table exists but the field to be controlled does not exist in it, it indicates that the control object of the control task has been deleted (that is, offline), so the target control task can be deleted from the data quality control pool.

[0154] In this way, the life cycle of the target control task is managed based on whether the control object of the control task is offline, avoiding unlimited expansion of control tasks and further avoiding waste of resources.

[0155] In a possible implementation of the present application, the method for generating the data quality control task may further include: storing the target control task in a data quality control pool; then executing the control task corresponding to the task execution operation in response to the user's task execution operation; then obtaining the task execution log of the data quality control pool after a first preset time period; finally, if there is no field to be controlled in the task execution log, deleting the target control task from the data quality control pool.

[0156] A task execution operation refers to a user triggering the execution of a required control task. For example, the data quality control pool includes control tasks 1, 2, and 3. If a user triggers the execution of both control tasks 1 and 2, then both tasks are executed.

[0157] The task execution log is used to record a series of activities and events during the execution of a control task. For example, the task execution log may include, but is not limited to, key information such as the field to be controlled, the task start time, the person executing the task, the control logic used, and the task completion status. If the field to be controlled does not exist in the task execution log, it indicates that the target control task has not been executed, which means that the user does not need the target control task at the moment and can delete it.

[0158] Exemplarily, the first preset time period may be 15 days, which is not limited in this application.

[0159] In this way, the user's demand for target control tasks can be judged through task execution logs to manage the life cycle of target control tasks, delete control tasks that the user does not need, avoid unlimited expansion of control tasks, and further avoid waste of resources.

[0160] In a possible implementation of the present application, the method for generating data quality control tasks may also include: storing the target control tasks in a data quality control pool; executing the control tasks included in the data quality control pool; if the task execution time of the control tasks included in the data quality control pool exceeds a time threshold, deleting the control tasks that match the preset redundant control tasks from the data quality control pool.

[0161] The task execution time of the control tasks included in the data quality control pool can be used to represent the execution performance of the control tasks in the data quality control pool. The preset redundant control tasks refer to pre-set redundant control tasks with low importance.

[0162] When the task execution time exceeds the time threshold, it indicates that too many control tasks have caused performance problems. Therefore, you can take down the control tasks with low importance in the data quality control pool and retain the control tasks with high importance.

[0163] In this way, the life cycle of control tasks is managed based on their importance, and control tasks with low importance are deleted to avoid unlimited expansion of control tasks, further improving the execution performance of control tasks with high importance.

[0164] In a possible implementation of the present application, the method for generating the data quality control task may further include: storing the target control task in a data quality control pool; then, after a second preset time period, obtaining subscription information of the control tasks included in the data quality control pool; if the subscription information of the target control task does not exist in the subscription information, deleting the target control task from the data quality control pool.

[0165] In some embodiments, subscription information refers to the user's subscription status to the control task. If there is no subscription information for the target control task in the subscription information, it means that no user has subscribed to the target control task, which means that the user does not need the target control task, so it can be deleted from the data quality control pool.

[0166] In addition, if there is subscription information of a target control task in the subscription information, it indicates that a user has subscribed to the target control task, and the user needs to know the task execution status of the target control task, such as task execution status, alarm information, etc.

[0167] In this way, the user's demand for target control tasks can be judged by subscription information, and the life cycle of target control tasks can be further managed. The control tasks that the user does not need can be deleted to avoid unlimited expansion of control tasks and further avoid waste of resources.

[0168] Furthermore, in some embodiments, the execution results of control tasks in the data quality control pool can be stored in a control result table, which users can view and access. This allows the control results to be shared through the data quality control pool, facilitating users' overall understanding and analysis of data quality.

[0169] As an example, see Figure 6a , which is a flowchart of a specific task uniqueness audit method provided by an embodiment of the present application, such as Figure 6a As shown, the following steps may be specifically included:

[0170] S601: For the to-be-controlled field in the target data table, obtain the task information of the to-be-generated control task input by the user.

[0171] S602: For the target data table, determine whether the data table information of the target data table exists in the data quality control pool. If yes, execute S604; otherwise, execute S603.

[0172] S603: Store the data table information of the target data table into the data quality control pool, and execute S604.

[0173] S604: Determine whether there is field information of the field to be controlled in the data quality control pool. If yes, execute S605; otherwise, execute S606.

[0174] S605: Store the task information of the to-be-generated control task into the data quality control pool, and execute S609.

[0175] S606: Determine whether the field to be controlled has a field name. If yes, execute S608; otherwise, execute S607.

[0176] S607: In response to the user's setting operation, obtain the field name.

[0177] S608: Store the field information of the field to be controlled into the data quality control pool and execute S605.

[0178] S609: Review the task information based on the uniqueness review rule to obtain the uniqueness review result of the control task to be generated.

[0179] As an example, see Figure 6b , which is a flowchart of a specific task generation and alarm information push method provided by an embodiment of the present application, such as Figure 6b As shown, the following steps may be specifically included:

[0180] S610: If the uniqueness review result is that the uniqueness review of the to-be-generated control task is passed, an alarm rule is configured for the to-be-generated control task based on the task information.

[0181] S611: Based on the task information and the alarm rules, a target control task is generated for controlling the data quality of the field to be controlled.

[0182] S612: If the uniqueness review result is that the uniqueness review of the to-be-generated control task fails, a pre-generated control task that matches the task information is determined.

[0183] S613: If it is determined that the pre-generated control task is not configured with an alarm rule, an alarm rule is configured for the pre-generated control task based on the task information of the pre-generated control task.

[0184] S614: If it is determined that the pre-generated control task has been configured with an alarm rule, the alarm threshold is modified in response to a modification operation on the alarm threshold of the alarm rule.

[0185] S615: Execute the task and generate an alarm message to push to the user.

[0186] Among them, the task can be a target control task or a pre-generated control task. When the alarm rules are met, an alarm can be triggered and an alarm message can be sent to users who have subscribed to the task.

[0187] In addition, it should be noted that the specific implementation of the above S601-S615 can refer to the specific implementation of S201-S204 in the above embodiment, and this application will not repeat them here.

[0188] Based on the method for generating data quality control tasks provided in the above embodiments, this application also provides a device for generating data quality control tasks. The device for generating data quality control tasks provided in the embodiments of this application is described in detail below.

[0189] See also Figure 7 , which is a structural diagram of a device for generating data quality control tasks provided by an embodiment of the present application. Figure 7 As shown, the data quality control task generation device 700 may specifically include:

[0190] The information acquisition module 710 is used to acquire task information of a control task to be generated for the field to be controlled in the target data table; the task information includes control indicators; the control indicators are used to indicate the control content related to the data quality of the field to be controlled;

[0191] The uniqueness review module 720 is used to review the task information based on the uniqueness review rules and obtain the uniqueness review result of the control task to be generated;

[0192] A rule configuration module 730 is configured to configure an alarm rule for the control task to be generated based on the task information if the uniqueness review result indicates that the uniqueness review of the control task to be generated has passed;

[0193] The task generation module 740 is used to generate a target control task for controlling the data quality of the field to be controlled based on the task information and the alarm rules.

[0194] As an implementation method, the uniqueness review rule may specifically include a first review rule and a second review rule;

[0195] The first review rule is to determine whether there is a logical expression in the data quality control pool that is the same as the target logical expression of the control indicator for the field to be controlled. The data quality control pool includes control tasks, and the control tasks have the logical expression of the control indicator.

[0196] The second review rule is to determine whether there is a similar indicator name with a mapping relationship with the target indicator name in the data quality control pool based on the mapping relationship between the target indicator name and the similar indicator name of the control indicator for the field to be controlled. The control indicator of the control task has an indicator name.

[0197] Accordingly, the uniqueness verification module 720 can be specifically used to:

[0198] Audit the task information based on the first audit rule and the second audit rule to obtain a unique audit result of the control task to be generated.

[0199] As an implementation method, the uniqueness review rule may further include a third review rule; the third review rule is to determine whether there is a target data sub-table storing the same data as the target data table in the data quality control pool, and to review whether the target data sub-table has the same control task as the control task to be generated in the target data sub-table based on the first review rule and the second review rule;

[0200] Accordingly, the uniqueness verification module 720 can be specifically used to:

[0201] Based on the first review rule, the second review rule and the third review rule, the task information is reviewed to obtain a unique review result of the control task to be generated.

[0202] As an implementation manner, the data quality control task generation device 700 may further include:

[0203] A first task determination module is configured to determine a pre-generated control task that matches the task information if the uniqueness review result indicates that the to-be-generated control task has failed the uniqueness review;

[0204] An alarm rule configuration module is used to configure an alarm rule for the pre-generated control task based on the task information of the pre-generated control task if it is determined that the pre-generated control task has no alarm rule configured;

[0205] The threshold modification module is used to modify the alarm threshold in response to a modification operation on the alarm threshold of the alarm rule if it is determined that the pre-generated control task has been configured with an alarm rule.

[0206] As an implementation manner, the data quality control task generation device 700 may further include:

[0207] A first information confirmation module is configured to, if it is determined that data table information of a target data table and field information of a field to be controlled exist in the data quality control pool, perform a step of reviewing the task information based on a uniqueness review rule to obtain a uniqueness review result of the control task to be generated;

[0208] The second information confirmation module is configured to, if it is determined that the data table information of the target data table and the field information of the field to be controlled do not exist in the data quality control pool, store the data table information of the target data table and the field information of the field to be controlled in the data quality control pool, and perform a step of reviewing the task information based on the uniqueness review rule to obtain a uniqueness review result of the control task to be generated;

[0209] The third information confirmation module is used to store the field information of the field to be controlled into the data quality control pool if it is determined that the data table information of the target data table exists in the data quality control pool and the field information of the field to be controlled does not exist, and to perform the steps of auditing the task information based on the uniqueness audit rules to obtain the uniqueness audit result of the control task to be generated.

[0210] As an implementation method, the task generation module 740 may be specifically used to:

[0211] Deployment processing is performed based on the control logic of the control indicators in the task information, and deployment processing is performed based on the alarm logic of the alarm rules to generate the target control task.

[0212] As an implementation manner, the data quality control task generation device 700 may further include:

[0213] A first task storage module is used to store target control tasks in a data quality control pool;

[0214] A second task determination module is used to determine a plurality of control tasks for controlling the data quality of a target data table from the data quality control pool;

[0215] The first task execution module is used to execute multiple management and control tasks simultaneously.

[0216] As an implementation manner, the target data table is stored in a database, and the data quality control task generation device 700 may further include:

[0217] The second task storage module is used to store the target control tasks in the data quality control pool;

[0218] A database update module, used to update the database according to a preset period;

[0219] The first task deletion module is used to delete the target control task from the data quality control pool if the target data table does not exist in the updated database, or if the target data table exists in the updated database but there is no field to be controlled.

[0220] As an implementation manner, the data quality control task generation device 700 may further include:

[0221] The third task storage module is used to store the target control tasks in the data quality control pool;

[0222] A second task execution module is used to execute the management and control task corresponding to the task execution operation in response to the task execution operation;

[0223] A log acquisition module is used to obtain the task execution log of the data quality control pool after a first preset time period;

[0224] The second task deletion module is used to delete the target control task from the data quality control pool if there is no field to be controlled in the task execution log.

[0225] As an implementation manner, the data quality control task generation device 700 may further include:

[0226] The fourth task storage module is used to store the target control tasks in the data quality control pool;

[0227] The third task execution module is used to execute the control tasks included in the data quality control pool;

[0228] The third task deletion module is configured to delete the control tasks that match the preset redundant control tasks from the data quality control pool if the task execution time of the control tasks included in the data quality control pool exceeds a time threshold.

[0229] As an implementation manner, the data quality control task generation device 700 may further include:

[0230] The fifth task storage module is used to store the target control tasks in the data quality control pool;

[0231] A subscription information acquisition module, configured to acquire subscription information of the control tasks included in the data quality control pool after a second preset time period;

[0232] The fourth task deletion module is configured to delete the target control task from the data quality control pool if the subscription information of the target control task does not exist in the subscription information.

[0233] Next, the structures of the generation devices for data quality control tasks are introduced in terms of server form and terminal device form respectively.

[0234] An embodiment of the present application provides a device for generating a data quality control task, and the device for generating the data quality control task may be a server. Figure 8 This is a schematic diagram of a server structure provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 922 (for example, one or more processors) and memories 932, and one or more storage media 930 (for example, one or more massive storage devices) for storing application programs 942 or data 944. Among them, the memories 932 and the storage media 930 may be temporary storage or persistent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 may be configured to communicate with the storage medium 930 to execute a series of instruction operations in the storage medium 930 on the server 900.

[0235] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems 941, such as Windows Server 2003 or Windows Server 2003R. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0236] The CPU 922 is configured to execute the following steps:

[0237] For the field to be controlled in the target data table, task information of a control task to be generated is obtained; the task information includes a control indicator; the control indicator is used to indicate the control content related to the data quality of the field to be controlled;

[0238] Review the task information based on the uniqueness review rule to obtain a uniqueness review result of the control task to be generated;

[0239] If the uniqueness review result is that the uniqueness review of the to-be-generated control task is passed, configuring an alarm rule for the to-be-generated control task based on the task information;

[0240] Based on the task information and the alarm rule, a target control task for controlling the data quality of the field to be controlled is generated.

[0241] The embodiment of the present application also provides another device for generating data quality control tasks, which can be a terminal device. Figure 9 For the sake of convenience, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. Take the terminal device as a mobile phone as an example:

[0242] Figure 9 The block diagram shows a partial structure of the mobile phone provided by the embodiment of the present application. Figure 9 The mobile phone includes components such as a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will appreciate that Figure 9 The mobile phone structure shown in the figure does not constitute a limitation to the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0243] The following combination Figure 9 A detailed introduction to the various components of a mobile phone:

[0244] RF circuitry 1010 can be used to receive and transmit signals during information transmission or calls. Specifically, it receives downlink information from the base station and transmits it to processor 1080 for processing. It also transmits uplink data to the base station. Typically, RF circuitry 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, and more. RF circuitry 1010 can also communicate with the network and other devices via wireless communications. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), email, Short Messaging Service (SMS), etc.

[0245] The memory 1020 can be used to store software programs and modules. The processor 1080 executes the various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 1020 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0246] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel 1031) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction and detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 1080. It can also receive commands sent by the processor 1080 and execute them. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.

[0247] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (English full name: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (English full name: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 9 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0248] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0249] Audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the phone. Audio circuit 1060 converts received audio data into electrical signals and transmits them to speaker 1061, which then converts them into sound signals for output. Microphone 1062, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 1060 and converted into audio data. The audio data is then processed by processor 1080 and transmitted to, for example, another phone via RF circuit 1010, or stored in memory 1020 for further processing.

[0250] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse the web and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 9 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0251] Processor 1080 is the control center of the phone, connecting all parts of the phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 1020 and accessing data stored in memory 1020, it executes various phone functions and processes data, thereby collecting data and information about the phone as a whole. Optionally, processor 1080 may include one or more processing units; preferably, processor 1080 may integrate an application processor and a modem processor, where the application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1080.

[0252] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, thereby managing charging, discharging, and power consumption through the power management system.

[0253] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0254] In the embodiment of the present application, the processor 1080 included in the mobile phone also has the following functions:

[0255] For the field to be controlled in the target data table, task information of a control task to be generated is obtained; the task information includes a control indicator; the control indicator is used to indicate the control content related to the data quality of the field to be controlled;

[0256] Review the task information based on the uniqueness review rule to obtain a uniqueness review result of the control task to be generated;

[0257] If the uniqueness review result is that the uniqueness review of the to-be-generated control task is passed, configuring an alarm rule for the to-be-generated control task based on the task information;

[0258] Based on the task information and the alarm rule, a target control task for controlling the data quality of the field to be controlled is generated.

[0259] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program. When the computer program is run on a device for generating a data quality control task, the device for generating the data quality control task executes any one of the implementation methods of a method for generating a data quality control task described in the aforementioned embodiments.

[0260] An embodiment of the present application also provides a computer program product including a computer program, which, when run on a device for generating data quality control tasks, enables the device for generating data quality control tasks to execute any one of the implementation methods of a method for generating a data quality control task described in the aforementioned embodiments.

[0261] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0262] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the system is merely a logical function division. In actual implementation, there may be other division methods, such as multiple systems can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0263] The systems described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of this embodiment as needed.

[0264] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0265] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), random access memory (English full name: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store computer programs.

[0266] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0267] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a data quality control task, characterized in that: include: For the fields to be controlled in the target data table, obtain the task information of the control tasks to be generated; The task information includes control indicators; The control indicator is used to indicate the control content related to the data quality of the field to be controlled; Review the task information based on the uniqueness review rule to obtain a uniqueness review result of the control task to be generated; If the uniqueness review result is that the uniqueness review of the to-be-generated control task is passed, configuring an alarm rule for the to-be-generated control task based on the task information; Based on the task information and the alarm rule, a target control task for controlling the data quality of the field to be controlled is generated.

2. The method according to claim 1, characterized in that The uniqueness audit rule includes a first audit rule and a second audit rule; The first audit rule is to determine, for the field to be controlled, whether there is a logical expression in the data quality control pool that is identical to the target logical expression of the control indicator; the data quality control pool includes a control task, and the control task has a logical expression of the control indicator; The second audit rule is to determine, for the field to be controlled, whether there is a similar indicator name in the data quality control pool that has a mapping relationship with the target indicator name based on the mapping relationship between the target indicator name and the similar indicator name of the control indicator; the control indicator of the control task has an indicator name; The step of reviewing the task information based on the uniqueness review rule to obtain the uniqueness review result of the control task to be generated includes: The task information is reviewed based on the first review rule and the second review rule to obtain a unique review result of the to-be-generated control task.

3. The method according to claim 2, characterized in that The uniqueness audit rule also includes a third audit rule; the third audit rule is to determine whether there is a target data sub-table in the data quality control pool that stores the same data as the target data table, and based on the first audit rule and the second audit rule, to check whether the field to be controlled in the target data sub-table has the same control task as the control task to be generated; The reviewing of the task information based on the first review rule and the second review rule to obtain a unique review result of the to-be-generated control task includes: The task information is reviewed based on the first review rule, the second review rule, and the third review rule to obtain a unique review result of the to-be-generated control task.

4. The method according to claim 1, wherein Also includes: If the uniqueness review result is that the uniqueness review of the to-be-generated control task fails, determining a pre-generated control task that matches the task information; If it is determined that the pre-generated control task is not configured with an alarm rule, configuring an alarm rule for the pre-generated control task based on the task information of the pre-generated control task; If it is determined that the pre-generated control task has been configured with an alarm rule, the alarm threshold is modified in response to a modification operation on the alarm threshold of the alarm rule.

5. The method according to claim 1, wherein Also includes: If it is determined that the data table information of the target data table and the field information of the field to be controlled exist in the data quality control pool, the step of reviewing the task information based on the uniqueness review rule to obtain the uniqueness review result of the control task to be generated is performed; If it is determined that the data table information of the target data table and the field information of the field to be controlled do not exist in the data quality control pool, the data table information of the target data table and the field information of the field to be controlled are stored in the data quality control pool, and the step of reviewing the task information based on the uniqueness review rule to obtain the uniqueness review result of the control task to be generated is performed; If it is determined that the data table information of the target data table exists in the data quality control pool and the field information of the field to be controlled does not exist, the field information of the field to be controlled is stored in the data quality control pool, and the step of auditing the task information based on the uniqueness audit rule is performed to obtain the uniqueness audit result of the control task to be generated.

6. The method according to claim 1, characterized in that The generating, based on the task information and the alarm rule, a target control task for controlling the quality of the data of the to-be-controlled field includes: Deployment processing is performed based on the control logic of the control indicators in the task information, and deployment processing is performed based on the alarm logic of the alarm rule to generate the target control task.

7. The method according to claim 1, characterized in that Also includes: Storing the target control task in a data quality control pool; Determining a plurality of control tasks for controlling the data quality of the target data table from the data quality control pool; The multiple control tasks are executed simultaneously.

8. The method according to any one of claims 1 to 7, characterized in that The target data table is stored in a database, and the method further includes: Storing the target control task in a data quality control pool; updating the database according to a preset period; If the target data table does not exist in the updated database, or if the target data table exists in the updated database but the field to be controlled does not exist, the target control task is deleted from the data quality control pool.

9. The method according to any one of claims 1 to 7, characterized in that Also includes: Storing the target control task in a data quality control pool; In response to a task execution operation, executing a management and control task corresponding to the task execution operation; After a first preset time period, obtaining a task execution log of the data quality control pool; If the field to be controlled does not exist in the task execution log, the target control task is deleted from the data quality control pool.

10. The method according to any one of claims 1 to 7, characterized in that Also includes: Storing the target control task in a data quality control pool; Execute the control tasks included in the data quality control pool; If the task execution time of the control tasks included in the data quality control pool exceeds a time threshold, the control tasks that are consistent with the preset redundant control tasks are deleted from the data quality control pool.

11. The method according to any one of claims 1 to 7, characterized in that: Also includes: Storing the target control task in a data quality control pool; After a second preset time period, obtaining subscription information of the control tasks included in the data quality control pool; If the subscription information of the target control task does not exist in the subscription information, the target control task is deleted from the data quality control pool.

12. A device for generating data quality control tasks, characterized in that: include: The information acquisition module is used to obtain the task information of the control task to be generated for the field to be controlled in the target data table; The task information includes control indicators; The control indicator is used to indicate the control content related to the data quality of the field to be controlled; A uniqueness review module, configured to review the task information based on uniqueness review rules and obtain a uniqueness review result of the control task to be generated; a rule configuration module, configured to configure an alarm rule for the to-be-generated control task based on the task information if the uniqueness audit result is that the uniqueness audit of the to-be-generated control task is passed; A task generation module is used to generate a target control task for controlling the data quality of the field to be controlled based on the task information and the alarm rule.

13. A device for generating data quality control tasks, characterized in that: The device includes a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the steps of the method for generating a data quality control task according to any one of claims 1 to 11 according to instructions in the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, which, when executed by a data quality control task generation device, implements the steps of the data quality control task generation method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a device for generating a data quality control task, implements the steps of the method for generating a data quality control task according to any one of claims 1 to 11.

Images