Data quality monitoring method and device and electronic equipment

Abstract

The invention relates to a data quality monitoring method and device and electronic equipment, and the method comprises the steps: obtaining a target downstream data task of a target data task; the target data task is a data task for generating a data triggering alarm; the target downstream data task depends on output data and is provided with delay configuration information; no intermediate data task exists between the target downstream data task and the target data task; and according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information, target monitoring rule delay execution processing is performed on the target downstream data task, and delay monitoring processing is performed on the target downstream data task, so that the target data task has time to be repaired and an alarm is released. The task responsible object of the target downstream data task is prevented from receiving the alarm caused by the target data task, the invalid alarm frequency is reduced, the alarm efficiency is improved, and the data monitoring efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data quality monitoring method, device and electronic equipment. Background Art

[0002] The current data quality monitoring method mainly triggers data quality monitoring after data task A is completed, and performs quality checks on the output data of data task A according to the monitoring rules; if there are quality problems, an alarm notification will be issued, and it will be determined whether the downstream data task B of data task A needs to be blocked; if necessary, blocking will be executed, and the downstream data task B will stop executing, causing an alarm notification to be issued for the monitoring rules of the downstream data task B.

[0003] In the above scheme, an alarm notification of an upstream data task may cause alarm notifications of multiple downstream data tasks, resulting in an excessive number of alarms, poor alarm efficiency, and poor data quality monitoring efficiency. Summary of the invention

[0004] The present disclosure provides a data quality monitoring method, device and electronic equipment.

[0005] According to a first aspect of an embodiment of the present disclosure, a data quality monitoring method is provided, the method comprising: obtaining a target downstream data task of a target data task; the target data task is a data task that triggers an alarm for output data; the target downstream data task depends on the output data and is provided with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; and performing target monitoring rule delayed execution processing on the target downstream data task according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task.

[0006] In one embodiment of the present disclosure, the target downstream data task of the target data task is obtained, including: obtaining the target data task; querying a dependency table according to the target data task to obtain a candidate downstream data task that depends on the output data of the target data task and has no intermediate data tasks between the target data task; determining the length of the time interval between the monitoring rule execution time of the candidate downstream data task and the monitoring rule execution time of the target data task; and when the length of the time interval is less than or equal to a time length threshold and the candidate downstream data task is set with delay configuration information, using the candidate downstream data task as the target downstream data task.

[0007] In one embodiment of the present disclosure, the delay configuration information of the target downstream data task includes a delay tolerance duration, or an emergency operation; the delay tolerance duration indicates the tolerable delayed execution duration based on the original execution time of the target monitoring rule of the target downstream data task; the emergency operation indicates that the emergency operation needs to be executed before executing the target monitoring rule.

[0008] In one embodiment of the present disclosure, the target monitoring rule delayed execution processing is performed on the target downstream data task based on the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task, including: determining the delayed execution condition of the target downstream data task based on the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task; and performing the target monitoring rule delayed execution processing on the target downstream data task based on the delayed execution condition.

[0009] In one embodiment of the present disclosure, the delayed execution condition of the target downstream data task is determined based on the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task, including: when the delay configuration information includes a delay tolerance duration, adding the original execution time and the delay tolerance duration to obtain the delayed execution time of the target downstream data task; using the delayed execution time as the delayed execution condition; when the delay configuration information includes an emergency job, using the successful execution of the emergency job as the delayed execution condition; the execution time of the emergency job is the original execution time.

[0010] In one embodiment of the present disclosure, the target downstream data task is subjected to delayed execution processing of target monitoring rules according to the delayed execution condition, including: when the delayed execution condition is a delayed execution time, the target downstream data task is subjected to target monitoring rule execution processing at the delayed execution time; when the delayed execution condition is successful execution of an emergency job, the emergency job is executed at the original execution time, and the target downstream data task is subjected to target monitoring rule execution processing after the successful execution of the emergency job.

[0011] In one embodiment of the present disclosure, the delayed execution condition also includes: an identifier of the target data task; the method also includes: when the delayed execution condition is a delayed execution time, monitoring an output data alarm release event of the target data task; when the output data alarm release event is monitored, obtaining the alarm duration of the target data task; based on the alarm duration, updating the delayed execution time in the delayed execution condition, and deleting the identifier of the target data task in the delayed execution condition.

[0012] In one embodiment of the present disclosure, the method further includes: when the original execution time of the target downstream data task is reached, generating delayed execution notification information according to the delayed execution condition; and providing the delayed execution notification information to the task responsible object of the target downstream data task.

[0013] According to the second aspect of the embodiment of the present disclosure, a data quality monitoring device is also provided, which includes: an acquisition module, used to acquire a target downstream data task of a target data task; the target data task triggers an alarm for output data; the target downstream data task depends on the output data and is provided with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; and a delayed execution module, used to perform target monitoring rule delayed execution processing on the target downstream data task according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task.

[0014] According to a third aspect of an embodiment of the present disclosure, there is also provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to: implement the steps of the data quality monitoring method as described above.

[0015] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is also provided. When instructions in the storage medium are executed by a processor, the processor is enabled to execute the data quality monitoring method as described above.

[0016] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0017] By obtaining the target downstream data task of the target data task; the target data task is the data task that triggers an alarm for the output data; the target downstream data task depends on the output data and is set with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task, the target downstream data task is subjected to delayed execution processing of the target monitoring rule, wherein the delayed monitoring processing of the target downstream data task allows the target data task to have time to repair and release the alarm, thereby preventing the task responsible object of the target downstream data task from receiving the alarm caused by the target data task, reducing the number of invalid alarms, improving the alarm efficiency, and improving the data monitoring efficiency.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0020] Figure 1 A flowchart of a data quality monitoring method according to an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of a data quality monitoring method according to another embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of the structure of a data quality monitoring device according to an embodiment of the present disclosure;

[0023] Figure 4 The figure is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0026] The current data quality monitoring method mainly triggers data quality monitoring after data task A is completed, and performs quality checks on the output data of data task A according to the monitoring rules; if there are quality problems, an alarm notification will be issued, and it will be determined whether the downstream data task B of data task A needs to be blocked; if necessary, blocking will be executed, and the downstream data task B will stop executing, causing an alarm notification to be issued for the monitoring rules of the downstream data task B.

[0027] In the above scheme, an alarm notification of an upstream data task may cause alarm notifications of multiple downstream data tasks, resulting in an excessive number of alarms, poor alarm efficiency, and poor data quality monitoring efficiency.

[0028] Figure 1 The flowchart of the data quality monitoring method of one embodiment of the present disclosure is as follows. It should be noted that the data quality monitoring method of the present embodiment can be applied to a data quality monitoring device, which can be configured in an electronic device so that the electronic device can perform a data quality monitoring function.

[0029] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0030] The data quality monitoring device may be one of the servers in the data quality monitoring system. The data quality monitoring system may include multiple servers for implementing multiple functions. Functions such as monitoring functions and alarm functions may be set according to actual needs. In the following embodiments, the execution subject is taken as an example of a data quality monitoring system.

[0031] like Figure 1 As shown, the method comprises the following steps:

[0032] Step 101, obtain the target downstream data task of the target data task; the target data task is a data task that triggers an alarm for output data; the target downstream data task depends on the output data and is set with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task.

[0033] In the disclosed embodiment, the process of the data quality monitoring system executing step 101 may be, for example, obtaining the target data task; querying the dependency table according to the target data task to obtain the output data of the target data task, and the candidate downstream data task that has no intermediate data tasks between the target data task and the candidate downstream data task; determining the time interval between the monitoring rule execution time of the candidate downstream data task and the monitoring rule execution time of the target data task; when the time interval is less than or equal to the time length threshold, and the candidate downstream data task is set with delay configuration information, the candidate downstream data task is used as the target downstream data task. Among them, the intermediate data task is a data task that needs to be executed after the target data task is executed and before the candidate downstream data task is executed.

[0034] In the disclosed embodiment, the number of data tasks may be multiple, and some of the multiple data tasks may have dependencies, while some of the data tasks may not have dependencies. For each data task, data quality monitoring will be triggered after the data task is completed, that is, according to the monitoring rules of the data task, the output data of the data task will be checked for quality to determine whether the output data has quality problems; if the output data has quality problems, it is determined that the output data of the data task triggers an alarm.

[0035] In the disclosed embodiment, the dependency table may include data dependency relationships between multiple data tasks. For example, if data task B needs to rely on the output data of data task A during execution, it means that data task B relies on the output data of data task A, and there is a data dependency relationship between data task B and data task A. The dependency table may be updated in real time according to the output data, execution process data, etc. of each data task.

[0036] In the disclosed embodiment, after the output data of the data task triggers an alarm, the data quality monitoring system will provide an alarm notification to the task responsible object corresponding to the data task; after receiving the alarm notification, the task responsible object will repair the data task until the output data clears the alarm. Among them, the repair time is limited, that is, in the process from triggering the alarm to clearing the alarm, the monitoring rule execution time of some candidate downstream data tasks may not have arrived, that is, it will not affect the execution of some candidate downstream data tasks. Therefore, the data quality monitoring system can select and determine the target downstream data tasks from the candidate downstream data tasks whose corresponding monitoring rule execution time is within a certain time period, reduce the number of candidate downstream data tasks that need to be processed, further improve the data quality monitoring speed, and improve the data quality monitoring efficiency.

[0037] The length of a period of time, that is, the time length threshold mentioned above, can be a period of time starting from the execution time of the monitoring rule of the target data task.

[0038] Among them, if a data task does not need to be delayed in execution, there is no need to set the delay configuration information. Among them, for a data task, the process of setting the delay configuration information can be, for example, that the data quality monitoring system provides three options to the task responsible object corresponding to the data task, namely, delay alarm, alternative solution and no optimization. In the case where the task responsible object selects the delay alarm option, the task responsible object is prompted to configure the delay tolerance time. The value range of the delay tolerance time can be, for example, a value between (0, 480), in minutes. In the case where the task responsible object selects an alternative solution, the task responsible object is prompted to provide emergency operations. In the case where the task responsible object selects no optimization, stop setting the delay configuration information.

[0039] In the embodiment of the present disclosure, if the data quality monitoring system fails to obtain the target data task, or fails to obtain the target downstream data task for a certain target data task, the execution of step 101 and subsequent steps will be stopped.

[0040] In the embodiment of the present disclosure, it should be noted that the multiple data tasks may be executed periodically, for example, once a day, once a week, etc. When the multiple data tasks are executed periodically, step 101 and subsequent steps may be executed in each cycle.

[0041] Step 102 , performing target monitoring rule delayed execution processing on the target downstream data task according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task.

[0042] In the disclosed embodiment, the delay configuration information of the target downstream data task may include a delay tolerance duration, or an emergency operation; the delay tolerance duration indicates the tolerable delay execution duration based on the original execution time of the target monitoring rule of the target downstream data task; the emergency operation indicates that an emergency operation needs to be performed before executing the target monitoring rule. Among them, the emergency operation, for example, can be a job processing based on the historical output data of the target data task, etc., which can be set according to actual needs.

[0043] In the embodiment of the present disclosure, the process of the data quality monitoring system executing step 102 may, for example, be to determine the delayed execution condition of the target downstream data task based on the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task; and perform delayed execution processing of the target monitoring rule on the target downstream data task based on the delayed execution condition.

[0044] Among them, the process of the data quality monitoring system determining the delayed execution condition of the target downstream data task can be, for example, that when the delay configuration information includes the delay tolerance duration, the original execution time and the delay tolerance duration are added together to obtain the delayed execution time of the target downstream data task; the delayed execution time is used as the delayed execution condition; when the delay configuration information includes an emergency job, the successful execution of the emergency job is used as the delayed execution condition; the execution time of the emergency job is the original execution time.

[0045] In the disclosed embodiment, the delayed execution condition is to delay the execution time of the target monitoring rule, or to execute the target monitoring rule on the target downstream data task after the emergency operation is completed, so as to postpone the execution time of the target monitoring rule and avoid that after the output data of the target data task triggers an alarm, the subsequent downstream data tasks of the target data task are alarmed in sequence, thereby effectively reducing the number of invalid alarms, avoiding the task responsible objects of the downstream data tasks from responding to alarms frequently, and reducing data monitoring costs.

[0046] In the data quality monitoring method of the disclosed embodiment, a target downstream data task of a target data task is obtained; the target data task is a data task that triggers an alarm for output data; the target downstream data task depends on the output data and is provided with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task, the target downstream data task is subjected to delayed execution processing of the target monitoring rule, wherein the delayed monitoring processing of the target downstream data task allows the target data task to have time to repair and release the alarm, thereby preventing the task responsible object of the target downstream data task from receiving the alarm caused by the target data task, reducing the number of invalid alarms, improving the alarm efficiency, and improving the data monitoring efficiency.

[0047] Figure 2 This is a flow chart of a data quality monitoring method according to another embodiment of the present disclosure. It should be noted that the data quality monitoring method according to this embodiment can be applied to a data quality monitoring device, which can be configured in an electronic device so that the electronic device can perform a data quality monitoring function.

[0048] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0049] The data quality monitoring device may be one of the servers in the data quality monitoring system. The data quality monitoring system may include multiple servers for implementing multiple functions. Functions such as monitoring functions and alarm functions may be set according to actual needs. In the following embodiments, the execution subject is taken as an example of a data quality monitoring system.

[0050] like Figure 2 As shown, the method comprises the following steps:

[0051] Step 201, obtain the target downstream data task of the target data task; the target data task is a data task that triggers an alarm for output data; the target downstream data task depends on the output data and is set with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task.

[0052] Step 202, determining the delayed execution condition of the target downstream data task according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task; the delayed execution condition includes the identifier of the target data task.

[0053] In the disclosed embodiment, the delayed execution condition includes the identification of the target data task, so that the data quality monitoring system can understand the reason for the delayed execution of the target monitoring rule of the target downstream data task, that is, to understand that the delayed execution of the target monitoring rule is caused by the output data of the target data task triggering an alarm.

[0054] Step 203: When the delayed execution condition is the delayed execution time, monitor the output data of the target data task to clear the alarm event.

[0055] In the disclosed embodiment, after the target data task triggers an alarm, the data quality monitoring system will provide an alarm notification to the task responsible object corresponding to the target data task; after receiving the alarm notification, the task responsible object will repair the target data task; after the modification is completed, when the monitoring rules are executed to determine that there are no quality problems with the output data, the data quality monitoring system can determine that the output data of the target data task has been monitored and the alarm event has been lifted.

[0056] In the embodiment of the present disclosure, when the delayed execution time indicates that the emergency operation is successfully executed, since the execution time of the emergency operation cannot be adjusted, the data quality monitoring system may not execute steps 203 to 205.

[0057] Step 204, when the output data alarm release event is monitored, the alarm duration of the target data task is obtained.

[0058] In the embodiment of the present disclosure, the alarm duration is the duration of the time interval from when the output data of the target data task triggers an alarm to when the output data is monitored to release the alarm.

[0059] Step 205: update the delayed execution time in the delayed execution condition according to the alarm duration, and delete the identifier of the target data task in the delayed execution condition.

[0060] In the disclosed embodiment, the process of updating the delayed execution time in the delayed execution condition by the data quality monitoring system may be, for example, obtaining the original execution time of the target monitoring rule of the target downstream data task; determining the sum of the original execution time and the alarm time; and using the sum as the latest delayed execution time. Among them, deleting the identifier of the target data task in the delayed execution condition facilitates the data quality monitoring system to understand that the output data of the target data task has been cleared.

[0061] Among them, steps 203 to 205 are optional. That is, after step 202, steps 203 to 205 can be executed, and then step 206 can be executed; or, after step 202, step 206 can be directly executed. Among them, whether steps 203 to 205 are executed is determined by whether the output data of the target data task is monitored to release the alarm event.

[0062] If there are multiple target downstream data tasks, that is, multiple target data tasks use the same downstream data task as the target downstream data task, then steps 203 to 205 may be executed multiple times. For example, target data task A, target data task B, and target data task C all use downstream data task D as the target downstream data task, that is, downstream data task D needs to rely on the output data of target data task A, the output data of target data task B, and the output data of target data task C at the same time.

[0063] Step 206 , when the delayed execution condition is the delayed execution time, the target monitoring rule execution processing is performed on the target downstream data task at the delayed execution time.

[0064] Step 207, when the delayed execution condition is that the emergency job is successfully executed, the emergency job is executed at the original execution time, and after the emergency job is successfully executed, the target downstream data task is processed according to the target monitoring rule execution.

[0065] In the disclosed embodiment, the data quality monitoring system may also perform the following process: when the original execution time of the target downstream data task is reached, delayed execution notification information is generated according to the delayed execution condition; the delayed execution notification information is provided to the task responsible object of the target downstream data task, so as to facilitate the task responsible object to understand the execution status of the target monitoring rules of the target downstream data task.

[0066] Wherein, when the delayed execution condition is the delayed execution time, the delayed execution notification information may be, for example, that the monitoring rule of the downstream data task XX is delayed to be executed at XX time. Where the delayed execution condition is the successful execution of the emergency operation, the delayed execution notification information may be, for example, that the monitoring rule of the downstream data task XX is delayed to be executed after the successful execution of the emergency operation.

[0067] In the data quality monitoring method of the disclosed embodiment, a target downstream data task of a target data task is obtained; the target data task is a data task that triggers an alarm for output data; the target downstream data task depends on the output data and is provided with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task, a delayed execution condition of the target downstream data task is determined; the delayed execution condition includes an identifier of the target data task; in the case where the delayed execution condition is the delayed execution time, the output data of the target data task is monitored for an alarm release event; in the case where the output data alarm release event is monitored, the alarm duration of the target data task is obtained; according to the alarm duration, update the delayed execution time in the delayed execution condition, and delete the identifier of the target data task in the delayed execution condition; when the delayed execution condition is the delayed execution time, execute the target monitoring rule for the target downstream data task at the delayed execution time; when the delayed execution condition is the successful execution of the emergency job, execute the emergency job at the original execution time, and execute the target monitoring rule for the target downstream data task after the successful execution of the emergency job; wherein, the delayed monitoring of the target downstream data task allows the target data task to have time to repair and clear the alarm, thereby preventing the task responsible object of the target downstream data task from receiving the alarm caused by the target data task, reducing the number of invalid alarms, improving the alarm efficiency, and improving the data monitoring efficiency.

[0068] The following example is used to illustrate. The process of the data quality monitoring function system executing the data quality monitoring method may include the following steps: (1) When an alarm occurs in any table (denoted as table x, for example, the output data of the target data task), all direct downstream tables of the table (downstream data tasks of the target data task) are retrieved from the blood relationship data (dependency table). (2) If the table has a downstream table, an alarm notification is sent to the data person in charge (the person responsible for the task). The data person in charge can independently choose "whether to delay the triggering of the downstream table monitoring rules"; if not selected or selected "no", the execution is stopped; if selected "yes", all downstream tables are screened out. The monitoring rules that will be executed in the next eight hours (that is, the monitoring rules of the candidate downstream data tasks) are screened out. (3) Traverse the filtered downstream table monitoring rules (monitoring rules of candidate downstream data tasks), and perform the following judgments for each monitoring rule: If no alternative plan (emergency operation) is set and no delay tolerance (delay tolerance duration) is set, stop execution; if delay tolerance is set, mark the start execution time of the monitoring rule in the cache as t+l, where t refers to the original execution time of the monitoring rule and l refers to the delay tolerance of the monitoring rule. If the mark already exists in the cache, modify the value of the mark to t+l. At the same time, record the alarm upstream table x (for example, the identifier of the target data task) in the cache in an appended manner. If an alternative plan is set, mark the configured emergency operation in the cache before executing the monitoring rule. At the same time, record the alarm upstream table x in the cache in an appended manner. (4) When the data owner of table x fixes the problem of the table, he can choose "end downstream table delay" (detect the output data of the target data task to release the alarm event), and record the repair duration of table x as m. Traverse all tags in the cache and filter out all downstream table monitoring rules that are affected by table x and configured with delay tolerance. Traverse the filtered monitoring rules and perform the following judgment: If the alarm upstream table of the monitoring rule contains table x, then modify the start execution time of the monitoring rule to t+m. Delete the records of table x in the alarm upstream table of the current rule from the cache. (5) When any table (denoted as table y) reaches the original execution time of the monitoring rule (denoted as z), take out the relevant records from the cache; if there is a delayed start time record of z in the cache, send a delayed execution notification (delayed execution notification information) to the table data person in charge of the arbitrary table. If there is an alternative plan record of z in the cache, execute the emergency operation and send the emergency operation execution result (delayed execution notification information) to the table data person in charge of the arbitrary table. After execution, the relevant records of z in the cache need to be deleted.

[0069] Figure 3 The present invention is a schematic diagram of the structure of a data quality monitoring device according to an embodiment of the present invention.

[0070] like Figure 3As shown, the data quality monitoring device may include: an acquisition module 301 and a delay execution module 302 .

[0071] Among them, the acquisition module 301 is used to acquire the target downstream data task of the target data task; the target data task is a data task that triggers an alarm for output data; the target downstream data task depends on the output data and is provided with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; the delayed execution module 302 is used to perform target monitoring rule delayed execution processing on the target downstream data task according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task.

[0072] In one embodiment of the present disclosure, the acquisition module 301 is specifically used to acquire the target data task; query the dependency table according to the target data task to acquire the candidate downstream data tasks that depend on the output data of the target data task; determine the length of the time interval between the monitoring rule execution time of the candidate downstream data task and the monitoring rule execution time of the target data task; and when the length of the time interval is less than or equal to a time length threshold and the candidate downstream data task is set with delay configuration information, use the candidate downstream data task as the target downstream data task.

[0073] In one embodiment of the present disclosure, the delay configuration information of the target downstream data task includes a delay tolerance duration, or an emergency operation; the delay tolerance duration indicates the tolerable delayed execution duration based on the original execution time of the target monitoring rule of the target downstream data task; the emergency operation indicates that the emergency operation needs to be executed before executing the target monitoring rule.

[0074] In one embodiment of the present disclosure, the delayed execution module 302 is specifically used to determine the delayed execution condition of the target downstream data task based on the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task; and perform delayed execution processing of the target monitoring rule on the target downstream data task based on the delayed execution condition.

[0075] In one embodiment of the present disclosure, the delayed execution module 302 is specifically used to, when the delay configuration information includes the delay tolerance duration, add the original execution time and the delay tolerance duration to obtain the delayed execution time of the target downstream data task; use the delayed execution time as the delayed execution condition; when the delay configuration information includes an emergency job, use the successful execution of the emergency job as the delayed execution condition; the execution time of the emergency job is the original execution time.

[0076] In one embodiment of the present disclosure, the delayed execution module 302 is specifically used to, when the delayed execution condition is the delayed execution time, perform target monitoring rule execution processing on the target downstream data task at the delayed execution time; when the delayed execution condition is the successful execution of the emergency job, execute the emergency job at the original execution time, and perform target monitoring rule execution processing on the target downstream data task after the successful execution of the emergency job.

[0077] In one embodiment of the present disclosure, the delayed execution condition also includes: an identifier of the target data task; the device also includes: a monitoring module and an update module; the monitoring module is used to monitor the output data alarm release event of the target data task when the delayed execution condition is the delayed execution time; the acquisition module 301 is also used to obtain the alarm duration of the target data task when the output data alarm release event is monitored; the update module is used to update the delayed execution time in the delayed execution condition according to the alarm duration, and delete the identifier of the target data task in the delayed execution condition.

[0078] In one embodiment of the present disclosure, the device also includes: a generation module and a providing module; the generation module is used to generate delayed execution notification information according to the delayed execution condition when the original execution time of the target downstream data task is reached; the providing module is used to provide the delayed execution notification information to the task responsible object of the target downstream data task.

[0079] In the data quality monitoring device of the disclosed embodiment, a target downstream data task of a target data task is obtained; the target data task is a data task that triggers an alarm for output data; the target downstream data task depends on the output data and is provided with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task, the target downstream data task is subjected to delayed execution processing of the target monitoring rule, wherein the delayed monitoring processing of the target downstream data task allows the target data task to have time to repair and release the alarm, thereby preventing the task responsible object of the target downstream data task from receiving the alarm caused by the target data task, reducing the number of invalid alarms, improving the alarm efficiency, and improving the data monitoring efficiency.

[0080] According to a third aspect of an embodiment of the present disclosure, there is also provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to: implement the data quality monitoring method as described above.

[0081] In order to implement the above embodiments, the present disclosure also proposes a storage medium.

[0082] When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data quality monitoring method as described above.

[0083] In order to implement the above embodiments, the present disclosure also provides a computer program product.

[0084] When the computer program product is executed by a processor of an electronic device, the electronic device is enabled to execute the above method.

[0085] Figure 4 The figure is a structural block diagram of an electronic device according to an exemplary embodiment. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0086] like Figure 4As shown, the electronic device 1000 includes a processor 111, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 112 or a program loaded from a memory 116 to a random access memory (RAM) 113. In the RAM 113, various programs and data required for the operation of the electronic device 1000 are also stored. The processor 111, the ROM 112, and the RAM 113 are connected to each other via a bus 114. An input / output (I / O) interface 115 is also connected to the bus 114.

[0087] The following components are connected to the I / O interface 115: a memory 116 including a hard disk, etc.; and a communication part 117 including a network interface card such as a local area network (LAN) card, a modem, etc., and the communication part 117 performs communication processing via a network such as the Internet; a drive 118 is also connected to the I / O interface 115 as needed.

[0088] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 117. When the computer program is executed by the processor 111, the above-mentioned functions defined in the method of the present disclosure are executed.

[0089] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by the processor 111 of the electronic device 1000 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0090] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0091] In addition, the word "exemplary" is used herein to indicate serving as an example, instance, or diagram. Any aspect or design described as "exemplary" in this article is not necessarily understood to be advantageous compared to other aspects or designs. On the contrary, the use of the word exemplary is intended to present concepts in a specific way. As used herein, the term "or" is intended to represent an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to represent any one of the natural inclusive arrangements. That is, if X applies A; X applies B; or X applies both A and B, "X applies A or B" is satisfied under any of the aforementioned examples. In addition, unless otherwise specified or clearly pointed to a singular form from the context, the articles "one" and "an" as used in this application and the appended claims are generally understood to mean "one or more".

[0092] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art after reading and understanding the specification and drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components (e.g., elements, resources, etc.) described above, unless otherwise noted, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific functions of the described components, even if the structure is not equivalent to the disclosed structure. In addition, although the specific features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and conducive to any given or specific application. In addition, with respect to "including", "having", "having", "having", or variations thereof used in the specific embodiments or claims, such terms are intended to be inclusive in a manner similar to the term "comprising".

[0093] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0094] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data quality monitoring method, characterized in that: The method comprises: Obtain a target downstream data task of a target data task; the target data task is a data task that triggers an alarm by output data; the target downstream data task depends on the output data and is provided with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; According to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task, the target monitoring rule delayed execution processing is performed on the target downstream data task.

2. The method according to claim 1, characterized in that The target downstream data task of the target data acquisition task includes: Acquiring the target data task; Querying a dependency table according to the target data task to obtain candidate downstream data tasks that are dependent on the output data of the target data task and have no intermediate data tasks between the target data task and the target data task; Determine the length of the time interval between the monitoring rule execution time of the candidate downstream data task and the monitoring rule execution time of the target data task; When the length of the time interval is less than or equal to the time length threshold and the candidate downstream data task is set with delay configuration information, the candidate downstream data task is used as the target downstream data task.

3. The method according to claim 1, characterized in that The delay configuration information of the target downstream data task includes delay tolerance duration, or emergency operation; The delay tolerance duration indicates the tolerable delay execution duration based on the original execution time of the target monitoring rule of the target downstream data task; The emergency operation indicates that the emergency operation needs to be performed before executing the target monitoring rule.

4. The method according to claim 1 or 3, characterized in that: The step of performing target monitoring rule delayed execution processing on the target downstream data task according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task includes: Determining a delayed execution condition of the target downstream data task according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task; According to the delayed execution condition, target monitoring rule delayed execution processing is performed on the target downstream data task.

5. The method according to claim 4, characterized in that The determining, according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task, the delayed execution condition of the target downstream data task comprises: In the case where the delay configuration information includes a delay tolerance duration, the original execution time and the delay tolerance duration are added together to obtain a delayed execution time of the target downstream data task; and the delayed execution time is used as the delayed execution condition; In the case where the delay configuration information includes an emergency operation, the successful execution of the emergency operation is used as the delay execution condition; and the execution time of the emergency operation is the original execution time.

6. The method according to claim 4, characterized in that The step of performing target monitoring rule delayed execution processing on the target downstream data task according to the delayed execution condition includes: In the case where the delayed execution condition is a delayed execution time, performing target monitoring rule execution processing on the target downstream data task at the delayed execution time; When the delayed execution condition is that the emergency job is successfully executed, the emergency job is executed at the original execution time, and after the emergency job is successfully executed, the target downstream data task is subjected to target monitoring rule execution processing.

7. The method according to claim 5, characterized in that The delayed execution condition further includes: an identifier of the target data task; and the method further includes: In the case where the delayed execution condition is a delayed execution time, monitoring an output data alarm release event of the target data task; When the output data alarm release event is detected, the alarm duration of the target data task is obtained; According to the alarm duration, the delayed execution time in the delayed execution condition is updated, and the identifier of the target data task in the delayed execution condition is deleted.

8. The method according to claim 6, characterized in that The method further comprises: When the original execution time of the target downstream data task is reached, generating delayed execution notification information according to the delayed execution condition; The delayed execution notification information is provided to the task responsible object of the target downstream data task.

9. A data quality monitoring device, characterized in that: The device comprises: An acquisition module is used to acquire a target downstream data task of a target data task; the target data task is a data task that triggers an alarm by output data; the target downstream data task depends on the output data and is provided with delay configuration information; there is no intermediate data task between the target downstream data task and the target data task; The delayed execution module is used to perform target monitoring rule delayed execution processing on the target downstream data task according to the original execution time of the target monitoring rule of the target downstream data task and the delay configuration information of the target downstream data task.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to: Implement the steps of the data quality monitoring method as described in any one of claims 1 to 8. 11 . A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor, enables the processor to perform the data quality monitoring method according to any one of claims 1 to 8.

Images