Data warehouse scheduling task blocking method, device and equipment and storage medium

Abstract

The invention relates to the technical field of data warehouses, in particular to a data warehouse scheduling task blocking method, device and equipment and a storage medium. The method comprises the steps that firstly, monitoring detection operation is executed based on a constructed target task import state table to obtain a task running state data set; performing first task abnormity judgment on the current task based on the task running state data set, when the current task is judged to be in an abnormal state, performing state updating processing on a target task import state table, performing downstream task blocking processing on the current task to obtain an updated task import state table and a failed task, obtaining a preset time frequency, and performing task processing on the failed task according to the updated task import state table and the updated task import state table; and triggering the failed task to run again based on the time frequency, performing second task abnormality judgment on the failed task based on the updated task import state table, and performing downstream task recovery processing on the failed task to complete task execution when the failed task returns to normal, so as to capture task report errors in real time and block downstream operation. And the manual operation and maintenance cost is saved.

Description

Technical Field

[0001] This invention relates to the field of data warehouse technology, and in particular to a data warehouse scheduling task blocking method, apparatus, device, and storage medium. Background Technology

[0002] In data warehouse systems that rely on Flink tasks for real-time data import, numerous operational management challenges often arise when setting up batch scheduling tasks. When a Flink task fails, failure to promptly halt downstream batch scheduling processes can lead to tasks running without data. Even after Flink tasks are repaired, the entire batch scheduling job must be rerun to reload the newly imported data. This rerun process relies on manual intervention, and any oversight or lack of oversight can result in data errors or delays. When the real-time data warehouse database undergoes upgrades, Flink tasks may briefly fail and automatically attempt restarts. While Flink tasks can automatically recover after the upgrade, the lack of effective management during this process can still potentially impact the overall data flow. Furthermore, when a Flink task fails to write data to the data warehouse due to its own malfunction, manual pausing of the data warehouse scheduling task is necessary to prevent meaningless empty runs. Manual operation carries a high risk of oversight; whether pausing a task is missed or restarting it later, both can compromise data accuracy or significantly delay the data cleaning process. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the present invention aims to provide a data warehouse scheduling task blocking method, apparatus, device and storage medium, which aims to capture Flink task errors in real time and block downstream operations, avoid empty runs and reduce the cost of full reruns, and save manual operation and maintenance costs.

[0004] The first aspect of this invention provides a data warehouse scheduling task blocking method, comprising: constructing a target task import status table; performing monitoring and detection operations based on the target task import status table to obtain a task running status dataset; performing a first task anomaly determination on the current task based on the task running status dataset; when the current task is determined to be in an abnormal state, performing a status update process on the target task import status table and performing downstream task blocking processing on the current task to obtain an updated task import status table and failed tasks; obtaining a preset time frequency, triggering the failed task to re-run based on the time frequency, and performing a second task anomaly determination on the failed task based on the updated task import status table; when the failed task is determined to have recovered to normal, performing downstream task recovery processing on the failed task to complete the task execution.

[0005] Optionally, in a first implementation of the first aspect of the present invention, constructing the target task import status table includes: obtaining a preset initial status value and a preset table field structure, wherein the table field structure includes historical error counts, task recovery status, and table update time; constructing an initial task import status table based on the table field structure; and performing data insertion processing on the initial task import status table based on the initial status value to obtain the target task import status table.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the task running status dataset includes the current number of errors, the number of tasks running, and the historical number of errors; the step of performing monitoring and detection operations based on the target task import status table to obtain the task running status dataset includes: obtaining a preset error record table and a preset task instance table; performing a current task error count statistics operation based on the error record table to obtain the current number of errors; performing a current task running count statistics operation based on the task instance table to obtain the number of tasks running; and performing a historical error count query and extraction operation based on the target task import status table to obtain the historical number of errors.

[0007] Optionally, in a third implementation of the first aspect of the present invention, a first task anomaly determination is performed on the current task based on the task running status dataset, including: obtaining a preset number of monitoring tasks; comparing the current number of error reports with the historical number of error reports, and simultaneously comparing the number of monitoring tasks with the number of tasks running; determining that the current task is in an abnormal state when the current number of error reports is greater than the historical number of error reports, or the number of tasks running is less than the number of monitoring tasks; and determining that the current task is in a normal state when the current number of error reports is equal to the historical number of error reports, and the number of tasks running is equal to the number of monitoring tasks.

[0008] Optionally, in the fourth implementation of the first aspect of the present invention, when the current task is determined to be in an abnormal state, the target task import status table is updated, and the current task is blocked from downstream tasks to obtain an updated task import status table and a failed task. This includes: when the current task is determined to be in an abnormal state, obtaining the execution result of a first script and a preset abnormal state update rule; assigning update fields to the historical error count, the task recovery status, and the table update time based on the abnormal state update rule to obtain the updated task import status table; and using a task scheduling tool to block downstream tasks of the current task based on the execution result of the first script to obtain the failed task.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of triggering the failed task to run again based on the time frequency and performing a second task anomaly determination on the failed task based on the update task import status table includes: performing a timed trigger operation on the failed task based on the time frequency to obtain a rerun task; obtaining the update error count based on the update task import status table; comparing the update error count with the current error count; and determining that the rerun task has recovered to normal when the update error count is equal to the current error count.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the step of performing downstream task recovery processing on the failed task to complete the task execution when the failed task is determined to have recovered to normal includes: when the failed task is determined to have recovered to normal, obtaining the execution result of the second script and a preset normal status update rule; performing status update processing on the import status table of the update task based on the normal status update rule to obtain a new task import status table; using the task scheduling tool, performing downstream task recovery processing on the failed task based on the execution result of the second script, and performing fast rerun processing on the downstream tasks to complete the task execution.

[0011] A second aspect of the present invention provides a data warehouse scheduling task blocking device, comprising: a status table construction module for constructing a target task import status table; a monitoring and detection module for performing monitoring and detection operations based on the target task import status table to obtain a task running status dataset; a task anomaly determination module for performing a first task anomaly determination on the current task based on the task running status dataset; a task blocking module for performing status update processing on the target task import status table and downstream task blocking processing on the current task when the current task is determined to be in an abnormal state, thereby obtaining an updated task import status table and a failed task; a task triggering module for obtaining a preset time frequency, triggering the failed task to run again based on the time frequency, and performing a second task anomaly determination on the failed task based on the updated task import status table; and a task recovery module for performing downstream task recovery processing on the failed task to complete the task execution when the failed task is determined to be recovered to normal.

[0012] A third aspect of the present invention provides a data warehouse scheduling task blocking device, the data warehouse scheduling task blocking device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the data warehouse scheduling task blocking device to execute the various steps of the data warehouse scheduling task blocking method described in any of the preceding claims.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the data warehouse scheduling task blocking method described in any of the preceding claims.

[0014] In the technical solution of this invention, a target task import status table is first constructed. Based on the target task import status table, a monitoring and detection operation is performed to obtain a task running status dataset. Then, based on the task running status dataset, a first task anomaly determination is performed on the current task. When the current task is determined to be in an abnormal state, the target task import status table is updated, and downstream task blocking is performed on the current task, resulting in an updated task import status table and a failed task. Next, a preset time frequency is obtained, and the failed task is triggered to run again based on the time frequency. A second task anomaly determination is performed on the failed task based on the updated task import status table. Finally, when the failed task returns to normal, downstream task recovery is performed on the failed task to complete the task execution. This aims to capture Flink task errors in real time and block downstream jobs, avoiding the costs of empty runs and full reruns, and saving manual operation and maintenance costs. Attached Figure Description

[0015] Figure 1 This is a first flowchart of a data warehouse scheduling task blocking method provided in an embodiment of the present invention; Figure 2 This is a second flowchart of the data warehouse scheduling task blocking method provided in an embodiment of the present invention; Figure 3 This is a third flowchart of the data warehouse scheduling task blocking method provided in an embodiment of the present invention; Figure 4 This is a fourth flowchart of the data warehouse scheduling task blocking method provided in an embodiment of the present invention; Figure 5 This is a fifth flowchart of the data warehouse scheduling task blocking method provided in an embodiment of the present invention; Figure 6 The sixth flowchart of the data warehouse scheduling task blocking method provided in the embodiments of the present invention; Figure 7 The seventh flowchart of the data warehouse scheduling task blocking method provided in the embodiments of the present invention; Figure 8 This is a schematic diagram of the data warehouse scheduling task blocking device provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a data warehouse scheduling task blocking device provided in an embodiment of the present invention. Detailed Implementation

[0016] The present invention provides a method, apparatus, device and storage medium for blocking data warehouse scheduling tasks. In the present invention, the terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order different from that illustrated or described here. In addition, the term "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0017] For ease of understanding, the specific process of the embodiments of the present invention will be described below. Please refer to Figure 1 , an embodiment of the method for blocking data warehouse scheduling tasks in the embodiments of the present invention includes: 101. Construct a target task import status table; In this embodiment, when constructing the target task import status table, it is necessary to complete the normalization design of the table structure within the real-time data warehouse system. The core of the target task import status table is used to save the statistical data of the entire life cycle of the Flink task, providing a stable data carrier and status benchmark for subsequent monitoring and detection operations. Its design needs to conform to the running characteristics and monitoring requirements of the Flink task to ensure accurate storage of key statistical indicators during the task running process, realizing the normalized storage and centralized management of Flink task statistical data, avoiding data inconsistency problems caused by scattered storage of monitoring data, and providing data support for the implementation of monitoring logic.

[0018] 102. Perform monitoring and detection operations based on the target task import status table to obtain a task running status data set; In this embodiment, when performing monitoring and detection operations based on the constructed target task import status table, it is necessary to use the target task import status table as the core data hub, extract the core indicators during the task running process and perform integration processing, and then generate a complete task running status data set. Through the target task import status table, the historical running statistical information of the task can be traced, and at the same time, combined with the real-time collected task running data, the linkage verification of the historical status and the real-time status is realized, ensuring that the obtained task running status data set has integrity, accuracy and relevance, and providing a reliable data basis for subsequent task anomaly determination, status update and other operations.

[0019] 103. Perform a first task anomaly determination on the current task based on the task running status data set; In this embodiment, the first task anomaly determination is performed on the current task based on the task running status dataset. The core of this process is to rely on key quantitative indicators within the dataset, such as the current number of errors, historical number of errors, and the number of tasks running, to conduct multi-dimensional and comprehensive verification and identification of the task running status. This process first extracts and standardizes the effective indicators within the task running status dataset, eliminating invalid and redundant data to ensure the completeness and accuracy of the indicators used for determination. Then, based on the standardized core indicators, threshold comparisons and running status deductions are performed to accurately determine whether the current task has new errors or missing running tasks. Finally, the quantified indicator data is transformed into clear conclusions about whether the task is in an abnormal or normal state, providing a core decision-making basis for subsequent task management operations.

[0020] 104. When the current task is determined to be in an abnormal state, the target task import status table is updated, and the downstream task is blocked for the current task, so as to obtain the updated task import status table and the failed task. In this embodiment, based on the anomaly determination result, the core fields related to the abnormal state of the task in the table are updated synchronously, including the task recovery status identifier, the latest error count statistics, and the status update timestamp, ensuring that the target task import status table can reflect the abnormal running status of the current task in real time and accurately. The downstream task blocking processing for the current task is achieved by triggering the task dependency control mechanism built into the scheduling system. This process blocks all associated batch scheduling tasks downstream of the current task, forcibly terminating their pending or ongoing processes. This fundamentally avoids the data-free running behavior of downstream tasks due to missing upstream data, effectively preventing the waste of system resources caused by data-free running and eliminating a series of derivative problems such as data errors and data delays caused by data-free running. The entire process, through the coordinated execution of status table updates and downstream task blocking, ultimately generates a task import status table with completed abnormal status updates and marks the current task as a failed task. By updating the target task import status table in real time, the consistency and traceability of task abnormal status data are ensured, providing solid data support for subsequent fully automated management. The overall operation automates and enables real-time management of abnormal tasks, significantly improving the efficiency and accuracy of task management, reducing the need for manual intervention, and enhancing the stability and controllability of the collaborative operation of the real-time data warehouse and batch scheduling system.

[0021] 105. Obtain a preset time frequency, trigger the failed task to run again based on the time frequency, and perform a second task anomaly determination on the failed task based on the status table imported by the updated task. In this embodiment, obtaining the preset time frequency is based on a fixed period preset according to the business scenario and fault recovery time requirements. This period is usually configured in the scheduling system or monitoring script to form a standardized automatic retry triggering mechanism. Triggering the rerun of failed tasks based on this time frequency essentially starts an automated cyclic detection process, repeatedly executing previously marked failed tasks according to the preset period, continuously probing whether the running status of Flink tasks has recovered, and achieving routine monitoring of fault recovery without manual intervention. After triggering a rerun, a second task anomaly determination is performed on the failed task based on the updated task import status table. This requires comparing and verifying the updated anomaly status data in the updated task import status table with the real-time operating metrics collected after the rerun to determine the recovery status. Using the historical error count and anomaly identifier stored in the table as a benchmark, combined with the current error count and task execution count obtained after the rerun, it is determined whether the task recovery conditions are met, forming a clear second task anomaly determination result. This achieves accurate verification of the task recovery status, reducing manual intervention costs and avoiding the risks caused by starting subsequent processes before the fault is fully recovered. It strengthens the rigor of task management, further ensures the stability and timeliness of real-time data warehouse data flow, and reduces data delays or errors caused by untimely fault recovery.

[0022] 106. When the failed task is determined to have recovered to normal, downstream task recovery processing is performed on the failed task to complete the task execution.

[0023] In this embodiment, based on the judgment result of task recovery, the task dependency control release mechanism built into the scheduling system is triggered. First, the status verification and priority sorting of the blocked downstream tasks are completed, clarifying the dependency links and pending execution nodes of each downstream task. Then, according to the preset scheduling order and business priority, all previously blocked downstream batch scheduling tasks are started sequentially. At the same time, the scheduling system's batch processing capability is activated to quickly make up for the delayed batch data caused by task blocking, ensuring that downstream tasks can not only resume normal operation, but also fully cover the unexecuted process during the blocking period. Finally, a closed-loop execution of the entire chain from failed task recovery to downstream task execution is achieved, completing the entire task process. The automated downstream task recovery processing realizes a seamless connection from abnormal blocking to normal operation of the task chain. The entire process recovery and make-up can be completed without manual intervention, improving the continuity and timeliness of task execution.

[0024] Please see Figure 2 In this embodiment of the invention, the construction of the target task import status table includes: 201. Obtain the preset initial state value and the preset table field structure, wherein the table field structure includes the number of historical error reports, task recovery status and table update time; 202. Construct an initial task import status table based on the table field structure; 203. Based on the initial state value, perform data insertion processing on the initial task import state table to obtain the target task import state table.

[0025] In this embodiment, a preset initial state value and a preset table field structure are obtained. The table field structure explicitly includes three core fields according to monitoring requirements: historical error count, task recovery status, and table update time. These correspond to the cumulative error statistics, fault recovery identifier, and latest data update time during task execution, respectively, forming a standardized data storage dimension adapted to Flink task monitoring. The initial state value is preset to baseline data of no errors, normal task recovery, and the current time, matching the default state of initial task execution. Constructing the initial task import state table based on the table field structure essentially involves the standardized creation of a physical table in the real-time data warehouse. Combining data warehouse storage standards with the characteristics of task monitoring data, the table structure ensures good compatibility and scalability, accurately carrying the full statistical data of subsequent task execution status, providing a structured data carrier for the implementation of monitoring logic. Data insertion processing is performed on the initial task import state table based on the preset initial state value. This involves writing the preset baseline data into an empty table, transforming the table from a structural definition into a functional table that can be practically applied, ultimately obtaining the target task import state table. This establishes an initial data baseline for the collection, comparison, and updating of subsequent task execution status.

[0026] Please see Figure 3 In this embodiment of the invention, the task running status dataset includes the current number of errors, the number of tasks running, and the historical number of errors; the step of performing monitoring and detection operations based on the target task import status table to obtain the task running status dataset includes: 301. Obtain the preset error log table and the preset task instance table; 302. Perform a current task error count statistics operation based on the error record table to obtain the current error count; 303. Perform a current task execution count operation based on the task instance table to obtain the number of tasks executed; 304. Based on the target task import status table, perform a historical error count query and extraction operation to obtain the historical error count.

[0027] In this embodiment, the task execution status dataset includes three core quantitative indicators: current error count, number of task executions, and historical error count. This dataset is the core data support for the monitoring system to conduct subsequent anomaly judgment. First, a preset error record table and a preset task instance table need to be obtained. Both tables are core business data carriers built into the Dinky application, possessing the characteristic of recording task execution error information and instance execution status in real time, serving as the basic data source for subsequent statistical query operations. Based on the error record table, the current task error count is statistically analyzed. The core is to limit data filtering and counting operations to the current day's time dimension, focusing on the error situation of core business chain tasks, and finally extracting a timely and targeted current error count indicator. Based on the task instance table, the current task execution count is statistically analyzed, filtering task instances in normal operation and counting them, ultimately extracting the task execution count indicator, accurately reflecting the current operational status of core tasks. The historical error count query and extraction operation based on the target task import status table relies on the standardized and normalized historical statistical data stored in the target task import status table to locate and extract the corresponding historical error count indicators. Finally, the three indicators are integrated to form a complete task operation status dataset, providing comprehensive and reliable data support for subsequent task anomaly judgment. At the same time, it realizes the traceability and verifiability of task operation status data, enhances the automation and standardization level of the monitoring system, reduces operational omissions and errors caused by manual intervention, and further ensures the stability and controllability of the collaborative operation of the real-time data warehouse and batch scheduling system.

[0028] Please see Figure 4 In this embodiment of the invention, the first task anomaly determination is performed on the current task based on the task running status dataset, including: 401. Obtain the preset number of monitoring tasks; 402. Compare the current number of error reports with the historical number of error reports, and at the same time, compare the number of monitoring tasks with the number of tasks running; 403. When the current number of error reports is greater than the historical number of error reports, or the number of tasks running is less than the number of monitored tasks, the current task is determined to be in an abnormal state. 404. When the current number of error reports equals the historical number of error reports, and the number of tasks running equals the number of monitoring tasks, the current task is determined to be in a normal state.

[0029] In this embodiment, the preset number of monitoring tasks is first obtained. This number is the total number of key tasks preset based on the core business link requirements and serves as the core reference for determining the integrity of task operation. The current number of error reports is compared with the historical number of error reports to identify whether new errors have occurred during task operation, thus accurately capturing abnormal increases in task operation. At the same time, the number of monitoring tasks is compared with the number of running tasks. By verifying the horizontal benchmark number with the actual number of running tasks, it is determined whether all core tasks are in an effective running state and whether there are gaps in the business link caused by task shutdowns. When the current number of error reports is greater than the historical number of error reports, or the number of running tasks is less than the number of monitoring tasks, the current task is determined to be in an abnormal state. The former corresponds to the task having generated new running errors, and the latter corresponds to an insufficient number of core running tasks. Either condition being met means that the task operation has deviated from the normal track, posing a potential risk to data flow and business progress. When the current number of error reports equals the historical number of error reports, and the number of running tasks equals the number of monitored tasks, the current task is considered to be in a normal state. This indicates that no new errors were generated during the task's execution, all core tasks are running effectively, and the business data flow is complete and smooth, meeting the normal expectations of the monitoring system and business operations. This two-way data comparison method accurately captures new errors in tasks while effectively verifying the integrity of core task operations, avoiding the bias and misjudgment risks associated with single-dimensional judgments, and significantly improving the accuracy and reliability of task status determination. Furthermore, it can quickly identify the type of anomaly in task execution, providing guidance for subsequent troubleshooting and efficient recovery, reducing the cost of manual intervention and operational oversights, further ensuring the stability and continuity of the collaborative operation of the real-time data warehouse and batch scheduling system, and reducing the probability of data errors and delays caused by unrecognized task anomalies.

[0030] Please see Figure 5 In this embodiment of the invention, when the current task is determined to be in an abnormal state, the target task import status table is updated, and the downstream task is blocked, resulting in an updated task import status table and failed tasks, including: 501. When the current task is determined to be in an abnormal state, obtain the execution result of the first script and the preset abnormal state update rules; 502. Based on the abnormal status update rules, update the fields of the historical error count, the task recovery status, and the table update time to obtain the update task import status table; 503. Using a task scheduling tool, based on the execution result of the first script, the downstream task of the current task is blocked to obtain the failed task.

[0031] In this embodiment, when the current task is determined to be in an abnormal state, the first script execution result obtained is the abnormal task operation data and status feedback information collected by the Shell script in the monitoring process. The preset abnormal state update rule is a table field update guideline formulated in accordance with task monitoring needs and data flow specifications, which clarifies the update logic and assignment standards of each core field, providing a standardized basis for subsequent status table updates. Based on this abnormal state update rule, the historical error count, task recovery status, and table update time are updated by assigning update field values. Specifically, the historical error count is updated to the latest captured task error count, the task recovery status is updated to the unrecovered flag, and the table update time is updated to the real-time operation timestamp. Through standardized field assignment and data persistence operations, a full update of the task import status table is completed, and the updated task import status table is finally obtained. This ensures that the table can retain the core statistical data of the task's abnormal state in real time and accurately, providing an authoritative data benchmark for subsequent fault recovery detection. By employing a task scheduling tool, and based on the error status feedback contained in the execution result of the first script, downstream task blocking is implemented for the current task. After the scheduling tool identifies the abnormal status returned by the script, it triggers the built-in task dependency control mechanism to block the execution flow of all related batch scheduling tasks downstream of the current task. At the same time, the current task is marked as a failed task, which realizes rapid control of abnormal tasks and timely blocking of downstream links. This fundamentally avoids the data-free running behavior of downstream tasks due to missing upstream data, greatly reduces the ineffective consumption of system resources, and eliminates derivative problems such as data errors and data delays caused by data-free running.

[0032] Please see Figure 6 In this embodiment of the invention, the step of triggering the failed task to re-run based on the time frequency, and performing a second task anomaly determination on the failed task based on the updated task import status table, includes: 601. Perform a timed trigger operation on the failed task based on the time frequency to obtain a rerun of the task; 602. Obtain the number of update errors based on the status table imported from the update task; 603. Compare the number of update error reports with the current number of error reports; 604. When the number of update errors equals the current number of errors, it is determined that the rerun task has returned to normal.

[0033] In this embodiment, the scheduling system automatically starts previously marked failed tasks at predetermined intervals, generating rerun tasks to achieve normalized and automated monitoring of fault recovery status. It continuously verifies whether the task's operational status has recovered without manual intervention, synchronously repeating core monitoring and detection processes to collect the latest operational data. A timed triggering mechanism constructs an automated fault recovery re-inspection system, improving the timeliness of fault recovery response and avoiding various problems caused by omissions or delayed interventions during manual monitoring. The system obtains the update error count based on the updated task import status table. This count is standardized historical error baseline data saved when the task is abnormal. Simultaneously, during the rerun task execution, the current error count, i.e., the error count in the target task import status table, is collected synchronously, ensuring the timeliness and correspondence of the two sets of comparison data, providing accurate data support for recovery determination. The update error count is compared with the current error count, and the number of updated task runs is compared with the preset number of monitoring tasks to form a two-dimensional recovery judgment logic. When the update error count equals the current error count and the number of task runs equals the number of monitoring tasks, it is determined that the task has been restored to normal after rerunning. This indicates that there are no new errors during the task run, and the core tasks are all in normal operation, and the overall operation chain has been restored to complete and smooth operation.

[0034] Please see Figure 7 In this embodiment of the invention, when the failed task is determined to have recovered to normal, downstream task recovery processing is performed on the failed task to complete the task execution, including: 701. When the failed task is determined to have recovered to normal, obtain the execution result of the second script and the preset normal status update rules; 702. Based on the normal state update rules, perform state update processing on the update task import state table to obtain a new task import state table; 703. Using the task scheduling tool, based on the execution result of the second script, perform downstream task recovery processing on the failed task, and perform rapid rerun processing on the downstream task to complete the task execution.

[0035] In this embodiment, when a failed task recovers normally, the obtained second script execution result is a success status feedback and a record of the entire task recovery data after the monitoring shell script has been successfully executed. The preset normal status update rule is a table field update guideline that conforms to the monitoring system and data flow specifications. It clarifies the verification logic and update standards for core fields such as task recovery status. A key aspect is the pre-verification requirement for the task recovery status field; subsequent update operations can only be performed if the field is still in the unrecovered state, ensuring the rigor and effectiveness of the status update. Based on this normal status update rule, the task import status table is updated. The core process is to first accurately verify the task recovery status field in the table to confirm whether its current status meets the update conditions. Then, the task recovery status fields that meet the conditions are updated to the recovered flag, and the latest update timestamp in the table is updated simultaneously. This completes the standardized data archiving of the normal task operation status, ultimately resulting in a new task import status table. This ensures that the table can retain the core status data after task recovery in real time and accurately. By employing a task scheduling tool and based on the success status feedback from the second script execution result, downstream task recovery processing is performed for failed tasks. After the scheduling tool recognizes the success status returned by the script, it triggers the built-in task dependency control release mechanism, restoring the normal execution permissions of all previously blocked downstream batch scheduling tasks. Downstream tasks are started sequentially according to preset business priorities and task dependency links. At the same time, the scheduling system's batch processing capability is activated to quickly re-run delayed batch data caused by abnormal task blocking, fully covering the unexecuted data flow during the blocking period. Ultimately, a complete closed loop is achieved for the entire task chain from abnormal blocking to normal operation, achieving full-process completion of task execution. This realizes a fully automated closed loop from abnormal task recovery judgment to status archiving, downstream recovery, and delayed re-run, completing the entire process control without manual intervention. This significantly reduces manual operation and maintenance costs and operational oversight risks, further strengthening the collaborative operation capability of Flink tasks and batch scheduling systems, and improving the stability and reliability of the overall data system operation.

[0036] The data warehouse scheduling task blocking method in the embodiments of the present invention has been described above. The data warehouse scheduling task blocking device in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 8 One embodiment of the data warehouse scheduling task blocking device in this invention includes: Status table construction module 801: Used to construct the target task import status table; Monitoring and detection module 802: used to perform monitoring and detection operations based on the target task's imported status table to obtain a task running status dataset; Task anomaly determination module 803: used to perform a first task anomaly determination on the current task based on the task running status dataset; Task blocking module 804: When the current task is determined to be in an abnormal state, it performs state update processing on the target task import status table and performs downstream task blocking processing on the current task to obtain the updated task import status table and the failed task. Task triggering module 805: used to obtain a preset time frequency, trigger the failed task to run again based on the time frequency, and perform a second task anomaly determination on the failed task based on the updated task import status table. Task recovery module 806: When the failed task is determined to have recovered normally, it performs downstream task recovery processing on the failed task to complete the task execution.

[0037] Based on the same ideas as the methods in the above embodiments, the apparatus provided in this application can implement the methods in the above embodiments.

[0038] above Figure 8 The data warehouse scheduling task blocking device in this embodiment of the invention is described in detail from the perspective of modular functional entities. The data warehouse scheduling task blocking device in this embodiment of the invention is described in detail below from the perspective of hardware processing.

[0039] Figure 9 This is a schematic diagram of a data warehouse scheduling task blocking device 900 provided in an embodiment of the present invention. The data warehouse scheduling task blocking device 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (e.g., one or more processors) and a memory 920, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 933 or data 932. The memory 920 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data warehouse scheduling task blocking device 900. Furthermore, the processor 910 may be configured to communicate with the storage media 930 and execute a series of instruction operations on the storage media 930 on the data warehouse scheduling task blocking device 900 to implement the steps of the data warehouse scheduling task blocking method provided in the above-described method embodiments.

[0040] The data warehouse scheduling task blocking device 900 may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input / output interfaces 960, and / or one or more operating systems 931, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 9 The data warehouse scheduling task blocking device structure shown does not constitute a limitation on the data warehouse scheduling task blocking device. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0041] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a data warehouse scheduling task blocking method.

[0042] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0043] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0044] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data warehouse scheduling task blocking method, characterized in that, include: Construct the target task import status table; Based on the target task, import the status table and perform monitoring and detection operations to obtain the task running status dataset. Based on the task running status dataset, perform a first task anomaly determination on the current task; When the current task is determined to be in an abnormal state, the target task import status table is updated, and the downstream task is blocked, resulting in an updated task import status table and a failed task. Obtain a preset time frequency, trigger the failed task to run again based on the time frequency, and perform a second task anomaly determination on the failed task based on the status table imported by the updated task. When the failed task is determined to have recovered to normal, downstream task recovery processing is performed on the failed task to complete the task execution.

2. The data warehouse scheduling task blocking method according to claim 1, characterized in that, The construction target task import status table includes: Obtain preset initial state values ​​and preset table field structures, wherein the table field structures include historical error counts, task recovery status, and table update time; Construct an initial task import status table based on the table field structure; Based on the initial state value, data insertion processing is performed on the initial task import state table to obtain the target task import state table.

3. The data warehouse scheduling task blocking method according to claim 1, characterized in that, The task running status dataset includes the current number of errors, the number of tasks running, and the historical number of errors; The step of performing monitoring and detection operations based on the target task import status table to obtain the task running status dataset includes: Retrieve the preset error log table and the preset task instance table; Based on the error record table, perform a current task error count statistics operation to obtain the current error count; Perform a current task execution count operation based on the task instance table to obtain the task execution count; Based on the target task's imported status table, a historical error count query and extraction operation is performed to obtain the historical error count.

4. The data warehouse scheduling task blocking method according to claim 3, characterized in that, Based on the task running status dataset, the current task is subjected to a first task anomaly determination, including: Get the preset number of monitoring tasks; The current number of errors is compared with the historical number of errors, and the number of monitoring tasks is compared with the number of tasks running. When the current number of error reports is greater than the historical number of error reports, or the number of tasks running is less than the number of monitored tasks, the current task is determined to be in an abnormal state. When the current number of error reports equals the historical number of error reports, and the number of tasks running equals the number of monitoring tasks, the current task is determined to be in a normal state.

5. The data warehouse scheduling task blocking method according to claim 2, characterized in that, When the current task is determined to be in an abnormal state, the target task import status table is updated, and the downstream task is blocked, resulting in an updated task import status table and failed tasks, including: When the current task is determined to be in an abnormal state, the execution result of the first script and the preset abnormal state update rules are obtained; Based on the abnormal status update rules, update fields are assigned to the historical error count, the task recovery status, and the table update time to obtain the update task import status table. Using a task scheduling tool, the downstream task is blocked based on the execution result of the first script, resulting in the failed task.

6. The data warehouse scheduling task blocking method according to claim 4, characterized in that, The step of triggering the failed task to run again based on the time frequency, and performing a second task anomaly determination on the failed task based on the updated task import status table, includes: Based on the time frequency, a timed trigger operation is performed on the failed task to allow the task to be rerun. The number of update errors is obtained based on the status table imported from the update task. Compare the number of update errors with the current number of errors; When the number of update errors equals the current number of errors, it is determined that the rerun task has returned to normal.

7. The data warehouse scheduling task blocking method according to claim 5, characterized in that, When the failed task is determined to have recovered normally, downstream task recovery processing is performed on the failed task to complete the task execution, including: When the failed task is determined to have recovered to normal, the execution result of the second script and the preset normal status update rules are obtained; Based on the normal state update rules, the update task import state table is updated to obtain a new task import state table. Using the task scheduling tool, the failed task is recovered by downstream task processing based on the execution result of the second script, and the downstream task is quickly re-run to complete the task execution.

8. A data warehouse scheduling task blocking device, characterized in that, include: Status table construction module: used to build the target task import status table; Monitoring and detection module: used to perform monitoring and detection operations based on the status table imported by the target task in order to obtain the task running status dataset; Task anomaly determination module: used to perform a first task anomaly determination on the current task based on the task running status dataset; Task blocking module: When the current task is determined to be in an abnormal state, the module updates the status of the target task import status table and blocks downstream tasks of the current task to obtain the updated task import status table and the failed task. Task triggering module: used to obtain a preset time frequency, trigger the failed task to run again based on the time frequency, and perform a second task anomaly determination on the failed task based on the updated task import status table; Task recovery module: When the failed task is determined to have recovered normally, it performs downstream task recovery processing on the failed task to complete the task execution.

9. A data warehouse scheduling task blocking device, characterized in that, The data warehouse scheduling task blocking device includes: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause the data warehouse scheduling task blocking device to perform the steps of the data warehouse scheduling task blocking method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the data warehouse scheduling task blocking method as described in any one of claims 1-7.

Images