ETL process scheduling method and device
A process and scheduling model technology, applied in the field of data processing, can solve problems such as inability to query process scheduling model objects, scheduling time interval errors, and insufficient real-time scheduling, so as to ensure timeliness, avoid scheduling errors, and avoid process real-time degradation Effect
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0029] When using ETL tools for process scheduling, for periodically scheduled scheduling tasks, it is necessary to perform timing and schedule and execute the process model object and scheduling object corresponding to the scheduling task when the preset scheduling time point is reached. However, the existing scheme of using a timer for timing is based on absolute time, does not catch exceptions, and is single-threaded scheduling, which will cause the process model object to fail to start executing at the correct time. Therefore, this embodiment provides a new scheduling method to avoid incorrect scheduling that may occur when using the timer for scheduling.
[0030] like figure 1 As shown, the specific steps of the method for ETL process scheduling provided by the embodiment of the present invention are as follows:
[0031] Step 101: Create a delay queue.
[0032] In the current process scheduling method, the basic processing model of Timer is the task queue model of singl...
Embodiment 2
[0064] Based on the ETL process scheduling method provided in Embodiment 1, in different specific application scenarios, it can also be supplemented and adjusted according to different usage requirements or actual scenarios.
[0065] In ETL process scheduling, a process model object can configure one or more scheduling tasks, and execute the same process model object at different scheduling times, such as executing file Load once and executing file Load every interval; a scheduling task can also It can be referenced by one or more process model object configurations, and execute multiple process model objects sequentially or concurrently at the same scheduling time, such as executing a preset ETL process at a fixed time point; multiple schedules can also be configured for multiple process model objects Tasks such as executing a pre-set ETL process once a day at a valid date time. In step 102, when encapsulating the scheduling task into a scheduling element, it is necessary to ...
Embodiment 3
[0080] On the basis of the method for ETL process scheduling provided by the above-mentioned embodiment 1 to embodiment 2, the present invention also provides a device for ETL process scheduling that can be used to implement the above method, such as Figure 4 Shown is a schematic diagram of the device architecture of the embodiment of the present invention. The apparatus for ETL process scheduling in this embodiment includes one or more processors 21 and memory 22 . in, Figure 4 A processor 21 is taken as an example.
[0081] Processor 21 and memory 22 can be connected by bus or other means, Figure 4 Take connection via bus as an example.
[0082] The memory 22 is a non-volatile computer-readable storage medium for an ETL process scheduling method, and can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as those in Embodiment 1 to Embodiment 2 ETL process scheduling method. The processor 21 runs the non-volat...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More