ETL task scheduling optimization method, system, equipment and medium

Abstract

The invention discloses an ETL task scheduling optimization method and system, equipment and a medium, and relates to the field of data processing. According to the method, an ETL task is divided into a plurality of execution stages, and pause time with preset duration is set between every two execution stages; executing the next execution stage of the current ETL task when the pause time is entered and the first resource currently usable by the platform is greater than or equal to the second resource required by the next execution stage of the current ETL task; when the first resource is smaller than the second resource, the current ETL task is converted into a pause state, a pause point is set, the current ETL task is added into a task waiting queue, and an ETL task with the priority higher than that of the current ETL task is executed or whether resource release exists or not is continuously monitored. By implementing the technical scheme provided by the invention, the ETL task can be reasonably scheduled when resources are insufficient, resource waste and task delay are avoided, and the execution efficiency of the ETL scheduling task is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to an optimization method, system, electronic device and storage medium for ETL task scheduling. Background Art

[0002] With the continuous development of big data technology, ETL (Extract, Transform, Load) tasks are increasingly used in the field of data processing. However, in practical applications, due to the limited resources of the data processing platform, there is often a problem of resource competition between multiple ETL tasks.

[0003] At present, in order to solve the problem of resource competition, some technical solutions try to improve resource utilization by optimizing resource allocation. For example, some technical solutions use static resource allocation to allocate fixed resources to each task before the task is executed. However, this method cannot be dynamically adjusted according to the real-time needs of the task and the resource status of the platform, which easily leads to waste or shortage of resources.

[0004] Therefore, a method is needed to dynamically manage resources and improve the efficiency of ETL task execution. Summary of the invention

[0005] The present application provides an optimization method, system, electronic device and storage medium for ETL task scheduling. By setting intermediate states and pause times for scheduling tasks, real-time perception and intelligent scheduling of system resources are achieved. This technology not only improves the efficiency and stability of the ETL process, but also can provide enterprises with more accurate and timely data support, and promote the optimization and upgrading of business decisions.

[0006] In a first aspect of the present application, an optimization method for ETL task scheduling is provided, which is applied to a task scheduling platform. The optimization method includes: Divide the ETL task into multiple execution stages and set a preset pause time between each execution stage; When entering the pause time, obtaining a first resource currently available on the platform and a second resource required for the next execution phase of the current ETL task; When the first resource is greater than or equal to the second resource, executing the next execution phase of the current ETL task; When the first resource is less than the second resource, the current ETL task is converted to a pause state and a pause point is set, the current ETL task is added to a task waiting queue and an ETL task with a higher priority than the current ETL task is executed or whether there is resource release is continuously monitored.

[0007] By adopting the above technical solution, the ETL task is divided into multiple execution stages, and a pause time of preset duration is set between each two stages. This method allows the system to dynamically evaluate the currently available resources (such as CPU, memory, disk IO, etc.) during the task execution process. This ensures that when the system resources are insufficient to support the next execution stage of the current task, the task can be paused and wait for the release of resources, thereby avoiding task failure or performance bottlenecks caused by insufficient resources. In this way, system resources can be used more effectively and the overall task execution efficiency can be improved. The embodiment of the present application makes task scheduling more flexible by introducing pause points and task waiting queues. When resources are insufficient, tasks can be paused and wait, and the system can process other tasks with higher priority at the same time. This mechanism enables the ETL task scheduling system to better adapt to different loads and resource conditions, and improves the scalability of the system and the ability to cope with sudden loads. Traditional ETL task scheduling often adopts a simple sequential execution method, ignoring the impact of dynamic changes in resources on task execution. The embodiment of the present application can significantly reduce the task failure rate caused by insufficient resources by monitoring the resource situation in real time and dynamically adjusting the task execution status. When resources meet the demand, the task can be smoothly executed to the next stage, thereby improving the overall success rate of the task. By optimizing resource allocation and task execution order, operation and maintenance operations such as system restart and task retry caused by resource conflicts and task failures are reduced. This not only reduces the workload of operation and maintenance personnel, but also reduces the risk of business interruption due to system instability, thereby reducing the overall operation and maintenance cost. The embodiment of the present application can adapt well to complex and changeable ETL task scenarios. Whether it is the execution of a large number of concurrent tasks or the scheduling of tasks in a resource-constrained environment, the efficient execution of tasks can be ensured by reasonably setting the pause time and dynamically adjusting the task status. This provides strong support for ETL task scheduling in scenarios such as enterprise data warehouses and big data analysis.

[0008] Optionally, executing an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release includes: Determine whether there is an ETL task with a higher priority than the current ETL task; When there is a first ETL task with a higher priority than the current ETL task, select a second ETL task with the highest priority from the first ETL task, and when the first resource can meet the resource requirement of the second ETL task, execute the second ETL task; When there is no first ETL task with a higher priority than the current ETL task, whether there is resource release is continuously monitored, and when resource release is detected, whether the third resource currently available to the platform is greater than the second resource is determined.

[0009] By adopting the above technical solution, by judging and giving priority to tasks with higher priority than the current ETL task, it is ensured that high-priority tasks can obtain system resources first, thereby improving the execution speed and success rate of key tasks. This is particularly important for business scenarios that require rapid response, such as real-time data analysis, emergency data repair, etc. When there are high-priority tasks, the embodiment of the present application will evaluate whether the system resources can meet the resource requirements of these tasks. Only when resources are sufficient will these tasks be executed, thereby avoiding task failure or performance degradation caused by insufficient resources. This decision-making mechanism based on resource availability helps to achieve efficient utilization of resources. When there is no task with higher priority than the current ETL task, the release of system resources will be continuously monitored. When resource release is detected, the resource requirements of the current task will be re-evaluated, and the execution of the task will be attempted to be restored. This dynamic monitoring and response mechanism enables the system to flexibly respond to resource changes and improves the flexibility and reliability of task execution. Through priority sorting and resource monitoring, resource competition and conflict between different tasks are reduced. High-priority tasks can obtain resources first, while low-priority tasks will be executed when resources are sufficient, thereby avoiding task delays or failures caused by resource competition. Through reasonable task scheduling and resource management, this method helps to improve the stability and reliability of the entire ETL task scheduling system. The system can better cope with resource fluctuations and task changes, and reduce system crashes or performance degradation caused by insufficient resources or task conflicts.

[0010] Optionally, executing an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release further includes: When the first resource cannot meet the resource demand of the second ETL task, determining whether there is a third ETL task with a lower priority than the second ETL task among the ETL tasks currently being asynchronously executed; When there is a third ETL task with a lower priority than the second ETL task among the ETL tasks currently being asynchronously executed, a fourth ETL task with the lowest priority is selected from the third ETL tasks, and the fourth ETL task is set to a pause state to release resources occupied by the fourth ETL task.

[0011] By adopting the above technical solution, when a high-priority task (such as the second ETL task) cannot be executed immediately due to insufficient resources, the method will check the low-priority tasks (such as the third ETL task) currently being executed asynchronously, and select the task with the lowest priority (such as the fourth ETL task) to pause to release the resources occupied by it. This dynamic resource adjustment mechanism helps to ensure that high-priority tasks can obtain the required resources as soon as possible, thereby improving the efficiency and success rate of task execution. By distinguishing the priorities of different tasks and giving priority to the execution of high-priority tasks when resources are insufficient, the embodiment of the present application realizes the refinement of priority management. This refinement not only helps to ensure the timely completion of key tasks, but also balances the allocation of system resources to a certain extent, avoiding low-priority tasks from occupying resources for a long time and affecting overall performance. By releasing the resources occupied by low-priority tasks in a timely manner, the embodiment of the present application helps to shorten the waiting time of high-priority tasks. This is particularly important for business scenarios that require rapid response, because it can reduce business delays and losses caused by resource waiting. By dynamically adjusting the task execution status and resource allocation, the embodiment of the present application helps to optimize the utilization of system resources. When resources are insufficient, pausing low-priority tasks to release resources can ensure that high-priority tasks can make full use of system resources, thereby improving overall resource utilization and efficiency.

[0012] Optionally, executing an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release further includes: When there is a third ETL task whose priority is lower than that of the second ETL task among the ETL tasks currently being asynchronously executed, the difference between the resource requirements of the first resource and the second ETL task is calculated, and it is determined whether there is an ETL task among the third ETL tasks that occupies resources greater than the difference; When there is a fifth ETL task in the third ETL tasks and the resources occupied are greater than the difference, the fifth ETL task is converted into a pause state to release the resources occupied by the fifth ETL task.

[0013] By adopting the above technical solution, the difference between the currently available resources (first resources) and the resources required by the high-priority task (second ETL task) is calculated, and the low-priority task that can release enough resources to meet the needs of the high-priority task is judged based on this, so as to achieve more accurate resource management. This accuracy helps to reduce unnecessary resource release and task pauses, thereby improving the resource utilization and task execution efficiency of the system as a whole. Compared with simply suspending the lowest priority task to release resources, the embodiment of the present application selectively suspends tasks by comparing resource differences, reducing the waste caused by excessive release of resources. Only those low-priority tasks that occupy more resources than the difference will be suspended, thereby ensuring the pertinence and effectiveness of resource release. Since the embodiment of the present application can manage resources more accurately and optimize the order of task execution, it helps to improve the response speed of the system. In an emergency, the system can respond to the needs of high-priority tasks more quickly, thereby ensuring the timely processing of key services. By introducing accurate resource management and task execution order optimization strategies, the embodiment of the present application enhances the stability and predictability of the ETL task scheduling system. The system can more accurately predict task execution time and resource requirements, thereby avoiding system instability and performance fluctuations caused by resource conflicts or task waiting.

[0014] Optionally, the optimization method further includes: When the first resource is less than a threshold and a new ETL task is received, adding the new ETL task to a message queue; Determine the priority of the new ETL task, and arrange the new ETL task to the last one of the priority in the message queue according to the priority; When an ETL task is converted to a paused state, it is determined whether the ETL task currently ranked first in the message queue can be executed.

[0015] By adopting the above technical solution, when the system resources (first resources) are lower than the preset threshold, the newly received ETL task will not be executed immediately, but will be added to the message queue. This helps to protect the currently executed task from the resource competition caused by the sudden addition of the new task, thereby ensuring the stable operation of the system. The newly added ETL task will be arranged at the end of the corresponding priority in the message queue according to its priority. This priority sorting mechanism ensures that high-priority tasks can get priority execution opportunities, thereby improving the response speed and success rate of key tasks. When there is an ETL task converted to a pause state, the system will check whether the task ranked first in the message queue can be executed. This dynamic inspection mechanism enables the system to flexibly respond to changes in resource release and task execution status, and timely schedule new task execution, thereby improving resource utilization and task execution efficiency. By adding a new task to the message queue instead of executing it immediately, the embodiment of the present application reduces the resource competition between the new task and the currently executed task. Only when the system resources are sufficient or there is resource release, the new task will be considered for execution, thereby avoiding task failure or performance degradation caused by insufficient resources. As a buffering mechanism, the message queue can smoothly handle the fluctuation of task requests. When the system load increases, new tasks can be temporarily stored in the queue waiting to be executed; when the load decreases, the tasks in the queue can be gradually processed. This mechanism improves the system's scalability and ability to cope with sudden loads.

[0016] Optionally, the optimization method further includes: Determine whether any resources are released by monitoring memory usage; When there are resources to be released, determining whether the first highest priority in the task waiting queue is greater than the second highest priority in the message queue; When the first highest priority in the task waiting queue is greater than the second highest priority in the message queue, the ETL task corresponding to the first highest priority is selected from the task waiting queue for awakening, and execution continues from the pause point of the ETL task.

[0017] By adopting the above technical solution, resource release is detected in time by monitoring memory usage, which enables the system to respond quickly to changes in resource status. Once resources are released, the system can reallocate resources based on the current task priority, thereby avoiding idleness and waste of resources and improving the overall utilization efficiency of resources. After the resources are released, the system does not simply execute the next task in the message queue, but first compares the priorities of the tasks in the task waiting queue and the message queue. This mechanism ensures that even if new tasks are added, high-priority suspended tasks have the opportunity to be executed first when resources are available, thereby achieving more reasonable and flexible task scheduling. For those high-priority tasks that are suspended due to insufficient resources, once resources are released and their priority is higher than the tasks in the message queue, they can be quickly awakened and continue to execute from the pause point. This significantly reduces the waiting time of these tasks, improves the execution efficiency of tasks and the overall system performance. By monitoring resource status in real time and flexibly adjusting the order of task execution, the system can respond to resource changes and task requirements more quickly, thereby improving the responsiveness and stability of the system. This ability of instant feedback and dynamic adjustment is particularly important for handling urgent tasks and high-load scenarios. Users usually expect the system to process the tasks they submit as soon as possible, especially those with high priority. The above optimization method significantly improves the user experience by reducing task waiting time and improving task execution efficiency, thereby increasing user satisfaction and trust in the system.

[0018] Optionally, setting a pause time of a preset duration between each two execution stages includes: Evaluate the complexity of ETL tasks, including the amount of data processing and the complexity of conversion logic, and predict the amount of resources required for each execution stage based on the evaluation results; The initial pause time is calculated according to the recovery speed of the system resources and the required resource amount, and the initial pause duration is adjusted according to influencing factors to obtain the pause time. The influencing factors include network delay and external data source update frequency.

[0019] By adopting the above technical solution, the complexity of the ETL task is evaluated, including the amount of data processing and the complexity of the transformation logic, and the system can more accurately predict the amount of resources required for each execution stage. This prediction helps to reasonably allocate resources before the task is executed, avoid excessive concentration or idleness of resources, and thus improve resource utilization. Setting a pause time between each two execution stages allows the system to have enough time to recover and adjust the resource status. This helps to reduce performance degradation caused by resource competition or conflict, and ensure that the subsequent execution stages can proceed smoothly. The setting of the pause time also takes into account factors such as network latency and the update frequency of external data sources. Consideration of these factors enables the system to better adapt to changes in the external environment, avoid task failures or interruptions caused by external factors, and thus enhance the stability and reliability of the system. If an error or failure occurs in a certain execution stage, the setting of the pause time also provides the system with a time window for troubleshooting and recovery. Administrators or automated tools can use this time to locate the problem, adjust the configuration, or restart the service, so as to resume normal execution of the task more quickly. The strategy also includes steps to calculate the initial pause time based on the system resource recovery speed and the required amount of resources, and adjust it based on the influencing factors. This dynamic adjustment capability enables the system to respond flexibly according to actual conditions and further optimize the task execution process.

[0020] In a second aspect of the present application, an optimization system for ETL task scheduling is provided, including a partitioning module, a resource module, an execution module and a scheduling module, wherein: A partitioning module is configured to divide the ETL task into multiple execution stages and set a preset pause time between each execution stage; A resource module configured to obtain, when entering the pause time, a first resource currently available to the platform and a second resource required for a next execution phase of the current ETL task; an execution module, configured to execute a next execution phase of the current ETL task when the first resource is greater than or equal to the second resource; The scheduling module is configured to, when the first resource is less than the second resource, convert the current ETL task into a pause state and set a pause point, add the current ETL task to a task waiting queue and execute an ETL task with a higher priority than the current ETL task or continuously monitor whether there is resource release.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By setting a preset pause time between each execution phase, the system has the opportunity to re-evaluate the relationship between the currently available resources (first resources) and the resources required for the task to be executed (second resources). This real-time evaluation ensures the efficient allocation of resources among tasks, avoids over-allocation or under-allocation of resources, and thus improves overall resource utilization; 2. When the system finds that the current resources are insufficient to support the next execution stage, it can flexibly convert the current task to a pause state and add it to the task waiting queue. This mechanism allows the system to adjust the order of task execution according to the dynamic changes in resource availability, enhancing the flexibility and responsiveness of task execution; 3. By giving priority to ETL tasks with higher priority and whose resource requirements are met, the system can ensure that critical or urgent tasks are processed in a timely manner. At the same time, continuous monitoring of resource release also provides timely opportunities for the execution of subsequent tasks, thereby reducing task waiting time and the overall execution cycle; 4. The pause time setting reduces the situation where multiple tasks compete for limited resources at the same time, reducing the risk of performance degradation or task failure due to resource competition. In addition, by accurately controlling the resource requirements of each execution stage, the system can manage resource allocation more effectively and reduce the possibility of resource conflicts; 5. Through reasonable resource management and task scheduling strategies, the risk of system crash or instability caused by insufficient or overused resources is reduced. At the same time, by giving priority to high-priority tasks and responding to resource changes in a timely manner, the system can maintain a high level of stability and reliability; 6. Allow the system to dynamically adjust and optimize according to real-time resource conditions and task requirements. Whether it is increasing or decreasing the pause time during the task execution phase, or adjusting the task execution order and priority, the system can respond flexibly to ensure the efficiency and smoothness of the task execution process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of the optimization method for ETL task scheduling disclosed in the embodiment of the present application; Figure 2 It is a schematic diagram of the implementation of the optimization method for ETL task scheduling disclosed in the embodiment of the present application; Figure 3 It is a module schematic diagram of the optimization system for ETL task scheduling disclosed in the embodiment of the present application; Figure 4 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0025] Explanation of the reference numerals: 301, partition module; 302, resource module; 303, execution module; 304, scheduling module; 401, processor; 402, communication bus; 403, user interface; 404, network interface; 405, memory. DETAILED DESCRIPTION

[0026] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0027] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0028] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0029] This embodiment discloses an optimization method for ETL task scheduling, which is applied to a task scheduling platform. Figure 1 is a flow chart of the optimization method for ETL task scheduling disclosed in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: S110, dividing the ETL task into multiple execution stages, and setting a pause time of a preset duration between each two execution stages; Extract is the first step of the ETL process, which involves reading and extracting data from various data sources (such as databases, file systems, applications, etc.). Transform is the intermediate link of the ETL process, which involves cleaning, converting and formatting the extracted data to meet the data standards and quality requirements of the target system or data warehouse. Load is the last step of the ETL process, which involves loading the transformed data into the target system or data warehouse. By setting multiple intermediate states for the scheduled tasks in ETL, the tasks in the intermediate states are accompanied by a short pause time, during which resources and tasks are tested. The length of the pause time should be determined according to the complexity of the ETL task, the amount of data and the system resources. A longer pause time can provide more sufficient resource release and error handling time, but it will also increase the overall execution time.

[0030] Optionally, setting a pause time of a preset duration between each two execution stages includes: Evaluate the complexity of ETL tasks, including the amount of data processing and the complexity of conversion logic, and predict the amount of resources required for each execution stage based on the evaluation results; The initial pause time is calculated according to the recovery speed of the system resources and the required resource amount, and the initial pause duration is adjusted according to influencing factors to obtain the pause time. The influencing factors include network delay and external data source update frequency.

[0031] The purpose of evaluating the complexity of an ETL task is to more accurately predict the amount of resources it requires at different execution stages. This helps the system allocate resources based on the actual needs of the task and decide whether to set a pause time and the length of the pause time. Data processing volume: refers to the amount of data that the ETL task needs to process. The larger the amount of data, the more computing resources and storage resources are usually required during the processing. Transformation logic complexity: refers to the complexity of the data transformation logic in the ETL task. Complex transformation logic may require more CPU time and memory resources to execute. The complexity of an ETL task can be evaluated using methods such as historical data, simulation tests, or expert experience. The evaluation results are usually expressed in numerical or graded form for subsequent resource prediction and pause time calculation. Based on the results of the complexity evaluation, the amount of resources required by the ETL task at each execution stage is predicted. This includes resources such as CPU time, memory, disk space, and network bandwidth. A resource consumption model can be established based on historical execution data, and resource requirements can be predicted by inputting the complexity evaluation results of the current task. Heuristic methods or rule-based reasoning can also be used to estimate resource requirements. A preliminary pause time is calculated based on the recovery speed of system resources and the amount of resources required for each execution stage. This time should be enough for the system to recover enough resources for the next execution stage. The calculation method can be as follows: Determine the recovery speed of system resources, that is, how long it takes for the system to recover from the current state to a state that meets the resource requirements of the next execution stage. Based on the amount of resources required for each execution stage and the speed of resource recovery, calculate the preliminary pause time. Consider various influencing factors and adjust the preliminary pause duration to obtain a more reasonable and accurate pause time.

[0032] By evaluating the complexity of ETL tasks, including the amount of data to be processed and the complexity of the transformation logic, the system can more accurately predict the amount of resources required for each execution phase. This prediction enables the system to plan resource allocation in advance, ensuring that resources are ready before the execution phase begins, avoiding task waiting or failure due to insufficient resources. Considering the recovery speed of system resources to calculate the initial pause time means that the system can make full use of the "window period" of resource recovery, that is, the time between the release of resources from the previous task and the use of the next task. Doing so helps reduce resource idle time and improve resource utilization efficiency. Reasonable pause time settings can reduce resource competition and conflicts between tasks, so that each execution phase can be carried out in the best state. In addition, by adjusting the pause time in a timely manner to adapt to the actual situation, the system can respond more flexibly to resource requirements and task changes, thereby improving the overall task execution efficiency.

[0033] S120, when entering the pause time, obtaining a first resource currently available on the platform and a second resource required for the next execution phase of the current ETL task; The first resource here refers to the sum of all available resources in the system platform that can be allocated to the ETL task execution phase at the current point in time. These resources may include but are not limited to computing resources (such as CPU, memory), storage resources (such as disk space), network resources (such as bandwidth), etc. The system needs to monitor the status of these resources in real time or near real time in order to accurately assess their availability. The second resource refers to the amount of resources expected to be required for the next execution phase that the current ETL task is about to enter. This amount is predicted based on the characteristics of the task, the complexity assessment results, and the possible execution path. Specifically, it may involve the computing power required to process a specific amount of data, the disk space required to store intermediate results, and possible data transmission requirements.

[0034] S130: When the first resource is greater than or equal to the second resource, executing the next execution phase of the current ETL task; When the first resource is greater than or equal to the second resource, the system considers that the current platform has sufficient resources to support the current ETL task to continue to the next stage. This judgment is based on a direct assessment of resource adequacy, which ensures that the task will not be interrupted or delayed due to insufficient resources during execution. After confirming that the resources are sufficient, the system will execute the next execution stage of the current ETL task. This step may include operations such as loading new batches of data, executing data transformation logic, and updating the database or data warehouse. During the execution process, the system will continuously monitor resource usage to ensure that the task can be completed smoothly, and make resource adjustments or task scheduling when necessary.

[0035] S140. When the first resource is less than the second resource, convert the current ETL task into a pause state and set a pause point, add the current ETL task to a task waiting queue, and execute an ETL task with a higher priority than the current ETL task or continuously monitor whether there is resource release.

[0036] When the first resource is less than the second resource, the system believes that the resources on the current platform are insufficient to support the ETL task to continue to the next stage. In order to avoid task failure or unnecessary resource waiting due to insufficient resources, the system will convert the current ETL task to a paused state and set a clear pause point. The pause point refers to the specific location or stage where the task is paused during execution so that it can be continued from the interruption point when sufficient resources are available. The advantage of this is that the execution progress and status of the task can be retained, reducing duplication of work and resource waste. After converting the task to a paused state, the system will add the task to the task waiting queue. The task waiting queue is a list of tasks sorted by a certain priority, which is used to manage tasks that cannot be executed temporarily but need to wait for resource release. By adding tasks to the queue, the system can ensure that when resources are released or available, the execution of tasks can be rescheduled in order of priority. In the waiting queue, the system may encounter multiple tasks waiting to be executed. In order to maximize resource utilization and meet business needs, the system will give priority to those ETL tasks with higher priority. Priority can be determined based on multiple factors such as the urgency of the task, business value, and resource requirements. By executing higher priority tasks, the system can respond to business needs more quickly and improve overall work efficiency. At the same time, the system will continue to monitor the resource usage on the platform, especially those resources occupied or locked by other tasks. When these resources are released or become available, the system will re-evaluate and allocate resources to the corresponding ETL tasks based on the priority order of the task waiting queue. This continuous monitoring and resource reallocation mechanism helps to achieve dynamic management and optimal utilization of resources.

[0037] Optionally, executing an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release includes: Determine whether there is an ETL task with a higher priority than the current ETL task; When there is a first ETL task with a higher priority than the current ETL task, select a second ETL task with the highest priority from the first ETL task, and when the first resource can meet the resource requirement of the second ETL task, execute the second ETL task; When there is no first ETL task with a higher priority than the current ETL task, whether there is resource release is continuously monitored, and when resource release is detected, whether the third resource currently available to the platform is greater than the second resource is determined.

[0038] The system needs to determine whether there are other ETL tasks with higher priority than the current ETL task (i.e., the current ETL task) that is paused due to insufficient resources. This step is the core of task priority management, which ensures that the system can give priority to tasks that are more important or more urgent to the business. If the system finds that there is an ETL task with higher priority than the current ETL task (i.e., the first ETL task), if there are multiple first ETL tasks, it will further select a task with the highest priority from these high-priority tasks to execute (i.e., the second ETL task). The selection criteria are based on the priority of the task, which may be determined based on a combination of factors such as business needs, urgency, and historical priority. After selecting the second ETL task, the system needs to evaluate whether the resources available on the current platform (i.e., the first resource) can meet the resource requirements required for the execution of the task. If the first resource can meet the resource requirements of the second ETL task, the system will execute the task. This step ensures efficient use of resources and avoids the situation where low-priority tasks occupy too many resources and high-priority tasks cannot be executed. If the system does not find a first ETL task with higher priority than the current ETL task, or if it is found but the current resources cannot meet its requirements, the system will enter a state of continuous monitoring of resource release. This means that the system will constantly check whether resources are released or become available on the platform. This monitoring is real-time or periodic, depending on the system design and resource management requirements. When the system detects that there is a resource release, it will re-evaluate the amount of resources that the platform can currently use (i.e., the third resource). The system will then determine whether the third resource is greater than the amount of resources required for the next execution stage of the current ETL task that was previously paused due to insufficient resources (i.e., the second resource). If the third resource is greater than or equal to the second resource, the system will resume the execution of the current ETL task and continue to execute the next stage from the previously set pause point. If the third resource is still insufficient, the system will continue to keep the current ETL task in a paused state and continue to monitor resource release.

[0039] By determining whether there are other tasks with higher priority than the current ETL task, the system can ensure that critical and urgent tasks are processed first. This priority management mechanism helps to respond quickly to business needs and improve overall work efficiency. When there are tasks with higher priority, the system will select the task with the highest priority to execute, and execute it only when the current resources can meet the needs of the task. This avoids waste of resources and unnecessary waiting, and ensures efficient use of resources. When there are no tasks with higher priority, the system will continuously monitor the release of resources. This dynamic monitoring mechanism enables the system to grasp the changes in resources in real time and adjust the task execution plan in time according to the release of resources. This helps to reduce the waiting time of tasks caused by insufficient resources and improve the flexibility and responsiveness of task execution. By pausing the current task and joining the waiting queue when resources are insufficient, and executing tasks according to priority and resource release, the system can reduce resource competition and conflicts between tasks. This helps to maintain the stability and reliability of the system and avoid task failures or system crashes caused by resource conflicts. Priority management and resource optimization strategies can ensure that critical and urgent tasks are processed in a timely manner, thereby improving user satisfaction and trust. At the same time, by reducing task waiting time and improving task execution efficiency, users can see results and feedback faster, further enhancing the user experience. Priority judgment, resource monitoring, and task execution in the above strategies can all be implemented through automation tools, thus supporting intelligent scheduling and automated management of ETL tasks. This not only reduces the burden of manual operations, but also improves the accuracy and efficiency of scheduling.

[0040] Optionally, executing an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release further includes: When the first resource cannot meet the resource demand of the second ETL task, determining whether there is a third ETL task with a lower priority than the second ETL task among the ETL tasks currently being asynchronously executed; When there is a third ETL task with a lower priority than the second ETL task among the ETL tasks currently being asynchronously executed, a fourth ETL task with the lowest priority is selected from the third ETL tasks, and the fourth ETL task is set to a pause state to release resources occupied by the fourth ETL task.

[0041] The system will first traverse all ETL tasks currently being executed asynchronously. These tasks may be in different execution stages and each occupy a different amount of resources. The system needs to understand the priority of these tasks and the resources they currently occupy. After evaluating all ETL tasks being executed asynchronously, the system will pay special attention to ETL tasks with a lower priority than the second ETL task (i.e., the third ETL task). Although these tasks are also being executed, their priorities are relatively low, so when resources are tight, they may be the objects that are given priority to suspend or release resources. If there are multiple third ETL tasks with a lower priority than the second ETL task, the system will further select the one with the lowest priority (i.e., the fourth ETL task) from these tasks for pause processing. This selection is based on the consideration of maximizing resource release and minimizing the impact on the overall task execution progress. When the fourth ETL task is selected, the system will set it to a paused state and release the resources it occupies. These resources can then be reallocated to the second ETL task or other tasks with higher priority, thereby ensuring that the execution of critical tasks is not affected. After releasing the resources of the fourth ETL task, the system will re-evaluate whether the currently available resources (i.e., the third resources) meet the resource requirements of the second ETL task. If the conditions are met, the system will immediately execute the second ETL task; if not, the system may continue to wait for resource release or execute other tasks with higher priority.

[0042] The system can dynamically evaluate the priority and resource usage of tasks currently being executed asynchronously, and select lower-priority tasks to pause as needed to release the resources they occupy. This flexible resource management method enables the system to better adapt to resource constraints and ensure the execution of critical and urgent tasks. When selecting tasks to pause, the system makes decisions based on priority, pausing the lowest-priority tasks first. This priority-driven decision-making mechanism helps maintain the priority order of task execution and ensure that important tasks can get resources first. By setting lower-priority tasks to a paused state, the system can prevent these tasks from continuing to occupy valuable resources, thereby reducing resource waste. These resources can then be reallocated to higher-priority tasks to improve overall resource utilization. By releasing resources occupied by low-priority tasks, the system can meet the resource requirements of high-priority tasks more quickly, thereby shortening the waiting time of these tasks and improving the overall efficiency of task execution. Through reasonable resource allocation and task management, the system can reduce the risk of task failure or system crash caused by resource conflicts and competition, thereby enhancing the stability and reliability of the system. Users can see the execution results of important and urgent tasks more quickly because the system can prioritize these tasks. This fast response and efficient execution can improve user satisfaction and trust. The implementation of this strategy relies on the system's real-time monitoring and evaluation of task priorities, resource usage, and execution status. This requires the system to have a high level of intelligence and be able to automatically schedule tasks and allocate resources. Therefore, the implementation of this strategy helps promote the intelligent development of ETL task scheduling.

[0043] Optionally, executing an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release further includes: When there is a third ETL task whose priority is lower than that of the second ETL task among the ETL tasks currently being asynchronously executed, the difference between the resource requirements of the first resource and the second ETL task is calculated, and it is determined whether there is an ETL task among the third ETL tasks that occupies resources greater than the difference; When there is a fifth ETL task in the third ETL tasks and the resources occupied are greater than the difference, the fifth ETL task is converted into a pause state to release the resources occupied by the fifth ETL task.

[0044] The system calculates the difference between the currently available resources (first resources) and the resources required by the second ETL task. This difference represents the amount of additional resources that need to be released in order to execute the second ETL task. The system traverses all tasks that are currently being executed asynchronously and have a lower priority than the second ETL task (i.e., the third ETL task) and checks the amount of resources occupied by each of these tasks. In the third ETL task, the system looks for tasks that occupy more resources than the resource difference calculated above (i.e., the fifth ETL task). These tasks become potential candidates for releasing resources to support the execution of the second ETL task because they occupy too many resources. When the fifth ETL task that occupies more resources than the difference is identified, the system converts these tasks to a paused state. This means that these tasks will pause execution and release the resources they currently occupy. These released resources can then be reallocated to the second ETL task to meet its resource needs.

[0045] By calculating the difference between the currently available resources (first resources) and the resources required by the high-priority task to be executed (second ETL task), it is possible to accurately determine how many resources need to be released. This precise resource management avoids excessive or insufficient release of resources and ensures efficient use of resources. After determining the amount of resources that need to be released, the system does not blindly pause all low-priority tasks, but selectively pauses those tasks that occupy more resources than the difference (fifth ETL task). This targeted task pause strategy reduces unnecessary task interruptions and reduces the impact on the overall system performance. Through precise calculation and selective pauses, the system can ensure that only necessary resources are released, thereby reducing resource waste. These released resources can be immediately used to execute high-priority tasks, improving resource utilization efficiency. Because the system can quickly release enough resources to support the execution of high-priority tasks, the waiting time of these tasks is reduced. This helps to improve the overall efficiency of task execution and enable critical and urgent tasks to be completed faster. Through precise resource management and task pause strategies, the system can avoid task failures or system crashes due to insufficient resources. This stability is critical to maintaining the normal operation of the system and ensuring business continuity. Users can see the results of high-priority tasks more quickly because the system can quickly allocate sufficient resources to them. This quick response and efficient execution can improve user satisfaction and trust.

[0046] Optionally, the optimization method further includes: When the first resource is less than a threshold and a new ETL task is received, adding the new ETL task to a message queue; Determine the priority of the new ETL task, and arrange the new ETL task to the last one of the priority in the message queue according to the priority; When an ETL task is converted to a paused state, it is determined whether the ETL task currently ranked first in the message queue can be executed.

[0047] When the system detects that the currently available resources (first resources) are lower than the preset threshold, it indicates that the system is in a state of resource shortage. At this time, if new ETL tasks are received, the system will not try to execute these tasks immediately, because direct execution may further aggravate the resource shortage and even cause the system to crash. In order to manage these newly arrived tasks in an orderly manner, the system adds them to a message queue. The message queue is used to store tasks waiting to be executed. This mechanism allows the system to keep track of and manage these tasks without executing them immediately. When adding new ETL tasks to the message queue, the system determines the priority of these tasks. Then, according to the priority, the tasks are arranged at the end of the corresponding priority in the message queue, that is, they are arranged in order from high to low priority, and the same priority is arranged in the order of entering the message queue. This arrangement ensures that high-priority tasks can be executed first when resources are released. The system continuously monitors the status of the currently executing tasks. When an ETL task is converted to a paused state due to insufficient resources, it means that some resources have been released. At this time, the system needs to evaluate whether these released resources can be used to execute the tasks waiting in the message queue. The system checks the ETL task currently ranked first in the message queue (i.e., the next task to be executed). If the current system resources are sufficient to support the execution of the task, the system will respond quickly, wake up the task and allocate the necessary resources to enable it to continue execution. In this way, the system can efficiently utilize the released resources and reduce the execution delay of waiting tasks. By adding new tasks to the message queue and arranging their priorities, the system actually implements a flow restriction mechanism. This mechanism can control the entry rate of new tasks and prevent a large number of new tasks from flooding into the system in a short period of time and causing overload. At the same time, by dynamically monitoring the task status and waking up waiting tasks, the system can also achieve load balancing to ensure that system resources are used reasonably.

[0048] When system resources are tight, by adding newly arrived ETL tasks to the message queue instead of executing them immediately, the system can avoid task failure or system crash due to insufficient resources. This strategy ensures the effective use of system resources so that resources can be allocated to tasks at a more appropriate time. After determining the priority of the new ETL task, the system arranges it in the message queue in order of priority. This priority management mechanism ensures that high-priority tasks can be executed first when resources are released, thereby improving the execution efficiency and response speed of key tasks. By adding new tasks to the message queue, the system actually implements a flow control mechanism. This mechanism can limit the entry rate of new tasks and prevent a large number of new tasks from flooding into the system in a short period of time and causing overload. At the same time, the system achieves load balancing by dynamically monitoring task status and waking up waiting tasks, ensuring balanced allocation and efficient use of system resources. When an ETL task is converted to a paused state, the system will immediately determine and try to execute the task ranked first in the message queue. This immediate response mechanism reduces the execution delay of waiting tasks and improves the overall execution efficiency of the system. Through the above optimization methods, the system can better cope with resource constraints and avoid system crashes or task failures due to insufficient resources. This stability is critical to maintaining the normal operation of the system and ensuring business continuity.

[0049] Optionally, the optimization method further includes: Determine whether any resources are released by monitoring memory usage; When there are resources to be released, determining whether the first highest priority in the task waiting queue is greater than the second highest priority in the message queue; When the first highest priority in the task waiting queue is greater than the second highest priority in the message queue, the ETL task corresponding to the first highest priority is selected from the task waiting queue for awakening, and execution continues from the pause point of the ETL task.

[0050] The system continuously monitors memory usage, which is one of the important indicators for judging resource status. Through real-time monitoring, the system can promptly discover resources (such as memory, CPU, etc.) that are released or become available. When an ETL task enters a paused state due to insufficient resources, the resources it occupies (such as part of the memory, CPU time slice, etc.) will be released back to the system. By monitoring the changes in these resources, the system can accurately identify events in which resources are released. The task waiting queue is a queue specifically used to store ETL tasks that are forced to pause due to insufficient resources. These tasks have the opportunity to be reawakened and continue to execute after the resources are released. The message queue is used to store newly arrived but not yet executed ETL tasks, which are arranged in order of priority. When the system detects that resources are released, it compares the highest priority task in the task waiting queue (the first highest priority) with the highest priority task in the message queue (the second highest priority). The purpose of this comparison is to determine which queue's tasks should be executed first. If the first highest priority in the task waiting queue is greater than the second highest priority in the message queue, this means that there is a higher priority task waiting to be executed in the task waiting queue, and now there is an opportunity to execute it due to the release of resources. In this case, the system will select the ETL task corresponding to the first highest priority from the task waiting queue to wake up. The wake-up operation usually involves restoring the state of the task, allocating necessary resources, and resuming execution from the point where the task was paused. The awakened task will continue execution from where it was previously paused due to insufficient resources. This ensures the continuity of task execution and data consistency. If the first highest priority in the task waiting queue and the second highest priority in the message queue are the same, the task in the task waiting queue will be processed first.

[0051] By monitoring memory usage, the system can sense the release of resources in real time. Once the release of resources is detected, the system can respond immediately to evaluate whether there are waiting tasks to be executed. This real-time performance ensures that resources can be used as soon as possible and reduces the idle time of resources. After the resources are released, the system does not blindly wake up the tasks in the waiting queue, but first compares the priorities of the tasks in the task waiting queue and the message queue. This priority-driven scheduling strategy ensures that high-priority tasks can be executed first, thereby improving the execution efficiency and response speed of key tasks. When a high-priority ETL task is selected from the task waiting queue for wake-up, the system will continue to execute from the pause point of the task. This design avoids repeated execution of tasks and waste of resources, while improving the continuity and efficiency of task execution. Through the above optimization strategy, the system can wake up and execute high-priority tasks faster, thereby reducing the waiting time of these tasks. At the same time, since the system can use resources more effectively, it also reduces the overall load of the system and improves the stability and performance of the system.

[0052] Figure 2 is a schematic diagram of the implementation of the optimization method for ETL task scheduling disclosed in the embodiment of the present application, such as Figure 2 As shown, the created task is in the initialization state, and the resource monitoring service is used to check whether the resources are sufficient. When the resources are insufficient, the task enters the pause state. When resources are released, the paused task is checked and executed. If the resources are sufficient, the task (including extracting data / filtering data / converting data) is executed. After executing one stage of the task, the task is guided to enter the intermediate state, and the resource monitoring service is used to check whether the resources are sufficient. If not, the task is still guided to enter the pause state. If sufficient, it is determined whether there is a paused task with a higher priority. If there is a paused task with a higher priority, the paused task with a higher priority is executed. If there is no paused task with a higher priority, the current task is executed.

[0053] The embodiment of the present application not only tracks the usage of system resources (such as CPU, memory, etc.) in real time, but also deeply monitors the resource consumption of the upstream and downstream of the ETL project to obtain more comprehensive and accurate resource usage information. Through comprehensive resource perception, the system can more accurately grasp the resource requirements of the entire data processing process. When a high-priority task appears, the system will comprehensively consider the resource status of the current system, the priority of the task, and the resource usage of the upstream and downstream projects to make the best scheduling decision. In this way, the system can ensure that key tasks are processed in a timely manner while avoiding excessive pressure on the entire data processing process. In order to achieve more intelligent scheduling, the embodiment of the present application adopts a scheduling task optimization method based on state and pause time. The system sets intermediate state and pause time checks for scheduling tasks. During the pause time, the system will make intelligent decisions based on the current resource status, task priority, and the usage of context resources. If the system detects that there are insufficient resources and high-priority tasks appear, the system will give priority to these tasks and suspend or adjust the execution of low-priority tasks. In addition, in order to cope with the situation of insufficient resources, the embodiment of the present application also adopts a flow restriction strategy. When system resources are tight, new asynchronous tasks will be placed in the message queue for blocking processing to avoid system overload. This mechanism can effectively control the concurrency of tasks, ensuring the rational use of system resources and timely processing of tasks.

[0054] This embodiment also discloses an optimization system for ETL task scheduling. Figure 3 Schematic diagram of the module of the optimization system for ETL task scheduling disclosed in the embodiment of the present application. Figure 3 As shown, the system includes a partitioning module 301, a resource module 302, an execution module 303 and a scheduling module 304, wherein: A division module 301 is configured to divide the ETL task into multiple execution stages and set a pause time of a preset duration between each two execution stages; The resource module 302 is configured to obtain the first resource currently available to the platform and the second resource required for the next execution phase of the current ETL task when entering the pause time; An execution module 303, configured to execute the next execution phase of the current ETL task when the first resource is greater than or equal to the second resource; The scheduling module 304 is configured to, when the first resource is less than the second resource, convert the current ETL task into a pause state and set a pause point, add the current ETL task to a task waiting queue and execute an ETL task with a higher priority than the current ETL task or continuously monitor whether there is resource release.

[0055] Optionally, the scheduling module 304 is configured to: Determine whether there is an ETL task with a higher priority than the current ETL task; When there is a first ETL task with a higher priority than the current ETL task, select a second ETL task with the highest priority from the first ETL task, and when the first resource can meet the resource requirement of the second ETL task, execute the second ETL task; When there is no first ETL task with a higher priority than the current ETL task, whether there is resource release is continuously monitored, and when resource release is detected, whether the third resource currently available to the platform is greater than the second resource is determined.

[0056] Optionally, the scheduling module 304 is further configured to: When the first resource cannot meet the resource demand of the second ETL task, determining whether there is a third ETL task with a lower priority than the second ETL task among the ETL tasks currently being asynchronously executed; When there is a third ETL task with a lower priority than the second ETL task among the ETL tasks currently being asynchronously executed, a fourth ETL task with the lowest priority is selected from the third ETL tasks, and the fourth ETL task is set to a pause state to release resources occupied by the fourth ETL task.

[0057] Optionally, the scheduling module 304 is further configured to: When there is a third ETL task whose priority is lower than that of the second ETL task among the ETL tasks currently being asynchronously executed, the difference between the resource requirements of the first resource and the second ETL task is calculated, and it is determined whether there is an ETL task among the third ETL tasks that occupies resources greater than the difference; When there is a fifth ETL task in the third ETL tasks and the resources occupied are greater than the difference, the fifth ETL task is converted into a pause state to release the resources occupied by the fifth ETL task.

[0058] Optionally, the optimization system further includes a queue module, and the queue module is configured to: When the first resource is less than a threshold and a new ETL task is received, adding the new ETL task to a message queue; Determine the priority of the new ETL task, and arrange the new ETL task to the last one of the priority in the message queue according to the priority; When an ETL task is converted to a paused state, it is determined whether the ETL task currently ranked first in the message queue can be executed.

[0059] Optionally, the optimization system further includes a judgment module, and the judgment module is configured to: Determine whether any resources are released by monitoring memory usage; When there are resources to be released, determining whether the first highest priority in the task waiting queue is greater than the second highest priority in the message queue; When the first highest priority in the task waiting queue is greater than the second highest priority in the message queue, the ETL task corresponding to the first highest priority is selected from the task waiting queue for awakening, and execution continues from the pause point of the ETL task.

[0060] Optionally, the division module 301 is configured to: Evaluate the complexity of ETL tasks, including the amount of data processing and the complexity of conversion logic, and predict the amount of resources required for each execution stage based on the evaluation results; The initial pause time is calculated according to the recovery speed of the system resources and the required resource amount, and the initial pause duration is adjusted according to influencing factors to obtain the pause time. The influencing factors include network delay and external data source update frequency.

[0061] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0062] This embodiment also discloses an electronic device, referring to Figure 4 The electronic device may include: at least one processor 401 , at least one communication bus 402 , a user interface 403 , a network interface 404 , and at least one memory 405 .

[0063] The communication bus 402 is used to realize the connection and communication between these components.

[0064] The user interface 403 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.

[0065] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0066] Among them, the processor 401 may include one or more processing cores. The processor 401 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 405, and calling data stored in the memory 405. Optionally, the processor 401 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 401 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 401, and it can be implemented separately through a chip.

[0067] Among them, the memory 405 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 405 includes a non-transitory computer-readable storage medium. The memory 405 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 405 may also be optionally at least one storage device located away from the aforementioned processor 401. As Figure 4 As shown, the memory 405 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of an optimization method for ETL task scheduling.

[0068] exist Figure 4 In the electronic device shown, the user interface 403 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 401 can be used to call the application program of the optimization method for ETL task scheduling stored in the memory 405. When executed by one or more processors 401, the electronic device executes one or more methods in the above-mentioned embodiments.

[0069] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0070] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0074] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory 405, including a number of instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory 405 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0075] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. An optimization method for ETL task scheduling, characterized in that: Applied to the task scheduling platform, the optimization method includes: Divide the ETL task into multiple execution stages and set a preset pause time between each execution stage; When entering the pause time, obtaining a first resource currently available on the platform and a second resource required for the next execution phase of the current ETL task; When the first resource is greater than or equal to the second resource, executing the next execution phase of the current ETL task; When the first resource is less than the second resource, the current ETL task is converted to a pause state and a pause point is set, the current ETL task is added to a task waiting queue and an ETL task with a higher priority than the current ETL task is executed or whether there is resource release is continuously monitored.

2. The optimization method for ETL task scheduling according to claim 1, characterized in that: The executing of an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release includes: Determine whether there is an ETL task with a higher priority than the current ETL task; When there is a first ETL task with a higher priority than the current ETL task, select a second ETL task with the highest priority from the first ETL task, and when the first resource can meet the resource requirement of the second ETL task, execute the second ETL task; When there is no first ETL task with a higher priority than the current ETL task, whether there is resource release is continuously monitored, and when resource release is detected, whether the third resource currently available to the platform is greater than the second resource is determined.

3. The optimization method for ETL task scheduling according to claim 2, characterized in that: The executing of an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release further includes: When the first resource cannot meet the resource requirement of the second ETL task, determining whether there is a third ETL task with a lower priority than the second ETL task among the ETL tasks currently being asynchronously executed; When there is a third ETL task with a lower priority than the second ETL task among the ETL tasks currently being asynchronously executed, a fourth ETL task with the lowest priority is selected from the third ETL tasks, and the fourth ETL task is set to a pause state to release resources occupied by the fourth ETL task.

4. The optimization method for ETL task scheduling according to claim 2, characterized in that: The executing of an ETL task with a higher priority than the current ETL task or continuously monitoring whether there is resource release further includes: When there is a third ETL task whose priority is lower than that of the second ETL task among the ETL tasks currently being asynchronously executed, the difference between the resource requirements of the first resource and the second ETL task is calculated, and it is determined whether there is an ETL task among the third ETL tasks that occupies resources greater than the difference; When there is a fifth ETL task in the third ETL task and the resources occupied are greater than the difference, the fifth ETL task is converted into a pause state to release the resources occupied by the fifth ETL task.

5. The optimization method for ETL task scheduling according to claim 1, characterized in that: The optimization method further comprises: When the first resource is less than a threshold and a new ETL task is received, adding the new ETL task to a message queue; Determine the priority of the new ETL task, and arrange the new ETL task to the last one of the priority in the message queue according to the priority; When an ETL task is converted to a paused state, it is determined whether the ETL task currently ranked first in the message queue can be executed.

6. The optimization method for ETL task scheduling according to claim 5, characterized in that: The optimization method further comprises: Determine whether any resources are released by monitoring memory usage; When there are resources to be released, determining whether the first highest priority in the task waiting queue is greater than the second highest priority in the message queue; When the first highest priority in the task waiting queue is greater than the second highest priority in the message queue, the ETL task corresponding to the first highest priority is selected from the task waiting queue for awakening, and execution continues from the pause point of the ETL task.

7. The optimization method for ETL task scheduling according to claim 1, characterized in that: The pause time of a preset duration is set between each two execution stages, including: Evaluate the complexity of ETL tasks, including the amount of data processing and the complexity of conversion logic, and predict the amount of resources required for each execution stage based on the evaluation results; The initial pause time is calculated according to the recovery speed of the system resources and the required resource amount, and the initial pause duration is adjusted according to influencing factors to obtain the pause time. The influencing factors include network delay and external data source update frequency.

8. An optimization system for ETL task scheduling, characterized in that: It includes a partitioning module, a resource module, an execution module, and a scheduling module, among which: A partitioning module is configured to divide the ETL task into multiple execution stages and set a preset pause time between each execution stage; A resource module configured to obtain, when entering the pause time, a first resource currently available to the platform and a second resource required for a next execution phase of the current ETL task; an execution module, configured to execute a next execution phase of the current ETL task when the first resource is greater than or equal to the second resource; The scheduling module is configured to, when the first resource is less than the second resource, convert the current ETL task into a pause state and set a pause point, add the current ETL task to a task waiting queue and execute an ETL task with a higher priority than the current ETL task or continuously monitor whether there is resource release.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images