Task submission optimization method and system

Abstract

The invention relates to the field of task processing optimization, in particular to a task submission optimization method and system, and the method comprises the steps: receiving an external task, and judging the interval between the execution time of the task and the current time; if the interval is greater than a first preset threshold value, writing the task into a persistent storage unit; if the interval is smaller than a first preset threshold value, directly writing into an asynchronous queue; task polling is independently and regularly carried out in a persistent storage unit, and when the execution time of a task is close to the current time, the task is migrated to an asynchronous queue; the asynchronous queue is divided into a fast queue and a slow queue, the fast queue is preferentially written into a task queue for processing, and then tasks in the slow queue are written into the task queue for processing; according to the method, the resource state is monitored, the queue write-in rate is dynamically adjusted, and the task execution efficiency is optimized, so that the system load is reduced and the execution efficiency is improved by dynamically adjusting the task write-in strategy on the premise of not changing the resource and task load.

Description

Technical Field

[0001] The present invention relates to the field of task processing optimization, and in particular to a task submission optimization method and system thereof. Background Art

[0002] Current Internet systems are plagued by a variety of scheduled tasks. To support this massive volume, a general-purpose distributed task scheduling system or centralized task scheduling system is typically developed. Its primary function is to collect and record various scheduled tasks and trigger them through an internal polling mechanism. With the increasing number of external business systems, business complexity is also increasing, leading to an increase in the number of scheduled tasks. This massive volume of scheduled tasks also puts pressure on distributed scheduling.

[0003] Traditional methods alleviate pressure by increasing the resources of the scheduling system to handle more tasks or reducing the amount of task input, but they cannot take into account both task execution efficiency and cost.

[0004] Therefore, an optimization method is urgently needed to reduce system load and improve execution efficiency by dynamically adjusting task writing strategies without changing resources and task volume. Summary of the Invention

[0005] In order to overcome the difficulty in the existing technology of increasing the resources of the scheduling system to process more tasks or reducing the task input volume to relieve pressure, but failing to take into account both task execution efficiency and cost, the purpose of the present invention is to provide a task submission optimization method and system thereof, which can reduce the system load and improve execution efficiency by dynamically adjusting the task writing strategy without changing the resources and task volume.

[0006] The present invention is implemented by the following scheme:

[0007] A method for optimizing task submission, the method steps are as follows:

[0008] Step 1: Receive external tasks and determine the interval between the task execution time and the current time;

[0009] Step 2: If the interval between the task execution time and the current time is greater than a first preset threshold, the task is written to the persistent storage unit; if the interval between the task execution time and the current time is less than the first preset threshold, the task is directly written to the asynchronous queue;

[0010] Step 3: By independently polling tasks in the persistent storage unit at regular intervals, when the execution time of a task approaches the current time, it is migrated to the asynchronous queue;

[0011] Step 4: Divide the asynchronous queue into a fast queue and a slow queue, set a second preset threshold, and if the interval between the execution time of the current task and the current time is greater than the second preset threshold, write the current task into the slow queue; if the interval between the execution time of the current task and the current time is less than the second preset threshold, write the current task into the fast queue, give priority to writing the fast queue into the task queue for processing, and then write the tasks in the slow queue into the task queue for processing;

[0012] Step 5: Monitor resource status, dynamically adjust the queue write rate, and optimize task execution efficiency.

[0013] Furthermore, the persistent storage unit is a database.

[0014] Furthermore, the asynchronous queue is the message middleware MQ.

[0015] Furthermore, the dynamic adjustment of the queue write rate in step 5 includes: increasing the fast queue consumption rate when resources are sufficient, and reducing the slow queue write rate when resources are scarce.

[0016] A task submission optimization system, comprising: a diversion module, a migration module, a queue management module, a dynamic control module and a task execution optimization module;

[0017] The diversion module is used to receive external tasks and determine the interval between the execution time of the task and the current time;

[0018] The migration module is used to write the task into the persistent storage unit if the interval between the execution time of the task and the current time is greater than a first preset threshold; if the interval between the execution time of the task and the current time is less than the first preset threshold, directly write the task into the asynchronous queue;

[0019] The queue management module is used to independently poll tasks in the persistent storage unit at regular intervals, and when the execution time of a task is close to the current time, it is migrated to the asynchronous queue;

[0020] The dynamic control module is used to divide the asynchronous queue into a fast queue and a slow queue, set a second preset threshold, and if the interval between the execution time of the current task and the current time is greater than the second preset threshold, the current task is written into the slow queue; if the interval between the execution time of the current task and the current time is less than the second preset threshold, the current task is written into the fast queue, and the fast queue is preferentially written into the task queue for processing, and then the tasks in the slow queue are written into the task queue for processing;

[0021] The task execution optimization module is used to monitor resource status, dynamically adjust queue write rate, and optimize task execution efficiency.

[0022] Furthermore, the persistent storage unit is a database.

[0023] Furthermore, the asynchronous queue is the message middleware MQ.

[0024] Furthermore, the dynamic adjustment of the queue writing rate in the dynamic control module includes: increasing the fast queue consumption rate when resources are sufficient, and reducing the slow queue writing rate when resources are scarce.

[0025] The beneficial effects of the present invention are:

[0026] The present invention provides a task submission optimization method and system, which can reduce the system polling pressure without increasing resources or reducing the task volume; improve task execution efficiency and resource utilization by dynamically adjusting strategies; and is suitable for distributed scenarios with high concurrency and massive scheduled tasks, and has strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 It is a structural block diagram of the system of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the accompanying drawings.

[0030] See also Figure 1 , a task submission optimization method, the method steps are as follows:

[0031] Step 1: Receive external tasks and determine the interval between the task execution time and the current time;

[0032] Step 2: If the interval between the task execution time and the current time is greater than a first preset threshold, the task is written to the persistent storage unit; if the interval between the task execution time and the current time is less than the first preset threshold, the task is directly written to the asynchronous queue;

[0033] Step 3: By independently polling tasks in the persistent storage unit at regular intervals, when the execution time of a task approaches the current time, it is migrated to the asynchronous queue;

[0034] Step 4: Divide the asynchronous queue into a fast queue and a slow queue, set a second preset threshold, and if the interval between the execution time of the current task and the current time is greater than the second preset threshold, write the current task into the slow queue; if the interval between the execution time of the current task and the current time is less than the second preset threshold, write the current task into the fast queue, give priority to writing the fast queue into the task queue for processing, and then write the tasks in the slow queue into the task queue for processing;

[0035] Step 5: Monitor resource status, dynamically adjust the queue write rate, and optimize task execution efficiency.

[0036] The present invention will be further described below with reference to a specific embodiment:

[0037] A method for optimizing task submission, comprising the following steps:

[0038] Step 1: When an external task arrives, determine the time interval between the task's execution time and the current time interval.

[0039] Step 2: If the interval is greater than 2 hours, the task is stored in the database; if the interval is less than or equal to 2 hours, it is directly written to the message middleware MQ.

[0040] Step 3: The tasks in the database are polled every 5 minutes by an independent scheduled task. If the difference between the task execution time and the current time is ≤ 2 hours, the task is migrated to the message middleware MQ.

[0041] Step 4: The message middleware MQ divides the queues according to the task delay time: if the interval between the execution time of the current task and the current time is less than 1 hour, the task enters the fast queue; if the interval between the execution time of the current task and the current time is ≥ 1 hour, the task enters the slow queue. Tasks in the fast queue are written into the task queue for processing first, and then tasks in the slow queue are written into the task queue for processing.

[0042] The task queue is the actual queue used for task execution. For example, the task queue can be divided into 60 grids, one grid per second, and each grid is filled with the number of tasks to be processed in that second. This method greatly reduces the amount of task data in each grid compared to the original method, improving efficiency.

[0043] Step 5: Monitor current resource usage, such as CPU and memory usage. When resource usage is less than 70%, schedule existing resources to speed up task queue consumption. When resource usage is ≥ 70%, limit the write rate of slow queues.

[0044] Through the above approach, the server can still efficiently process high-priority tasks when resources are limited, while avoiding task accumulation.

[0045] See also Figure 2 , a task submission optimization system, the system includes: a diversion module, a migration module, a queue management module, a dynamic control module and a task execution optimization module;

[0046] The diversion module is used to receive external tasks and determine the interval between the execution time of the task and the current time;

[0047] The migration module is used to write the task into the persistent storage unit if the interval between the execution time of the task and the current time is greater than a first preset threshold; if the interval between the execution time of the task and the current time is less than the first preset threshold, directly write the task into the asynchronous queue;

[0048] The queue management module is used to independently poll tasks in the persistent storage unit at regular intervals, and when the execution time of a task is close to the current time, it is migrated to the asynchronous queue;

[0049] The dynamic control module is used to divide the asynchronous queue into a fast queue and a slow queue, set a second preset threshold, and if the interval between the execution time of the current task and the current time is greater than the second preset threshold, the current task is written into the slow queue; if the interval between the execution time of the current task and the current time is less than the second preset threshold, the current task is written into the fast queue, and the fast queue is preferentially written into the task queue for processing, and then the tasks in the slow queue are written into the task queue for processing;

[0050] The task execution optimization module is used to monitor resource status, dynamically adjust queue write rate, and optimize task execution efficiency.

[0051] In one embodiment of the present invention, the persistent storage unit is a database.

[0052] In one embodiment of the present invention, the asynchronous queue is a message middleware MQ.

[0053] In one embodiment of the present invention, the dynamic adjustment of queue write rates in the dynamic control module includes increasing the consumption rate of the fast queue when resources are sufficient and reducing the write rate of the slow queue when resources are scarce. This is achieved by monitoring resource conditions in real time and writing them to a unified cache. By querying resource conditions within a certain time range and setting a threshold, when resource utilization is less than the threshold, the number of consumers can be increased, thereby increasing the consumption rate; otherwise, the consumption rate can be reduced.

[0054] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A task submission optimization method, characterized in that: The method steps are as follows: Step 1: Receive external tasks and determine the interval between the task execution time and the current time; Step 2: If the interval between the task execution time and the current time is greater than a first preset threshold, the task is written to the persistent storage unit; if the interval between the task execution time and the current time is less than the first preset threshold, the task is directly written to the asynchronous queue; Step 3: By independently polling tasks in the persistent storage unit at regular intervals, when the execution time of a task approaches the current time, it is migrated to the asynchronous queue; Step 4: Divide the asynchronous queue into a fast queue and a slow queue, set a second preset threshold, and if the interval between the execution time of the current task and the current time is greater than the second preset threshold, write the current task into the slow queue; if the interval between the execution time of the current task and the current time is less than the second preset threshold, write the current task into the fast queue, give priority to writing the fast queue into the task queue for processing, and then write the tasks in the slow queue into the task queue for processing; Step 5: Monitor resource status, dynamically adjust the queue write rate, and optimize task execution efficiency.

2. The task submission optimization method according to claim 1, characterized in that: The persistent storage unit is a database.

3. The task submission optimization method according to claim 1, characterized in that: The asynchronous queue is the message middleware MQ.

4. The method for optimizing task submission according to claim 1, wherein: The dynamic adjustment of the queue write rate in step 5 includes: increasing the fast queue consumption rate when resources are sufficient, and reducing the slow queue write rate when resources are scarce.

5. A task submission optimization system, characterized in that: The system includes: a diversion module, a migration module, a queue management module, a dynamic control module and a task execution optimization module; The diversion module is used to receive external tasks and determine the interval between the execution time of the task and the current time; The migration module is used to write the task into the persistent storage unit if the interval between the execution time of the task and the current time is greater than a first preset threshold; if the interval between the execution time of the task and the current time is less than the first preset threshold, directly write the task into the asynchronous queue; The queue management module is used to independently poll tasks in the persistent storage unit at regular intervals, and when the execution time of a task is close to the current time, it is migrated to the asynchronous queue; The dynamic control module is used to divide the asynchronous queue into a fast queue and a slow queue, set a second preset threshold, and if the interval between the execution time of the current task and the current time is greater than the second preset threshold, the current task is written into the slow queue; if the interval between the execution time of the current task and the current time is less than the second preset threshold, the current task is written into the fast queue, and the fast queue is preferentially written into the task queue for processing, and then the tasks in the slow queue are written into the task queue for processing; The task execution optimization module is used to monitor resource status, dynamically adjust queue write rate, and optimize task execution efficiency.

6. The task submission optimization system according to claim 5, characterized in that: The persistent storage unit is a database.

7. The task submission optimization system according to claim 5, characterized in that: The asynchronous queue is the message middleware MQ.

8. The task submission optimization system according to claim 5, characterized in that: The dynamic adjustment of the queue writing rate in the dynamic control module includes: increasing the fast queue consumption rate when resources are sufficient, and reducing the slow queue writing rate when resources are scarce.

Images