A task scheduling method with variable partitioning strategy

Abstract

The present invention relates to a task scheduling method with a variable table partitioning strategy. The method includes three table partitioning strategies, which are respectively used to cope with task scheduling requirements of different business data volumes and concurrency volumes; according to the actual requirements of the target task, the business data volume and concurrency volume are estimated, and the corresponding strategy is selected for deployment. The task scheduling method with a variable table partitioning strategy of the present invention can achieve 0-100,000-level tasks, 0-100 QPS concurrency per second; 100-500 QPS concurrency per second for 10 million-level tasks; and 500 QPS or more concurrency per second for 10 million-level tasks by adopting three table partitioning strategies, which can meet the project application requirements of 100,000-level, 1 million-level, and 10 million-level data volumes and concurrency pressures. At the same time, the method of the present invention also has the advantages of low resource occupancy, high resource utilization rate, and low maintenance cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of database and table partitioning methods of task scheduling systems, and in particular relates to a task scheduling method with a variable table partitioning strategy. Background Art

[0002] In order to solve the performance bottleneck problem of a single database or table in the scenario of high concurrency and massive stock data, the current task scheduling system will use the strategy of sharding databases and tables, for example, splitting the existing database according to different businesses, or horizontally splitting the business table according to specified indicators, etc. However, the current sharding database and table strategies still have many shortcomings. For example, once the technical solution is determined, it is difficult to adjust the sharding solution; overly simple sharding database and table strategies cannot effectively cope with the use scenarios of massive data; overly complex sharding database and table solutions, if used in small data scenarios, will have problems such as excessive resource usage (excessive consumption of CPU, memory and other resources), high usage and maintenance costs, etc. Summary of the invention

[0003] In order to solve the above-mentioned problems existing in the existing database and table sharding strategies, and to meet the task scheduling requirements of different scenarios with the smallest possible resource occupation (such as 0-100,000-level tasks, 0-100QPS concurrency per second; 1-million-level tasks, 100-500QPS concurrency per second; 10-million-level tasks, more than 500QPS concurrency per second), the present invention proposes a new task scheduling method with a variable table sharding strategy.

[0004] Explanation of terms

[0005] Task scheduling system: A task scheduling system is a system used to automatically manage task execution. It can automatically execute a series of tasks at a specific time or event trigger. Common task scheduling systems are as follows: Figure 1 shown.

[0006] The task scheduling system is generally composed of WEB / API, task management, instance management, instance executor, and business library.

[0007] WEB / API module: the system's product interface function module and the interface module exposed by the API; responsible for operations such as adding, deleting, modifying, and checking tasks or instances.

[0008] Task management module: registers the tasks sent by the WEB / API module and stores the task data in the task table of the database. Performs a scheduled scan on the data in the task table and can trigger the task on time according to the task's timing parameters.

[0009] Instance management module: Receives task trigger requests sent by the task management module and starts the life cycle of the instance. The life cycle generally consists of triggering, startup verification, execution, and termination. All business data of the life cycle is recorded in the instance table of the database. Each step in the life cycle corresponds to the addition, deletion, modification, and query of instance table data.

[0010] Instance executor module: responsible for executing instances in the execution phase of the instance life cycle and running the business included in the actual task, such as: sql, spark, shell scripts, etc.

[0011] Business database: responsible for storing the business data of the task scheduling system, including tasks, instances, etc. The database types used vary according to the actual scenario, such as MySQL, Oracle, and DAMO.

[0012] Instance life cycle: An instance is an object in each cycle corresponding to a periodic task, and each instance has its own life cycle, including triggering, startup verification, execution, and termination steps.

[0013] In order to effectively solve the database performance bottleneck of the task scheduling system in various concurrency and data volume scenarios, the present invention proposes a solution of variable table partitioning strategy.

[0014] Specifically, the present invention provides a task scheduling method with a variable table partitioning strategy, such as Figure 5 As shown, this method includes three table partitioning strategies: strategy 1, strategy 2, and strategy 3. The three table partitioning strategies are respectively used to cope with task scheduling requirements with different business data volumes and concurrency volumes. According to the actual requirements of the target task, its business data volume and concurrency volume are estimated, and the corresponding strategy is selected for deployment; wherein:

[0015] The life cycle of a task instance in strategy 1 includes four stages: triggering, start verification, execution, and termination. The four stages flow in sequence. When a task instance is triggered, an instance table is created. The status of the instance at each stage of the life cycle needs to be updated. All instance data is stored in the same instance table.

[0016] The life cycle of a task instance in strategy 2 includes four stages: triggering, startup verification, execution, and termination. The four stages flow in sequence. When a task instance is triggered, an instance table is created. The instance table is partitioned according to the creation time. All logic that interacts with the instance table is partitioned according to the creation time, and the instance table of the corresponding interval is operated.

[0017] The life cycle of a task instance in strategy 3 includes four stages: triggering, start-up verification, execution, and end. The four stages flow in sequence. When a task instance is triggered, four instance tables are created and instance data with the same primary key are inserted into the four instance tables, namely, the triggering stage instance table, the start-up verification stage instance table, the execution stage instance table, and the end stage instance table. Among them, the triggering stage instance table, the start-up verification stage instance table, and the execution stage instance table are created directly, and the end stage instance table is partitioned according to the creation time. In the triggering, start-up verification, and execution stages, the status of the instance in the stage is updated in the instance table corresponding to the stage, and the instance data in the instance table corresponding to the previous stage is deleted when the life cycle of the task instance flows to the next stage. In the end stage, the status of the instance in the stage is updated in the end stage instance table according to the creation time partition.

[0018] Furthermore, in the task scheduling method of the variable table partitioning strategy of the present invention, the three table partitioning strategies can be dynamically upgraded in sequence, from strategy 1 to strategy 2, and from strategy 2 to strategy 3, according to the business data volume and concurrency volume of the target task.

[0019] Furthermore, among the three table partitioning strategies described in the task scheduling method of the variable table partitioning strategy of the present invention:

[0020] Strategy 1 is used to meet the task scheduling requirements of business data volume of 100,000 and / or concurrent volume of 0-100 QPS;

[0021] Strategy 2 is used to meet the task scheduling requirements of business data volume of millions and / or concurrent volume of 100-500QPS;

[0022] Strategy 3 is used to meet the task scheduling requirements with business data volume of tens of millions and / or concurrent volume of more than 500 QPS.

[0023] Furthermore, the strategy 1 in the task scheduling method of the variable table partitioning strategy of the present invention includes:

[0024] S11. Triggering stage: triggering the task instance and creating an instance table, setting and updating the status of the instance in this stage;

[0025] S12. Startup verification phase: determine the startup of the instance, verify whether the instance can be executed, and update the status of the instance in this phase;

[0026] S13. Execution phase: responsible for the execution of the instance and updating the status of the instance in this phase;

[0027] S14. End stage: Update the status of the stage instance and trigger downstream tasks.

[0028] Furthermore, the strategy 2 in the task scheduling method of the variable table strategy of the present invention includes:

[0029] S21. Triggering stage: triggering the task instance and creating an instance table, partitioning the instance table according to the creation time, and updating the status of the instance in this stage according to the creation time partition;

[0030] S22. Startup verification phase: determine the startup of the instance, verify whether the instance can be executed, and update the status of the instance in this phase according to the creation time partition;

[0031] S23. Execution phase: responsible for the execution of the instance and updates the status of the instance in this phase according to the creation time partition;

[0032] S24. End stage: Update the status of the instance in this stage according to the creation time partition and trigger downstream tasks.

[0033] Furthermore, the task scheduling method of the variable table partitioning strategy of the present invention, as described in S21, partitions the instance table according to the creation time, and updates the state of the instance at this stage according to the creation time partition, wherein the granularity of the creation time partition is configured according to the actual data increment (including but not limited to days, weeks, months, etc.) to ensure that the data volume of a single table is between 5 and 10 million.

[0034] Furthermore, the strategy 3 in the task scheduling method of the variable table strategy of the present invention includes:

[0035] S31. Triggering stage: trigger the task instance and create four instance tables, namely, the triggering stage instance table, the startup verification stage instance table, the execution stage instance table, and the end stage instance table. Insert instance data with the same primary key into the four instance tables, and divide the end stage instance table according to the creation time; update the status of the triggering stage instance in the triggering stage instance table, and delete the instance data in the triggering stage instance table when the task instance life cycle flows to the startup verification stage;

[0036] S32. Startup verification phase: judge the startup of the instance and verify whether the instance can be executed; update the status of the startup verification phase instance in the startup verification phase instance table, and delete the instance data in the startup verification phase instance table when the task instance life cycle flows to the execution phase;

[0037] S33. Execution phase: responsible for instance execution; updating the status of the execution phase instance in the execution phase instance table, and deleting the instance data in the execution phase instance table when the task instance life cycle flows to the end phase;

[0038] S34. Ending stage: Update the status of the ending stage instance in the ending stage instance table according to the creation time partition, and trigger the downstream task.

[0039] Furthermore, among the three table partitioning strategies described in the task scheduling method of the variable table partitioning strategy of the present invention:

[0040] The running configuration of strategy 1 is 8 cores and 16 GB for the application service and 4 cores and 8 GB for the database.

[0041] The running configuration of strategy 2 is 16 cores and 32 GB for the application service and 4 cores and 8 GB for the database.

[0042] The running configuration of strategy 3 is 16 cores and 64 GB for the application service on a single machine and 8 cores and 16 GB for the database on a single machine.

[0043] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the task scheduling method of the variable partitioning strategy are implemented.

[0044] In summary, the task scheduling method of the variable table partitioning strategy of the present invention can achieve 0-100,000-level tasks with 0-100QPS concurrency per second; 1-million-level tasks with 100-500QPS concurrency per second; and 10-million-level tasks with more than 500QPS concurrency per second by adopting three table partitioning strategies. It can meet the project application requirements of 100,000-level, 1-million-level, and 10-million-level data volumes and concurrency pressures. At the same time, the method of the present invention also has the advantages of lower resource occupancy, higher resource utilization rate, and lower maintenance cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is the basic architecture diagram of the existing task scheduling system.

[0047] Figure 2 The figure is a flow chart for implementing the table partitioning strategy 1 in the method of the present invention.

[0048] Figure 3 This is a flow chart for implementing the table partitioning strategy 2 in the method of the present invention.

[0049] Figure 4 This is a flow chart for implementing the table partitioning strategy 3 in the method of the present invention.

[0050] Figure 5 The figure is a schematic diagram of the overall implementation process of the method of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. The present invention can also be implemented or applied through other different specific implementation methods. The details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0052] At the same time, it should be understood that the protection scope of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the protection scope of the present invention.

[0053] Embodiment: A task scheduling method with variable table partitioning strategy

[0054] The present invention provides three table partitioning strategies to cope with the needs of database and table partitioning in three scenarios: 100,000-level tasks, 1 million-level tasks, and 10 million-level tasks. According to the actual needs of the project, estimate its concurrency requirements and business data volume, and select the corresponding strategy for deployment. In addition, strategy 1 can be dynamically upgraded to strategies 2 and 3.

[0055] Strategy 1: To cope with 100,000-level tasks and 0-100 QPS concurrent scenarios, the specific implementation plan is as follows Figure 2 shown.

[0056] When a task instance is triggered, the data of the instance table is created, and the instance status of the triggering phase is set and updated. After the trigger is successful, the instance life cycle will flow to the startup verification phase, which is responsible for judging the startup of the instance and verifying whether the instance can be executed; and updating the instance status of this phase. After the startup verification is successful, the instance life cycle will flow to the execution phase, which is responsible for the execution of the instance and the update of the instance status in this phase. After the execution is completed, the instance life cycle flows to the end phase, which is responsible for updating the instance status and triggering downstream tasks. Among them, if any stage sends an exception or interruption, the instance life cycle will end. It can be seen that this strategy does not split the database business table. All instance data is in the same instance table. Strategy 1 can generally support the task scheduling requirements of low-concurrency scenarios with hundreds of thousands of tasks under the condition of small resource occupation. Strategy 1 has a low operation and maintenance cost. The recommended configuration is 8 cores and 16G for a single application service machine and 4 cores and 8G for a single database machine. In the application scenario of strategy 1, when the business concurrency, stock data, and concurrency requirements have changed substantially, that is, the stock tasks exceed 100,000 and the system's continuous concurrent task requirements exceed 100QPS, then the table sharding strategy 1 can be changed to strategy 2 through the configuration file to complete the smooth upgrade of the system's table sharding strategy to meet the corresponding business needs.

[0057] Strategy 2: To cope with millions of tasks and 100-500 QPS concurrent scenarios, the specific implementation plan is as follows Figure 3 shown.

[0058] There is no difference in the life cycle flow between Strategy 2 and Strategy 1, except for the interaction with the business database table. The main difference is that the instance table is partitioned according to the creation time. All logic that interacts with the instance table needs to be partitioned according to the creation time and operate the instance table in the corresponding interval. The granularity of the instance table partition by time can be configured according to the actual data increment, such as day, week, month, etc., to ensure that the data volume of a table is between 5 million and 10 million. Strategy 2 can generally complete millions of tasks under medium resource occupation and support task scheduling requirements in medium concurrency scenarios (100-500QPS). The recommended system configuration is 16 cores and 32G for a single application service machine and 4 cores and 8G for a single database machine. The related maintenance cost is slightly higher than Strategy 1, but still remains at a low level. In the application scenario of strategy 2, when the business concurrency, stock data, and concurrency requirements have changed substantially, that is, the stock tasks exceed one million and the system's continuous concurrent task requirements exceed 500QPS, then the table sharding strategy 2 can be changed to strategy 3 through the configuration file to complete the smooth upgrade of the system's table sharding strategy to meet the corresponding business needs.

[0059] Strategy 3: To cope with tens of millions of tasks and more than 500 QPS of concurrency, the specific implementation plan is as follows Figure 4 shown.

[0060] Strategy 3 is an improved version of Strategy 2. Based on Strategy 2, Strategy 3 divides the process flow involved in the life cycle of the instance into cold and hot data. Triggering, startup verification, and execution are divided into hot data, and the end stage is divided into cold data. Because the first three stages of the instance will be quickly converted or interrupted, only when the execution of the actual business of the instance is uncontrollable (such as the time to extract 1 million or 10 million data), it will flow to the end state after the execution is completed. It can be expected that according to the concurrency of 1000 per second, the instance state will not accumulate too much in the triggering and startup verification stages, but will only accumulate periodically in the execution state (because the execution time is uncontrollable), and finally accumulate in the end state. Then according to this logic, the instance table is split into the four stages of triggering, startup verification, execution, and end. In this way, the table data of the triggering, startup verification, and execution stage operations will be relatively small, which can reduce the time consumption of the instance life cycle and database interaction; while the data volume of the instance end table does not change, but the instance end table can be divided by creation time to reduce the data volume of each partition table. To handle the changes in the table partitioning scheme, the data flow process between the four tables has also changed. At the beginning of the trigger, instance data with the same primary key will be inserted into the four tables at the same time, but the changes in the instance status of each stage will be limited to the corresponding table. When the instance life cycle flows to the next stage, the instance data in the instance table corresponding to the previous stage is deleted. Strategy 3 can generally complete tens of millions of tasks under high resource usage, and support task scheduling requirements in high concurrency scenarios (more than 500QPS). The recommended system configuration is 16 cores and 64G for a single application service machine and 8 cores and 16G for a single database machine.

[0061] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession can make some changes or modifications to the technical contents disclosed above without departing from the scope of the technical solution of the present invention to obtain equivalent embodiments of equivalent changes. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A task scheduling method with a variable table partitioning strategy, characterized in that: The method includes three table partitioning strategies: strategy 1, strategy 2, and strategy 3. The three table partitioning strategies are respectively used to cope with task scheduling requirements with different business data volumes and concurrency volumes. According to the actual requirements of the target task, its business data volume and concurrency volume are estimated, and the corresponding strategy is selected for deployment. Among the three table partitioning strategies: Strategy 1 is used to meet the task scheduling requirements of business data volume of 100,000 and / or concurrent volume of 0-100 QPS; Strategy 2 is used to meet the task scheduling requirements of business data volume of millions and / or concurrent volume of 100-500QPS; Strategy 3 is used to meet the task scheduling requirements with business data volume of tens of millions and / or concurrent volume of more than 500 QPS; in: The life cycle of a task instance in strategy 1 includes four stages: triggering, start verification, execution, and termination. The four stages flow in sequence. When a task instance is triggered, an instance table is created. The status of the instance at each stage of the life cycle needs to be updated. All instance data is stored in the same instance table. The life cycle of a task instance in strategy 2 includes four stages: triggering, startup verification, execution, and termination. The four stages flow in sequence. When a task instance is triggered, an instance table is created. The instance table is partitioned according to the creation time. All logic that interacts with the instance table is partitioned according to the creation time, and the instance table of the corresponding interval is operated. The life cycle of a task instance in strategy 3 includes four stages: triggering, start-up verification, execution, and end. The four stages flow in sequence. When a task instance is triggered, four instance tables are created and instance data with the same primary key are inserted into the four instance tables, namely, the triggering stage instance table, the start-up verification stage instance table, the execution stage instance table, and the end stage instance table. Among them, the triggering stage instance table, the start-up verification stage instance table, and the execution stage instance table are created directly, and the end stage instance table is partitioned according to the creation time. In the triggering, start-up verification, and execution stages, the status of the instance in the stage is updated in the instance table corresponding to the stage, and the instance data in the instance table corresponding to the previous stage is deleted when the life cycle of the task instance flows to the next stage. In the end stage, the status of the instance in the stage is updated in the end stage instance table according to the creation time partition.

2. The task scheduling method of the variable table partitioning strategy according to claim 1 is characterized in that: According to the business data volume and concurrency of the target task, the three table partitioning strategies can be dynamically upgraded in sequence, from strategy 1 to strategy 2, and from strategy 2 to strategy 3.

3. The task scheduling method of the variable table partitioning strategy according to claim 1 is characterized in that: The strategy 1 includes: S11. Triggering stage: triggering the task instance and creating an instance table, setting and updating the status of the instance in this stage; S12. Startup verification phase: determine the startup of the instance, verify whether the instance can be executed, and update the status of the instance in this phase; S13. Execution phase: responsible for the execution of the instance and updating the status of the instance in this phase; S14. End stage: Update the status of the stage instance and trigger downstream tasks.

4. The task scheduling method of the variable table partitioning strategy according to claim 1 is characterized in that: Strategy 2 includes: S21. Triggering stage: triggering the task instance and creating the instance table, partitioning the instance table according to the creation time, and updating the status of the instance in this stage according to the creation time partition; S22. Startup verification phase: determine the startup of the instance, verify whether the instance can be executed, and update the status of the instance in this phase according to the creation time partition; S23. Execution phase: responsible for the execution of instances and updates the status of instances in this phase according to the creation time partition; S24. End stage: Update the status of the instance in this stage according to the creation time partition and trigger downstream tasks.

5. The task scheduling method of the variable table partitioning strategy according to claim 4 is characterized in that: The instance table described in S21 is partitioned according to the creation time, and the state of the instance at this stage is updated according to the creation time partition. The granularity of the creation time partition is configured according to the actual data increment to ensure that the data volume of a single table is 5-10 million.

6. The task scheduling method of the variable table partitioning strategy according to claim 1 is characterized in that: Strategy 3 includes: S31. Triggering stage: trigger the task instance and create four instance tables, namely, the triggering stage instance table, the startup verification stage instance table, the execution stage instance table, and the end stage instance table. Insert instance data with the same primary key into the four instance tables, and divide the end stage instance table according to the creation time; update the state of the triggering stage instance in the triggering stage instance table, and delete the instance data in the triggering stage instance table when the task instance life cycle flows to the startup verification stage; S32. Start verification phase: judge the startup of the instance to verify whether the instance can be executed; update the status of the startup verification phase instance in the startup verification phase instance table, and delete the instance data in the startup verification phase instance table when the task instance life cycle flows to the execution phase; S33. Execution phase: responsible for instance execution; updates the status of the execution phase instance in the execution phase instance table, and deletes the instance data in the execution phase instance table when the task instance life cycle flows to the end phase; S34. Ending stage: Update the status of the ending stage instance in the ending stage instance table according to the creation time partition, and trigger downstream tasks.

7. The task scheduling method of the variable table partitioning strategy according to claim 1 is characterized in that: Among the three table partitioning strategies: The running configuration of strategy 1 is 8 cores and 16 GB for the application service and 4 cores and 8 GB for the database. The running configuration of strategy 2 is 16 cores and 32 GB for the application service and 4 cores and 8 GB for the database. The running configuration of strategy 3 is 16 cores and 64 GB for the application service on a single machine and 8 cores and 16 GB for the database on a single machine.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the task scheduling method of the variable partitioning strategy according to any one of claims 1 to 7 are implemented.

Images