Task scheduling method and system

Abstract

The present invention discloses a task scheduling method and system, relating to the field of computer technology. A specific implementation of the method includes: allocating a scheduling thread of a scheduling application to each shard item in at least one shard item of the current task type; wherein the scheduling thread is used to select an execution application to process the corresponding shard item, and send a scheduling policy pre-set for the current task type and the identifier of the shard item to the selected execution application; when receiving a to-be-executed task belonging to the current task type: the execution application obtains the business data of the shard item of the to-be-executed task based on the scheduling policy and the identifier of the shard item to process the shard item of the to-be-executed task. This implementation can achieve high-availability task scheduling that supports multi-task scheduling and task sharding.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task scheduling method and device. Background Art

[0002] Scheduled tasks are a common use case in application services, such as calculating user phone bills at the end of each month in the telecommunications industry. In practice, scheduled tasks can also be used to handle some asynchronous processing scenarios, such as sending an email to a user after they place an order. Task scheduling systems are a specific solution to the scheduled task problem. Existing task scheduling systems are mainly divided into the following three categories: stand-alone scheduled task systems represented by Timer, distributed cluster multi-task scheduling systems represented by Quartz, and distributed scheduling systems that support task sharding, represented by TBSchedule and Elastic-Job.

[0003] In the process of realizing the present invention, the inventors found that: the above-mentioned first type of system cannot achieve high availability and high performance processing of the system, and the above-mentioned second type of system cannot achieve sharding processing of tasks and elastic expansion of task scheduling. As for the above-mentioned third type of system, since it relies more on ZooKeeper (a distributed application coordination service) in design, only one node provides services at the same time, so it cannot cope with situations where the scale of task scheduling is large; at the same time, its operating status monitoring of the job node is based on the Transmission Control Protocol TCP (Transmission Control Protocol) connection. Since the network often has jitter in a distributed environment, its business stability is poor; finally, this type of system is designed to place the task scheduling logic and business processing logic on one job node, thereby failing to guarantee the elastic expansion of business processing capabilities and task scheduling capabilities. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a task scheduling method and system, which can realize high-availability task scheduling that supports multi-task scheduling and task slicing.

[0005] To achieve the above object, according to one aspect of the present invention, a task scheduling method is provided.

[0006] The task scheduling method of an embodiment of the present invention includes: allocating a scheduling thread of a scheduling application to each shard item in at least one shard item of the current task type; wherein the scheduling thread is used to select an execution application to process the corresponding shard item, and send the scheduling policy pre-set for the current task type and the identifier of the shard item to the selected execution application; when receiving a task to be executed belonging to the current task type: the execution application obtains the business data of the shard item of the task to be executed based on the scheduling policy and the identifier of the shard item to process the shard item of the task to be executed.

[0007] Optionally, the method further includes: when the execution application is started, registering the task type corresponding to the execution application to the registration center, and setting a scheduling strategy for the registered task type; wherein the scheduling strategy includes: the number of shard items of the task type, the scheduling interval and the maximum amount of data obtained at one time.

[0008] Optionally, the method of allocating a scheduling thread of a scheduling application to each shard item in at least one shard item of the current task type specifically includes: for any shard item of the current task type: multiple preemptive threads of scheduling applications attempt to obtain a pre-set data lock, the scheduling application that obtains the data lock starts a scheduling thread, and allocates the scheduling thread to the shard item; wherein, after the scheduling application starts the scheduling thread, it releases the data lock; the method further includes: when the scheduling application corresponding to the scheduling thread is unavailable, reallocating a scheduling thread to the shard item corresponding to the scheduling thread.

[0009] Optionally, the scheduling thread may select the execution application to process the corresponding shard item according to the following steps: obtaining the currently available execution application from the registration center; wherein the registration center determines whether the execution application is available through the heartbeat signal sent by each execution application; and determining the execution application to process the shard item from the currently available execution applications based on the load balancing strategy.

[0010] Optionally, the executing application can obtain the business data of the shard item corresponding to the executing application of the task to be executed according to the following steps: in the data table where the business data of the task to be executed is located, the primary key data of each record is modulo the number of shard items in the scheduling strategy, and the records whose modulo result is the identifier of the shard item and the number is not greater than the maximum amount of data obtained at one time in the scheduling strategy are determined as the business data of the shard item of the task to be executed.

[0011] Optionally, the method further includes: when the amount of business data of the shard items of the task to be executed obtained by the execution application is less than the total amount of business data of the shard items of the task to be executed, reselecting the execution application to process the shard item, and sending the scheduling policy of the current task type and the identifier of the shard item to the execution application; when another task to be executed belonging to the current task type is received, reselecting the execution application to process each shard item, and sending the scheduling policy of the current task type and the identifier of the corresponding shard item to the execution application; the scheduling application and the execution application are set in different computer clusters.

[0012] To achieve the above object, according to another aspect of the present invention, a task scheduling system is provided.

[0013] A task scheduling system according to an embodiment of the present invention may include: a scheduling system for setting up at least one scheduling application and an execution system for setting up at least one execution application; wherein the scheduling application can be used to start a scheduling thread and assign the scheduling thread to a shard item of the current task type; wherein the scheduling thread is used to select an execution application for processing the shard item, and send the scheduling policy pre-set for the current task type and the identifier of the shard item to the selected execution application; the execution application can be used to: upon receiving a task to be executed belonging to the current task type, obtain the business data of the shard item of the task to be executed based on the scheduling policy and the identifier of the shard item to process the shard item of the task to be executed.

[0014] Optionally, the system may further include: a registration center, used to register the task type corresponding to the execution application and the calling application when the execution application and the calling application are started; determine whether the execution application is available through the heartbeat signal sent by each execution application; determine whether the scheduling application is available through the heartbeat signal sent by each scheduling application; a control center, used to set a scheduling policy for the task type registered in the registration center; wherein the scheduling policy includes: the number of shard items of the task type, the scheduling interval and the maximum amount of data to be obtained at one time.

[0015] Optionally, multiple scheduling applications of the scheduling system can be used to: for any shard item of the current task type, use the multiple preemptive threads started to attempt to obtain a pre-set data lock; the scheduling application that obtains the data lock can be used to: start the scheduling thread, assign the scheduling thread to the shard item, and release the data lock; the scheduling thread can be further used to: obtain the currently available execution application from the registration center, and determine the execution application to process the corresponding shard item based on the load balancing strategy; the execution application can be further used to: in the data table where the business data of the task to be executed is located, the primary key data of each record is modulo the number of shard items in the scheduling strategy, and the records whose modulo result is the identifier of the shard item and the number is not greater than the maximum amount of data obtained at one time in the scheduling strategy are determined as the business data of the shard item of the task to be executed.

[0016] Optionally, the registration center can be further used to: store data locks and their current status; store the correspondence between task types and implementation interfaces; store the correspondence between task types and scheduling strategies; store the correspondence between execution applications, server addresses and the running status of execution applications; store the correspondence between scheduling applications, server addresses, scheduling threads and the running status of scheduling threads; store the correspondence between task types, shard item identifiers, scheduling threads and the current status of shard items.

[0017] To achieve the above objective, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0018] A computer-readable storage medium of the present invention stores a computer program, which implements the task scheduling method provided by the present invention when executed by a processor.

[0019] According to the technical solution of the present invention, one embodiment of the above invention has the following advantages or beneficial effects:

[0020] First, after each task type is initiated, a scheduling thread is assigned to each shard item. This scheduling thread selects an execution application that meets the preset criteria for the shard item and sends the corresponding scheduling policy and shard item identifier to the execution application, allowing the execution application to obtain the business data of the shard item in the task to be executed. This setup achieves the separation of task scheduling logic (based on the scheduling application) and business processing capabilities (based on the execution application), ensuring the flexible expansion of both business processing and task scheduling capabilities.

[0021] Secondly, the present invention can achieve highly available task scheduling through multiple scheduling applications in the scheduling system, solving the problems caused by TBSchedule, Elastic-Job and other heavy reliance on ZooKeeper; at the same time, the present invention uses a database as a registration center to store task types, execution applications, scheduling applications, sharding items, scheduling threads, scheduling strategies, data locks and related data. When the amount of data is large, the database and table can be divided as needed, thereby improving the horizontal scalability of the system and helping to support larger-scale task scheduling scenarios.

[0022] Third, the present invention adopts a heartbeat mechanism to replace the TCP connection in the prior art, thereby being able to accurately reflect the running status of the application, thereby ensuring the stability of the system; by effectively responding to the scheduling of applications and the online and offline execution of applications, the scheduling strategy does not change with the number of applications, further improving the high availability of the system.

[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0025] Figure 1 Schematic diagram of the main steps of the task scheduling method according to an embodiment of the present invention;

[0026] Figure 2 2 is a schematic diagram of components of a task scheduling system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] It should be pointed out that, in the absence of conflict, the embodiments of the present invention and the technical features therein may be combined with each other.

[0029] Figure 1 1 is a schematic diagram of the main steps of the task scheduling method according to an embodiment of the present invention.

[0030] like Figure 1 As shown, the task scheduling method of the embodiment of the present invention can be specifically performed according to the following steps:

[0031] Step S101: Allocate a scheduling thread of a scheduling application to each slice item in at least one slice item of the current task type; wherein the scheduling thread is used to select an execution application to process the corresponding slice item, and send the scheduling policy pre-set for the current task type and the identifier of the slice item to the selected execution application.

[0032] In this embodiment of the present invention, a task type refers to a category of tasks with similar characteristics. In real-world applications, a task type corresponds to multiple tasks. The following are some examples of different task types: sending an email to a user, sending a text message to a user, and making a phone call to a user. In a specific application, if the task to be executed is to send an email to 100 specified users, then this task belongs to the task type "sending an email to a user."

[0033] It is understandable that task types are often associated with one or more front-end servers that execute business processing logic, that is, the above-mentioned servers have the ability to execute specific tasks belonging to the above-mentioned task types. Generally, one or more execution applications are set in the above-mentioned servers to directly execute tasks. Among them, the execution application refers to a computer program that executes business processing logic and then processes tasks. In actual applications, it is necessary to adopt a certain strategy to send tasks to suitable execution applications, which requires a calling unit to schedule tasks. It is understandable that the scheduling unit refers to a computer program that executes task scheduling logic and assigns tasks to reasonable execution applications. It is generally deployed in the back-end server responsible for task scheduling. The scheduling application and the execution application can be set in different computer clusters to realize the separation of task call logic and business processing logic.

[0034] With the rapid development of internet technology, executing a specific task in real-world scenarios often requires acquiring and processing large amounts of data. If this task is sent to a single execution application for processing, efficiency is extremely low and cannot meet business requirements. Therefore, it is necessary to divide a task into multiple shard items, with each execution application processing each shard item. This can significantly improve task execution efficiency. Generally, shard items can be divided based on the primary key value of the task-related data. For example, if Task A involves sending emails to 100 users, the relevant data required for executing the task includes the email addresses of the 100 users and the content of each email. If the task needs to be divided into 10 shard items, the division can be based on the primary key (user ID) of the user data table. Specifically, the user ID can be modulo the number of shard items (10). Users whose modulo result is 0 (i.e., ending in 0) will be designated as users of the first shard item, users whose modulo result is 1 (i.e., ending in 1) will be designated as users of the second shard item, and users whose modulo result is 2 (i.e., ending in 2) will be designated as users of the third shard item. In practical applications, these modulo results can serve as identifiers for the corresponding shard items.

[0035] In specific application scenarios, in order to uniformly manage tasks belonging to the same task type, a scheduling strategy can be set for each task type. The scheduling strategy may include: the number of shard items of the task type, the scheduling interval, and the maximum amount of data to be obtained at one time. Among them, the number of shard items limits the total number of shards of the task type; the scheduling interval refers to the time interval between two consecutive schedulings of the scheduling application, that is, the time interval between two consecutive processing tasks of the executing application. In actual applications, the thread that executes the scheduling logic (that is, the scheduling thread to be introduced later) initiates task scheduling according to the scheduling interval; the maximum amount of data to be obtained at one time refers to the maximum amount of data that the executing application can obtain at one time when processing a task. It can be understood that in transactions involving massive data, even for sharded items, the amount of data is still huge. In order to further improve data processing efficiency, the maximum amount of data to be obtained at one time for each task type can be configured to limit the amount of data processed at a single time.

[0036] In this step, a scheduling thread can be assigned to each shard item of the current task type. The scheduling thread is used to execute the task scheduling logic and select the execution application to process the shard item of the current task type. In specific applications, the following steps can be used to assign a scheduling thread to a shard item:

[0037] For any shard item of the current task type, multiple preemptive threads from scheduling applications attempt to acquire a pre-set data lock, essentially issuing a preemptive action on the data lock. The scheduling application that ultimately acquires the data lock initiates a scheduling thread, which is then assigned to the shard item. The scheduling application then releases the data lock. The data lock can be a global lock applied to the data row. The data lock and its current status data can be stored in a separate data table. The current status data can contain two values, one indicating whether the data lock is preempted and the other indicating whether it is released.

[0038] In an embodiment of the present invention, after assigning a scheduling thread to a shard item, the scheduling thread can select an execution application suitable for processing the shard item from the currently available execution applications. Specifically, the scheduling thread first obtains a list of all currently available execution applications and then selects an execution application from these applications based on a load balancing algorithm, such as a random algorithm or a consistent hashing algorithm. In practical applications, when an execution application starts, it registers its associated task type with a registration center, which can be a computer cluster including a database. Each execution application then periodically checks its operating status and sends a heartbeat signal to the registration center. The registration center determines whether the execution application is available based on the received heartbeat signal and ultimately stores the execution unit's operating status in a data table. When the scheduling application's scheduling thread starts, the scheduling application can initiate an information synchronization thread. This thread accesses the registration center's data table storing the execution application's operating status and synchronizes the available execution application information with the scheduling thread. In practical applications, the scheduling thread can also use a load balancing algorithm to obtain the least loaded execution server and then select an idle execution application from the execution server as the execution application to process the shard item.

[0039] In specific application scenarios, the scheduling server where the scheduling thread resides may crash. In this case, the scheduling thread terminates and its corresponding relationship with the shard item also stops. In this case, the aforementioned locking mechanism can be used to reassign the scheduling thread to the shard item, thereby achieving failover of the scheduling application.

[0040] After selecting an execution application, the scheduling thread can send the execution application the scheduling policy for the corresponding task type and the identifier of the corresponding shard item. After the execution application obtains this data, it can use it to determine the business data of the shard item.

[0041] Step S102: When receiving a pending task belonging to the current task type, the executing application obtains the business data of the pending task's fragment item according to the scheduling policy and the identifier of the fragment item to process the pending task's fragment item.

[0042] In this step, the execution application can obtain the task to be executed by monitoring the business database, and obtain the business data of the shard item of the task to be executed according to the corresponding scheduling strategy and the identifier of the shard item, thereby realizing the processing of the task to be executed. For example, the shard item identifier corresponding to an execution application is 2, and the scheduling strategy is: the number of shard items is 10, the scheduling interval is 3s, and the maximum amount of data obtained at one time is 100. Then, after the task to be executed is issued, the execution application can perform a modulo operation on the primary key data (user ID) of each record in the corresponding data table with the number of shard items 10 in the scheduling strategy, and determine the records with a modulo result of 2 and a number not greater than 100 as the business data of the shard item of the task to be executed, that is, the first 100 records with the user ID ending in 2 as the required data.

[0043] In actual applications, it's often difficult for the executing application to retrieve and process the data for a single shard item of a pending task all at once. If the amount of business data for a shard item retrieved by the executing application is less than the total amount of business data for that shard item, after the first execution, the scheduling thread reselects the executing application to process the shard item based on the load balancing policy and sends the scheduling policy for the current task type and the identifier of the shard item to the executing application. Furthermore, upon receiving another pending task of the current task type, the scheduling thread reselects the executing application to process each shard item and sends the scheduling policy for the current task type and the identifier of the corresponding shard item to the executing application.

[0044] In the technical solution of the embodiment of the present invention, after each task type is started, a scheduling thread is assigned to each shard item thereof. The scheduling thread is used to select an execution application that meets the preset conditions for the shard item, and send the corresponding scheduling policy and shard item identifier to the execution application so that the execution application can obtain the business data of the shard item in the task to be executed. Through the above-mentioned setting, the separation of task scheduling logic (based on the scheduling application) and business processing capability (based on the execution application) is achieved, thereby ensuring the elastic expansion of business processing capability and task scheduling capability. In addition, the present invention can achieve highly available task scheduling through multiple scheduling applications in the scheduling system, solving the problems caused by TBSchedule, Elastic-Job and the like that are heavily dependent on ZooKeeper; at the same time, the present invention uses a database as a registration center to store task types, execution applications, scheduling applications, shard items, scheduling threads, scheduling policies, data locks and related data. When the amount of data is large, it can be divided into libraries and tables as needed, thereby improving the horizontal expansion capability of the system, which helps to support larger-scale task scheduling scenarios. Finally, the present invention adopts a heartbeat mechanism to replace the TCP connection in the prior art, thereby being able to accurately reflect the running status of the application, thereby ensuring the stability of the system; by effectively responding to the scheduling of applications and the online and offline execution of applications, the scheduling strategy does not change with the number of applications, further improving the high availability of the system.

[0045] Figure 2 3 is a schematic diagram of components of a task scheduling system according to an embodiment of the present invention. This task scheduling system can be used as a specific system for implementing the above-mentioned task scheduling method.

[0046] like Figure 2 As shown, the task scheduling system may include a registration center, a control center, a scheduling system and an execution system.

[0047] The execution system can include multiple front-end servers, each hosting multiple execution applications. The execution system handles specific business logic and exposes Remote Procedure Call (RPC) services for invocation by the scheduling system. Upon receiving an RPC request from a scheduling application within the scheduling system, the execution application automatically retrieves the required shard data based on the scheduling policy and shard item identifiers contained in the request.

[0048] The registration center may include a computer cluster equipped with multiple registration applications and a database including multiple data tables. The registration center can be used to register task types and scheduling applications, and can also obtain the running status of the application based on the heartbeat signals sent periodically by the execution application and the scheduling application. At the same time, the registration center can provide an interface to the control center so that the staff of the control center can configure, issue (to the registration center), start or stop the scheduling strategy for the registered task type. It can be understood that after the staff starts the task type, they can execute the above-mentioned steps of allocating scheduling threads and selecting execution applications. In addition, multiple data tables can be maintained in the database of the registration center to store relevant data in the task scheduling process. Specifically, the Work table can store the name of the task type, the task type identifier, and the implementation interface; the Schedule table can store the task type and the scheduling policy configured by the staff, and can also store the running status of the corresponding scheduling thread; the Schedule global lock table can store data locks and their current status; the Manager table can store the scheduling application, the corresponding server Internet Protocol IP (Internet Protocol) address, the scheduling thread, and the scheduling application running status; the Server table can store the task type, the shard item identifier, the scheduling thread and the current status of the shard item (for example, to be started or started), and can also store the corresponding scheduling policy; the Execute table can store the execution application, server address, and the execution application running status. It is worth mentioning that when dealing with the situation where the amount of data is constantly increasing, the above tables can be divided into sub-libraries and sub-tables to achieve horizontal expansion, thereby improving data processing efficiency.

[0049] The scheduling system can include multiple backend servers, each hosting multiple scheduling applications. The scheduling system uses a data lock mechanism to control preemptive scheduling of these applications, ensuring that only one scheduling application can be assigned to a shard item of a given task type at a time. After acquiring the data lock, the scheduling application can start a scheduling thread to schedule tasks.

[0050] The control center sends scheduling policies to the registration center by calling the RPC service interface provided by the registration center, and controls the start and stop of scheduling tasks.

[0051] The above are the basic functions of each subsystem of the task scheduling system. The following describes the workflow of the task scheduling system based on the above content. The steps are as follows:

[0052] 1. When each execution application in the execution system starts, the relevant task type is registered with the registration center, and a record is stored in the Work table for each task type.

[0053] 2. The dispatch center queries the task types registered with the registration center and displays them on the web page. Staff can query the system's default dispatch policy.

[0054] 3. The staff can configure the scheduling policy for the newly registered task type and start the task type allocation. The configured scheduling policy will be stored in the Schedule table, and each shard item of a task type will be stored as a record in the Server table.

[0055] 4. The scheduling system starts multiple scheduling applications and registers them with the registration center. After that, the relevant data of each scheduling application is stored as a record in the Manager table.

[0056] 5. Each scheduling application uses a preemptive thread to attempt to obtain the data lock in the Schedule global lock table. The scheduling application that finally obtains the data lock starts the scheduling thread for the current shard item. After that, the scheduling application releases the data lock and continues to start the scheduling thread by preempting the data lock.

[0057] 6. The scheduling thread started in the previous step retrieves the available execution applications from the Execute table in the registry and selects an appropriate execution application based on the load balancing algorithm to process the corresponding shard item. The scheduling policy, including the number of shard items, scheduling interval, and maximum data size to be retrieved at one time, along with the identifier of the corresponding shard item, is sent to the selected execution application. Upon receiving a task to be executed, the execution application retrieves the corresponding shard data and executes it.

[0058] Through the above steps, high-availability task scheduling and execution can be achieved. It can be understood that when an execution application becomes unavailable, the execution thread for the shard item it is processing will select another application from the currently available execution applications to process the shard item. When a scheduling application becomes unavailable, its scheduling thread and the corresponding shard item are deleted from the Server table. At this point, multiple scheduling applications can reassign scheduling threads to the shard item through the preemptive data lock mechanism.

[0059] Thus, in an embodiment of the present invention, a complete distributed, highly available task scheduling system that supports multi-task scheduling, task sharding, and batch processing is provided. Compared with the prior art, the present invention implements a database-based registration center service, and supports massive Internet concurrent scheduling scenarios through database sharding technology, successfully solving the problem of limited processing capacity of the centralized registration center Zookeeper. At the same time, the present invention does not rely on Zookeeper's TCP connection as the basis for re-sharding the scheduling tasks, thereby improving the stability of the scheduling center. Finally, the present invention places the task scheduling logic and business processing logic in the existing solution in two independent clusters for processing, thereby realizing elastic expansion of scheduling logic nodes and business logic nodes and improving the flexibility of the system.

[0060] In addition, although the above Figure 2 The task scheduling system shown is used as an example to illustrate the task scheduling method of the present invention, but this does not impose any limitation on the application scenarios of the present invention.

[0061] It should be noted that for the aforementioned method embodiments, for ease of description, they are expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited to the order of the actions described, and certain steps can actually be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required to implement the present invention.

[0062] In order to better implement the above solution of the embodiment of the present invention, the following further describes Figure 2 The task scheduling system shown.

[0063] A task scheduling system provided by an embodiment of the present invention may include: a scheduling system for setting up at least one scheduling application and an execution system for setting up at least one execution application; wherein, the scheduling application can be used to start a scheduling thread and assign the scheduling thread to a shard item of the current task type; wherein, the scheduling thread can be used to select an execution application to process the shard item, and send the scheduling policy pre-set for the current task type and the identifier of the shard item to the selected execution application; the execution application can be used to: when receiving a to-be-executed task belonging to the current task type, obtain the business data of the shard item of the to-be-executed task based on the scheduling policy and the identifier of the shard item to process the shard item of the to-be-executed task.

[0064] In an embodiment of the present invention, the system may further include: a registration center, which is used to register the task type corresponding to the execution application and the calling application when the execution application and the calling application are started; determine whether the execution application is available through the heartbeat signal sent by each execution application; determine whether the scheduling application is available through the heartbeat signal sent by each scheduling application; a control center, which is used to set a scheduling policy for the task type registered in the registration center; wherein the scheduling policy includes: the number of shard items of the task type, the scheduling interval and the maximum amount of data obtained at one time.

[0065] As a preferred solution, multiple scheduling applications of the scheduling system can be used to: for any shard item of the current task type, use the multiple preemptive threads started to try to obtain a pre-set data lock; the scheduling application that obtains the data lock can be used to: start the scheduling thread, assign the scheduling thread to the shard item, and release the data lock; the scheduling thread can be further used to: obtain the currently available execution application from the registration center, and determine the execution application to process the corresponding shard item based on the load balancing strategy; the execution application can be further used to: in the data table where the business data of the task to be executed is located, the primary key data of each record is modulo the number of shard items in the scheduling strategy, and the records whose modulo result is the identifier of the shard item and the number is not greater than the maximum amount of data obtained at one time in the scheduling strategy are determined as the business data of the shard item of the task to be executed.

[0066] Preferably, in an embodiment of the present invention, the registration center can be further used to: store data locks and their current status; store the correspondence between task types and implementation interfaces; store the correspondence between task types and scheduling strategies; store the correspondence between execution applications, server addresses and the running status of execution applications; store the correspondence between scheduling applications, server addresses, scheduling threads and the running status of scheduling threads; store the correspondence between task types, shard item identifiers, scheduling threads and the current status of shard items.

[0067] In the technical solution of the embodiment of the present invention, after each task type is started, a scheduling thread is assigned to each shard item thereof. The scheduling thread is used to select an execution application that meets the preset conditions for the shard item, and send the corresponding scheduling policy and shard item identifier to the execution application so that the execution application can obtain the business data of the shard item in the task to be executed. Through the above-mentioned setting, the separation of task scheduling logic (based on the scheduling application) and business processing capability (based on the execution application) is achieved, thereby ensuring the elastic expansion of business processing capability and task scheduling capability. In addition, the present invention can achieve highly available task scheduling through multiple scheduling applications in the scheduling system, solving the problems caused by TBSchedule, Elastic-Job and the like that are heavily dependent on ZooKeeper; at the same time, the present invention uses a database as a registration center to store task types, execution applications, scheduling applications, shard items, scheduling threads, scheduling policies, data locks and related data. When the amount of data is large, it can be divided into libraries and tables as needed, thereby improving the horizontal expansion capability of the system, which helps to support larger-scale task scheduling scenarios. Finally, the present invention adopts a heartbeat mechanism to replace the TCP connection in the prior art, thereby being able to accurately reflect the running status of the application, thereby ensuring the stability of the system; by effectively responding to the scheduling of applications and the online and offline execution of applications, the scheduling strategy does not change with the number of applications, further improving the high availability of the system.

[0068] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the steps executed by the device include: allocating a scheduling thread of a scheduling application to each slice item in at least one slice item of the current task type; wherein the scheduling thread is used to select an execution application to process the corresponding slice item, and send the scheduling policy pre-set for the current task type and the identifier of the slice item to the selected execution application; when receiving a to-be-executed task belonging to the current task type: the execution application obtains the business data of the slice item of the to-be-executed task based on the scheduling policy and the identifier of the slice item to process the slice item of the to-be-executed task.

[0069] In the technical solution of the embodiment of the present invention, after each task type is started, a scheduling thread is assigned to each shard item thereof. The scheduling thread is used to select an execution application that meets the preset conditions for the shard item, and send the corresponding scheduling policy and shard item identifier to the execution application so that the execution application can obtain the business data of the shard item in the task to be executed. Through the above-mentioned setting, the separation of task scheduling logic (based on the scheduling application) and business processing capability (based on the execution application) is achieved, thereby ensuring the elastic expansion of business processing capability and task scheduling capability. In addition, the present invention can achieve highly available task scheduling through multiple scheduling applications in the scheduling system, solving the problems caused by TBSchedule, Elastic-Job and the like that are heavily dependent on ZooKeeper; at the same time, the present invention uses a database as a registration center to store task types, execution applications, scheduling applications, shard items, scheduling threads, scheduling policies, data locks and related data. When the amount of data is large, it can be divided into libraries and tables as needed, thereby improving the horizontal expansion capability of the system, which helps to support larger-scale task scheduling scenarios. Finally, the present invention adopts a heartbeat mechanism to replace the TCP connection in the prior art, thereby being able to accurately reflect the running status of the application, thereby ensuring the stability of the system; by effectively responding to the scheduling of applications and the online and offline execution of applications, the scheduling strategy does not change with the number of applications, further improving the high availability of the system.

[0070] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A task scheduling method, characterized in that: include: Allocating a scheduling thread of the scheduling application to each shard item in at least one shard item of the current task type; wherein the scheduling thread is used to select an execution application to process the corresponding shard item and send a scheduling policy pre-set for the current task type and an identifier of the shard item to the selected execution application; When receiving a task to be executed that belongs to the current task type: the execution application obtains the business data of the shard item of the task to be executed according to the scheduling policy and the identifier of the shard item to process the shard item of the task to be executed; When the amount of business data of the shard item of the task to be executed obtained by the execution application is less than the total amount of business data of the shard item of the task to be executed, reselect the execution application to process the shard item, and send the scheduling policy of the current task type and the identifier of the shard item to the execution application; Upon receiving another pending task of the current task type, reselect an execution application to process each shard item, and send the scheduling policy of the current task type and the identifier of the corresponding shard item to the execution application; and The scheduling application and the execution application are set in different computer clusters; When an execution application is started, the task type corresponding to the execution application is registered with a registration center, and a scheduling policy is set for the registered task type. The registration center includes a computer cluster with multiple registration applications and a database including multiple data tables. The registration center is used to register task types and scheduling applications, and obtains the application's running status based on the heartbeat signals sent periodically by the execution application and the scheduling application.

2. The method according to claim 1, characterized in that The scheduling strategy includes: the number of shard items for this task type, the scheduling interval, and the maximum amount of data to be obtained at one time.

3. The method according to claim 1, characterized in that Allocating a scheduling thread of the scheduling application to each shard item of at least one shard item of the current task type specifically includes: for any shard item of the current task type: multiple preemptive threads of the scheduling applications attempt to acquire a pre-set data lock, the scheduling application that acquires the data lock starts a scheduling thread and allocates the scheduling thread to the shard item; wherein, after the scheduling application starts the scheduling thread, the data lock is released; and The method further includes: when a scheduling application corresponding to the scheduling thread is unavailable, reallocating a scheduling thread for a slice item corresponding to the scheduling thread.

4. The method according to claim 2, characterized in that The scheduling thread selects the execution application for the corresponding shard item according to the following steps: Obtaining currently available execution applications from the registration center; wherein the registration center determines whether the execution application is available through a heartbeat signal sent by each execution application; The execution application that processes the shard item is determined from the currently available execution applications based on the load balancing strategy.

5. The method according to claim 2, characterized in that The executing application obtains the business data of the shard item corresponding to the executing application for the task to be executed according to the following steps: In the data table where the business data of the task to be executed is located, the primary key data of each record is modulo the number of shard items in the scheduling strategy, and the records whose modulo result is the identifier of the shard item and the number is not greater than the maximum amount of data to be obtained at one time in the scheduling strategy are determined as the business data of the shard item of the task to be executed.

6. A task scheduling system, characterized in that: include: At least one scheduling system for scheduling applications and at least one execution system for executing applications are provided; wherein, The scheduling application is used to start a scheduling thread and assign the scheduling thread to a slice item of the current task type; wherein the scheduling thread is used to select an execution application to process the slice item and send a scheduling policy pre-set for the current task type and an identifier of the slice item to the selected execution application; The execution application is used to: when receiving a task to be executed belonging to the current task type, obtain the business data of the shard item of the task to be executed according to the scheduling policy and the identifier of the shard item, so as to process the shard item of the task to be executed; The scheduling application is further configured to: when the amount of business data of the shard items of the task to be executed obtained by the execution application is less than the total amount of business data of the shard items of the task to be executed, reselect the execution application to process the shard item, and send the scheduling policy of the current task type and the identifier of the shard item to the execution application; when another task to be executed belonging to the current task type is received, reselect the execution application to process each shard item, and send the scheduling policy of the current task type and the identifier of the corresponding shard item to the execution application; A registration center is configured to register the task types and calling applications corresponding to the executing applications when the executing applications and calling applications are started; determine whether the executing applications are available through the heartbeat signals sent by each executing application; and determine whether the scheduling applications are available through the heartbeat signals sent by each scheduling application. The registration center includes a computer cluster equipped with multiple registration applications and a database including multiple data tables. The control center is used to set scheduling policies for task types registered in the registration center; The scheduling application and the execution application are set in different computer clusters.

7. The system according to claim 6, characterized in that The scheduling strategy includes: the number of shard items for this task type, the scheduling interval, and the maximum amount of data to be obtained at one time.

8. The system according to claim 7, characterized in that Multiple scheduling applications of the scheduling system are used to: for any shard item of the current task type, use multiple preemptive threads started to try to obtain a pre-set data lock; The scheduling application that obtains the data lock is used to: start the scheduling thread, assign the scheduling thread to the shard item, and release the data lock; The scheduling thread is further used to: obtain the currently available execution applications from the registration center, and determine the execution application to process the corresponding shard item based on the load balancing strategy; and The execution application is further used to: in the data table where the business data of the task to be executed is located, perform a modulo operation on the primary key data of each record with the number of shard items in the scheduling strategy, and determine the records whose modulo result is the identifier of the shard item and the number of which is not greater than the maximum amount of data to be obtained at one time in the scheduling strategy as the business data of the shard item of the task to be executed.

9. The system according to claim 8, characterized in that The registry is further used to: Stores data locks and their current status; The correspondence between storage task types and implementation interfaces; The correspondence between storage task types and scheduling strategies; Storing the corresponding relationship between the executed application, the server address and the running status of the executed application; Store the corresponding relationship between the scheduling application, server address, scheduling thread and the running status of the scheduling thread; as well as The correspondence between the storage task type, shard item ID, scheduling thread and the current status of the shard item.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images