A distributed task scheduling method and system based on an xxl-job framework

Abstract

The application discloses a distributed task scheduling method based on an xxl-job framework, which comprises the following steps: building a scheduling system of an xxl-job server, setting the id of the xxl-job server, and obtaining a distributed lock through the id configured by the xxl-job server; building an xxl-job execution end, establishing the connection between the xxl-job server and the xxl-job execution end, and enabling the xxl-job servers with the same id to compete for a distributed lock; after successfully obtaining the distributed lock, searching for a scheduling task corresponding to the id of the xxl-job server, and executing the corresponding task scheduling through the xxl-job execution end. The application can improve the execution efficiency, increase the amount of tasks that can be run, improve the operation performance of the server, and ensure the instant and efficient operation of the task system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer task scheduling, and more particularly to a distributed task scheduling method and system based on an xxl-job framework. BACKGROUND

[0002] At present, most distributed task scheduling frameworks, such as quartz, xxl-job, etc., in order to ensure that they can be used in actual production environment, are deployed in clusters, and the problem that follows is the consistency of scheduling, and it is possible that a task is scheduled multiple times at the same time. Generally, the scheduling end avoids conflicts by using a distributed lock, but the distributed lock is generally locked as a whole, and only one scheduling module can obtain the lock at the same time, and the remaining modules must wait, which affects the overall concurrency and efficiency. SUMMARY

[0003] In view of at least one defect or improvement demand of the prior art, the present application provides a distributed task scheduling method and system based on an xxl-job framework, which can improve the execution efficiency, increase the number of tasks that can be run, improve the running performance of the server, and ensure the instant and efficient operation of the task system.

[0004] To achieve the above-mentioned purpose, according to the first aspect of the present application, a distributed task scheduling method based on an xxl-job framework is provided, which comprises:

[0005] Building a scheduling system of an xxl-job server, setting the id of the xxl-job server, and obtaining a distributed lock through the id configured by the xxl-job server;

[0006] Building an xxl-job execution end, and establishing a connection between the xxl-job server and the xxl-job execution end;

[0007] The xxl-job servers with the same id compete for a distributed lock, and after successfully obtaining the distributed lock, find the scheduling task corresponding to the id of the xxl-job server, and execute the corresponding task scheduling through the xxl-job execution end.

[0008] Further, the above-mentioned distributed task scheduling method based on an xxl-job framework, wherein the scheduling system of the xxl-job server comprises at least one xxl-job scheduling end cluster and at least one mysql data unit; the xxl-job scheduling end cluster comprises at least four nodes, two of which are master nodes, and the other two nodes are corresponding slave nodes of the master nodes.

[0009] Further, the xxl-job framework-based distributed task scheduling method, wherein the master node and the corresponding slave node have the same id value.

[0010] Further, the xxl-job framework-based distributed task scheduling method, wherein the xxl-job execution end is built to establish the connection between the xxl-job server and the xxl-job execution end, and specifically includes:

[0011] The xxl-job execution end is built, and the appName parameter of the xxl-job execution end is set to be the same as the name of the xxl-job server.

[0012] Further, the xxl-job framework-based distributed task scheduling method further includes creating an xxl_job_lock table and an xxl_job_info table in the mysql storage unit, and inserting the id values of the master node and the slave node into the xxl_job_lock table and the xxl_job_info table in the mysql storage unit.

[0013] Further, the xxl-job framework-based distributed task scheduling method, wherein the xxl_job_info table is used to store the time of the timing scheduling task and the execution method of the scheduling task.

[0014] Further, the xxl-job framework-based distributed task scheduling method, wherein a relation_id field is added in the xxl_job_info table, the relation_id field is used to store the result value of the scheduling task taking the modulus of the primary key, and the number of the xxl-job scheduling end is used as the modulus base.

[0015] Further, the xxl-job framework-based distributed task scheduling method, wherein the scheduling task corresponding to the id of the xxl-job server specifically includes:

[0016] The relation_id value equal to the id value of the xxl-job server is found in the xxl_job_info table, and the scheduling task corresponding to the id of the xxl-job server is acquired.

[0017] According to a second aspect of the present application, a xxl-job framework-based distributed task scheduling system is also provided, which includes:

[0018] The task scheduling module at least includes one task scheduling cluster, and the task scheduling cluster at least includes four xxl-job scheduling ends.

[0019] The task execution module comprises at least two xxl-job executors, and is configured to execute specific scheduling tasks when the xxl-job scheduling end is scheduled.

[0020] The mysql data unit is configured to store scheduling task information, and at least comprises one of scheduling task execution time, scheduling task trigger time and execution type.

[0021] Further, the above xxl-job framework-based distributed task scheduling system comprises four xxl-job scheduling ends as four nodes, two of which are master nodes, and the other two are slave nodes corresponding to the master nodes, and the master nodes and the corresponding slave nodes have the same id value.

[0022] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0023] (1) The xxl-job framework-based distributed task scheduling method provided by the present application modifies the source code of the xxl-job service end, acquires a distributed lock through the id configured by the xxl-job service end, and competes for a distributed lock with xxl-job service ends having the same id. After successfully acquiring the distributed lock, the xxl-job service end searches for a scheduling task corresponding to the id of the xxl-job service end, and executes the corresponding task scheduling through the xxl-job execution end. This method can generate multiple distributed locks, different service ends can acquire the distributed lock at the same time, can increase the amount of running tasks, and improve the running performance of the server.

[0024] (2) The xxl-job framework-based distributed task scheduling method provided by the present application takes the scheduling task as a primary key to perform modulo operation, saves the value, and the value corresponds to the id of the scheduling end. When scheduling, scheduling ends with the same id will acquire a distributed lock, and after successfully acquiring the lock, the scheduling end searches for a task corresponding to the id in the mysql unit. In this way, the scheduling task is dispersed to multiple scheduling units, and the execution efficiency of the task is improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1A flowchart of a distributed task scheduling method based on an xxl-job framework provided by an embodiment of the present application is shown in the figure.

[0027] Figure 2 A structural diagram of a distributed task scheduling system based on an xxl-job framework provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0029] The terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units, but can optionally include steps or units that are not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0030] In one aspect, the present application provides a distributed task scheduling method based on an xxl-job framework, Figure 1 A flowchart of a distributed task scheduling method based on an xxl-job framework provided by an embodiment of the present application is shown in the figure. Figure 1 The method comprises:

[0031] (1) Build a scheduling system of the xxl-job server, set the id of the xxl-job server, and obtain a distributed lock through the id configured by the xxl-job server;

[0032] Specifically, the source code of the xxl-job server is modified, the JobScheduleHelper.java file under the xxl-job-admin module is found, the start method in the file is found, and the traditional method is to obtain a unique lock in the table through the mysql statement select for update where lock_name=".

[0033] That is, modify the sql statement in the statement preparedStatement = conn.prepareStatement() as follows:

[0034] "select*from xxl_job_lock where lock_name=#{id}for update"

[0035] Where #{id} is the id of the xxl-job server read from the configuration file.

[0036] Modify the sql statement passed in by scheduleJobQuery by adding a parameter, add #{id} in the query filter condition, to ensure that the xxl-job server only selects the scheduling task with the same id.

[0037] The scheduling system of the xxl-job server includes at least one xxl-job scheduling end cluster and at least one mysql data unit; the xxl-job scheduling end cluster includes at least four scheduling ends: serverA, serverB, serverC, and serverD. The four scheduling ends are taken as four nodes, two of which are master nodes, i.e., master nodes, and the other two are slave nodes corresponding to the master nodes, i.e., slave nodes. The master node and its corresponding slave node have the same id value. The id is equivalent to the basis for grouping the xxl-job scheduling end, and the xxl-job scheduling ends with the same id are a group, which compete for the same distributed lock.

[0038] In a specific embodiment, the id values of serverA and serverB are set to 1, and the id values of serverC and serverD are set to 2.

[0039] (2) build an xxl-job execution end, and establish a connection between the xxl-job server and the xxl-job execution end;

[0040] The xxl-job execution end includes at least two xxl-job executors, which perform specific scheduling tasks when the xxl-job scheduling end performs scheduling. For the xxl-job execution end, a spring project is generally used for startup without modification, and configuration can be performed according to the default example. The appName parameter is used to return the name of the xxl-job execution end. The xxl-job execution end is registered in the xxl-job service end as the basis for establishing the connection between the xxl-job service end and the xxl-job execution end by setting the appName parameter of the xxl-job execution end to be the same as the name of the xxl-job service end, so that the xxl-job service end can mobilize the xxl-job execution end to perform corresponding task scheduling.

[0041] (3) The xxl-job service ends with the same id compete for a distributed lock, and after successfully obtaining the distributed lock, the scheduling tasks corresponding to the id of the xxl-job service end are found, and the corresponding task scheduling is performed by the xxl-job execution end.

[0042] The xxl_job_lock table and the xxl_job_info table are created in the mysql storage unit, and the id values of the master node and the slave node are inserted into the xxl_job_lock table and the xxl_job_info table in the mysql storage unit.

[0043] Specifically, the xxl_job_lock table originally has only one data, representing a global distributed lock, and the id value of the xxl-job service end is inserted. In a specific embodiment, the id values corresponding to serverA-serverD are inserted into the table to insert the corresponding distributed lock. This step is performed by the following code:

[0044] INSERT INTO xxl_job_lock (lock_name) VALUES ('1');

[0045] INSERT INTO xxl_job_lock (lock_name) VALUES ('2');

[0046] Specifically, the xxl_job_info table is used to store the time of a scheduled task and the execution method of the scheduled task, and the scheduled task is identified by a primary key value. A relation_id field is added to the xxl_job_info table, and the relation_id field is used to store the result value of taking the primary key as a modulus remainder. The number of xxl-job scheduling ends is used as the modulus, and the value of the relation_id field is equal to the id value of the xxl-job scheduling node.

[0047] In a specific embodiment, the relation_id field is added by modifying a trigger in the xxl_job_info table, and the trigger implements the following logic:

[0048] TRIGGER "TRG" AFTER INSERT ON xxl_job_info

[0049] FOREACH ROW

[0050] DECLARE

[0051] v_newVal NUMBER(12) := 0;

[0052] BEGIN

[0053] v_newVal := select LAST_INSERT_ID();

[0054] update xxl_job_info set relation_id = mod(v_newVal, 2) where id = v_newVal;

[0055] END;

[0056] The xxl_job service ends with the same id first compete for the same lock in the xxl_job_lock table by using the select for update method, find the relation_id value equal to the id value of the xxl-job service end in the xxl_job_info table, and obtain the scheduled task corresponding to the id of the xxl-job service end, and execute the corresponding task scheduling by the xxl-job execution end.

[0057] On the other hand, the embodiment of the application provides a distributed task scheduling system based on an xxl-job framework, Figure 2 The structure of the distributed task scheduling system based on the xxl-job framework provided by the embodiment of the application is shown in the figure. Please refer to Figure 2The system comprises a task scheduling module, a task execution module and a mysql data unit.

[0058] The task scheduling module comprises at least one task scheduling cluster, and the task scheduling cluster comprises at least four xxl-job scheduling ends.

[0059] The source code of the xxl-job service end is modified, the JobScheduleHelper.java file under the xxl-job-admin module is found, the start method in the file is found, and the traditional method is to acquire the unique lock in the table by using a mysql statement select for update where lock_name=".

[0060] The four xxl-job scheduling ends are used as four nodes, two nodes are master nodes, and the other two nodes are slave nodes corresponding to the master nodes, and the master nodes and the slave nodes corresponding thereto have the same id value.

[0061] The task execution module comprises at least two xxl-job executors, and is used for performing specific scheduling task execution when the xxl-job scheduling end performs scheduling.

[0062] The appName parameter of the xxl-job execution end is set to be the same as the name of the xxl-job service end, is used as the basis for the xxl-job execution end to register to the xxl-job service end, and is used for establishing the connection between the xxl-job service end and the xxl-job execution end, so that the xxl-job service end can mobilize the xxl-job execution end to execute corresponding task scheduling.

[0063] The mysql data unit is used for storing timing scheduling task information, and comprises at least one of scheduling task execution time, scheduling task trigger time and execution class.

[0064] The xxl_job_lock table and the xxl_job_info table are created in the mysql storage unit, and the id values of the master node and the slave node are inserted into the xxl_job_lock table and the xxl_job_info table in the mysql storage unit. The relation_id field is added in the xxl_job_info table, and the relation_id field is used to store the result value of the scheduling task by taking the modulus of the primary key. The number of xxl-job scheduling ends is used as the modulus base, and the value of the relation_id field is equal to the id value of the xxl-job scheduling end node. The xxl_job service ends with the same id first compete for the same lock in the xxl_job_lock table by using the select for update mode, and find the relation_id value equal to the id value of the xxl-job service end in the xxl_job_info table, so as to obtain the scheduling task corresponding to the id of the xxl-job service end, and execute the corresponding task scheduling through the xxl-job execution end.

[0065] In one specific embodiment, there is a certain regional governance project task to be executed, and the xxl_job_info table is created to create the timing scheduling task to be executed. There are nearly 50 timing scheduling tasks in total, and most of the timing task times overlap. There are about 30 tasks to be executed within 5 minutes. If the original framework is used, only one execution task is executed for multiple scheduling ends, and the 50 scheduling tasks need to be logically processed and then executed. If the simultaneous scheduling task is time-consuming, the task will be delayed.

[0066] The improved xxl-job framework is used to build the scheduling system of the xxl-job service end, four scheduling ends are established, and the four scheduling ends are used as four nodes. Two nodes are master nodes, and the other two are slave nodes corresponding to the master nodes. The id of the scheduling end is set, and the master node and the corresponding slave node have the same id value. The id is equivalent to the basis for grouping the scheduling end. The scheduling ends with the same id are a group, and compete for the same distributed lock.

[0067] The execution end is built, and the execution end includes two executors. The appName parameter of the execution end is set to be the same as the name of the service end, which is used as the basis for registering the execution end to the service end, so as to establish the contact between the service end and the execution end, so that the service end can mobilize the execution end to execute the corresponding task scheduling.

[0068] The xxl_job_lock table and the xxl_job_info table are created in the mysql storage unit, and the id values of the master node and the slave node are inserted into the xxl_job_lock table and the xxl_job_info table in the mysql storage unit. The relation_id field is added in the xxl_job_info table, and the relation_id field is used to store the result value of the scheduling task by taking the modulus of the primary key. The number of xxl-job scheduling ends is used as the modulus base, and the value of the relation_id field is equal to the id value of the xxl-job server node. The servers with the same id first compete for the same lock in the xxl_job_lock table by using the select for update mode, and the relation_id value equal to the id value of the xxl-job server is searched in the xxl_job_info table, so as to obtain the scheduling task corresponding to the id of the xxl-job server, and the corresponding task scheduling is executed by the execution end.

[0069] The four scheduling ends are divided into two categories to execute a certain regional governance project task, and then 50 scheduling tasks are divided into two types, and each scheduling category executes 25 scheduling tasks, and the corresponding scheduling task is executed by the executor. In this way, the delay probability of the regional governance project task is greatly reduced.

[0070] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the application is not limited to the action sequence described, because according to the application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the application.

[0071] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0072] In several embodiments provided in the present application, it should be understood that the disclosed device can be implemented by other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0073] In addition, each of the function units in the embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0074] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned memory includes: a U disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0075] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be instructed by a program to be completed by relevant hardware, and the program can be stored in a computer readable memory, which can include a flash disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0076] The above is only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. A person of ordinary skill in the art will easily think of embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

[0077] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope recorded in the present disclosure.

[0078] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A distributed task scheduling method based on an xxl-job framework, characterized in that, The application comprises: building an xxl-job service end scheduling system, setting the id of the xxl-job service end, and obtaining a distributed lock through the id configured by the xxl-job service end; building an xxl-job execution end and establishing a connection between the xxl-job service end and the xxl-job execution end; xxl-job service ends with the same id compete for a distributed lock, and after successfully obtaining the distributed lock, the xxl-job service end finds a scheduling task corresponding to the id of the xxl-job service end and executes the corresponding task scheduling through the xxl-job execution end; The xxl-job service end scheduling system comprises at least one xxl-job scheduling end cluster and at least one mysql data unit; the xxl-job scheduling end cluster comprises at least four nodes, two of which are master nodes and the other two are slave nodes corresponding to the master nodes; the master nodes and their corresponding slave nodes have the same id value.

2. The distributed task scheduling method based on xxl-job framework of claim 1, wherein, The application builds an xxl-job execution end and establishes a connection between the xxl-job service end and the xxl-job execution end, which specifically comprises: building an xxl-job execution end and setting the appName parameter of the xxl-job execution end to be the same as the name of the xxl-job service end.

3. The distributed task scheduling method based on the xxl-job framework according to claim 1, further comprising creating an xxl_job_lock table and an xxl_job_info table in the mysql storage unit, and inserting the id values of the master nodes and the slave nodes into the xxl_job_lock table and the xxl_job_info table in the mysql storage unit.

4. The distributed task scheduling method based on xxl-job framework of claim 3, wherein, The xxl_job_info table is used to store the time of the timing scheduling task and the execution method of the scheduling task.

5. The distributed task scheduling method based on xxl-job framework of claim 4, wherein, A relation_id field is added to the xxl_job_info table, which is used to store the result value of the scheduling task taking the modulus of the primary key, and the number of xxl-job scheduling ends is used as the modulus base.

6. The distributed task scheduling method based on xxl-job framework of claim 5, wherein, The xxl_job_info table is used to store the time of the timing scheduling task and the execution method of the scheduling task. The application comprises:

7. A distributed task scheduling system based on an xxl-job framework, configured to implement the distributed task scheduling method based on the xxl-job framework according to any one of claims 1-6. a task scheduling module comprising at least one task scheduling cluster, the task scheduling cluster comprising at least four xxl-job scheduling ends; a task execution module comprising at least two xxl-job executors, used to execute specific scheduling tasks when the xxl-job scheduling ends perform scheduling; a mysql data unit used to store timing scheduling task information, comprising at least one of scheduling task execution time, scheduling task trigger time and execution type. ​ 8. The distributed task scheduling system based on xxl-job framework of claim 7, wherein, The four xxl-job scheduling ends are used as four nodes, two nodes are master nodes, and the other two nodes are slave nodes corresponding to the master nodes, and the master nodes and the corresponding slave nodes have the same id value.

Images