A task scheduling method and device
By introducing adapters into the task scheduling framework, using the data lock of the target database to achieve a pessimistic lock mechanism, the problem that task scheduling methods in the existing technology cannot meet high performance and high reliability at the same time, and the efficiency of the task processing system of financial institutions is achieved Run.
Patent Information
- Application Number
- CN202010218602.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-03-25
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-03-25
AI Technical Summary
Existing task scheduling methods cannot meet both high performance and high reliability, especially in the task processing systems of financial institutions, poor horizontal scalability of relational databases leads to poor performance, while the optimistic locking mechanism of new databases may lead to repeated tasks. Causes reliability issues.
By introducing adapters in the scheduling framework, the pessimistic lock mechanism is realized by using the data lock of the target database, the holding identifier of the data lock is obtained to judge the lock status, and the exclusiveness of the data lock is ensured through locking transactions, thus achieving high quality in the new database Performance and high reliability task scheduling.
It realizes high-performance and high-reliability task scheduling in the task processing system of financial institutions, avoids repeated task execution and performance bottlenecks, and meets the efficient operation needs of financial institutions for task processing.
Smart Images

Figure CN111444002B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the database field in the field of financial technology (Fintech), and in particular to a task scheduling method and device. Background Art
[0002] With the development of computer technology, more and more technologies are being applied in the financial field. Traditional finance is gradually transforming to Fintech. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology. The task processing systems of financial institutions are faced with a large number of tasks and massive amounts of data that need to be processed.
[0003] For tasks of financial institutions, tasks are generally scheduled through a scheduling framework. For example, the scheduling framework is Quartz. A basic feature of the scheduling framework is that for the same task, within any trigger time window, by locking the data used by the task, there is only one application instance that can execute the task, thereby preventing the same task from being repeatedly executed in the same time window and causing errors. However, the mutex lock mechanism of this task scheduling relies on the combination of the scheduling framework and the relational database, which is essentially achieved by locking data through the pessimistic lock mechanism of the relational database. The horizontal scalability of the relational database is poor, and when the amount of data is large, it will lead to poor performance of task scheduling. Although the current new database can be horizontally expanded, it uses an optimistic lock mechanism, which may cause the same task to be repeatedly executed by multiple application instances in the same time window and cause errors, that is, poor reliability. Therefore, the current task scheduling method cannot meet both high performance and high reliability at the same time, which is a problem that needs to be solved urgently. Summary of the invention
[0004] The present invention provides a task scheduling method and device, which solves the problem that the task scheduling method in the prior art cannot simultaneously meet high performance and high reliability.
[0005] In the first aspect, the present invention provides a task scheduling method, comprising: obtaining a first adapter in a scheduling framework; the first adapter is adapted to the target database; the first adapter is configured to replace the second adapter in the scheduling framework; the second adapter is adapted to a relational database; the first adapter and the second adapter are both used to seize a data lock of data to be called in the target database; the data lock is used to lock the data to be called; the data to be called is data required for the task to be scheduled; the data lock is determined through the first adapter; through the first adapter, a holding identifier of the data lock is obtained; the holding identifier is used to indicate that the data lock is in an idle state or an occupied state; if it is determined that the holding identifier indicates that the data lock is in an idle state, a lock grabbing transaction is submitted to the target database; the lock grabbing transaction is used to seize the data lock; if it is determined that the lock grabbing transaction is submitted successfully, the task to be scheduled is scheduled according to the data to be called; the successful submission of the lock grabbing transaction indicates that the holding identifier indicates that the data lock is in an occupied state.
[0006] In the above method, the data to be called is the data required for scheduling the task to be scheduled, and the data lock is used to lock the data to be called; the first adapter and the second adapter are both used to seize the data lock of the data to be called in the target database; but the second adapter is adapted to the relational database. In order to adapt to the target database, the first adapter is configured to replace the second adapter in the scheduling framework. After obtaining the first adapter in the scheduling framework, the data lock is determined through the first adapter, and the holding identifier of the data lock is obtained through the first adapter; the holding identifier is used to indicate that the data lock is in an idle state or an occupied state. After adding the holding identifier, it is still necessary to first determine whether the data lock is in an idle state before seizing the data lock. When When the holding flag indicates that the data lock is in an idle state, a lock grabbing transaction is submitted to the target database, but the implemented lock grabbing mechanism is a pessimistic lock mechanism, that is, it is first determined whether the lock is occupied. After it is determined that it is not occupied and the lock grabbing transaction is successfully submitted, it is determined that the data lock that locks the data to be called has been acquired, and the successful submission of the lock grabbing transaction indicates that the holding flag is modified to indicate that the data lock is in an occupied state. After it is determined that the data lock locks the data to be called, the scheduled task is scheduled according to the data to be called that has been locked by the data lock, so that the performance can be expanded through the target database, and the pessimistic lock grabbing mechanism can be implemented by adding a holding flag to the data lock, thereby providing a task scheduling method that meets both high performance and high reliability.
[0007] Optionally, the target database corresponds to a first lock record in the scheduling framework; the relational database corresponds to a second lock record in the scheduling framework; the determining of the data lock through the first adapter includes: determining the data lock from the first lock record through the first adapter according to the task to be scheduled, the data to be called and the second lock record; obtaining the holding identifier of the data lock through the first adapter includes: obtaining the holding identifier from the first lock record through the mutex lock acquisition interface of the first adapter; the mutex lock acquisition interface defines an acquisition function for the data lock in the target database, and the acquisition function defines a mechanism for the holding identifier of the data lock.
[0008] In the above method, the target database corresponds to the first lock record in the scheduling framework; the relational database corresponds to the second lock record in the scheduling framework, so the first adapter can be used to determine the data lock from the first lock record of the target database in the scheduling framework according to the task to be scheduled, the data to be called and the second lock record; further, since the mutex lock acquisition interface in the first adapter defines the acquisition function of the data lock in the target database, and the acquisition function defines the mechanism of the holding identification of the data lock, the mutex lock acquisition interface of the first adapter can be used to mark the data lock as held, and the holding identification of the data lock can be obtained from the first lock record, thereby obtaining the holding identification of the data lock through the first lock record, thereby providing a method for determining the data lock and obtaining the holding identification of the data lock through the first lock record dedicated to the target database and the mutex lock acquisition interface of the first adapter.
[0009] Optionally, the data lock is determined from the first lock record through the first adapter according to the task to be scheduled, the data to be called and the second lock record, including: determining the scheduler corresponding to the task to be scheduled in the scheduling framework through the first adapter, and determining the first lock of the data to be called from the second lock record; determining the name of the second lock of the data to be called in the first lock record through the first adapter according to the name of the scheduler and the name of the first lock; the second lock is the data lock of the data to be called.
[0010] In the above method, the scheduler corresponding to the task to be scheduled in the scheduling framework is first determined through the first adapter, and the first lock is determined from the original second lock record, so that the name of the second lock can be determined from the first lock record corresponding to the newly defined first adapter according to the name of the scheduler and the name of the first lock, thereby realizing the determination of the data lock of the data to be called in the new database by introducing the first adapter in the scheduling framework.
[0011] Optionally, the lock grabbing transaction is created through the mutex lock acquisition interface; the lock grabbing transaction is used to modify the indication of the holding identifier in the first lock record to: the data lock is in an occupied state; and the lock grabbing transaction is submitted through the mutex lock acquisition interface.
[0012] In the above manner, a lock grabbing transaction can be created through the mutex acquisition interface, thereby implementing transaction packaging through the mutex acquisition interface of the first adapter, and further submitting the lock grabbing transaction through the mutex acquisition interface to modify the indication of the holding identifier in the first lock record.
[0013] Optionally, after obtaining the holding identifier of the data lock, it also includes: if the holding identifier indicates that the data lock is in an occupied state, or the holding identifier indicates that the data lock is in an idle state and the lock-grabbing transaction fails to be submitted, then determining the cumulative number of times the holding identifier has been obtained as of the time the holding identifier is obtained this time; if the cumulative number of times the holding identifier has been obtained is less than a preset number, then executing the step of obtaining the holding identifier of the data lock according to a preset period.
[0014] In the above manner, the holding identifier indicates that the data lock is in an occupied state, or when the data lock is in an idle state and the lock-grabbing transaction fails to be submitted, the cumulative number of times the data lock has been acquired is continuously determined. If the cumulative number of times is less than the preset number of times, the holding identifier of the data lock is continuously acquired, thereby providing a method for periodically competing for data locks.
[0015] Optionally, if the cumulative number of times the holding identifier has been obtained is greater than or equal to the preset number of times, it is determined whether the following situation exists in the cumulative number of holding identifier acquisition situations: the holding identifier obtained this time indicates that the data lock is in an idle state, and the submission of a lock grabbing transaction to the target database after the holding identifier is obtained this time fails; if so, the cumulative number of times is updated to 0; if not, an unlocking transaction is submitted; the unlocking transaction is used to release the lock of the data lock on the data to be called; the indication of the holding identifier is modified to: the data lock is in an idle state.
[0016] In the above manner, if the cumulative number of times the holding identifier has been obtained is greater than or equal to the preset number of times, different measures can be taken according to whether there are specific circumstances. If the following circumstances exist: the obtained holding identifier indicates that the data lock is in an idle state, and the submission of a lock grabbing transaction to the target database fails after the holding identifier is obtained this time, it means that the data lock is still released, but the competition is too fierce, resulting in the failure to submit the lock grabbing transaction. Therefore, the competition can continue, and the cumulative number of times is updated to 0. Or if the above situation does not exist, it means that the data lock has not been released, then an unlocking transaction is submitted to forcibly release the lock of the data to be called by the data lock; the indication of the holding identifier is modified to: the data lock is in an idle state, so that the data lock is restored to an idle state that can be contested.
[0017] Optionally, after submitting the lock grabbing transaction to the target database, the method further includes: if the lock grabbing transaction fails to be submitted, waiting for a first preset time period, and re-executing the step of obtaining the holding identifier of the data lock.
[0018] In the above method, if the lock-grabbing transaction fails to be submitted, it means that the data lock is in an idle state, but the data lock has not been competed for. Then, after waiting for a preset time, the step of obtaining the holding identifier of the data lock can be re-executed, thereby providing a method for waiting for a preset time to compete for the data lock.
[0019] Optionally, after determining that the lock-grabbing transaction is successfully submitted, it also includes: recording the application instance and scheduling thread that successfully occupy the data lock; if the occupancy time of the application instance and the scheduling thread occupying the data lock is greater than a second preset time, then modifying the indication of the holding identifier to: the data lock is in an idle state.
[0020] Under the above method, after determining that the lock-grabbing transaction has been successfully submitted, the application instance and scheduling thread that have successfully occupied the data lock can also be recorded, and when the occupation time is greater than the second preset time, the indication of the holding flag can be modified to: the data lock is in an idle state, allowing other application instances to also obtain the competing data lock, thereby avoiding the application instance and the scheduling thread from occupying the data lock all the time.
[0021] Optionally, after scheduling the task to be scheduled according to the data to be called, the method further includes: if it is determined that the task to be scheduled is completely scheduled, submitting an unlock transaction; the unlock transaction is used to modify the indication of the holding flag to: the data lock is in an idle state.
[0022] Under the above method, after it is determined that the scheduled task has been scheduled, the unlocking transaction is submitted. Since the unlocking transaction is used to modify the indication of the holding identifier to: the data lock is in an idle state, the data lock is recovered in time after the scheduled task has been scheduled, so that the data lock can continue to be used by competition.
[0023] In a second aspect, the present invention provides a task scheduling device, comprising: an acquisition module, used to acquire a first adapter in a scheduling framework; the first adapter is adapted to the target database; the first adapter is configured to replace the second adapter in the scheduling framework; the second adapter is adapted to a relational database; the first adapter and the second adapter are both used to seize a data lock of data to be called in the target database; the data lock is used to lock the data to be called; the data to be called is data required for the task to be scheduled; a determination module, used to determine the data lock through the first adapter; through the first adapter, obtain a holding identifier of the data lock; the holding identifier is used to indicate that the data lock is in an idle state or an occupied state; if it is determined that the holding identifier indicates that the data lock is in an idle state, submit a lock grabbing transaction to the target database; the lock grabbing transaction is used to seize the data lock; a scheduling module, used to schedule the task to be scheduled according to the data to be called if it is determined that the lock grabbing transaction is submitted successfully; the successful submission of the lock grabbing transaction indicates that the holding identifier indicates that the data lock is in an occupied state.
[0024] Optionally, the target database corresponds to a first lock record in the scheduling framework; the relational database corresponds to a second lock record in the scheduling framework; the determination module is specifically used to: determine the data lock from the first lock record through the first adapter according to the task to be scheduled, the data to be called and the second lock record; obtain the holding identifier from the first lock record through the mutex lock acquisition interface of the first adapter; the mutex lock acquisition interface defines an acquisition function for the data lock in the target database, and the acquisition function defines a mechanism for the holding identifier of the data lock.
[0025] Optionally, the determination module is specifically used to: determine the scheduler corresponding to the task to be scheduled in the scheduling framework through the first adapter, and determine the first lock of the data to be called from the second lock record; determine the name of the second lock of the data to be called in the first lock record through the first adapter according to the name of the scheduler and the name of the first lock; the second lock is the data lock of the data to be called.
[0026] Optionally, the determination module is specifically used to: create the lock grabbing transaction through the mutex lock acquisition interface; the lock grabbing transaction is used to modify the indication of the holding identifier in the first lock record to: the data lock is in an occupied state; submit the lock grabbing transaction through the mutex lock acquisition interface.
[0027] Optionally, the determination module is also used to: if the holding identifier indicates that the data lock is in an occupied state, or the holding identifier indicates that the data lock is in an idle state and the lock-grabbing transaction fails to be submitted, then determine the cumulative number of times the holding identifier has been obtained as of this time when the holding identifier is obtained; if the cumulative number of times the holding identifier has been obtained is less than a preset number, then execute the step of obtaining the holding identifier of the data lock according to a preset period.
[0028] Optionally, the determination module is also used to: if the cumulative number of times the holding identifier has been obtained is greater than or equal to the preset number of times, then determine whether the following situation exists in the cumulative number of holding identifier acquisition situations: the holding identifier obtained this time indicates that the data lock is in an idle state, and the submission of a lock-grabbing transaction to the target database after the holding identifier is obtained this time fails; if so, update the cumulative number of times to 0; if not, submit an unlocking transaction; the unlocking transaction is used to release the lock of the data lock on the data to be called; modify the indication of the holding identifier to: the data lock is in an idle state.
[0029] Optionally, the determination module is further configured to: if the lock-grabbing transaction fails to be submitted, re-execute the step of obtaining the holding identifier of the data lock after waiting for a first preset time period.
[0030] Optionally, the determination module is also used to: record the application instance and scheduling thread that successfully occupy the data lock; if the occupancy time of the application instance and the scheduling thread for the data lock is greater than a second preset time, modify the indication of the holding identifier to: the data lock is in an idle state.
[0031] Optionally, the determination module is further used to: submit an unlock transaction if it is determined that the to-be-scheduled task has been scheduled; the unlock transaction is used to modify the indication of the holding flag to: the data lock is in an idle state.
[0032] The beneficial effects of the above-mentioned second aspect and each optional method of the second aspect can refer to the beneficial effects of the above-mentioned first aspect and each optional method of the first aspect, and will not be repeated here.
[0033] In a third aspect, the present invention provides a computer device, including a program or an instruction, which, when executed, is used to execute the above-mentioned first aspect and each optional method of the first aspect.
[0034] In a fourth aspect, the present invention provides a storage medium, including a program or an instruction, which, when executed, is used to execute the above-mentioned first aspect and each optional method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A schematic diagram of the steps of a task scheduling method provided in an embodiment of the present application;
[0036] Figure 2 A schematic diagram of a process of seizing an optimistic lock in a task scheduling method provided in an embodiment of the present application;
[0037] Figure 3 A schematic diagram of the structure of a task scheduling device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] In order to better understand the above-mentioned technical scheme, the above-mentioned technical scheme will be described in detail below in combination with the accompanying drawings and specific implementation methods of the specification. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments may be combined with each other.
[0039] Atomicity, consistency, durability, and isolation (ACID) of transactions.
[0040] Pessimistic lock: A mechanism used by a database to ensure transaction isolation and consistency. It means that before a thread modifies data, it must first lock the data to prevent other threads from modifying the data at the same time. If other threads have already locked the data, the current thread must be blocked and wait until the holding thread releases the lock and the current thread successfully locks the data, or the thread is blocked until the transaction times out.
[0041] Optimistic locking: A mechanism used by a database to ensure transaction isolation and consistency. It means that a thread does not need to lock the data before modifying it. The database only detects whether there is a conflict when the transaction is committed. If there is a conflict, the transaction is rolled back and an error is reported.
[0042] Relational database: has ACID characteristics, but poor horizontal scalability, such as Oracle, DB2, MySQL, etc.
[0043] Non-relational databases (Not Only SQL, NoSQL) do not have the ACID characteristics of traditional relational databases, but have the characteristics of unlimited horizontal expansion, such as: HBase, Hive, MongoDB, etc.
[0044] New SQL databases have the characteristics of both relational and non-relational databases, such as VoltDB, ScaleDB, TiDB, etc. New SQL databases have the ACID characteristics of traditional relational databases and the unlimited horizontal expansion characteristics of non-relational databases. They are also compatible with MySQL protocols and syntax, such as TiDB. However, the difference is that their lock type is optimistic, while the lock type of relational databases is pessimistic.
[0045] Mutex lock: also known as exclusive lock, is a mutual exclusion mechanism compiled in an application system, which means that before a thread performs a specific operation or modifies a specific piece of data, it must first lock a unique and exclusive object. This lock is used to ensure that at any time, only one thread can perform a specific operation or modify a specific piece of data. If and only if the thread that owns the object lock releases the lock, other threads are allowed to compete to lock the object again. This mechanism of mutual exclusion of each thread in the application system is called a mutual exclusion lock, which is generally implemented through the pessimistic lock mechanism of the database.
[0046] Quartz: A well-known and widely used task scheduling framework based on the Java language.
[0047] Redis: An in-memory database.
[0048] When financial institutions (banking institutions, insurance institutions or securities institutions) are operating their businesses (such as bank loan business, deposit business, etc.), the task processing systems of Internet banks are faced with a large number of tasks and massive amounts of data to be processed. In order to improve the efficiency of task processing, leaving aside the database, the deployment of its application layer is composed of multiple hardware (physical machines, virtual machines, or containers), each of which contains one or more application instances, and these application instances may schedule tasks. For task scheduling: some tasks may be triggered by requests submitted by peripheral systems, or they may be triggered by the task processing system itself, such as: various transaction flow processing tasks, reconciliation tasks, end-of-day tasks, etc.
[0049] For tasks of financial institutions, the scheduling framework is generally used to schedule tasks. A basic feature of the scheduling framework is that for the same task, within any trigger time window, by locking the data used by the task, only one application instance can execute the task, thereby preventing the same task from being executed repeatedly within the same time window and causing errors.
[0050] In the prior art, the use of relational databases will lead to poor performance of task scheduling due to poor horizontal scalability. If a new database is used, its optimistic locking mechanism will cause poor reliability of task scheduling. Therefore, the current task scheduling method cannot meet both high performance and high reliability. This situation does not meet the needs of financial institutions such as banks and cannot ensure the efficient operation of various businesses of financial institutions.
[0051] For this reason, Figure 1 As shown, the present application provides a task scheduling method.
[0052] Step 101: Obtain the first adapter in the scheduling framework.
[0053] Step 102: Determine the data lock through the first adapter.
[0054] Step 103: Obtain the ownership identifier of the data lock through the first adapter.
[0055] The holding flag is used to indicate that the data lock is in an idle state or an occupied state.
[0056] Step 104: If it is determined that the holding flag indicates that the data lock is in an idle state, a lock grabbing transaction is submitted to the target database.
[0057] The lock grabbing transaction is used to grab the data lock.
[0058] Step 105: If it is determined that the lock-grabbing transaction is submitted successfully, the task to be scheduled is scheduled according to the data to be called.
[0059] The successful submission of the lock-grabbing transaction indicates that the holding flag indicates that the data lock is in an occupied state.
[0060] The method in steps 101 to 105 can be implemented by a modified scheduling framework, such as Quartz with an optimistic lock holding identification mechanism added, but it cannot be implemented in the original Quartz because the original Quartz applies the pessimistic lock mechanism of the relational database. There are many options for the target database in steps 101 to 105, such as a new database, specifically TiDB.
[0061] It should be noted that the first adapter in steps 101 to 105 is adapted to the target database; the first adapter is set to replace the second adapter in the scheduling framework, and the second adapter is adapted to the relational database; the first adapter and the second adapter are both used to seize the data lock of the data to be called in the target database; the data lock is used to lock the data to be called; the data to be called is the data required by the task to be scheduled. Therefore, the first adapter can be used to replace the second adapter to achieve adaptation with the target database, and the first adapter can be used to add a holding flag through steps 101 to 105 to achieve the lock grabbing mechanism of the pessimistic lock.
[0062] It should be noted that the second adapter is used to obtain the data lock through the second lock record corresponding to the relational database in the scheduling framework, that is, the second adapter is the original adapter in the scheduling framework, and what is implemented in the second adapter is to obtain the data lock through the second lock record corresponding to the relational database in the scheduling framework. In the embodiment of the present application, it is necessary to obtain the data lock through the first lock record corresponding to the target database in the scheduling framework. Therefore, it is necessary to create a first adapter that can realize the above function, and use the first adapter to replace the second adapter in the scheduling framework.
[0063] In an optional implementation, the data lock can be determined and the data lock holding identifier can be obtained by adding the first lock record corresponding to the target database in the scheduling framework. Step 101 can be specifically performed in the following manner:
[0064] The data lock is determined from the first lock record through the first adapter according to the task to be scheduled, the data to be called and the second lock record.
[0065] The target database corresponds to the first lock record in the scheduling framework; the relational database corresponds to the second lock record in the scheduling framework.
[0066] For example, the first lock record may actually be a mutex definition table, the structure of which may be as shown in Table 1, and the table name may be recorded as QUARTZ_LOCKS_TIDB.
[0067]
[0068] Table 1
[0069] The first lock record may be pre-placed with a mutex lock. For example, when the scheduling framework is Quartz, two mutex locks are pre-defined in the table:
[0070] Mutex 1:
[0071] lock_name: scheduler-name.STATE_ACCESS;
[0072] host_ip: 0;
[0073] thread_id: 0;
[0074] update_datetime: 0.
[0075] Mutex 2:
[0076] lock_name: scheduler-name.TRIGGER_ACCESS;
[0077] host_ip: 0;
[0078] thread_id: 0;
[0079] update_datetime: 0.
[0080] Therefore, mutex 1 and mutex 2 can correspond to the two original mutexes in Quartz. The purpose of creating the first lock record is to replace the original QUARTZ_LOCKS table (the second lock record) in Quartz. In addition, the scheduler name (scheduler-name) and the names of the two mutexes (STATE_ACCESS and TRIGGER_ACCESS) in the Quartz context can be placed in it for backup.
[0081] For the first adapter and the second adapter, for example, when the scheduling framework is Quartz, Semaphore is a pessimistic lock acquisition and release interface inside Quartz. The default implementation class of this interface (the second adapter) is a Semaphore implementation class suitable for relational databases, such as: DBSemaphore, StdRowLockSemaphore, UpdateLockRowSemaphore. The second adapter includes one or more implementation classes in the Semaphore implementation class. Semaphore contains three methods (functions):
[0082] (1) boolean obtainLock(Connection conn, String lockName), used to obtain a mutex lock;
[0083] (2) void releaseLock(String lockName), used to release the mutex lock;
[0084] (3) boolean requiresConnection(), indicating whether the above obtainLock database connection is required.
[0085] Based on the second adapter, the present invention adds a new interface: PessimisticLockFetcher, and the corresponding implementation class (first adapter):
[0086] TiDBLockFetcher and QuartzSemaphoreTiDBAdapter, the first adapter includes one or more of TiDBLockFetcher and QuartzSemaphoreTiDBAdapter.
[0087] The first adapter can be placed in the Quartz runtime environment to replace the second adapter, so as to obtain the data lock of the target database and implement the data lock mechanism of the target database together with the holding identification mechanism. For example, the QuartzSemaphoreTiDBAdapter adapter can be placed in the Quartz runtime environment to replace its default StdRowLockSemaphore implementation class, so as to achieve the purpose of running Quartz on TiDB.
[0088] Specifically, the replacement method can be: adding the following two lines to the quartz.properties configuration file to instruct the Quartz framework to replace its default first adapter with the first adapter written by the present invention at runtime:
[0089] "org.quartz.jobStore.acquireTriggersWithinLock=true
[0090] org.quartz.jobStore.lockHandler.class=com.mycompany.QrtzSemaphoreTiD
[0091] BAdapter".
[0092] It can be seen that based on the present invention, the scheduling framework (such as Quartz) will ignore its internal default mutex acquisition method when running, and use QuartzSemaphoreTiDBAdapter to acquire the mutex; and QuartzSemaphoreTiDBAdapter ignores Quartz's default mutex definition table, and uses the QUARTZ_LOCKS_TIDB table instead. By operating QUARTZ_LOCKS_TIDB, the pessimistic lock behavior based on TiDB's optimistic lock is realized, achieving the purpose of mutual exclusion under multi-instance multi-threading, and not intervening in the transactions originally existing in Quartz. In this way, the purpose of running Quartz on TiDB is achieved.
[0093] The following describes the TiDBLockFetcher implementation class and the QuartzSemaphoreTiDBAdapter implementation class in combination with specific implementation methods.
[0094] It should be noted that, in step 101, more specifically, the method of determining the data lock from the first lock record through the first adapter according to the task to be scheduled, the data to be called and the second lock record may be as follows:
[0095] Determine the scheduler corresponding to the task to be scheduled in the scheduling framework through the first adapter, and determine the first lock of the data to be called from the second lock record; determine the name of the second lock of the data to be called in the first lock record according to the name of the scheduler and the name of the first lock through the first adapter.
[0096] The second lock is a data lock of the data to be called.
[0097] The above method of determining the data lock of the data to be called in the target database can be implemented by the QuartzSemaphoreTiDBAdapter implementation class. The QuartzSemaphoreTiDBAdapter implementation class is extended from the TiDBLockFetcher implementation class and implements the methods in the Semaphore interface at the same time. Specifically:
[0098] (1) Implement the data lock acquisition method boolean obtainLock(Connection conn,String lockName):
[0099] From the Quartz context, obtain the scheduler corresponding to the task to be scheduled in the scheduling framework, and then concatenate it with the first lock (lockName) of the data to be called in the second lock record, and concatenate the name of the scheduler with the name of the first lock; use the concatenated name to call the obtain(String lockName) method of TiDBLockFetcher to obtain the second lock from the first lock record, thereby obtaining the data lock (expressed as a mutex lock in the scheduling framework).
[0100] (2) Implement the optimistic lock release method void releaseLock(String lockName):
[0101] From the Quartz context, obtain the scheduler corresponding to the task to be scheduled in the scheduling framework, and then concatenate it with the first lock (lockName) of the data to be called determined in the second lock record, and concatenate the name of the scheduler with the name of the first lock; use the concatenated name to call the release(String lockName) method of TiDBLockFetcher to release the data lock (expressed as a mutex lock in the scheduling framework).
[0102] (3) Implement the database connection method boolean requiresConnection()
[0103] The backend database is usually accessed using frameworks such as MyBatis, so an error (false) is returned directly here, that is, the database connection passed in by Quartz is not used.
[0104] In the above optional implementation of "determining the data lock by adding the first lock record corresponding to the target database in the scheduling framework in the scheduling framework", corresponding to the execution mode of step 101, step 102 can be executed in the following manner:
[0105] The holding identifier is obtained from the first lock record through the mutex lock acquisition interface of the first adapter.
[0106] The acquisition function of the optimistic lock in the new database is defined in the mutex acquisition interface, and the mechanism of the holding mark of the optimistic lock is defined in the acquisition function. Therefore, the mechanism of holding mark can be implemented for the data lock through the mutex acquisition interface of the first adapter, and the holding mark of the data lock can be obtained from the first lock record, so as to obtain the holding mark of the data lock through the first lock record.
[0107] For example, the mutex acquisition interface (which can be expressed as boolean obtain(String lockName)) can be implemented by the TiDBLockFetcher implementation class in the first adapter. The mutex acquisition interface can convert the optimistic lock mechanism of the target database (such as TiDB) into a pessimistic lock behavior by retrying transaction packaging and setting a holding mark, thereby achieving the purpose of the mutex. Specifically, the mutex acquisition interface can define the following steps:
[0108] (i) Create a new synchronous waiting thread to completely isolate the new transaction from the original Quartz transaction.
[0109] (ii) Start a transaction in a new thread: start transaction.
[0110] (iii) Execution Order:
[0111] select * from QUARTZ_LOCKS_TIDB where lock_name = 'lockName' and host_ip = 0 for update; lockName is the parameter passed in by the method; this command will instruct TiDB to add an optimistic lock to the 'lockName' record in the QUARTZ_LOCKS_TIDB table.
[0112] If the result set is not empty (the holding flag indicates that the optimistic lock is in the idle state), proceed to the next step. Otherwise (the holding flag indicates that the optimistic lock is in the occupied state), it means that the specified type of lock is being held by other scheduling threads. Roll back the transaction and wait for a certain period of time before returning to (ii).
[0113] (iv) Execute the command: update QUARTZ_LOCKS_TIDB set host_ip = 'current instance ip', thread_id = 'Quartz scheduling thread id', update_datetime = 'current date time' where lock_name = 'lockName' and host_ip = 0; this command sets the instance ip and scheduling thread id as holding tags on the 'lockName' record.
[0114] If the result is 1, proceed to the next step. Otherwise, it means that another thread has acquired the lock first, so roll back the transaction, wait for a certain period of time, and then return to (ii).
[0115] (v) Commit the transaction: commit; if the transaction is successfully committed, it means that the mutex lock is successfully acquired. Otherwise, it means that there is a transaction conflict with other scheduled threads and the mutex lock acquisition fails. After waiting for a certain period of time, return to (ii).
[0116] Therefore, after submitting the lock grabbing transaction to the new database in step 103, an optional implementation method may be as follows:
[0117] If the lock grabbing transaction fails to be submitted, the step of obtaining the holding identifier of the data lock is performed again after waiting for a first preset time. This implementation method can be defined in the TiDBLockFetcher implementation class.
[0118] Correspondingly, there is also a mutex release interface (expressed as void release(String lockName)), which can define the following steps:
[0119] (i) Also create a new synchronous waiting thread.
[0120] (ii) Start a transaction: begin transaction;
[0121] (iii) Execute the command: update QUARTZ_LOCKS_TIDB set host_ip=0, thread_id=0, update_datetime='current date and time' where lock_name='lockName' and host_ip='current instance ip' and thread_id='Quartz scheduling thread ID'; this command will clear the holding mark of the current thread on the lock.
[0122] (iv) Commit the transaction: commit.
[0123] Based on the execution mode of step 102, the mode of submitting the lock grabbing transaction to the target database in step 103 may be: creating the lock grabbing transaction through the mutex acquisition interface; submitting the lock grabbing transaction through the mutex acquisition interface.
[0124] The lock grabbing transaction is used to modify the indication of the holding flag in the first lock record to: the data lock is in an occupied state.
[0125] After step 102, the holding flag indicates that the optimistic lock may still be in an occupied state. Based on this, an optional implementation is as follows:
[0126] If the holding identifier indicates that the data lock is in an occupied state, or the holding identifier indicates that the data lock is in an idle state and the lock-grabbing transaction fails to be submitted, then the cumulative number of times the holding identifier has been obtained as of the time the holding identifier is obtained this time is determined; if the cumulative number of times the holding identifier has been obtained is less than the preset number, then the step of obtaining the holding identifier of the data lock (i.e., step 102) is executed according to the preset period.
[0127] The above implementation can be implemented by adding the number of acquisitions (or the number of retries) and the preset period (T) to the boolean obtain(String lockName) method of the TiDBLockFetcher implementation class. If the cumulative number is less than the preset number, when the timer is equal to T, the process returns to execute step (ii) in the boolean obtain(String lockName) method of the TiDBLockFetcher implementation class.
[0128] Based on the above method of indicating that the optimistic lock is in an occupied state by the holding flag, when a Quartz scheduling thread acquires the mutex, if an abnormal situation such as a sudden crash occurs, the mutex will not be released. For this situation, a possible implementation method can be as follows:
[0129] If the cumulative number of times the holding identifier has been obtained is greater than or equal to the preset number of times, determine whether the following situation exists in the cumulative number of holding identifier acquisition situations: the holding identifier obtained this time indicates that the data lock is in an idle state, and the lock grabbing transaction submitted to the target database after the holding identifier is obtained this time fails; if so, update the cumulative number of times to 0; if not, submit an unlocking transaction; the unlocking transaction is used to release the lock of the data lock on the data to be called; modify the indication of the holding identifier to: the data lock is in an idle state.
[0130] The above implementation can be implemented by adding the acquisition times (or the retry times) and the preset period (T) to the boolean obtain(String lockName) method of the TiDBLockFetcher implementation class. For example, in the mutex acquisition interface in the TiDBLockFetcher implementation class, the return result of each execution of step (iii) is empty, indicating that the specified lock has been occupied by an instance and has not been released for a long time. At this time, the command update QUARTZ_LOCKS_TIDB set host_ip=0,thread_id=0where lock_name='lockName' can be executed asynchronously to force the release of the occupied mutex.
[0131] Furthermore, the mutex lock can be released according to conditions.
[0132] Asynchronously execute the command: update QUARTZ_LOCKS_TIDB set host_ip=0,thread_id=0where lock_name='lockName'and update_datetime<'current date and time minus unlocking time', which will force the release of the mutex lock that is held and occupied for a period longer than the unlocking time. If the unlocking time is 2 minutes, the purpose of subtracting 2 minutes is to prevent forced unlocking due to misjudgment, because the time for each lock acquisition, query modification submission task, and lock release will not exceed 2 minutes in principle. If the maximum number of acquisitions (or retries), that is, the preset number of times, is exceeded, and the return result of each execution of step (iii) is not completely empty, it means that the competition is fierce. At this time, there is no need to force unlocking, and only need to wait for the next trigger of the scheduling thread.
[0133] The above-mentioned recording of the cumulative number of times the holder of the identification has been obtained can be combined with the optional implementation of the entire process Figure 2 understand. Figure 2 The maximum number of retries in is the preset number, and the unlocking time is 2 minutes as an example. After the lock grabbing transaction is started, the lock with the specified name and free is queried. If the result is empty, the transaction is rolled back directly; if the result is not empty, the data lock is grabbed to put the Internet protocol (IP) address of the application instance that started the lock grabbing transaction and the identification (ID) of the scheduling thread into it, and the lock grabbing transaction is submitted. If the submission of the lock grabbing transaction fails, or the result is empty, check whether the cumulative number of times exceeds the maximum number of retries. If it does not exceed, reopen the lock grabbing transaction. If it exceeds, check whether the results obtained each time in the cumulative number are empty. If so, it is unlocked asynchronously, and the data lock (2 minutes ago) can be released through the mutex release interface; otherwise, the synchronous waiting thread ends and the lock grabbing failure is returned.
[0134] After determining in step 105 that the lock-grabbing transaction is successfully submitted, an optional implementation is as follows:
[0135] Record the duration of the application instance and the scheduling thread occupying the data lock; if the duration of the application instance and the scheduling thread occupying the data lock is greater than a second preset duration, modify the indication of the holding flag to: the data lock is in an idle state. This implementation method can be defined in the TiDBLockFetcher implementation class.
[0136] After step 105 is executed, an optional implementation is as follows:
[0137] If it is determined that the scheduled task is completed, an unlock transaction is submitted; the unlock transaction is used to modify the indication of the holding flag to: the data lock is in an idle state. This implementation method can be implemented in the mutex release interface in the TiDBLockFetcher implementation class.
[0138] like Figure 3 As shown, the present invention provides a task scheduling device, including: an acquisition module 301, used to acquire a first adapter in a scheduling framework; the first adapter is adapted to the target database; the first adapter is set to replace the second adapter in the scheduling framework; the second adapter is adapted to a relational database; the first adapter and the second adapter are both used to seize the data lock of the data to be called in the target database; the data lock is used to lock the data to be called; the data to be called is the data required by the task to be scheduled; a determination module 302, used to determine the data lock through the first adapter; through the first adapter, obtain the holding identifier of the data lock; the holding identifier is used to indicate that the data lock is in an idle state or an occupied state; if it is determined that the holding identifier indicates that the data lock is in an idle state, submit a lock grabbing transaction to the target database; the lock grabbing transaction is used to seize the data lock; a scheduling module 303, used to schedule the task to be scheduled according to the data to be called if it is determined that the lock grabbing transaction is submitted successfully; the successful submission of the lock grabbing transaction indicates that the holding identifier indicates that the data lock is in an occupied state.
[0139] Optionally, the target database corresponds to a first lock record in the scheduling framework; the relational database corresponds to a second lock record in the scheduling framework; the determination module 302 is specifically used to: determine the data lock from the first lock record through the first adapter according to the task to be scheduled, the data to be called and the second lock record; obtain the holding identifier from the first lock record through the mutex lock acquisition interface of the first adapter; the mutex lock acquisition interface defines an acquisition function for the data lock in the target database, and the acquisition function defines a mechanism for the holding identifier of the data lock.
[0140] Optionally, the determination module 302 is specifically used to: determine the scheduler corresponding to the task to be scheduled in the scheduling framework through the first adapter, and determine the first lock of the data to be called from the second lock record; determine the name of the second lock of the data to be called in the first lock record through the first adapter according to the name of the scheduler and the name of the first lock; the second lock is the data lock of the data to be called.
[0141] Optionally, the determination module 302 is specifically used to: create the lock grabbing transaction through the mutex lock acquisition interface; the lock grabbing transaction is used to modify the indication of the holding identifier in the first lock record to: the data lock is in an occupied state; submit the lock grabbing transaction through the mutex lock acquisition interface.
[0142] Optionally, the determination module 302 is also used to: if the holding identifier indicates that the data lock is in an occupied state, or the holding identifier indicates that the data lock is in an idle state and the lock-grabbing transaction fails to be submitted, then determine the cumulative number of times the holding identifier has been obtained as of this time the holding identifier is obtained; if the cumulative number of times the holding identifier has been obtained is less than a preset number, then execute the step of obtaining the holding identifier of the data lock according to a preset period.
[0143] Optionally, the determination module 302 is also used to: if the cumulative number of times the holding identifier has been obtained is greater than or equal to the preset number of times, then determine whether the following situation exists in the cumulative number of holding identifier acquisition situations: the holding identifier obtained this time indicates that the data lock is in an idle state, and the submission of a lock grabbing transaction to the target database after the holding identifier is obtained this time fails; if so, update the cumulative number of times to 0; if not, submit an unlocking transaction; the unlocking transaction is used to release the lock of the data lock on the data to be called; modify the indication of the holding identifier to: the data lock is in an idle state.
[0144] Optionally, the determination module 302 is further configured to: if the lock grabbing transaction fails to be submitted, wait for a first preset time period and then re-execute the step of obtaining the holding identifier of the data lock.
[0145] Optionally, the determination module 302 is also used to: record the application instance and scheduling thread that successfully occupy the data lock; if the occupancy time of the application instance and the scheduling thread for the data lock is greater than a second preset time, the indication of the holding identifier is modified to: the data lock is in an idle state.
[0146] Optionally, the determination module 302 is further used to: submit an unlock transaction if it is determined that the to-be-scheduled task has been scheduled; the unlock transaction is used to modify the indication of the holding flag to: the data lock is in an idle state.
[0147] An embodiment of the present application provides a computer device, including a program or an instruction. When the program or the instruction is executed, it is used to execute a task scheduling method and any optional method provided in the embodiment of the present application.
[0148] An embodiment of the present application provides a storage medium, including a program or an instruction, which, when executed, is used to execute a task scheduling method and any optional method provided in an embodiment of the present application.
[0149] Finally, it should be noted that: those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.
[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0152] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A task scheduling method, It is characterized in that include: Get the first adapter in the scheduling framework; The first adapter is adapted to a target database, and the target database corresponds to a first lock record in the scheduling framework; The first adapter is configured to replace the second adapter in the scheduling framework; The second adapter is adapted to a relational database, and the relational database corresponds to a second lock record in the scheduling framework; the first adapter and the second adapter are both used to seize a data lock of the data to be called in the target database; The data lock is used to lock the data to be called; The data to be called is the data required by the task to be scheduled; Determining the data lock from the first lock record through the first adapter according to the task to be scheduled, the data to be called and the second lock record; Acquiring, through the first adapter, a holding identifier of the data lock; the holding identifier is used to indicate whether the data lock is in an idle state or an occupied state; If it is determined that the holding flag indicates that the data lock is in an idle state, submitting a lock grabbing transaction to the target database; the lock grabbing transaction is used to grab the data lock; If it is determined that the lock-grabbing transaction is submitted successfully, scheduling the task to be scheduled according to the data to be called; The successful submission of the lock-grabbing transaction indicates that the holding flag indicates that the data lock is in an occupied state.
2. The method according to claim 1, It is characterized in that The acquiring, through the first adapter, the holding identifier of the data lock includes: The holding identifier is obtained from the first lock record through the mutex lock acquisition interface of the first adapter; the mutex lock acquisition interface defines an acquisition function for the data lock in the target database, and the acquisition function defines a mechanism for the holding identifier of the data lock.
3. The method according to claim 2, It is characterized in that The step of determining the data lock from the first lock record according to the task to be scheduled, the data to be called and the second lock record through the first adapter includes: Determine, through the first adapter, a scheduler corresponding to the task to be scheduled in the scheduling framework, and determine, from the second lock record, a first lock of the data to be called; The name of the second lock of the data to be called in the first lock record is determined through the first adapter according to the name of the scheduler and the name of the first lock; the second lock is the data lock of the data to be called.
4. The method according to claim 2, It is characterized in that Submitting the lock grabbing transaction to the target database includes: The lock grabbing transaction is created through the mutex lock acquisition interface; the lock grabbing transaction is used to modify the indication of the holding flag in the first lock record to: the data lock is in an occupied state; The lock grabbing transaction is submitted through the mutex acquisition interface.
5. The method according to any one of claims 1 to 4, It is characterized in that After obtaining the holding identifier of the data lock, the method further includes: If the holding identifier indicates that the data lock is in an occupied state, or the holding identifier indicates that the data lock is in an idle state and the lock grabbing transaction fails to be submitted, then determining the cumulative number of times the holding identifier has been acquired by the time the holding identifier is acquired this time; If the cumulative number of times the holding identifier has been obtained is less than a preset number, the step of obtaining the holding identifier of the data lock is performed according to a preset period.
6. The method according to claim 5, It is characterized in that Also includes: If the cumulative number of times the holding identifier has been obtained is greater than or equal to the preset number of times, determining whether the following situation exists in the cumulative number of times the holding identifier is obtained: the holding identifier obtained this time indicates that the data lock is in an idle state, and the lock grabbing transaction submitted to the target database fails after the holding identifier is obtained this time; If it exists, update the cumulative number to 0; If it does not exist, submit the unlock transaction; The unlocking transaction is used to release the lock of the data to be called by the data lock; The indication of the holding flag is modified to: the data lock is in an idle state.
7. The method according to any one of claims 1 to 4, It is characterized in that After submitting the lock grabbing transaction to the target database, the method further includes: If the lock-grabbing transaction fails to be submitted, the step of obtaining the holding identifier of the data lock is performed again after waiting for a first preset time period.
8. The method according to any one of claims 1 to 4, It is characterized in that After determining that the lock-grabbing transaction is successfully submitted, the method further includes: Record the application instance and scheduling thread that successfully occupies the data lock; If the duration of the occupation of the data lock by the application instance and the scheduling thread is greater than a second preset duration, the indication of the holding flag is modified to: the data lock is in an idle state.
9. The method according to any one of claims 1 to 4, It is characterized in that After scheduling the task to be scheduled according to the data to be called, the method further includes: If it is determined that the to-be-scheduled task has been scheduled, an unlock transaction is submitted; the unlock transaction is used to modify the indication of the holding flag to: the data lock is in an idle state.
10. A task scheduling device, It is characterized in that include: An acquisition module, used for acquiring a first adapter in a scheduling framework; The first adapter is adapted to a target database, and the target database corresponds to a first lock record in the scheduling framework; The first adapter is configured to replace the second adapter in the scheduling framework; The second adapter is adapted to a relational database, and the relational database corresponds to a second lock record in the scheduling framework; the first adapter and the second adapter are both used to seize a data lock of the data to be called in the target database; The data lock is used to lock the data to be called; The data to be called is the data required by the task to be scheduled; a determination module, configured to determine the data lock from the first lock record through the first adapter according to the task to be scheduled, the data to be called and the second lock record; Obtaining, through the first adapter, a holding identifier of the data lock; the holding identifier is used to indicate whether the data lock is in an idle state or an occupied state; if it is determined that the holding identifier indicates that the data lock is in an idle state, submitting a lock grabbing transaction to the target database; the lock grabbing transaction is used to grab the data lock; A scheduling module, configured to schedule the task to be scheduled according to the data to be called if it is determined that the lock-grabbing transaction is submitted successfully; The successful submission of the lock-grabbing transaction indicates that the holding flag indicates that the data lock is in an occupied state.
11. A computer device, It is characterized in that The method comprises a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 9 is executed.
12. A storage medium, It is characterized in that The method comprises a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 9 is executed.
Citation Information
Patent Citations
Methods and devices of avoiding concurrently accessing database by threads
CN107229632A
Container resource scheduling method and device and computer readable storage medium
CN109753356A
Task scheduling method and system and terminal equipment
CN110162388A
Distributed lock implementation method in cloud computing environment
CN110445864A
Data operation method and device, storage medium and processor
CN110704198A
