A method and device for dynamic expansion of high-performance distributed scheduler
By adopting the service registration center, task sharding and thread pool/buffer queue mode in the distributed scheduler, the problem of insufficient dynamic expansion performance in traditional technology is solved, and high-performance and efficient task scheduling is achieved.
Patent Information
- Application Number
- CN202411905829.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-04
AI Technical Summary
It is difficult for the existing technology to achieve high-performance dynamic expansion of distributed schedulers, especially in Internet finance services. Traditional horizontal horizontal expansion solutions have problems such as business single lock competition conflict and insufficient server resource utilization.
By registering the service node of the task scheduler to the service registration center, obtaining the IP list of the surviving service nodes, database task sharding is performed based on the service number, and task processing is performed using the execution thread pool and thread buffer queue mode.
It realizes high-performance dynamic expansion of task schedulers, improves the throughput performance of single-node task scheduling services, avoids performance problems caused by waiting for task locks, and supports rapid dynamic expansion or shrinkage.
Smart Images
Figure CN119759530A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of dynamic capacity expansion technology, and in particular to a method and device for dynamic capacity expansion of a high-performance distributed scheduler. Background Art
[0002] At present, due to the rapid development of Internet finance, more and more financial businesses have realized online services, such as online account opening and online business handling in the securities and financial industry. Once the bull market comes, brokerage economic businesses will attract customers through various channels to meet the daily opening of 100,000 to 500,000 accounts, etc., which puts forward the demand for high-performance dynamic expansion of financial business online systems. Traditional technical means generally adopt ordinary horizontal solutions, which generally have problems such as business single lock competition conflicts and insufficient server resource utilization, and cannot meet the requirements of high-performance dynamic expansion.
[0003] If you want to implement a high-performance distributed scheduler, you need to increase its concurrent processing capabilities as much as possible to increase the transaction throughput per unit time. Since concurrency will cause task lock contention conflicts, if you cannot reduce lock conflicts by processing tasks in large batches during the window time, the scheduler task thread will keep trying to acquire locks, causing a performance bottleneck. Summary of the invention
[0004] To this end, the present invention aims to solve the deficiencies in the prior art to at least a certain extent, thereby proposing a method and device for dynamic capacity expansion of a high-performance distributed scheduler.
[0005] In a first aspect, the present invention provides a method for dynamic capacity expansion of a high-performance distributed scheduler, the method comprising:
[0006] Register the service nodes of the multiple task schedulers to the service registration center respectively, and obtain the IP list of all surviving service nodes through the service registration center, wherein the IP list of the service nodes includes the IPs of the multiple service nodes;
[0007] Based on the plurality of task schedulers, each task scheduler obtains a sequence of the IP of its own service node in the IP list of the service node to obtain a corresponding service number;
[0008] Based on the plurality of task schedulers, a shard page query is performed on the list of pending tasks in the database according to the shard logic algorithm through the corresponding service numbers, and a plurality of corresponding pending tasks are obtained respectively;
[0009] A portion of the pending tasks corresponding to each task scheduler is placed into an execution thread pool for processing, and another portion of the pending tasks is placed into a thread buffer queue for queue execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue.
[0010] In a second aspect, the present invention provides a device for dynamic expansion of a high-performance distributed scheduler, the device comprising:
[0011] Registration module: used to register the service nodes of multiple task schedulers to the service registration center respectively, and obtain the IP list of all surviving service nodes through the service registration center, wherein the IP list of the service node includes the IPs of multiple service nodes;
[0012] A sorting module: used for respectively obtaining the sequence of the IP of the service node of the task scheduler in the IP list of the service node based on the plurality of task schedulers, so as to obtain the corresponding service number;
[0013] Sharding module: used to perform shard page query on the list of pending tasks in the database based on the corresponding service numbers of the multiple task schedulers according to the sharding logic algorithm, and obtain multiple corresponding pending tasks respectively;
[0014] Execution module: used to put a part of the pending tasks corresponding to each task scheduler into the execution thread pool for processing, and put another part of the pending tasks into the thread buffer queue for queue execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue.
[0015] In a third aspect, the present invention also provides a device for dynamic expansion of a high-performance distributed scheduler, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements each step of the method for dynamic expansion of a high-performance distributed scheduler as described in the first aspect.
[0016] In a fourth aspect, the present invention further provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the various steps in the method for dynamic expansion of a high-performance distributed scheduler as described in the first aspect are implemented.
[0017] The present invention provides a method and device for dynamic capacity expansion of a high-performance distributed scheduler, the method comprising: registering the service nodes of multiple task schedulers to a service registration center respectively, and obtaining an IP list of all surviving service nodes through the service registration center, wherein the IP list of the service nodes includes the IPs of multiple service nodes; obtaining the sequence of the IPs of the service nodes of the multiple task schedulers in the IP list of the service nodes respectively, so as to obtain corresponding service numbers; performing shard page query on a list of pending tasks in a database according to a shard logic algorithm based on the corresponding service numbers of the multiple task schedulers, and obtaining multiple corresponding pending tasks respectively; placing a part of the pending tasks corresponding to each task scheduler into an execution thread pool for processing, and placing another part of the pending tasks into a thread buffer queue for queued execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue. Through the method provided by the present invention, the service registration center is responsible for registering the services of each dynamically expanded task scheduler, and the service node execution of the task scheduler adopts the service execution thread pool + service execution thread buffer queue mode, which can greatly improve the throughput performance of the single-node task scheduling service execution, can give full play to its concurrent processing capability, increase the transaction throughput per unit time, and can also protect the system from overload, to prevent the task service execution buffer queue from being too long due to the slow execution performance of the service node of the task scheduler, resulting in its memory leakage problem; the task scheduler can be quickly and dynamically expanded or shrunk according to the business load pressure without downtime, and the list of pending tasks in the database is sliced, which can realize the horizontal expansion of the service node of the task scheduler without lock waiting, improve the efficiency of the horizontal expansion shard service execution, and fundamentally solve the performance problem caused by the database task lock competition waiting. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0019] Figure 1 A schematic diagram of a process of a method for dynamic capacity expansion of a high-performance distributed scheduler of the present invention;
[0020] Figure 2 A schematic diagram of a sub-process of a method for dynamic capacity expansion of a high-performance distributed scheduler of the present invention;
[0021] Figure 3A logic block diagram of acquiring a lock in the method for dynamic expansion of a high-performance distributed scheduler of the present invention;
[0022] Figure 4 It is another sub-flow diagram of the method for dynamic capacity expansion of a high-performance distributed scheduler of the present invention;
[0023] Figure 5 A block diagram of sharding processing of a method for dynamic capacity expansion of a high-performance distributed scheduler of the present invention;
[0024] Figure 6 It is another sub-process diagram of the method for dynamic capacity expansion of a high-performance distributed scheduler of the present invention;
[0025] Figure 7 It is another sub-process diagram of the method for dynamic capacity expansion of a high-performance distributed scheduler of the present invention;
[0026] Figure 8 The overall logic block diagram of the method for dynamic capacity expansion of a high-performance distributed scheduler of the present invention;
[0027] Fig. 9 A schematic diagram of a program module of a device for dynamically expanding the capacity of a high-performance distributed scheduler of the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, features and advantages of the invention more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0029] Please refer to Figure 1 , Figure 1 This is a flow chart of a method for dynamic expansion of a high-performance distributed scheduler in an embodiment of the present application. In this embodiment, the method for dynamic expansion of a high-performance distributed scheduler includes:
[0030] Step S10: register the service nodes of the multiple task schedulers to the service registration center respectively, and obtain the IP list of all surviving service nodes through the service registration center, wherein the IP list of the service nodes includes the IPs of the multiple service nodes.
[0031] In this embodiment, the pending business is processed separately by multiple task schedulers, and each task scheduler registers its own service node to the service registration center, so that the service registration center saves all currently surviving and available batch scheduling service nodes, that is, the service registration center has an IP list of all surviving service nodes, wherein the IP list of service nodes includes the IPs of multiple service nodes.
[0032] Specifically, each task scheduler registers the service through the service registration center, and the service registration center can use Nacos, Redis, or Zookeeper to register the task scheduler.
[0033] For further information, see Figure 7 , Figure 7 This is another sub-flow diagram of the method for dynamic expansion of a high-performance distributed scheduler in an embodiment of the present application, wherein the service nodes of multiple task schedulers are registered to a service registration center respectively, and an IP list of all surviving service nodes is obtained through the service registration center, including:
[0034] Step 101: registering a plurality of dynamically expanded service nodes of the task scheduler through the service registration center;
[0035] Step 102: Perform heartbeat detection on the service nodes of the multiple task schedulers registered by the service registration center to respectively verify the survival status of the service nodes of the multiple task schedulers;
[0036] Step 103: Cache the IP addresses of the service nodes of the task scheduler that are in a live state, and obtain an IP list of all the live service nodes.
[0037] In this embodiment, the service registration center is responsible for registering the service nodes of multiple task schedulers that are dynamically expanded. The service registration center performs heartbeat detection on the service nodes of all registered task schedulers, thereby detecting the survival status of the service nodes of the registered task schedulers, and then caches the IPs of the service nodes of all surviving task schedulers, thereby obtaining an IP list of all surviving service nodes. The IP of the service node refers to the IP address. The IP list of the service node can be IP0, IP1, IP2, ... IPN.
[0038] Step S20: Based on the plurality of task schedulers, each of the task schedulers obtains the sequence of the IP of its own service node in the IP list of the service node to obtain a corresponding service number.
[0039] In this embodiment, each task scheduler has its own service node, and the IPs of the service nodes of multiple task schedulers form an IP list of the service nodes. The task scheduler can know the service number of the corresponding task scheduler according to the order of the IP of its own service node in the IP list of the service node.
[0040] For further information, see Figure 6 , Figure 6 It is another sub-process diagram of the dynamic expansion of the high-performance distributed scheduler in the embodiment of the present application. In this embodiment, the sequence of the IP of the service node of the service node in the IP list of the service node is obtained by the task schedulers based on the plurality of task schedulers to obtain the corresponding service number, including:
[0041] Step 201: sort the IP list of the service nodes according to a sorting algorithm for the IPs of the multiple service nodes, and obtain, through each task scheduler, a sequence of the service node in the sorted IP list of the service node according to the IP of its own service node;
[0042] Step 202: The sequence of the service node in the sorted IP list of the service node is used as the service number of the corresponding task scheduler.
[0043] In this embodiment, the IPs of the service nodes of multiple task schedulers themselves form an IP list of the service nodes, and the IP list of the service nodes is sorted by a sorting algorithm of the IPs of multiple service nodes in the IP list of the service nodes, wherein the sorting algorithm may adopt a dictionary order or a numerical order, the dictionary order is: the IP address is regarded as a string and sorted in dictionary order, this method is simple and direct, because the string comparison is performed character by character; the numerical order is: each IP address is split into four octets, i.e., four integers, and then sorted according to the numerical size, for example, 192.168.0.2 and 192.168.0.10 will be converted to (192, 168, 0, 2) and (192, 168, 0, 10), and then compared byte by byte and sorted from left to right.
[0044] When the IP list of the service nodes is sorted, each task scheduler can obtain the sequence of its own service node in the sorted IP list of the service nodes according to the IP of its own service node. The sequence of the IP of each task scheduler's own service node in the sorted IP list of the service nodes is used as the service number of the task scheduler. For example, the service number of the task scheduler's service node IP0 is 0, the service number of IP1 is 1, the service number of IP2 is 2, ..., and the service number of IPN is N.
[0045] Step S30: Based on the plurality of task schedulers, a shard page query is performed on the list of pending tasks in the database through the corresponding service numbers according to a shard logic algorithm to obtain a plurality of corresponding pending tasks.
[0046] In this embodiment, multiple task schedulers can perform shard page query on the list of pending tasks in the database through the corresponding service numbers and according to the shard logic algorithm, so that each task scheduler can obtain multiple corresponding pending tasks. Among them, the database is responsible for storing pending tasks for the task scheduler to perform task scheduling and service execution.
[0047] For further information, see Figure 4 , Figure 4 This is another sub-flow diagram of the method for dynamic expansion of a high-performance distributed scheduler in an embodiment of the present application. In this embodiment, the service number of each task scheduler is a task ID, and the task ID of each task scheduler is a self-incrementing sequence. The task schedulers perform a shard page query on the list of pending tasks in the database according to the shard logic algorithm through the corresponding service numbers, and obtain multiple corresponding pending tasks, including:
[0048] Step 301: Slice the list of pending tasks in the database according to the task ID modulo algorithm to distinguish between a plurality of different service nodes of the task scheduler itself;
[0049] Step 302: different service nodes are dynamically scalable to obtain different tasks to be processed in the list of tasks to be processed according to the corresponding task IDs.
[0050] The traditional mode is to lock tasks through the database row lock mechanism to ensure data consistency and avoid multiple batch programs processing the same task. However, each service node has the same logic and the same processing strategy, which will lead to performance bottlenecks caused by lock competition. Figure 5 , Figure 5 This is a block diagram of the sharding processing of the method for dynamic expansion of a high-performance distributed scheduler in an embodiment of the present application. Unlike the traditional model, the embodiment of the present application uses an efficient algorithm to logically distinguish the service nodes of multiple different task schedulers that are expanded horizontally, so that the service nodes of different task schedulers can dynamically scale to obtain data slices for different tasks to be processed. That is, the embodiment of the present application includes multiple different task schedulers, each of which includes a service node. Different tasks to be processed in the database are assigned to service nodes in different task schedulers for execution and processing through an efficient algorithm, thereby realizing lock-free waiting for the scheduling and execution of tasks to be processed.
[0051] Among them, the efficient algorithm specifically adopted in the embodiment of the present application is the task ID modulo algorithm, and the service number of each task scheduler is the task ID, that is, the number of task schedulers, and the task ID of each task scheduler is a self-increasing sequence. According to the task ID modulo algorithm, the service nodes of multiple task schedulers extract and process the pending tasks in the list of pending tasks in the database in their respective slices, and directly dynamically bind the allocation relationship between different pending tasks and different service nodes registered for dynamic expansion. The algorithm is simpler and more efficient, and can realize the horizontal expansion of the service nodes of the task scheduler without lock waiting, thereby improving the efficiency of the execution of the horizontal expansion shard service, and fundamentally solving the performance problems caused by the database task lock competition waiting.
[0052] Step S40, putting a part of the pending tasks corresponding to each task scheduler into the execution thread pool for processing, and putting another part of the pending tasks into the thread buffer queue for queue execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue.
[0053] In this embodiment, after the pending tasks in the database are divided into pieces and assigned to the corresponding task schedulers according to the task ID modulo algorithm, each task scheduler includes an execution thread pool and a thread buffer queue, and each task scheduler is assigned multiple corresponding pending tasks. The multiple corresponding pending tasks of each task scheduler are divided into two parts, wherein a part of the pending tasks are placed in the execution thread pool for processing, and the other part of the pending tasks are cached in the task service execution buffer queue for queued execution, which can ensure that the task scheduler can make full use of its multi-threaded CPU scheduling resources when executing pending tasks.
[0054] Furthermore, the step of placing a portion of the tasks to be processed corresponding to each task scheduler into an execution thread pool for processing, and placing another portion of the tasks to be processed into a thread buffer queue for queue execution, comprises:
[0055] The number of the tasks to be processed put into the thread buffer queue is configured according to the number of the tasks to be processed in the execution thread pool.
[0056] In this embodiment, the number of pending tasks in the thread buffer queue needs to limit its length. For example, the number of pending tasks in the thread buffer queue can be configured to be 1 times the number of pending tasks processed in the execution thread pool. This can avoid the problem of memory leakage caused by the thread buffer queue being too long due to the slow performance of the task scheduler's service node when executing pending tasks.
[0057] For further information, see Figure 2 , Figure 2 This is a sub-flow diagram of a method for dynamic expansion of a high-performance distributed scheduler in an embodiment of the present application. In this embodiment, the process of placing a portion of the tasks to be processed corresponding to each task scheduler into an execution thread pool for processing includes:
[0058] Step 401: Obtain a data lock from the database.
[0059] Step 402: If the data lock is acquired successfully, a portion of the tasks to be processed is placed in the execution thread pool and executed.
[0060] In this example, see Figure 3 , according to the task ID modulo algorithm, different multiple pending tasks are assigned to different task schedulers from the pending task list of the database, so that the service nodes in different task schedulers execute the assigned pending tasks. The task scheduler first puts the pending tasks into the thread buffer queue, and at the same time obtains the data lock in the pending task list in the database. The data lock is implemented based on the database row lock mechanism. The row lock (Row Lock) in the database is a mechanism for ensuring data security and consistency. Especially in a concurrent environment, the row lock allows multiple transactions to access the database at the same time, but ensures that each transaction has exclusive access to the data it is processing, thereby preventing data conflicts and inconsistencies, and realizing that when a transaction updates data, other transactions cannot modify the same data at the same time, thereby avoiding data loss. For example: in a transaction, execute UPDATE task table SET status = 'executing' WHERE task ID = xxx AND status = 'to be executed', so that the transaction that obtains the lock returns 1, and the transaction that does not obtain the lock returns 0.
[0061] If the data lock is acquired successfully, a portion of the pending tasks will be put into the execution thread pool for execution, so that when the pending tasks are finally executed, the database row lock can guarantee the security of their execution.
[0062] An embodiment of the present application provides a method for dynamic capacity expansion of a high-performance distributed scheduler, the method comprising: registering the service nodes of multiple task schedulers to a service registration center respectively, and obtaining an IP list of all surviving service nodes through the service registration center, wherein the IP list of the service nodes includes the IPs of multiple service nodes; obtaining the sequence of the IPs of the service nodes of the multiple task schedulers in the IP list of the service nodes respectively to obtain corresponding service numbers; performing shard page query on a list of pending tasks in a database according to a shard logic algorithm based on the corresponding service numbers of the multiple task schedulers, and obtaining multiple corresponding pending tasks respectively; placing a portion of the pending tasks corresponding to each task scheduler into an execution thread pool for processing, and placing another portion of the pending tasks into a thread buffer queue for queued execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue. Through the method provided by the present invention, the service registration center is responsible for registering the services of each dynamically expanded task scheduler, and the service node execution of the task scheduler adopts the service execution thread pool + service execution thread buffer queue mode, which can greatly improve the throughput performance of the single-node task scheduling service execution, can give full play to its concurrent processing capability, increase the transaction throughput per unit time, and can also protect the system from overload, to prevent the task service execution buffer queue from being too long due to the slow execution performance of the service node of the task scheduler, resulting in its memory leakage problem; the task scheduler can be quickly and dynamically expanded or shrunk according to the business load pressure without downtime, and the list of pending tasks in the database is sliced, which can realize the horizontal expansion of the service node of the task scheduler without lock waiting, improve the efficiency of the horizontal expansion shard service execution, and fundamentally solve the performance problem caused by the database task lock competition waiting.
[0063] For further information, see Figure 8 , Figure 8 This is an overall logic block diagram of a method for dynamic expansion of a high-performance distributed scheduler in an embodiment of the present application. In this embodiment, the specific steps of the method are as follows:
[0064] 1. Each task scheduler registers its service through the service registration center and registers its own service node to the service registration center. The service registration center also performs heartbeat detection on the service nodes of each registered task scheduler to verify its survival status. The service registration center can be implemented in various ways such as Nacos, Redis, and Zookeeper. The service registration center saves all currently registered surviving and available batch scheduling service nodes (for example: IP0, IP1, ... IPN);
[0065] 2. Each task scheduler obtains the IP list of all currently registered surviving and available service nodes through the service registration center, and sorts the IP list of the service nodes according to the sorting algorithm of the service node IP. Then, according to the IP of the service node of each task scheduler, the sequence of the IP list of the service node is obtained as the service number of the task scheduler. For example, the service number of IP0 is 0, the service number of IP1 is 1..., and the service number of IPN is N;
[0066] 3. The task ID of each task scheduler is a self-incrementing sequence, where the task ID is the number of schedulers, that is, the service number. According to the task ID modulus algorithm, the list of pending tasks in the database is queried in slices and pages, so that different task schedulers obtain different multiple pending tasks, and the service nodes in the task scheduler process the obtained pending tasks. At the same time, it is necessary to obtain the data lock in the list of pending tasks in the database. After the data lock is successfully obtained, the pending tasks are executed;
[0067] 4. Each task scheduler includes an execution thread pool and a thread buffer queue. It puts a part of the pending tasks into the execution thread pool for processing through its own service node, and caches the remaining pending tasks into the thread buffer queue for queued execution. The task service execution buffer queue limits its length (for example, it is configured to be 1 times the number of threads in the execution thread pool). This can ensure that the task scheduler can fully utilize its multi-threaded CPU scheduling resources during service execution, and avoid memory leakage caused by the task service execution buffer queue being too long due to the slow execution performance of the task scheduler service.
[0068] In summary, the embodiment of the present application combines the pending tasks and the execution service into one. When scheduling the pending tasks, the service node of each dynamically expanded task scheduler extracts the task fragments from the list of pending tasks in the database through a high-performance extraction algorithm, directly and dynamically binds the allocation relationship between the pending tasks and the dynamically expanded service nodes, and when the pending tasks are finally executed, the database task lock guarantees their execution security. In addition, the service node introduces the execution thread pool and thread buffer queue mechanism during execution, and the thread buffer queue limits its length. On the one hand, it can ensure that the task scheduler fully utilizes its multi-threaded CPU scheduling resources, and on the other hand, it can avoid the problem of memory leakage caused by the thread buffer queue being too long due to the slow execution of the task scheduler; it solves the data lock competition pressure brought about by horizontal expansion, thereby solving the performance problem between a single node obtaining a data lock and scheduling tasks, and logically separating the lock data and the scheduling tasks to form a buffer mechanism, thereby realizing that the execution thread pool of the task scheduler is fully loaded.
[0069] Furthermore, the embodiment of the present application also provides a device 600 for dynamically expanding the capacity of a high-performance distributed scheduler. Fig. 9 Schematic diagram of a program module of a device for dynamic expansion of a high-performance distributed scheduler in an embodiment of the present application. In this embodiment, the device 600 for dynamic expansion of a high-performance distributed scheduler includes:
[0070] Registration module 601: used to register the service nodes of multiple task schedulers to the service registration center respectively, and obtain the IP list of all surviving service nodes through the service registration center, wherein the IP list of the service nodes includes the IPs of multiple service nodes;
[0071] Sorting module 602: used for respectively obtaining the sequence of the IP of the service node of the task scheduler in the IP list of the service node based on the plurality of task schedulers, so as to obtain the corresponding service number;
[0072] Slicing module 603: used to perform slicing page query on the list of pending tasks in the database according to the slicing logic algorithm based on the corresponding service numbers of the plurality of task schedulers, and obtain a plurality of corresponding pending tasks respectively;
[0073] Execution module 604: used to put a part of the pending tasks corresponding to each task scheduler into the execution thread pool for processing, and put another part of the pending tasks into the thread buffer queue for queue execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue.
[0074] In an embodiment of the present application, a device 600 for dynamically expanding the capacity of a high-performance distributed scheduler is provided, which can achieve: registering the service nodes of multiple task schedulers to a service registration center respectively, and obtaining the IP list of all surviving service nodes through the service registration center, wherein the IP list of the service node includes the IPs of multiple service nodes; obtaining the sequence of the IPs of the service nodes of the multiple task schedulers in the IP list of the service nodes respectively to obtain the corresponding service numbers; performing shard page query on the list of pending tasks in the database according to the sharding logic algorithm based on the corresponding service numbers of the multiple task schedulers, and obtaining multiple corresponding pending tasks respectively; putting a part of the pending tasks corresponding to each task scheduler into an execution thread pool for processing, and putting another part of the pending tasks into a thread buffer queue for queued execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue. Through the method provided by the present invention, the service registration center is responsible for registering the services of each dynamically expanded task scheduler, and the service node execution of the task scheduler adopts the service execution thread pool + service execution thread buffer queue mode, which can greatly improve the throughput performance of the single-node task scheduling service execution, can give full play to its concurrent processing capability, increase the transaction throughput per unit time, and can also protect the system from overload, to prevent the task service execution buffer queue from being too long due to the slow execution performance of the service node of the task scheduler, resulting in its memory leakage problem; the task scheduler can be quickly and dynamically expanded or shrunk according to the business load pressure without downtime, and the list of pending tasks in the database is sliced, which can realize the horizontal expansion of the service node of the task scheduler without lock waiting, improve the efficiency of the horizontal expansion shard service execution, and fundamentally solve the performance problem caused by the database task lock competition waiting.
[0075] Furthermore, the present application also provides a device for dynamic expansion of a high-performance distributed scheduler, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps in the method for dynamic expansion of a high-performance distributed scheduler as described above.
[0076] Furthermore, the present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the above-mentioned method for dynamic expansion of a high-performance distributed scheduler is implemented.
[0077] Each functional module in each embodiment of the present invention may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium.
[0078] Based on this understanding, the technical solution of the specification of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.
[0079] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0080] For those skilled in the art, according to the concept of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A method for dynamic capacity expansion of a high-performance distributed scheduler, characterized in that: The method comprises: Register the service nodes of the multiple task schedulers to the service registration center respectively, and obtain the IP list of all surviving service nodes through the service registration center, wherein the IP list of the service nodes includes the IPs of the multiple service nodes; Based on the plurality of task schedulers, each task scheduler obtains a sequence of the IP of its own service node in the IP list of the service node to obtain a corresponding service number; Based on the plurality of task schedulers, a shard page query is performed on the list of pending tasks in the database according to the shard logic algorithm through the corresponding service numbers, and a plurality of corresponding pending tasks are obtained respectively; A portion of the pending tasks corresponding to each task scheduler is placed into an execution thread pool for processing, and another portion of the pending tasks is placed into a thread buffer queue for queue execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue.
2. The method according to claim 1, characterized in that The step of placing a portion of the tasks to be processed corresponding to each task scheduler into an execution thread pool for processing, and placing another portion of the tasks to be processed into a thread buffer queue for queue execution, comprises: The number of the tasks to be processed put into the thread buffer queue is configured according to the number of the tasks to be processed in the execution thread pool.
3. The method according to claim 1, characterized in that The step of placing a portion of the tasks to be processed corresponding to each of the task schedulers into an execution thread pool for processing includes: Acquire a data lock from the database; If the data lock is acquired successfully, a portion of the pending tasks are placed in the execution thread pool and executed.
4. The method according to claim 1, characterized in that: The service number of each task scheduler is a task ID, and the task ID of each task scheduler is a self-incrementing sequence. The plurality of task schedulers are based on the corresponding service numbers and perform a shard page query on the list of pending tasks in the database according to the shard logic algorithm to obtain a plurality of corresponding pending tasks, including: Slice the list of pending tasks in the database according to a task ID modulo algorithm to distinguish between a plurality of different service nodes of the task scheduler; So that different service nodes can dynamically scale to obtain different tasks to be processed in the list of tasks to be processed according to the corresponding task IDs.
5. The method according to claim 4, characterized in that The acquiring, based on the plurality of task schedulers, the sequence of the IP of the service node in the IP list of the service node to obtain the corresponding service number includes: Sorting the IP list of the service nodes according to a sorting algorithm for the IPs of the multiple service nodes, and obtaining, by each task scheduler according to the IP of its own service node, a sequence of the sorted IP list of the service nodes in which its own service node is located; And the sequence of the service nodes in the sorted IP list of the service nodes is used as the service number of the corresponding task scheduler.
6. The method according to claim 5, characterized in that The process of registering the service nodes of the multiple task schedulers to the service registration center respectively and obtaining the IP list of all the surviving service nodes through the service registration center includes: Registering a plurality of dynamically expanded service nodes of the task scheduler through the service registration center; Performing heartbeat detection on the service nodes of the multiple task schedulers registered by the service registration center to respectively verify the survival status of the service nodes of the multiple task schedulers; The IP addresses of the service nodes of the task scheduler that are alive are cached, and a list of the IP addresses of all the alive service nodes is obtained.
7. The method according to claim 6, characterized in that The service registration center registers multiple service nodes of the task scheduler in a manner including Nacos, Redis, and Zookeeper.
8. A device for dynamic expansion of a high-performance distributed scheduler, characterized in that: The device comprises: Registration module: used to register the service nodes of multiple task schedulers to the service registration center respectively, and obtain the IP list of all surviving service nodes through the service registration center, wherein the IP list of the service nodes includes the IPs of multiple service nodes; A sorting module: used for respectively obtaining the sequence of the IP of the service node of the task scheduler in the IP list of the service node based on the plurality of task schedulers, so as to obtain the corresponding service number; Sharding module: used to perform shard page query on the list of pending tasks in the database based on the corresponding service numbers of the multiple task schedulers according to the sharding logic algorithm, and obtain multiple corresponding pending tasks respectively; Execution module: used to put a part of the pending tasks corresponding to each task scheduler into the execution thread pool for processing, and put another part of the pending tasks into the thread buffer queue for queue execution, wherein any task scheduler includes the execution thread pool and the thread buffer queue.
9. A device for dynamic expansion of a high-performance distributed scheduler, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step of the method for dynamic capacity expansion of a high-performance distributed scheduler as described in any one of claims 1 to 7 is implemented.
10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method for dynamic expansion of a high-performance distributed scheduler as described in any one of claims 1 to 7 is implemented.