A resource adaptive based task scheduling method and system

Abstract

The present application relates to the field of information technology, and provide a kind of based on resource self-adapting task scheduling method and system, the method of the present application includes: task data, executor data, task instance data are distributed cache;Task is evenly distributed to each executor by hash calculation to scheduling executor and task instance;By configuring adaptive, adjustable resource pool, automatically allocate resource pool and provide configurable resource;According to the data source information configured, automatically adapt multiple data sources.The system of the present application is composed of scheduling center and scheduling executor, wherein, scheduling center includes: distributed cache center, routing management module, resource center and data source center.According to the task scheduling method and system based on resource self-adapting of exemplary embodiment of the present application, the efficiency of task scheduling can be improved, and the resource occupation and error rate of task scheduling are reduced.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a resource-adaptive task scheduling method and system. Background Technology

[0002] Scheduling tools play an indispensable role in big data platform operation, software development and other environments. Currently, the mainstream scheduling tools on the market include DolphinScheduler, Azkaban, Airflow, Oozie and Xxl-job.

[0003] In practical applications, existing scheduling tools cannot simultaneously solve the following problems: 1. The problem of adaptive or configurable resource pool settings: They cannot adaptively configure the scheduling resource pool according to the on-site resources or provide configurable parameters for manual adjustment, resulting in a mismatch between the resource pool and the available server resources, leading to resource waste or shortage; 2. The problem of multi-data source adaptation: The scheduling business library cannot quickly adapt to different data sources; 3. Reliable sharding routing strategies: Most open-source products only provide routing strategies such as sequential, round-robin, and random. When network jitter or other factors cause node unavailability, node downtime may occur due to the full redistribution of tasks and the skewed distribution of tasks received by the node; 4. Data loss caused by single-node memory caching and consistency issues in distributed deployment: Since the cache is stored in the node's memory, when the service crashes or the server restarts, the data in the memory cache will be lost, leading to task interruption. In addition, in distributed deployment, inconsistencies in cache information between nodes can cause system errors.

[0004] Therefore, how to provide an efficient, resource-saving, and accurate task scheduling method has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, in order to overcome the shortcomings of the existing technology, improve the efficiency of task scheduling, and reduce the probability of task scheduling failure, this invention provides a resource-adaptive task scheduling method.

[0006] On one hand, the present invention provides a resource-adaptive task scheduling method, comprising:

[0007] Distributed caching is implemented for task data, executor data, and task instance data;

[0008] Perform hash calculations on the scheduler and task instances to evenly distribute tasks to each scheduler;

[0009] By configuring adaptive and adjustable resource pools, resource pools are automatically allocated and configurable resources are provided.

[0010] It automatically adapts to multiple data sources based on the configured data source information.

[0011] Furthermore, in the resource-adaptive task scheduling method of the present invention, task data, executor data, and task instance data are distributedly cached, including:

[0012] When writing data to the distributed cache database, the cache is refreshed first, and then the database write operation is performed. If a write exception occurs, the cache is rolled back.

[0013] When performing a data query, the distributed cache database is queried first. If the data exists, the queried data is returned; if the data does not exist, the data center is queried, the data is cached in the distributed cache system, and then the queried data is returned.

[0014] Furthermore, in the resource-adaptive task scheduling method of the present invention, a hash calculation is performed on the scheduling executor and the task instance to evenly distribute the task to each executor, including:

[0015] Based on the consistent hash algorithm, the scheduler is abstracted as a ring, and the numbers from 0 to 2^32 are evenly distributed on the ring. The server IP where the executor is located is hashed, and the hash value is distributed within the range formed by the numbers on the ring.

[0016] For a scheduling instance, a hash is calculated based on the instance ID and the resulting hash value is projected onto the aforementioned ring. If no node is found at the projection point, the first node in the clockwise direction is the executor corresponding to that instance.

[0017] For a newly added node, tasks from that node counterclockwise to the first nearest node will be redistributed to the new node.

[0018] For nodes that need to be deleted or faulty nodes, the tasks from that node in a counter-clockwise direction to the nearest node will be reassigned to the first node in a clockwise direction starting from that node.

[0019] Furthermore, in the resource-adaptive task scheduling method of the present invention, by configuring an adaptive and adjustable resource pool, the resource pool is automatically allocated and configurable resources are provided, including:

[0020] Initialize the parameters of the scheduling resource pool based on the available memory capacity or CPU parameter information of the system.

[0021] The configuration information is exposed in the local configuration file, configuration center, or startup script parameters. Users can choose to modify the resource pool parameters in the local configuration file, configuration center, or startup script parameters as needed. The system will obtain the highest priority configuration according to the configuration priority and set the resource pool parameters accordingly.

[0022] Furthermore, in the resource-adaptive task scheduling method of the present invention, the parameters of the scheduling resource pool are initialized according to the available memory capacity or CPU parameter information of the system, including calculating the core thread number and maximum thread number of the scheduling resource pool using the following formula:

[0023] N Core_threads =N cpu *(1+M ava / M total )

[0024] N Max_threads =N cpu *(1+2*M ava / M total )

[0025] Where, N Core_threads N represents the core thread count of the resource pool. Max_threads N represents the maximum number of threads in the resource pool. cpu M represents the number of server CPUs. ava M represents the amount of available memory on the server. total This represents the total memory usage of the server.

[0026] Furthermore, in the resource-adaptive task scheduling method of the present invention, multiple data sources are automatically adapted according to the configured data source information, including:

[0027] The persistence layer uses the JPA framework based on the ORM specification to shield database dialect differences through object-relational mapping;

[0028] The database Dialect is built into the database to implement specific SQL statement conversion;

[0029] The data source configuration is exposed in the system configuration file. Users can configure and schedule database connection information according to the actual situation. The system automatically switches to the corresponding database Dialect class based on the database connection information, database type and corresponding database version information in the configuration file, and converts the query into the corresponding database SQL statement to achieve adaptation to different database types.

[0030] On the other hand, the present invention provides a resource-adaptive task scheduling system, comprising a scheduling center and a scheduling executor, wherein the scheduling center includes:

[0031] A distributed cache center is used to perform distributed caching of task data, executor data, and task instance data.

[0032] The routing management module is used to perform hash calculations on the scheduler and task instances to distribute tasks evenly to each executor.

[0033] The resource center is used to automatically allocate resource pools and provide configurable resources by configuring adaptive and adjustable resource pools;

[0034] The data source center is used to automatically adapt to multiple data sources based on the configured data source information.

[0035] Furthermore, in the resource-adaptive task scheduling system of this invention, the distributed cache center is specifically used for:

[0036] When writing data to the distributed cache database, the cache is refreshed first, and then the database write operation is performed. If a write exception occurs, the cache is rolled back.

[0037] When performing a data query, the distributed cache database is queried first. If the data exists, the queried data is returned; if the data does not exist, the data center is queried, the data is cached in the distributed cache system, and then the queried data is returned.

[0038] Furthermore, in the resource-adaptive task scheduling system of the present invention, the routing management module is specifically used for:

[0039] Based on the consistent hash algorithm, the scheduler is abstracted as a ring, and the numbers from 0 to 2^32 are evenly distributed on the ring. The server IP where the executor is located is hashed, and the hash value is distributed within the range formed by the numbers on the ring.

[0040] For a scheduling instance, a hash is calculated based on the instance ID and the resulting hash value is projected onto the aforementioned ring. If no node is found at the projection point, the first node in the clockwise direction is the executor corresponding to that instance.

[0041] For a newly added node, tasks from that node counterclockwise to the first nearest node will be redistributed to the new node.

[0042] For nodes that need to be deleted or faulty nodes, the tasks from that node in a counter-clockwise direction to the nearest node will be reassigned to the first node in a clockwise direction starting from that node.

[0043] Furthermore, in the resource-adaptive task scheduling system of the present invention, the resource center is specifically used for:

[0044] Initialize the parameters of the scheduling resource pool based on the available memory capacity or CPU parameter information of the system.

[0045] The configuration information is exposed in the local configuration file, configuration center, or startup script parameters. Users can choose to modify the resource pool parameters in the local configuration file, configuration center, or startup script parameters as needed. The system will obtain the highest priority configuration according to the configuration priority and set the resource pool parameters accordingly.

[0046] Furthermore, in the resource-adaptive task scheduling system of this invention, the resource center calculates the core thread count and maximum thread count of the scheduling resource pool using the following formula:

[0047] N Core_threads =N cpu *(1+M ava / M total )

[0048] N Max_threads =N cpu *(1+2*M ava / M total )

[0049] Where, N Core_threads N represents the core thread count of the resource pool. Max_threads N represents the maximum number of threads in the resource pool. cpu M represents the number of server CPUs. ava M represents the amount of available memory on the server. total This represents the total memory usage of the server.

[0050] Furthermore, in the resource-adaptive task scheduling system of this invention, the data source center is specifically used for:

[0051] The persistence layer uses the JPA framework based on the ORM specification to shield database dialect differences through object-relational mapping;

[0052] The database Dialect is built into the database to implement specific SQL statement conversion;

[0053] The data source configuration is exposed in the system configuration file. Users can configure and schedule database connection information according to the actual situation. The system automatically switches to the corresponding database Dialect class based on the database connection information, database type and corresponding database version information in the configuration file, and converts the query into the corresponding database SQL statement to achieve adaptation to different database types.

[0054] The present invention provides a resource-adaptive task scheduling method and system, which has the following beneficial effects:

[0055] 1) This invention is based on a distributed cache database, which caches task data, executor data and task instance data that require frequent database interactions in the distributed cache database. This solves the efficiency problem of frequent database interactions such as task verification. While improving the concurrency and query efficiency of the scheduling center, it also solves the problems of data loss caused by the restart of the scheduling center and inconsistent cache data in distributed deployment.

[0056] 2) This invention employs a routing sharding strategy based on a consistent hash algorithm. It performs hash calculations on both the scheduler executor and task instances, mapping tasks clockwise to the nearest node. If a node is added or deleted, only the task allocation on adjacent nodes is affected, avoiding the recalculation and reassignment of all scheduled tasks. This solves the problems of task allocation skew caused by sequential or random sharding in existing scheduling tools, as well as sharding fluctuations due to registration machine fluctuations. While ensuring even task allocation, the idle resources of the executor nodes are considered. This guarantees a balanced task load across executor nodes, improves resource utilization, and reduces the task execution failure rate.

[0057] 3) This invention, through adaptive and adjustable resource pool configuration, can automatically adapt to available server resources while providing a more refined resource configuration scheme. Within the limits of service resource capacity, it improves the task response efficiency and service throughput of the scheduling center by increasing the capacity and processing efficiency of the resource pool.

[0058] 4) This invention supports multi-data source adaptation based on the ORM specification. It automatically adapts to defined data sources according to the configured data source information, enabling rapid switching of scheduling business databases and reducing the adaptation cost of the scheduling center. It can adapt to common databases such as MySQL, Oracle, PostgreSQL, SQL Server, and DM, solving the problem of database incompatibility caused by different database SQL dialects in existing scheduling tools. Furthermore, to achieve configurable data source parameter adjustment and expose new data source access interfaces, only database-related information needs to be modified in the configuration file during on-site deployment, allowing adaptation to the specified database with minimal development effort. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1This is a schematic diagram of a consistent hash algorithm in a resource-adaptive task scheduling method according to an exemplary first embodiment of the present invention.

[0061] Figure 2 This is an architecture diagram of a resource-adaptive task scheduling system according to an exemplary second embodiment of the present invention;

[0062] Figure 3 This is an architecture diagram of a resource-adaptive task scheduling system according to an exemplary third embodiment of the present invention;

[0063] Figure 4 This is a flowchart illustrating the operation of a resource-adaptive task scheduling system according to an exemplary third embodiment of the present invention. Detailed Implementation

[0064] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0065] It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0066] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0067] The first exemplary embodiment of the present invention provides a resource-adaptive task scheduling method. This task scheduling method includes:

[0068] Distributed caching is implemented for task data, executor data, and task instance data;

[0069] Perform hash calculations on the scheduler and task instances to evenly distribute tasks to each scheduler;

[0070] By configuring adaptive and adjustable resource pools, resource pools are automatically allocated and configurable resources are provided.

[0071] It automatically adapts to multiple data sources based on the configured data source information.

[0072] In this embodiment of the resource-adaptive task scheduling method, distributed caching of task data, executor data, and task instance data is implemented, including:

[0073] When writing data to the distributed cache database, the cache is refreshed first, and then the database write operation is performed. If a write exception occurs, the cache is rolled back.

[0074] When performing a data query, the distributed cache database is queried first. If the data exists, the queried data is returned; if the data does not exist, the data center is queried, the data is cached in the distributed cache system, and then the queried data is returned.

[0075] In this embodiment of the resource-adaptive task scheduling method, a hash calculation is performed on the scheduling executor and task instances to evenly distribute tasks to each executor, including:

[0076] Based on the consistent hash algorithm, the scheduler is abstracted as a ring, and the numbers from 0 to 2^32 are evenly distributed on the ring. The server IP where the executor is located is hashed, and the hash value is distributed within the range formed by the numbers on the ring.

[0077] For a scheduling instance, a hash is calculated based on the instance ID and the resulting hash value is projected onto the aforementioned ring. If no node is found at the projection point, the first node in the clockwise direction is the executor corresponding to that instance.

[0078] For a newly added node, tasks from that node counterclockwise to the first nearest node will be redistributed to the new node.

[0079] For nodes that need to be deleted or faulty nodes, the tasks from that node in a counter-clockwise direction to the nearest node will be reassigned to the first node in a clockwise direction starting from that node.

[0080] In this embodiment, the resource-adaptive task scheduling method automatically allocates resources and provides configurable resources by configuring an adaptive and adjustable resource pool, including:

[0081] Based on the available memory capacity or CPU parameter information of the system, initialize the parameters of the scheduling resource pool to achieve adaptive resource pool configuration. Without any configuration, the parameters of the resource pool can be configured reasonably to avoid resource shortage or resource contention.

[0082] Based on the adaptive resource configuration strategy, adjustable resource pool configuration is performed, and reasonable resource allocation is carried out according to the actual situation on site. By exposing configuration information in local configuration files, configuration centers, or startup script parameters, users can choose to modify resource pool parameters in local configuration files, configuration centers, or startup script parameters as needed. The system obtains the highest priority configuration according to the configuration priority and sets the resource pool parameters, thereby ensuring the stable and efficient operation of the server system.

[0083] Specifically, based on the system's available memory capacity or CPU parameter information, initialize the parameters of the scheduling resource pool, including calculating the core thread count and maximum thread count of the scheduling resource pool using the following formula:

[0084] N Core_threads =N cpu *(1+M ava / M total )

[0085] N Max_threads =N cpu *(1+2*M ava / M total )

[0086] Where, N Core_threads N represents the core thread count of the resource pool. Max_threads N represents the maximum number of threads in the resource pool. cpu M represents the number of server CPUs. ava M represents the amount of available memory on the server. total This represents the total memory usage of the server.

[0087] In this embodiment, without the need for manual configuration, the resource utilization rate of the scheduling center can be improved while ensuring service stability by setting reasonable resource pool parameters.

[0088] In this embodiment of the resource-adaptive task scheduling method, multiple data sources are automatically adapted based on the configured data source information, including:

[0089] The persistence layer uses the JPA framework based on the ORM specification to shield database dialect differences through object-relational mapping;

[0090] The database Dialect is built into the database to implement specific SQL statement conversion;

[0091] The data source configuration is exposed in the system configuration file. Users can configure and schedule database connection information according to the actual situation. The system automatically switches to the corresponding database Dialect class based on the database connection information, database type and corresponding database version information in the configuration file, and converts the query into the corresponding database SQL statement to achieve adaptation to different database types.

[0092] An exemplary second embodiment of the present invention provides a resource-adaptive task scheduling system. Figure 2 This is an architectural diagram of a resource-adaptive task scheduling system according to an exemplary second embodiment of the present invention, as shown below. Figure 2 As shown, the system in this embodiment includes a scheduling center and a scheduling executor, wherein the scheduling center includes:

[0093] A distributed cache center is used to perform distributed caching of task data, executor data, and task instance data.

[0094] The routing management module is used to perform hash calculations on the scheduler and task instances to distribute tasks evenly to each executor.

[0095] The resource center is used to automatically allocate resource pools and provide configurable resources by configuring adaptive and adjustable resource pools;

[0096] The data source center is used to automatically adapt to multiple data sources based on the configured data source information.

[0097] In this embodiment of the task scheduling system, the distributed cache center is used to perform distributed caching of task data, executor data, and task instance data, including:

[0098] When writing data to the distributed cache database, the cache is refreshed first, and then the database write operation is performed. If a write exception occurs, the cache is rolled back.

[0099] When performing a data query, the distributed cache database is queried first. If the data exists, the queried data is returned; if the data does not exist, the data center is queried, the data is cached in the distributed cache system, and then the queried data is returned.

[0100] In practical applications, after a task is assigned to an executable thread, it enters the task verification phase. Before a task is triggered, it needs to verify whether the current task's execution conditions, such as the success of upstream tasks, are met. Currently, most of this verification information is stored in a database or local cache. This inevitably leads to frequent database interactions, which limits the verification efficiency of the task. Furthermore, information stored in the local cache may be lost or inconsistent due to service downtime or distributed deployment.

[0101] The distributed cache center in this embodiment is based on a distributed cache database. It caches task data, executor data, and task instance data that require frequent database interactions in the distributed cache database, which solves the efficiency problem of frequent database interactions such as task verification. While improving the concurrency and query efficiency of the scheduling center, it also solves the problems of data loss caused by scheduling center restart and inconsistent cache data in distributed deployment.

[0102] In this embodiment of the task scheduling system, the routing management module is used to perform hash calculations on the scheduling executors and task instances, and to evenly distribute tasks to each executor, including:

[0103] Based on the consistent hash algorithm, the scheduler is abstracted as a ring, and the numbers from 0 to 2^32 are evenly distributed on the ring. The server IP where the executor is located is hashed, and the hash value is distributed within the range formed by the numbers on the ring.

[0104] For a scheduling instance, a hash is calculated based on the instance ID and the resulting hash value is projected onto the aforementioned ring. If no node is found at the projection point, the first node in the clockwise direction is the executor corresponding to that instance.

[0105] For a newly added node, tasks from that node counterclockwise to the first nearest node will be redistributed to the new node.

[0106] For nodes that need to be deleted or faulty nodes, the tasks from that node in a counter-clockwise direction to the nearest node will be reassigned to the first node in a clockwise direction starting from that node.

[0107] In this embodiment, as Figure 1 As shown, the consistent hashing algorithm maps hash values ​​to a virtual ring, with the entire hash space ranging from 0 to 2^32. First, the IP address of the executor registered with the scheduling center is hashed to obtain a 32-bit binary value. The executor nodes are then configured on the ring, as shown in "Executor 1, Executor 2, Executor 3" in the diagram. Next, the runtime instance ID of the scheduled task is hashed to obtain the corresponding hash value. Based on the hash value, the task instance is projected onto the hash ring, as shown in "Task 1, Task 2" in the diagram. Finally, based on the position of the task's hash value, the process is performed clockwise along the ring; the first executor encountered is the one that will handle the task.

[0108] In practical applications, after a task passes verification, the scheduling center selects an executor based on the executors registered with the center and the routing strategy. Current routing strategies include round-robin or random routing. However, these strategies become inadequate when faced with numerous scheduling tasks, dynamic additions and removals of executor nodes, and limited executor resources. This leads to skewed task allocation and polarized resource utilization. Executors with a large number of tasks are under resource strain, causing tasks on their current nodes to fail or wait for extended periods due to insufficient resources, while executors with fewer tasks remain idle.

[0109] The routing management module of the task scheduling system in this embodiment uses a routing sharding strategy based on a consistent hash algorithm. It performs hash calculations on both the scheduling executor and the task instance, mapping tasks clockwise to the nearest node. If a node is added or deleted, only the task allocation on adjacent nodes is affected, avoiding the recalculation and reassignment of all scheduled tasks. This solves the problems of task allocation skew caused by sequential or random sharding in existing scheduling tools, as well as sharding fluctuations due to registration machine fluctuations. While ensuring even task allocation, the idle resources of the executor nodes are considered. This guarantees a balanced task load across executor nodes, improves resource utilization, and reduces the task execution failure rate.

[0110] In this embodiment of the task scheduling system, the resource center is used to automatically allocate resource pools and provide configurable resources by configuring adaptive and adjustable resource pools, including:

[0111] Based on the available memory capacity or CPU parameter information of the system, initialize the parameters of the scheduling resource pool to achieve adaptive resource pool configuration. Without any configuration, the parameters of the resource pool can be configured reasonably to avoid resource shortage or resource contention.

[0112] Based on the adaptive resource configuration strategy, configuration information is exposed in local configuration files, configuration centers, or startup script parameters. Users can choose to modify resource pool parameters in local configuration files, configuration centers, or startup script parameters as needed. The system obtains the highest priority configuration based on configuration priority and sets the resource pool parameters accordingly, thereby ensuring the stable and efficient operation of the server system.

[0113] Specifically, based on the system's available memory capacity or CPU parameter information, initialize the parameters of the scheduling resource pool, including calculating the core thread count and maximum thread count of the scheduling resource pool using the following formula:

[0114] N Core_threads =N cpu *(1+M ava / M total )

[0115] N Max_threads =N cpu *(1+2*M ava / M total )

[0116] Where, N Core_threads N represents the core thread count of the resource pool. Max_threads N represents the maximum number of threads in the resource pool. cpu M represents the number of server CPUs. ava M represents the amount of available memory on the server. total This represents the total memory usage of the server.

[0117] In practical applications, the resource pool is the core location for the scheduling center to process scheduling tasks. All scheduled tasks are triggered by retrieving available resources from the resource pool and then proceeding to their respective business processing. Task triggering is asynchronous; task triggering and verification are handled in separate threads, and tasks do not affect each other because their processing threads are different. Therefore, the available threads, maximum number of threads, and queue capacity of the scheduling resource pool determine the scheduling center's ability to concurrently process tasks, accommodate a certain number of tasks, and handle a maximum number of tasks.

[0118] In this embodiment, the adaptive and adjustable resource pool configurations aim to automatically adapt to available server resources while providing a more refined resource configuration scheme. Within the limits of service resource capacity, by increasing the capacity and processing efficiency of the resource pool, the task response efficiency and service throughput of the scheduling center are improved.

[0119] In this embodiment of the task scheduling system, the data source center is used to automatically adapt to multiple data sources based on the configured data source information, including:

[0120] The persistence layer uses the JPA framework based on the ORM specification to shield database dialect differences through object-relational mapping;

[0121] The database Dialect is built into the database to implement specific SQL statement conversion;

[0122] The data source configuration is exposed in the system configuration file. Users can configure and schedule database connection information according to the actual situation. The system automatically switches to the corresponding database Dialect class based on the database connection information, database type and corresponding database version information in the configuration file, and converts the query into the corresponding database SQL statement to achieve adaptation to different database types.

[0123] Data persistence is a necessary step for scheduling centers to store scheduling information. Most existing persistence layer solutions for scheduling centers are implemented using specific databases and dialects, which fails to enable rapid switching between business databases.

[0124] The task scheduling system in this embodiment supports multiple data source adaptation based on the ORM specification. It automatically adapts to defined data sources according to the configured data source information, enabling rapid switching of scheduling business databases and reducing the adaptation cost of the scheduling center. This task scheduling system can adapt to common databases such as MySQL, Oracle, PostgreSQL, SQL Server, and DM, solving the problem of database incompatibility caused by different database SQL dialects in existing scheduling tools. Furthermore, to achieve configurable data source parameter adjustment and expose new data source access interfaces, only the database-related information in the configuration file needs to be modified during on-site deployment, allowing adaptation to the specified database with minimal development effort.

[0125] The data source center of the task scheduling system in this embodiment also supports secondary development and expansion. Users only need to develop and expand the Dialect class that needs to support database types. In order to improve query efficiency, a cache center can also be introduced to cache hot data.

[0126] The third exemplary embodiment of the present invention provides a resource-adaptive task scheduling system. Figure 3 This is an architectural diagram of a resource-adaptive task scheduling system according to an exemplary third embodiment of the present invention, as shown below. Figure 3 As shown, the system in this embodiment includes:

[0127] The scheduling center includes a task management module, an executor management module, a distributed cache center, a routing management module, a data source center, a resource center, a log management module, a monitoring center, a data center, a registration center, a scheduler, a callback service module, and a log information module. The distributed cache center is used for distributed caching of task data, executor data, and task instance data. The routing management module performs hash calculations on the scheduled executors and task instances to evenly distribute tasks among the executors. The resource center automatically allocates and provides configurable resources by configuring an adaptive and adjustable resource pool. The data source center automatically adapts to multiple data sources based on the configured data source information.

[0128] The scheduler executor includes an executor registration module, an executor service module, a callback thread module, and a log service module. The executor registration module is used to register executor information with the scheduling center's registration center via API. The executor service module is used to execute tasks. The callback thread module is used to return task execution information to the callback service module after the task is completed. The log service module is used to record logs of the task execution process and pass the logs to the log information module.

[0129] like Figure 4 As shown, in practical applications, the resource-adaptive task scheduling system in this embodiment performs the following steps during task scheduling:

[0130] Step 1: Register the executor. After the node hosting the executor starts, the executor registers its name, IP address, port, executor code, and other information with the scheduling center's registration center via the API interface. This embodiment supports cluster deployment of the executor. When the executor is in a cluster, the scheduling center receives multiple nodes with the same executor code and distributes the nodes according to a consistent hash algorithm for subsequent task allocation. After registration, the executor can be viewed and edited in the executor management interface.

[0131] Step 2: Create a scheduling task. Create a scheduling task in the task management module of the scheduling center or through the API interface. The created scheduling task can be viewed and edited in the scheduling management module.

[0132] Step 3: Scheduled Scanning and Task Triggering. The scheduling center periodically scans for scheduled tasks, retrieving tasks scheduled to be triggered within the last 5 minutes from the distributed cache center (if the cache center data is null, the data from the data center is loaded into the cache center first). If the task triggering conditions are met (e.g., upstream tasks have been completed), the task is triggered at the specified time. After successful triggering, the next trigger time for the task is synchronously modified in the cache and database according to the task's triggering rules. After a task is triggered, the task instance ID generated by the task is hashed using a consistent hash algorithm, and the task is assigned to the corresponding executor for execution based on the calculated hash value.

[0133] Step 4: Task Execution. The scheduling center scheduler obtains the tasks to be executed and sends them to the corresponding executor service via the RPC protocol. The executor service receives the task from the scheduling center, adds it to the execution queue, and the executor resource pool consumes tasks from the queue for execution. When the executor resource pool consumes a task, it creates or obtains a thread corresponding to that task instance for asynchronous execution. Each thread has a bound task handler, and the specific task execution logic is handled in the corresponding handler.

[0134] Step 5: Record task logs. During task execution, the executor will record task logs to both the local log file and the log database, which will facilitate the investigation of abnormal tasks and the tracing of the source of problems.

[0135] Step 6: Callback Task. After the scheduler completes its task, it triggers a callback function, placing the callback information into a callback queue to await consumption by the callback resource pool. Once the callback information is retrieved from the queue and consumed, it triggers an asynchronous thread to call the scheduling center's callback service, uploading relevant parameters such as task execution status and execution time to the scheduling center for viewing, summarizing, and statistically analyzing the tasks. This completes the task scheduling process.

[0136] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for resource adaptive task scheduling, characterized in that, The method comprises: Distributed caching of task data, executor data, and task instance data; Hash calculation of the scheduling executor and the task instance to uniformly distribute tasks to each executor; Automatic allocation of a resource pool and provision of configurable resources through configuration of an adaptive and adjustable resource pool; Automatic adaptation of multiple data sources according to configured data source information; Automatic allocation of a resource pool and provision of configurable resources through configuration of an adaptive and adjustable resource pool, comprising: Initialization of parameters of the scheduling resource pool according to system available memory capacity or CPU parameter information; Exposure of configuration information in a local configuration file, a configuration center, or a startup script parameter, so that a user can modify resource pool parameters in the local configuration file, the configuration center, or the startup script parameter according to needs, and the system sets resource pool parameters according to the highest priority configuration according to a configuration priority; Initialization of parameters of the scheduling resource pool according to system available memory capacity or CPU parameter information, comprising calculation of the core thread number and the maximum thread number of the scheduling resource pool by using the following formula: N Core_threads =N cpu *(1+M ava / M total ) N Max_threads =N cpu *(1+2*M ava / M total ) where N Core_threads is the number of resource pool core threads, N Max_threads is the number of resource pool maximum threads, N cpu is the number of server CPUs, M ava is the number of server available memory, M total is the number of server total memory.

2. The method of claim 1, wherein, Distributed caching of task data, executor data, and task instance data, comprising: When data is written into a distributed cache database, the cache is first flushed, and then a database write operation is performed; if the write operation is abnormal, a cache rollback operation is performed; When data is queried, the distributed cache database is preferentially queried; if the data exists, the queried data is returned; if the data does not exist, the data center is queried, the data is cached into the distributed cache system, and the queried data is returned.

3. The method of claim 1, wherein, Hash calculation of the scheduling executor and the task instance to uniformly distribute tasks to each executor, comprising: According to a consistent hash algorithm, the scheduling executor is abstracted as a ring, and numbers from 0 to 2^32 are uniformly distributed on the ring; the server IP of the executor is hash calculated, and the hash value is distributed to the interval formed by the numbers on the ring; For a scheduling instance, the instance ID is hash calculated, and the obtained hash value is projected onto the ring; if no node is found at the projection, the first node in the clockwise direction is the executor corresponding to the instance; For a newly added node, the tasks between the node and the first adjacent node in the counterclockwise direction are redistributed to the newly added node; For a node to be deleted or a fault node, the tasks between the node and the first adjacent node in the counterclockwise direction are redistributed to the first node in the clockwise direction starting from the node.

4. The method of claim 1, wherein, Automatic adaptation of multiple data sources according to configured data source information, comprising: A JPA framework based on an ORM specification is used in a persistent layer to shield the differences between database dialects through object relationship mapping; A database Dialect is built in the database to implement specific sql statement conversion; The configuration of the data source is exposed in a system configuration file, a user configures database connection information according to actual conditions, and the system automatically switches to a corresponding database Dialect class according to the database connection information, the database type and the corresponding database version information in the configuration file, converts a query into a sql statement of the corresponding database, and realizes the adaptation of different database types.

5. A resource adaptive based task scheduling system, characterized in that, The system comprises a scheduling center and a scheduling executor, wherein the scheduling center comprises: A distributed cache center for caching task data, executor data and task instance data in a distributed manner; A routing management module for performing hash calculation on the scheduling executor and the task instance, and uniformly distributing tasks to each executor; A resource center for automatically allocating a self-adaptive and adjustable resource pool and providing configurable resources; A data source center for automatically adapting multiple data sources according to configured data source information; The resource center calculates the core thread number and the maximum thread number of the scheduling resource pool by using the following formula: N Core_threads =N cpu *(1+M ava / M total ) N Max_threads =N cpu *(1+2*M ava / M total ) where N Core_threads is the number of resource pool core threads, N Max_threads is the number of resource pool maximum threads, N cpu is the number of server CPUs, M ava is the number of server available memory, M total is the number of server total memory.

6. The resource adaptive based task scheduling system of claim 5, wherein, The distributed cache center is specifically configured to: When data is written into the distributed cache database, the cache is first flushed, and then the database write operation is performed; if the write operation is abnormal, the cache rollback operation is performed; When data is queried, the distributed cache database is preferentially queried; if the data exists, the queried data is returned; if the data does not exist, the data center is queried, the data is cached into the distributed cache system, and the queried data is returned.

7. The resource adaptive based task scheduling system of claim 5, wherein, The routing management module is specifically configured to: According to the consistent hash algorithm, the scheduling executor is abstracted as a ring, and the numbers from 0 to 2^32 are uniformly distributed on the ring; the server IP of the executor is calculated by using the hash algorithm, and the hash value is distributed to the interval formed by the numbers on the ring; For a scheduling instance, the instance ID is calculated by using the hash algorithm, and the obtained hash value is projected onto the ring; if no node is found at the projection, the first node in the clockwise direction is the executor corresponding to the instance; For a newly added node, the tasks between the first node adjacent to the node in the counterclockwise direction are redistributed to the newly added node; For a node to be deleted or a fault node, the tasks between the first node adjacent to the node in the counterclockwise direction are redistributed to the first node in the clockwise direction.

8. The resource adaptive based task scheduling system of claim 5, wherein, The resource center is specifically configured to: According to the available memory capacity or CPU parameter information of the system, the parameters of the scheduling resource pool are initialized; The configuration information is exposed in a local configuration file, a configuration center or a startup script parameter; a user can modify the resource pool parameters in the local configuration file, the configuration center or the startup script parameter according to requirements; and the system sets the resource pool parameters according to the configuration priority to obtain the configuration with the highest priority.

9. The resource adaptive based task scheduling system of claim 5, wherein, The data source center is specifically configured to: A JPA framework based on an ORM specification is used in the persistent layer to shield the differences between database dialects; A database Dialect is built in the database to realize specific sql statement conversion. The configuration of the data source is exposed in a system configuration file, user configures the database connection information according to actual conditions, and the system automatically switches to the corresponding database Dialect class according to the database connection information, the database type and the corresponding database version information in the configuration file, converts the query into the sql statement of the corresponding database, and realizes the adaptation of different database types.

Images