Distributed computing engine task queue submission method and system based on database

Abstract

The invention discloses a distributed computing engine task queue submission method and system based on a database, belongs to the technical field of distributed computing, and aims to solve the technical problems of complexity of an independent queue component in a task queue of a distributed computing engine, burden of reliability guarantee and separation from a storage system. Comprising the steps of writing task data in a task request into a task table through database atomic writing, and awakening a scheduler through a database native notification function; the scheduler preempts an unlocked and executable task, updates a task state and sets a distributed lock; after the Worker receives the task, the task state is updated in the database transaction, and the task is submitted to the distributed computing engine to be executed; the task state is updated according to the task execution result, if task execution fails, the task state is converted into RETRYING based on the retry strategy, the scheduling time is reset, and if the maximum retry number is reached, the task state is permanently marked as FAILED.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology, specifically to a database-based distributed computing engine task queue submission method and system. Background Technology

[0002] With the rapid development of big data, artificial intelligence, and high-performance computing, distributed computing engines (such as Spark, Flink, Presto / Trino, Ray, etc.) have become the core infrastructure for processing massive amounts of data and complex computing tasks. In these engines, the task queue plays a crucial role, responsible for receiving, buffering, and scheduling user-submitted jobs or tasks, and efficiently and reliably distributing them to worker nodes in the cluster for execution.

[0003] However, traditional distributed computing engines typically face the following challenges in implementing task queues:

[0004] 1. Complexity of independent queue components: Many engines rely on dedicated message queues (such as Kafka, RabbitMQ, Pulsar) or self-developed queue services that are independent of the computing engine and storage system to implement task submission and management. This introduces additional operational complexity (deployment, monitoring, scaling), system dependencies, and potential points of failure.

[0005] 2. Burden of reliability assurance: To achieve highly reliable task submission (such as Exactly-Once semantics), persistent storage, fault recovery (queue service downtime, network partitioning), and error handling (task retries, dead letter queues), the task queue component itself usually needs to have a strong fault tolerance mechanism, which increases the difficulty of implementation.

[0006] 3. Decoupling from the storage system: Computational tasks typically need to process data stored in databases or data warehouses. When the task queue is independent of the underlying storage system, task submission logic, metadata management, permission verification, data location awareness, etc., often require additional bridging logic, which may lead to inefficiency (such as data migration) or consistency issues.

[0007] The complexity of independent queue components in the task queue of a distributed computing engine, the burden of ensuring reliability, and the separation from the storage system are technical problems that need to be solved. Summary of the Invention

[0008] The technical objective of this invention is to address the above-mentioned shortcomings by providing a database-based distributed computing engine task queue submission method and system, thereby resolving the technical problems of complexity, reliability assurance burden, and separation from the storage system in the task queue of a distributed computing engine.

[0009] In a first aspect, the present invention provides a database-based distributed computing engine task queue submission method, comprising the following steps:

[0010] Task submission: Receive task requests submitted by producers, start database transactions, write task data from task requests to the task table through database atomic write, and wake up the scheduler through the database native notification function. The fields in the task table include task ID, task status, task priority information, number of retries, and lock information.

[0011] Task scheduling: The scheduler uses SELECT...FOR UPDATE SKIP LOCKED to preempt unlocked and executable tasks, updates the task status to SUBMITTED, and sets a distributed lock;

[0012] Task execution: After receiving the task, the Worker updates the task status to RUNNING in the database transaction and submits the task to the distributed computing engine for execution;

[0013] Task Management: Update the task status to SUCCESS or FAILED based on the task execution result. If the task execution fails, change the task status to RETRYING and reset the scheduling time based on the retry policy. If the maximum number of retries is reached, permanently mark it as FAILED.

[0014] Preferably, when writing task data from the task request to the task table via atomic database write,

[0015] The task data is transformed into a standard task queue object. The attributes of the task queue object have a one-to-one mapping relationship with the fields of the task table. A database transaction is started, and within the database transaction, the task queue object is written to the task table, and the task status is changed to PENDING.

[0016] As a preferred method, when waking up the scheduler using native database functions, the NOTIFY statement in PostgreSQL should be used.

[0017] As a preferred option, the task table is set with a locked_by field and a locked_until field. The locked_by field is used to record the Worker identifier of the currently occupied task, and the locked_until field is used to write the absolute expiration timestamp.

[0018] Distributed locks use the `locked_by` field to identify the Worker ID and the `locked_until` field to set the lock expiration time. Locks that are not released after the expiration time can be preempted by other schedulers.

[0019] Secondly, the present invention provides a database-based distributed computing engine task queue submission system, comprising a task submission module, a scheduler, a worker, and a task management module.

[0020] The task submission module is used to perform the following: receive task requests submitted by producers, start database transactions, write task data in the task request to the task table through database atomic write, and wake up the scheduler through the database native notification function. The fields in the task table include task ID, task status, task priority information, number of retries, and lock information.

[0021] The scheduler is used to perform the following: preempting unlocked and executable tasks using SELECT...FOR UPDATE SKIP LOCKED, updating the task status to SUBMITTED, and setting a distributed lock;

[0022] The Worker is used to perform the following: after receiving a task, update the task status to RUNNING in a database transaction, and submit the task to the distributed computing engine for execution;

[0023] The task management module is used to perform the following: update the task status to SUCCESS or FAILED based on the task execution result; if the task execution fails, change the task status to RETRYING and reset the scheduling time based on the retry policy; if the maximum number of retries is reached, permanently mark it as FAILED.

[0024] Preferably, when writing task data from the task request to the task table via atomic database write,

[0025] The task submission module performs the following actions: converts task data into standard task queue objects, with a one-to-one mapping between the attributes of the task queue objects and the fields of the task table; initiates a database transaction; writes the task queue objects into the task table within the database transaction; and changes the task status to PENDING.

[0026] Preferably, when waking up the scheduler using native database functionality, the task submission module is used to wake up the scheduler via the NOTIFY statement in PostgreSQL.

[0027] As a preferred option, the task table is set with a locked_by field and a locked_until field. The locked_by field is used to record the Worker identifier of the currently occupied task, and the locked_until field is used to write the absolute expiration timestamp.

[0028] Distributed locks use the `locked_by` field to identify the Worker ID and the `locked_until` field to set the lock expiration time. Locks that are not released after the expiration time can be preempted by other schedulers.

[0029] The database-based distributed computing engine task queue submission method and system of the present invention have the following advantages:

[0030] 1. Simplified architecture: Replacing message queues with databases to achieve a "zero middleware architecture" reduces operational costs;

[0031] 2. Unified Ecosystem: Simplifies complex distributed systems into database operation problems;

[0032] 3. Enhanced Reliability: Tasks are persisted to the database, ensuring that pending tasks are preserved even during system crashes, significantly improving reliability. With its reasonable design, simple structure, ease of manufacturing, small size, user-friendliness, and multi-functionality, it possesses excellent potential for widespread adoption. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] The invention will be further described below with reference to the accompanying drawings.

[0035] Figure 1 This is a flowchart of a database-based distributed computing engine task queue submission method, as described in Example 1. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0037] This invention provides a database-based task queue submission method for distributed computing engines, which addresses the technical problems of complexity, reliability assurance burden, and separation from storage systems in the task queues of distributed computing engines.

[0038] Example 1:

[0039] This invention provides a database-based distributed computing engine task queue submission method, comprising four steps: task submission, task scheduling, task execution, and task management.

[0040] Step S100 Task Submission: Receive the task request submitted by the producer, start a database transaction, write the task data in the task request into the task table through database atomic write, and wake up the scheduler through the database native notification function. The fields in the task table include task ID, task status, task priority information, number of retries, and lock information.

[0041] As a specific implementation of task submission, when writing task data from the task request to the task table via atomic database write, the task data is transformed into a standard task queue object. The attributes of the task queue object have a one-to-one mapping relationship with the fields of the task table. A database transaction is started, and within the database transaction, the task queue object is written to the task table, and the task status is changed to PENDING.

[0042] As a specific implementation of the task table, its structure is shown in Table 1.

[0043] Table 1, Task List

[0044]

[0045] Step S200 Task Scheduling: The scheduler uses SELECT...FOR UPDATE SKIP LOCKED to preempt unlocked and executable tasks, updates the task status to SUBMITTED, and sets a distributed lock. When waking the scheduler using native database functionality, the PostgreSQL NOTIFY statement is used.

[0046] The task table contains two fields: locked_by and locked_until. The locked_by field records the Worker ID of the currently holding task, and the locked_until field is used to write the absolute expiration timestamp. The distributed lock identifies the Worker ID through the locked_by field and sets the lock expiration time through the locked_until field. Locks that are not released after the timeout can be preempted by other schedulers.

[0047] As a specific implementation of task scheduling, this step includes the following operations:

[0048] (1) Use transactions and pessimistic locking to query pending tasks and encapsulate them into standard Task objects. The specific method for obtaining them is as follows:

[0049] SELECT * FROM task_queue

[0050] WHERE status = 'PENDING'

[0051] AND scheduled_at <= NOW()

[0052] AND(locked_until IS NULL OR locked_until <NOW())

[0053] ORDER BY priority DESC,created_at

[0054] LIMIT 1

[0055] FOR UPDATE SKIP LOCKED -- Key: Skips rows that are already locked;

[0056] (2) If the task is locked (to prevent it from being preempted by other schedulers):

[0057] UPDATE task_queue

[0058] SET status = 'SUBMITTED',

[0059] locked_by='xxx',

[0060] locked_until=NOW()+INTERVAL'5MINUTE'

[0061] WHERE task_id = 'xxxx';

[0062] (3) Send tasks (RPC / message) to the Worker.

[0063] Step S300 Task Execution: After receiving the task, the Worker updates the task status to RUNNING in the database transaction and submits the task to the distributed computing engine for execution.

[0064] The update status is RUNNING, and the corresponding code is as follows:

[0065] UPDATE task_queue

[0066] SET status = 'RUNNING'

[0067] WHERE task_id='xxxx'.

[0068] Step S400 Task Management: Update the task status to SUCCESS or FAILED based on the task execution result. If the task execution fails, change the task status to RETRYING and reset the scheduling time based on the retry policy. If the maximum number of retries is reached, permanently mark it as FAILED.

[0069] The code for updating the status after successful task execution is as follows:

[0070] UPDATE task_queue

[0071] SET status = 'SUCCESS',

[0072] locked_by = NULL,

[0073] locked_until = NULL

[0074] WHERE task_id = ?

[0075] When a task fails, a retry mechanism is triggered: the number of retries is checked and the status is updated.

[0076] UPDATE task_queue

[0077] SET retry_count=retry_count+1,

[0078] status = CASE

[0079] WHEN retry_count <max_retries THEN'RETRYING'

[0080] ELSE'FAILED'

[0081] END

[0082] WHERE task_id='xxx'.

[0083] If the task status is retry, add the task to the queue.

[0084] The method in this embodiment leverages the powerful data storage, transaction processing, concurrency control, high availability, and query capabilities of the database itself to "internalize" core functions such as task queue state management, persistent storage, and commit interfaces into the database. This not only simplifies the system architecture and reduces operational complexity, but also fully utilizes the inherent characteristics of the database (such as ACID transactions) to ensure the consistency and reliability of task commits and state management. Simultaneously, it achieves tight integration with the underlying storage data, improving overall efficiency and ease of use.

[0085] Example 2:

[0086] This invention discloses a database-based distributed computing engine task queue submission system, comprising a task submission module, a scheduler, a worker, and a task management module.

[0087] The task submission module is used to perform the following: receive task requests submitted by producers, start database transactions, write the task data in the task request to the task table through database atomic write, and wake up the scheduler through the database native notification function. The fields in the task table include task ID, task status, task priority information, number of retries, and lock information.

[0088] As a specific implementation of the task submission module, when writing task data from a task request to the task table via atomic database write, the task submission module performs the following: converting the task data into a standard task queue object, where the attributes of the task queue object have a one-to-one mapping with the fields of the task table; initiating a database transaction; writing the task queue object to the task table within the database transaction; and changing the task status to PENDING. When waking up the scheduler using native database functionality, the task submission module wakes up the scheduler using the PostgreSQL NOTIFY statement.

[0089] The scheduler is used to perform the following: preempting unlocked and executable tasks using SELECT...FOR UPDATE SKIP LOCKED, updating the task status to SUBMITTED, and setting a distributed lock.

[0090] The task table contains two fields: locked_by and locked_until. The locked_by field records the Worker ID of the currently holding task, and the locked_until field is used to write the absolute expiration timestamp. The distributed lock identifies the Worker ID through the locked_by field and sets the lock expiration time through the locked_until field. Locks that are not released after the timeout can be preempted by other schedulers.

[0091] As a specific implementation of the scheduler, it performs the following operations:

[0092] (1) Use transactions and pessimistic locking to query pending tasks and encapsulate them into standard Task objects. The specific method for obtaining them is as follows:

[0093] SELECT * FROM task_queue

[0094] WHERE status = 'PENDING'

[0095] AND scheduled_at <= NOW()

[0096] AND(locked_until IS NULL OR locked_until <NOW())

[0097] ORDER BY priority DESC,created_at

[0098] LIMIT 1

[0099] FOR UPDATE SKIP LOCKED -- Key: Skips rows that are already locked;

[0100] (2) If the task is locked (to prevent it from being preempted by other schedulers):

[0101] UPDATE task_queue

[0102] SET status = 'SUBMITTED',

[0103] locked_by='xxx',

[0104] locked_until=NOW()+INTERVAL'5MINUTE'

[0105] WHERE task_id = 'xxxx';

[0106] (3) Send tasks (RPC / message) to the Worker.

[0107] The Worker is used to perform the following: after receiving a task, it updates the task status to RUNNING in a database transaction and submits the task to the distributed computing engine for execution.

[0108] The update status is RUNNING, and the corresponding code is as follows:

[0109] UPDATE task_queue

[0110] SET status = 'RUNNING'

[0111] WHERE task_id='xxxx'.

[0112] Step S300 Task Execution: After receiving the task, the Worker updates the task status to RUNNING in the database transaction and submits the task to the distributed computing engine for execution.

[0113] The update status is RUNNING, and the corresponding code is as follows:

[0114] UPDATE task_queue

[0115] SET status = 'RUNNING'

[0116] WHERE task_id='xxxx'.

[0117] The task management module is used to perform the following: update the task status to SUCCESS or FAILED based on the task execution result; if the task execution fails, change the task status to RETRYING and reset the scheduling time based on the retry policy; if the maximum number of retries is reached, permanently mark it as FAILED.

[0118] The code for updating the status after successful task execution is as follows:

[0119] UPDATE task_queue

[0120] SET status = 'SUCCESS',

[0121] locked_by = NULL,

[0122] locked_until = NULL

[0123] WHERE task_id = ?

[0124] When a task fails, a retry mechanism is triggered: the number of retries is checked and the status is updated.

[0125] UPDATE task_queue

[0126] SET retry_count=retry_count+1,

[0127] status = CASE

[0128] WHEN retry_count <max_retries THEN'RETRYING'

[0129] ELSE'FAILED'

[0130] END

[0131] WHERE task_id='xxx'.

[0132] If the task status is retry, add the task to the queue.

[0133] The system in this embodiment can execute the method disclosed in Embodiment 1 to submit the task queue of the distributed computing engine.

[0134] The above provides a detailed description of the database-based distributed computing engine task queue submission method and system provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A database-based distributed computing engine task queue submission method, characterized in that, Includes the following steps: Task submission: Receive task requests submitted by producers, start database transactions, write task data from task requests to the task table through database atomic write, and wake up the scheduler through the database native notification function. The fields in the task table include task ID, task status, task priority information, number of retries, and lock information. Task scheduling: The scheduler uses SELECT...FOR UPDATE SKIP LOCKED to preempt unlocked and executable tasks, updates the task status to SUBMITTED, and sets a distributed lock; Task execution: After receiving the task, the Worker updates the task status to RUNNING in the database transaction and submits the task to the distributed computing engine for execution; Task Management: Update the task status to SUCCESS or FAILED based on the task execution result. If the task execution fails, change the task status to RETRYING and reset the scheduling time based on the retry policy. If the maximum number of retries is reached, permanently mark it as FAILED.

2. The database-based distributed computing engine task queue submission method according to claim 1, characterized in that, When writing task data from a task request to a task table via atomic database write, the task data is transformed into a standard task queue object. The attributes of the task queue object have a one-to-one mapping relationship with the fields of the task table. A database transaction is started, and within the database transaction, the task queue object is written to the task table, and the task status is changed to PENDING.

3. The database-based distributed computing engine task queue submission method according to claim 1, characterized in that, When waking up the scheduler using native database functions, the scheduler is woken up using the NOTIFY statement in PostgreSQL.

4. The database-based distributed computing engine task queue submission method according to claim 1, characterized in that, The task table has two fields: locked_by and locked_until. The locked_by field is used to record the Worker identifier of the currently occupied task, and the locked_until field is used to write the absolute expiration timestamp. Distributed locks use the `locked_by` field to identify the Worker ID and the `locked_until` field to set the lock expiration time. Locks that are not released after the expiration time can be preempted by other schedulers.

5. A database-based distributed computing engine task queue submission system, characterized in that, It includes a task submission module, a scheduler, a worker, and a task management module; The task submission module is used to perform the following: receive task requests submitted by producers, start database transactions, write task data in the task request to the task table through database atomic write, and wake up the scheduler through the database native notification function. The fields in the task table include task ID, task status, task priority information, number of retries, and lock information. The scheduler is used to perform the following: preempting unlocked and executable tasks using SELECT...FOR UPDATE SKIP LOCKED, updating the task status to SUBMITTED, and setting a distributed lock; The Worker is used to perform the following: after receiving a task, update the task status to RUNNING in a database transaction, and submit the task to the distributed computing engine for execution; The task management module is used to perform the following: update the task status to SUCCESS or FAILED based on the task execution result; if the task execution fails, change the task status to RETRYING and reset the scheduling time based on the retry policy; if the maximum number of retries is reached, permanently mark it as FAILED.

6. The database-based distributed computing engine task queue submission system according to claim 5, characterized in that, When writing task data from a task request to the task table via atomic database write, the task submission module performs the following: converts the task data into a standard task queue object, with a one-to-one mapping between the attributes of the task queue object and the fields of the task table; initiates a database transaction; writes the task queue object to the task table within the database transaction; and changes the task status to PENDING.

7. The database-based distributed computing engine task queue submission system according to claim 5, characterized in that, When waking up the scheduler using native database functionality, the task submission module is used to wake up the scheduler via the NOTIFY statement in PostgreSQL.

8. The database-based distributed computing engine task queue submission system according to claim 5, characterized in that, The task table has two fields: locked_by and locked_until. The locked_by field is used to record the Worker identifier of the currently occupied task, and the locked_until field is used to write the absolute expiration timestamp. Distributed locks use the `locked_by` field to identify the Worker ID and the `locked_until` field to set the lock expiration time. Locks that are not released after the expiration time can be preempted by other schedulers.

Images