A shared task queue management system and method under a Kubernetes environment

Abstract

The application discloses a shared task queue management system and method in a Kubernetes environment, and relates to the field of computers.The system comprises a kafka message module, a local task queue module and a kafka topic unit.The local task queue module comprises a task production unit and a task processing unit, and the task processing unit comprises a plurality of subscription groups corresponding to the plurality of kafka topic units.The task production unit is used for receiving a task request, generating a task and publishing the task to a corresponding kafka topic unit.The kafka topic unit is used for inserting the task into a to-be-assigned queue thereof.Each pod node of each subscription group is used for periodically acquiring task information in a to-be-assigned queue of a corresponding partition in a corresponding kafka topic unit, and actively acquiring a task suitable for a load condition of the pod node according to the load condition.The application solves the problem that a traditional single-machine task queue cannot assign tasks across pods.

Description

Technical Field

[0001] This invention relates to the field of task queue management technology, and in particular to a shared task queue management system and method in a Kubernetes environment. Background Technology

[0002] Some medical image data processing procedures deploy multi-instance task processing services in a Kubernetes containerized environment, utilizing the pod's local cache queue to process tasks. Each application instance independently maintains its in-memory task queue, and task generation, allocation, and execution are all completed within a single node. While this approach is feasible in a single-node system, in a Kubernetes multi-node environment, because pod instances are unaware of each other's load status, task allocation is typically based on round-robin or random strategies. This leads to some instances becoming overloaded due to processing time-consuming tasks, while other instances remain idle, resulting in low resource utilization and limited system throughput. Furthermore, because task queues are strongly bound to instances, new instances cannot immediately share tasks, and tasks cannot be smoothly migrated during scaling down, making it difficult to fully leverage Kubernetes' elastic scaling advantages and thus solidifying performance bottlenecks. Summary of the Invention

[0003] To address the technical problems existing in the background art, this invention proposes a shared task queue management system and method in a Kubernetes environment.

[0004] In a first aspect, the present invention proposes a shared task queue management system in a Kubernetes environment, comprising: a local task queue module and a Kafka message module; The Kafka messaging module includes multiple Kafka topic units, each of which contains multiple partitions, and each partition contains queues to be assigned. The local task queue module includes a task production unit and a task processing unit. The task processing unit includes multiple pod nodes, which are divided into multiple subscription groups. Each subscription group subscribes to multiple Kafka topic units in a one-to-one correspondence. The task production unit is used to receive user task requests, generate tasks according to the requests, and publish the tasks to the corresponding Kafka topic unit; Each Kafka topic unit is used to insert a new task into the assignment queue based on its priority attribute value when a new task is published. Each pod node in each subscription group is used to periodically obtain task information from the task queue of the corresponding partition in the corresponding Kafka topic unit, and determine whether it can receive tasks based on its load. If so, the pod node actively obtains tasks from the task queue that are appropriate for its load and processes them.

[0005] Preferably, each partition also includes an allocated queue and a completed queue; Each Kafka topic unit is also used to move a task in the pending queue to the assigned queue after it is actively acquired by a Pod node, and to determine whether the task completion information has been received after a predetermined time. If yes, the task is considered to have been successfully executed by the Pod node and is moved to the completed queue. If no, the task is considered to have failed and is moved back from the assigned queue to the pending queue.

[0006] Preferably, each pod node has a built-in local task queue, and each local task queue includes multiple independent task queues; After each Pod node actively retrieves tasks appropriate to its load from the task queue, it also adds the task to the corresponding task queue in its local task queue based on the category of the retrieved task.

[0007] Preferably, it also includes a first Redis caching middleware; Each Pod node is also used to store the processing result of a task in the first Redis cache middleware and return the task ID of the task to the user when a task is completed.

[0008] Preferably, after returning the task ID to the user, the method further includes: Users retrieve the corresponding task results from the first Redis cache middleware based on the task ID.

[0009] Preferably, it also includes a task status module and a second Redis cache middleware; the task status module is used to periodically obtain the task status of each task in each local task queue of each Pod node according to a second time interval, and store the task status of each task in each local task queue of each Pod node in the second Redis cache middleware.

[0010] Preferably, the task status module is also used to monitor the CPU and memory usage of each Pod node, and the local task queue module is also used to automatically reject new task requests when the CPU and memory usage exceed the corresponding thresholds.

[0011] Preferably, the task processing unit is also used to periodically clear failed or expired tasks from the local task queue of each pod node.

[0012] Secondly, the present invention also proposes a shared task queue management method in a Kubernetes environment, applicable to the shared task queue management system in the Kubernetes environment described in any one of the first aspects, comprising: Get the task request submitted by the user; Tasks are generated based on task requests and then published to the corresponding Kafka topic units. When a new task is published, each Kafka topic unit inserts the task into the queue to be assigned based on its priority attribute value. Each pod node in each subscription group periodically retrieves task information from the waiting queue of the corresponding Kafka topic unit and determines whether it can receive tasks based on its load. If so, the pod node actively retrieves tasks from the waiting queue of the Kafka topic unit that are appropriate for its load and processes them.

[0013] Preferably, each partition also includes an allocated queue and a completed queue; After the Pod node actively retrieves tasks appropriate to its load from the Kafka topic unit's assignment queue, the following steps are also included: The Kafka topic unit moves the task to the assigned queue and checks after a predetermined time whether it has received the task completion information. If it does, the task is considered to have been executed successfully on the Pod node, and the task is moved to the completed queue. If it does not, the task is considered to have failed, and the task is moved from the assigned queue to the pending queue.

[0014] This invention presents a shared task queue management system and method for Kubernetes environments. Through the collaboration of a local task queue module and a Kafka messaging module, it achieves task distribution and coordination among multiple Pod instances in a Kubernetes environment. The Kafka messaging module in this invention adopts a publish-subscribe pattern, uniformly coordinating the task production unit (Producer) and task processing unit (Consumer) within the local task queue module. The task production unit serializes newly generated tasks and sends them to designated Kafka topic units. The task processing unit subscribes to corresponding Kafka topic units through different subscription groups, achieving automatic task allocation and load balancing, thus solving the problem that traditional single-machine task queues cannot allocate tasks across Pods. Furthermore, dedicated topics for different task types enable task classification and load balancing, ensuring system availability. Attached Figure Description

[0015] Figure 1This is a schematic diagram of the structure of a shared task queue management system in a Kubernetes environment according to an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] Firstly, referring to Figure 1 The present invention proposes a shared task queue management system in a Kubernetes environment, comprising: a local task queue module and a Kafka message module; The Kafka messaging module includes multiple Kafka topic units, each of which contains multiple partitions, and each partition contains queues to be assigned. The local task queue module includes a task production unit and a task processing unit. The task processing unit includes multiple pod nodes, which are divided into multiple subscription groups. Each subscription group subscribes to multiple Kafka topic units in a one-to-one correspondence. The task production unit is used to receive user task requests, generate tasks according to the requests, and publish the tasks to the corresponding Kafka topic unit; Each Kafka topic unit is used to insert a new task into the assignment queue based on its priority attribute value when a new task is published. Each pod node in each subscription group is used to periodically retrieve task information from the task queue of the corresponding partition in the corresponding Kafka topic unit, and determine whether it can receive tasks based on its load. If not, the pod node will not retrieve tasks; if so, the pod node will actively retrieve tasks from the task queue that are appropriate for its load and process them.

[0018] This invention achieves task distribution and coordination among multiple Pod instances in a Kubernetes environment through the cooperation of a local task queue module, a Kafka message module, and a first Redis caching middleware. The Kafka message module in this invention adopts a publish-subscribe pattern, using a KafkaManager singleton instance to uniformly coordinate the task producer and consumer units in the local task queue module. The task producer serializes newly generated tasks and sends them to the designated Kafka topic unit. The task consumer subscribes to corresponding task consumer units through different subscription groups, achieving automatic task allocation and load balancing, solving the problem that traditional single-machine task queues cannot allocate tasks across Pods. Furthermore, dedicated topics for different task types enable task classification and load balancing.

[0019] In this embodiment, each partition also includes an allocated queue and a completed queue; Each Kafka topic unit is also used to move a task in the pending queue to the assigned queue after it is actively acquired by a Pod node, and to determine whether the task completion information has been received after a predetermined time. If yes, the task is considered to have been successfully executed by the Pod node and is moved to the completed queue. If no, the task is considered to have failed and is moved back from the assigned queue to the pending queue.

[0020] With this configuration, when a Pod node fails, the corresponding Kafka topic unit determines that the task completion information of the task assigned to that Pod node has not been received after a predetermined time. If so, the task execution of that Pod node is considered to have failed, and the task is readjusted from the assigned queue to the pending queue. By leveraging Kafka's partitioning mechanism and message persistence features, it is ensured that the unfinished tasks in the failed Pod node can be automatically reassigned in the event of a failure, thus guaranteeing the high availability of the system.

[0021] In this embodiment, each pod node has multiple built-in local task queues, each local task queue includes multiple task queues, and each task queue is configured with independent thread pool resources; After each Pod node actively retrieves tasks appropriate to its load from the task allocation queue, it also adds the task to the corresponding task queue in its local task queue according to the category of the retrieved task, in order to classify and process the task.

[0022] It's important to understand that each pod node can obtain its own load information, which includes CPU utilization, disk utilization, memory usage, and local task size.

[0023] It is important to understand that tasks are divided into real-time tasks and asynchronous tasks. In this embodiment, the task processed by the Kafka message module is an asynchronous task.

[0024] In one specific embodiment, the task production unit is used to serialize the task and then send it to the corresponding Kafka topic unit.

[0025] In this embodiment, a first Redis cache middleware is also included; each Pod node is also used to store the processing result of a task in the first Redis cache middleware when a task is completed, and return the task ID of the task to the user.

[0026] In a further embodiment, after returning the task ID of the task to the user, the method further includes: Users retrieve the corresponding task results from the first Redis cache middleware based on the task ID.

[0027] In this embodiment, a task status module and a second Redis cache middleware are also included. The task status module is used to periodically obtain the task status of each task in each local task queue of each Pod node, and store the task status of each task in each local task queue of each Pod node on the second Redis cache middleware.

[0028] With this configuration, the task status module can track changes in task status, ensuring reliable task execution and reasonable resource utilization. It also periodically synchronizes local task status to the shared storage of the second Redis cache middleware in batches, enabling cross-Pod status information sharing and maintaining the consistency and observability of global task status, so that users can obtain the corresponding task status in the second Redis cache middleware.

[0029] Specifically, the task status module and the local task queue module are connected through a heartbeat mechanism.

[0030] In a further embodiment, the task status module is also used to monitor the CPU and memory usage of each Pod node, and the local task queue module is also used to automatically reject new task requests when the CPU and memory usage exceed the corresponding thresholds to prevent system overload.

[0031] In this embodiment, the task processing unit is also used to periodically clear failed or expired tasks from each pod node.

[0032] It's important to understand that the local task queue module in this embodiment is responsible for localized task scheduling and concurrent execution. Specifically, this module maintains task lifecycle management through TaskManager and uses the TaskFactory pattern to create different types of task instances. Furthermore, it supports concurrent processing across multiple queues, with each queue configured with independent thread pool resources. Upon receiving a task request, the local queue module is responsible for allocating the task to the corresponding task queue for execution and tracking task status changes in real time to ensure reliable task execution and efficient resource utilization.

[0033] In this embodiment, the Kafka messaging module is used to implement task distribution and coordination among multiple Pod instances in a Kubernetes environment. This Kafka messaging module adopts a publish-subscribe pattern, using a KafkaManager singleton instance to uniformly coordinate the Producer and Consumer components. During processing, the task production unit serializes newly generated tasks and sends them to the designated Kafka topic unit. The task processing unit subscribes to the corresponding Kafka topic units through different subscription groups, achieving automatic task allocation and load balancing. This Kafka messaging module leverages Kafka's partitioning mechanism and message persistence features to ensure that tasks can be automatically redistributed in the event of a Pod failure, guaranteeing high system availability.

[0034] In this embodiment, the task status module can periodically synchronize local task status to the second Redis shared storage in batches, realizing cross-Pod status information sharing, thereby maintaining the consistency and observability of global task status; moreover, by monitoring CPU and memory usage, it can automatically reject new task requests when the system load is too high, preventing system overload.

[0035] Secondly, the present invention proposes a shared task queue management method in a Kubernetes environment, applied to the shared task queue management system in the Kubernetes environment described in any one of the first aspects, comprising: Get the task request submitted by the user; Tasks are generated based on task requests and then published to the corresponding Kafka topic units. When a new task is published, each Kafka topic unit inserts the task into the queue to be assigned based on its priority attribute value. Each pod node in each subscription group periodically retrieves task information from the waiting queue of the corresponding Kafka topic unit and determines whether it can receive tasks based on its load. If so, the pod node actively retrieves tasks from the waiting queue of the Kafka topic unit that are appropriate for its load and processes them.

[0036] In this embodiment, each partition also includes an allocated queue and a completed queue; After the Pod node actively retrieves tasks appropriate to its load from the Kafka topic unit's assignment queue, the following steps are also included: The Kafka topic unit moves the task to the assigned queue and checks after a predetermined time whether it has received the task completion information. If it does, the task is considered to have been executed successfully on the Pod node, and the task is moved to the completed queue. If it does not, the task is considered to have failed, and the task is moved from the assigned queue to the pending queue.

[0037] In this embodiment, each pod node has a built-in local task queue, and each local task queue includes multiple independent task queues. After each pod node actively obtains a task that is suitable for its load from the queue to be assigned, it also includes adding the task to the corresponding task queue in its local task queue according to the category of the obtained task.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A shared task queue management system in a Kubernetes environment, characterized in that, include: Local task queue module and Kafka message module; The Kafka messaging module includes multiple Kafka topic units, each containing multiple partitions. Each partition includes a queue to be assigned, a queue of assigned tasks, and a queue of completed tasks. The local task queue module includes a task production unit and a task processing unit. The task processing unit includes multiple pod nodes, which are divided into multiple subscription groups. Each subscription group subscribes to multiple Kafka topic units in a one-to-one correspondence. The number of pod nodes in each subscription group is less than or equal to the number of partitions in the corresponding Kafka topic unit, ensuring that each pod node corresponds to at least one partition. The task production unit is used to receive user task requests, generate tasks according to the requests, and publish the tasks to the corresponding Kafka topic unit; Each Kafka topic unit is used to determine the partition of a new task based on its preset attribute values ​​and insert the task into the waiting queue of that partition based on its priority attribute value when a new task is published. Each pod node in each subscription group is used to periodically retrieve task information from the task queue of the corresponding partition in the corresponding Kafka topic unit, and determine whether it can receive tasks based on its load. If so, the pod node actively retrieves tasks from the task queue that are appropriate for its load. Each Kafka topic unit is also used to move a task in the pending queue to the assigned queue after it is actively acquired by a Pod node, and to determine whether the task completion information has been received after a predetermined time. If yes, the task is considered to have been successfully executed by the Pod node and is moved to the completed queue. If no, the task is considered to have failed and is moved back from the assigned queue to the pending queue.

2. The shared task queue management system in a Kubernetes environment according to claim 1, characterized in that, Each pod node has a built-in local task queue, and each local task queue includes multiple independent task queues; After each Pod node actively retrieves tasks appropriate to its load from the task queue, it also adds the task to the corresponding task queue in its local task queue based on the category of the retrieved task.

3. The shared task queue management system in a Kubernetes environment according to claim 1, characterized in that, It also includes a first Redis cache middleware; each Pod node is also used to store the processing result of a task in the first Redis cache middleware when a task is completed, and return the task ID of the task to the user.

4. The shared task queue management system in a Kubernetes environment according to claim 3, characterized in that, After returning the task ID to the user, the following is also included: Users retrieve the corresponding task results from the first Redis cache middleware based on the task ID.

5. The shared task queue management system in a Kubernetes environment according to claim 1, characterized in that, It also includes a task status module and a second Redis cache middleware; the task status module is used to periodically obtain the task status of each task on each Pod node and store the task status of each task on each Pod node in the second Redis cache middleware.

6. The shared task queue management system in a Kubernetes environment according to claim 5, characterized in that, The task status module is also used to monitor the CPU and memory usage of each Pod node, and the local task queue module is used to automatically reject new task requests when the CPU and memory usage exceed the corresponding thresholds.

7. The shared task queue management system in a Kubernetes environment according to claim 1, characterized in that, The task processing unit is also used to periodically remove failed or expired tasks from each pod node.

8. A shared task queue management method in a Kubernetes environment, applied to the shared task queue management system in a Kubernetes environment as described in any one of claims 1-7, characterized in that, include: Get the task request submitted by the user; Tasks are generated based on task requests and then published to the corresponding Kafka topic units. When a new task is published, each Kafka topic unit inserts the task into the queue to be assigned based on its priority attribute value. Each pod node in each subscription group sequentially retrieves task information from the waiting queue of the corresponding Kafka topic unit at regular intervals, and determines whether it can receive tasks based on its load. If so, the pod node actively retrieves a task that matches its load from the waiting queue of the Kafka topic unit, moves the task to the assigned queue, and determines whether it has received task completion information after a predetermined time. If yes, the task on the Pod node is determined to have been executed successfully, and the task is moved to the completed queue; if no, the task on the Pod node is determined to have failed, and the task is moved from the assigned queue to the pending queue.

Images