Task management method and device, computer equipment and readable storage medium

Abstract

The invention relates to a task management method and device, computer equipment and a readable storage medium. The method comprises the following steps: adding each to-be-executed task to a target task queue according to task information of each to-be-executed task and queue information of the target task queue; determining a task score of each to-be-executed task according to the execution waiting time required by each to-be-executed task; and under the condition that the thread pool meets the execution condition, according to the task score of each to-be-executed task, allocating each to-be-executed task to a corresponding thread in the thread pool for execution. By adopting the method, the enqueue process of the to-be-executed task can be completed based on the task information and the queue information, the execution waiting time of the to-be-executed task is considered, the corresponding task score is determined, accurate management of the to-be-executed task is facilitated, the task management effect is improved, and the task management efficiency is improved according to the task score. And the to-be-executed task is allocated to the corresponding thread for execution, so that a distributed computing service can be provided, and the task execution efficiency and reliability are improved.

Description

Technical Field

[0001] The present application relates to the technical field of pipeline task scheduling, and in particular to a task management method, apparatus, computer equipment and readable storage medium. Background Art

[0002] The CI / CD (Continuous Integration / Continuous Delivery) platform is a tool that simplifies the code integration and software construction development process. It can frequently build, test, and release software in an automated way, which is conducive to improving the speed and quality of software delivery. In the CI / CD platform, it is necessary to package and build tasks for applications based on the pipeline engine, and deploy tasks in combination with resources to achieve safe scheduling of tasks.

[0003] However, the current scheduling method for pipeline tasks in CI / CD platforms is relatively simple, which makes it difficult to adapt to application scenarios with increasing traffic and cannot support and ensure efficient management of tasks. Summary of the invention

[0004] Based on this, it is necessary to provide a task management method, apparatus, computer equipment and readable storage medium that can ensure high performance and stability of task management in response to the above technical problems.

[0005] In a first aspect, the present application provides a task management method, comprising:

[0006] According to the task information of each task to be executed and the queue information of the target task queue, each task to be executed is added to the target task queue;

[0007] Determine the task score of each task to be executed according to the waiting execution time required for each task to be executed;

[0008] When the thread pool meets the execution conditions, each task to be executed is assigned to the corresponding thread in the thread pool for execution according to the task score of each task to be executed.

[0009] In one embodiment, adding each to-be-executed task to the target task queue according to the task information of each to-be-executed task and the queue information of the target task queue includes:

[0010] For each task to be executed, a hash operation is performed on the task information to be executed to obtain a hash value corresponding to the task to be executed;

[0011] Find the hash value corresponding to the task to be executed from the queue information of the target task queue;

[0012] If the hash value corresponding to the task to be executed is found, it is determined that the task to be executed is not waiting for execution for the first time, and the task to be executed is directly added to the target task queue.

[0013] If the hash value corresponding to the task to be executed is not found, it is determined that the task to be executed is waiting for execution for the first time, the task information of the task to be executed is initialized, and the Redis transaction is started to add the task to be executed to the target task queue.

[0014] In one embodiment, determining the task score of each task to be executed according to the waiting execution time required for each task to be executed includes:

[0015] For each task to be executed, the task score of the task to be executed is determined according to the waiting execution time and the queue entry time of the task to be executed; wherein the task score is directly proportional to the waiting execution time.

[0016] In one embodiment, when the thread pool meets the execution condition, each task to be executed is assigned to a corresponding thread in the thread pool for execution according to the task score of each task to be executed, including:

[0017] When the thread pool meets the execution conditions, the task with the smallest task score among all tasks to be executed is taken as the target task;

[0018] Assign the target task to the corresponding thread in the thread pool for execution, and return the operation of taking the task with the smallest task score among the tasks to be executed as the target task when the thread pool meets the execution conditions.

[0019] In one embodiment, assigning the target task to a corresponding thread in a thread pool for execution includes:

[0020] Encapsulate the target task into a target event;

[0021] According to the event type of the target event, determine the target event handler from the candidate event handlers corresponding to the thread pool;

[0022] The target event handler calls the corresponding thread in the thread pool to execute the target event.

[0023] In one embodiment, the method further comprises:

[0024] When it is determined based on the task information of the target task that a new task to be executed is generated after the target task is completed, the new task to be executed is added to the target task queue based on the task information of the new task to be executed and the queue information of the target task queue; wherein the target task and the new task to be executed belong to the same pipeline.

[0025] In one embodiment, the task information of the task to be executed includes the number of times the task is executed, and the method further includes:

[0026] For each task to be executed in the failed task queue, compare the task execution count of the task to be executed with the preset allowed execution count;

[0027] If the number of task executions is less than the allowed number of executions, the task to be executed is transferred from the failed task queue to the target task queue according to the task information of the task to be executed and the queue information of the target task queue;

[0028] If the task execution count is not less than the allowed execution count, the pending task is deleted from the failed task queue.

[0029] In a second aspect, the present application also provides a task management device, including:

[0030] A task adding module, used for adding each to-be-executed task to the target task queue according to the task information of each to-be-executed task and the queue information of the target task queue;

[0031] A score determination module, used to determine the task score of each task to be executed according to the waiting execution time required for each task to be executed;

[0032] The task execution module is used to assign each task to be executed to the corresponding thread in the thread pool for execution according to the task score of each task to be executed when the thread pool meets the execution conditions.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of the first aspect when executing the computer program.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method of the first aspect when the computer program is executed by a processor.

[0035] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the method of the first aspect when executed by a processor.

[0036] The above-mentioned task management method, device, computer equipment and readable storage medium add each task to be executed to the target task queue according to the task information of each task to be executed and the queue information of the target task queue; determine the task score of each task to be executed according to the waiting execution time required for each task to be executed; when the thread pool meets the execution conditions, according to the task score of each task to be executed, assign each task to be executed to the corresponding thread in the thread pool for execution. This application completes the queue entry process of the task to be executed based on the task information and queue information, and determines the corresponding task score taking into account the waiting execution time of the task to be executed, which is conducive to the precise management of the tasks to be executed and improves the task management effect, and then assigns it to the corresponding thread for execution according to the task score of the task to be executed, which can provide distributed computing services and improve the efficiency and reliability of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 An application environment diagram of a task management method in an embodiment;

[0039] Figure 2 is a flowchart of a task management method in one embodiment;

[0040] Figure 3 A schematic diagram of a process of adding a task to be executed to a target task queue in one embodiment;

[0041] Figure 4 A schematic diagram of a flow chart of allocating tasks to be executed to corresponding threads for execution in one embodiment;

[0042] Figure 5 A schematic diagram of a process for managing pending tasks that have failed in execution in one embodiment;

[0043] Figure 6 is a flowchart of a task management method in another embodiment;

[0044] Figure 7 is a structural block diagram of a task management device in one embodiment;

[0045] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The task management method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can add each task to be executed to the target task queue according to the task information of each task to be executed and the queue information of the target task queue; determine the task score of each task to be executed according to the waiting execution time required for each task to be executed; when the thread pool meets the execution conditions, according to the task score of each task to be executed, each task to be executed is assigned to the corresponding thread in the thread pool for execution. Among them, the server 104 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server that provides cloud computing services.

[0048] In an exemplary embodiment, Figure 2 As shown, a task management method is provided, which is applied to Figure 1 The following is an example of a server in the example, including:

[0049] S201, adding each to-be-executed task to a target task queue according to the task information of each to-be-executed task and the queue information of the target task queue.

[0050] Among them, the tasks to be executed can be the bottom-level tasks scheduled by the pipeline engine in the CI / CD platform. Each task to be executed can be executed in series or in parallel, or nested. The pipeline engine uses the Zset ordered set as a task queue to store the tasks to be executed, and uses the Redis Hash table to store the relevant information of the tasks to be executed, such as the hash value used for indexing.

[0051] When enqueuing a task to be executed, the corresponding enqueuing method can be determined based on the task information of the task to be executed and the queue information of the target task queue that it is expected to enter, as well as the appropriate position of the task to be executed in the target task queue, so that the task to be executed can be added to the appropriate position in the target task queue using the corresponding enqueuing method.

[0052] Among them, the task information of the task to be executed may include information such as task name, task type and task status, and the queue information of the target task queue may include information such as queue name, queue type and number of tasks stored in the queue, which is not limited in this embodiment.

[0053] Optionally, for each task to be executed, the task information of the task to be executed is matched with the queue information of the target task queue, and if the match is successful, the task to be executed is added to the target task queue. If the match is unsuccessful, the task to be executed is processed accordingly and then added to the target task queue.

[0054] It is understandable that when adding tasks to be executed to the target task queue, not only do the tasks to be executed need to be stored in the target task queue, that is, the corresponding Zset ordered set, but also the relevant information of the tasks to be executed needs to be stored in the list associated with the target task queue, such as storing the hash value used for indexing in the Hash table associated with the target task queue.

[0055] S202: Determine the task score of each task to be executed according to the waiting execution time required for each task to be executed.

[0056] Due to the task attributes of the task to be executed, or the requirements of other tasks or devices associated with the task to be executed, the task to be executed scheduled by the pipeline engine needs to wait for a period of time before being executed. This period of time is the waiting execution time required for the task to be executed. Generally, when a task to be executed is created, the waiting execution time corresponding to the task to be executed can be determined, and the waiting execution time can be a preset value.

[0057] Optionally, for each task to be executed, the waiting execution time required for the task to be executed is substituted into the corresponding delay time scoring algorithm to obtain the task score of the task to be executed. The task score of the task to be executed can represent the urgency of scheduling the task to be executed to a certain extent.

[0058] It can be understood that when adding a task to be executed to a target task queue, the waiting execution time required for the task to be executed can be stored in a list associated with the target task queue as relevant information of the task to be executed, or after adding the task to be executed to the target task queue, the task score of the task to be executed can be stored in a list associated with the target task queue as relevant information of the task to be executed.

[0059] S203, when the thread pool meets the execution condition, each task to be executed is assigned to a corresponding thread in the thread pool for execution according to the task score of each task to be executed.

[0060] Among them, the thread pool meets the execution condition, which may mean that there are schedulable free threads in the thread pool, which are sufficient for executing the tasks to be executed. Exemplarily, it is determined whether there are schedulable free threads in the thread pool; if so, the tasks to be executed are assigned to the corresponding free threads one by one according to the task scores of the tasks to be executed, until the tasks to be executed in the target task queue are assigned, or there are no schedulable free threads in the thread pool; if not, wait until there are schedulable free threads in the thread pool, and then execute the above steps.

[0061] Specifically, according to the task scores of each task to be executed, the order of assigning each task to be executed to the corresponding thread in the thread pool is determined, and then each task to be executed is assigned to the thread one by one according to the corresponding order. The lower the task score, the shorter the waiting time required for the task to be executed, and accordingly, the task to be executed is scheduled earlier.

[0062] In the above-mentioned task management method, each task to be executed is added to the target task queue according to the task information of each task to be executed and the queue information of the target task queue; the task score of each task to be executed is determined according to the waiting execution time required for each task to be executed; when the thread pool meets the execution conditions, each task to be executed is assigned to the corresponding thread in the thread pool for execution according to the task score of each task to be executed. This embodiment completes the queue entry process of the task to be executed based on the task information and queue information, and determines the corresponding task score taking into account the waiting execution time of the task to be executed, which is conducive to the precise management of the tasks to be executed and improves the task management effect, and then assigns the task to be executed to the corresponding thread for execution according to the task score of the task to be executed, which can provide distributed computing services and improve the efficiency and reliability of task execution.

[0063] In an exemplary embodiment, Figure 3 As shown, the above S201 includes:

[0064] S301, for each task to be executed, performing a hash operation on the task information of the task to be executed to obtain a hash value corresponding to the task to be executed.

[0065] The corresponding hash algorithm is used to calculate the task information of the task to be executed, and the corresponding hash value is obtained, thereby obtaining the index information of the task to be executed in the target task queue.

[0066] The hash algorithm may be an MD4 algorithm (Message Digest 4), an MD5 algorithm, etc., which is not limited in this embodiment.

[0067] S302, determining whether a hash value corresponding to the task to be executed is found in the queue information of the target task queue.

[0068] If yes, execute S303; if no, execute S304.

[0069] Optionally, the Zrank function of Redis is called to find the hash value corresponding to the task to be executed from the queue information of the target task queue. Exemplarily, the Zrank function of Redis is called to find the hash value corresponding to the task to be executed from the queue information of the target task queue. The parameter passed may include the team name of the target task queue.

[0070] Optionally, call the Zrank function of Redis to calculate the index value of the hash value corresponding to the task to be executed, and then search for the index value in the queue information of the target task queue.

[0071] S303: Determine that the task to be executed is not the first time to be executed, and directly add the task to be executed to the target task queue.

[0072] If the hash value corresponding to the task to be executed is found, it is determined that the task to be executed is not waiting for execution for the first time, that is, it is not added to the target task queue for the first time, and the task to be executed can be directly added to the target task queue.

[0073] Specifically, the Zadd function can be called to update the hash value and task score of the task to be executed to the list associated with the target task queue.

[0074] S304, determining that the task to be executed is waiting for execution for the first time, initializing the task information of the task to be executed, and starting a Redis transaction to add the task to be executed to a target task queue.

[0075] If the hash value corresponding to the task to be executed is not found, it is determined that the task to be executed is waiting for execution for the first time, that is, it is added to the target task queue for the first time. It needs to be initialized first, and then copied, and then the Redis transaction is opened to call the Hset function to store the relevant information of the task to be executed in the Hash table associated with the target task queue. Further, the Zadd function is called to update the hash value and task score of the task to be executed to the list associated with the target task queue.

[0076] In this embodiment, the hash value corresponding to the task to be executed is used to determine whether the task to be executed is added to the target task queue for the first time, and then different enqueuing methods are adopted in different situations. It has a wider application range, can improve the reliability of task enqueuing, and is conducive to the subsequent accurate management of the tasks to be executed.

[0077] In an exemplary embodiment, the above S202 includes:

[0078] For each task to be executed, the task score of the task to be executed is determined according to the waiting execution time and the queue entry time of the task to be executed; wherein the task score is directly proportional to the waiting execution time.

[0079] Specifically, the Score algorithm is called, the waiting execution time and the queue time of the task to be executed are substituted into the function, and the score of the task to be executed is calculated. The shorter the waiting execution time, the lower the task score, and vice versa, the longer the waiting execution time, the higher the task score.

[0080] Optionally, the predicted execution time of the task to be executed is determined based on the waiting execution time and the queue entry time, and then the corresponding task score is determined based on the predicted execution time.

[0081] In this embodiment, the task score of the task to be executed is calculated based on the waiting execution time and the queue entry time, so that the shorter the waiting execution time, the lower the task score, and the earlier the task to be executed is scheduled. The tasks to be executed can be accurately managed in a digitized form to ensure the sequential execution of the entire pipeline.

[0082] In an exemplary embodiment, Figure 4 As shown, the above S203 includes:

[0083] S401, when the thread pool meets the execution condition, the task to be executed with the smallest task score among the tasks to be executed is taken as the target task.

[0084] Among them, the thread pool meets the execution condition, which may mean that there are schedulable free threads in the thread pool. First, determine whether the thread pool meets the execution condition. If it does, then further determine the pending task with the smallest task score among the pending tasks in the target task queue as the target task that needs to be assigned currently. The task with the smallest task score indicates that the scheduling urgency is the highest and needs to be assigned first for processing.

[0085] Optionally, different tasks to be executed may require different thread resources. The task to be executed with the smallest task score in the target task queue may be used as the target task, and then it is determined whether the thread pool meets the execution conditions of the target task, that is, whether there is an idle thread matching the schedulable target task in the thread pool.

[0086] S402, assigning the target task to the corresponding thread in the thread pool for execution, and returning to execute the operation of taking the task with the smallest task score among the tasks to be executed as the target task when the thread pool meets the execution condition.

[0087] After the taken target task is assigned to the corresponding thread in the thread pool, if the thread pool still meets the execution conditions, the task with the smallest task score among the remaining tasks in the target task queue can be used as the new target task and assigned to the corresponding thread for execution.

[0088] Optionally, when the thread pool no longer meets the execution condition, wait until the thread pool meets the execution condition, then continue to execute the task with the smallest task score among the tasks to be executed as the target task, and assign the target task to the corresponding thread for execution.

[0089] Optionally, when all the tasks to be executed in the target task queue are allocated, the above allocation process is ended until there is a new task to be executed that is added to the target task queue.

[0090] It should be noted that the tasks to be executed can be executed in parallel, so there is no need to wait for the previous target task to be completed before returning to assign a new target task.

[0091] Exemplarily, the ZrangeByScore function is called to obtain the to-be-executed task with the smallest task score in the target task queue, that is, the hash value of the target task, and then the SetNx function is called to obtain the distribution lock of the hash value of the target task. Further, the Hget function is called to obtain the task information of the target task according to the hash value of the target task, so as to execute the Zrem function to remove the hash value of the target task from the Hash table associated with the target task queue, thereby completing the dequeue process of the target task.

[0092] Optionally, for the target task to be dequeued, the task information of the target task can be updated by calling the Hset function, such as adding 1 to the task execution count of the target task.

[0093] Optionally, encapsulate the target task into a target event; determine the target event handler from the candidate event handlers corresponding to the thread pool according to the event type of the target event; and call the corresponding thread in the thread pool through the target event handler to execute the target event.

[0094] Task events in the pipeline can include multiple types, such as StartPipeline, CancelStage, and SuccessTask. Each thread in the thread pool asynchronously executes task events and processes related internal logic to execute each task to be executed.

[0095] Specifically, the target task is encapsulated into a corresponding task event, namely, a target event. After the listener of the pipeline engine listens to the target event, it can match a suitable event handler, namely, a target event handler, for it according to the event type of the target event. Furthermore, the thread pool is used to asynchronously execute the target event to execute the target task.

[0096] Among them, each candidate event processor not only needs to process its own business logic, but also needs to combine external state influences to form a complex state machine transition definition.

[0097] In this embodiment, each task to be executed is assigned to the corresponding thread one by one according to the task score of each task to be executed, so that tasks that need to be executed as soon as possible are scheduled first, and tasks that need to wait for a while before execution are scheduled later, which adapts to the execution requirements of different tasks and optimizes the task scheduling process by taking into account the resource situation of the thread pool.

[0098] In an exemplary embodiment, the task management method further includes:

[0099] When it is determined based on the task information of the target task that a new task to be executed is generated after the target task is completed, the new task to be executed is added to the target task queue based on the task information of the new task to be executed and the queue information of the target task queue; wherein the target task and the new task to be executed belong to the same pipeline.

[0100] Specifically, after the current target task is completed, the entire pipeline may be completed, or the pipeline may not be completed and still need to wait for the next task to be executed. Exemplarily, whether the pipeline is completed can be set as the task information of the target task when creating the target task, and then the target task is added to the target task queue and stored in the list associated with the target task queue.

[0101] Furthermore, when executing the target task, it can be determined based on the task information of the target task whether the pipeline to which the target task belongs has been completed after the target task is completed, that is, whether a new task to be executed is generated after the target task is completed; if so, the corresponding task enqueuing algorithm is executed, and the newly generated task to be executed is added to the target task queue, waiting for the pipeline engine to schedule execution, and the enqueuing method is consistent with other tasks to be executed; if not, the status information of the pipeline to which the target task belongs is modified to end the pipeline.

[0102] In this embodiment, different processes after the task is completed can be determined based on the task information, thereby ensuring the stable execution of each pipeline, providing high-performance, distributed and scalable computing node services, and facilitating deployment and maintenance.

[0103] In an exemplary embodiment, the task information of the task to be executed includes the number of times the task is executed, such as Figure 5 As shown, the above task management method further includes:

[0104] S501 : for each to-be-executed task in the failed task queue, determining whether the task execution times of the to-be-executed task are less than a preset allowed execution times.

[0105] If yes, execute S502; if no, execute S503.

[0106] The failed task queue is another task queue different from the target task queue, which is used to store pending tasks that failed to execute. For example, taking the pending tasks as target tasks and assigning them to the corresponding nodes of the node pool terminal for execution may result in execution failures, such as task failures caused by network interruptions. These pending tasks that failed to execute can be stored in the failed task queue.

[0107] Further, according to the task execution times of the tasks to be executed in the failed task queue, it is determined whether it needs to be executed again. Specifically, for each task to be executed, the task execution times of the task to be executed and the allowed execution times are compared. Among them, the task execution times is one of the task information of the task to be executed. Each time the task to be executed is executed, the task execution times are increased by 1, and the allowed execution times are the corresponding preset values.

[0108] Optionally, by calling the ZrangeByScore function, a set of hash values ​​of all failed tasks to be executed is taken out from the failed task queue, and then the set is traversed to obtain the task information corresponding to the hash value by calling the Hget function to obtain the task execution count of each task to be executed.

[0109] S502 : Transferring the to-be-executed task from the failed task queue to the target task queue according to the task information of the to-be-executed task and the queue information of the target task queue.

[0110] When the number of task executions is less than the allowed number of executions, the task to be executed is transferred from the failed task queue to the target task queue to reallocate the task to be executed to the corresponding thread, and its queue entry method is consistent with other tasks to be executed.

[0111] Exemplarily, a Redis transaction is started, the task score of the task to be executed is determined by calling the Score algorithm, and the hash value and task score of the task to be executed are added to a list associated with the target task queue.

[0112] S503, deleting the to-be-executed task from the failed task queue.

[0113] When the number of task executions is not less than the allowed number of executions, the task to be executed is directly deleted from the failed task queue, and the task information of the task to be executed is deleted simultaneously.

[0114] Optionally, a dead task event corresponding to the task to be executed is sent to manage the linkage status of the pipeline.

[0115] In this embodiment, a retry algorithm for failed tasks is provided, that is, when it is determined that the task to be executed can be attempted again based on the number of task executions, it is re-added to the target task queue, thereby avoiding task execution failure or failure to respond normally caused by problems such as network interruption, ensuring that each task is safely executed within the allowed number of executions, thereby improving the effect of task management.

[0116] In an exemplary embodiment, Figure 6 As shown, a task management method is provided, including:

[0117] S601, performing a hash operation on the task information of each task to be executed to obtain a hash value corresponding to each task to be executed;

[0118] S602, determining whether the hash value corresponding to each task to be executed is found in the queue information of the target task queue;

[0119] If yes, execute S603; if no, execute S604.

[0120] S603, determining that the task to be executed is not the first time to be executed, and directly adding the task to be executed to the target task queue.

[0121] S604, determining that the task to be executed is waiting for execution for the first time, initializing the task information of the task to be executed, and starting a Redis transaction to add the task to be executed to a target task queue.

[0122] S605, determining the task score of each task to be executed according to the waiting execution time and the queue entry time of each task to be executed.

[0123] Among them, the task score is directly proportional to the waiting execution time.

[0124] S606, taking the task with the smallest task score among the tasks to be executed as the target task;

[0125] S607: Allocate the target task to the corresponding thread in the thread pool for execution.

[0126] After S607 is executed, the process returns to step S606. S607 specifically includes encapsulating the target task into a target event; determining the target event handler from the candidate event handlers corresponding to the thread pool according to the event type of the target event; and calling the corresponding thread in the thread pool through the target event handler to execute the target event.

[0127] S608: Determine whether a new task to be executed is generated after the target task is completed according to the task information of the target task.

[0128] If yes, the process returns to step S601; if no, the process proceeds to step S609.

[0129] S609, modify the status information of the pipeline to which the target task belongs, and end the pipeline.

[0130] Optionally, for each task to be executed in the failed task queue, compare the task execution count of the task to be executed with the preset allowed execution count; if the task execution count is less than the allowed execution count, transfer the task to be executed from the failed task queue to the target task queue based on the task information of the task to be executed and the queue information of the target task queue; if the task execution count is not less than the allowed execution count, delete the task to be executed from the failed task queue.

[0131] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0132] Based on the same inventive concept, the embodiment of the present application also provides a task management device for implementing the task management method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more task management device embodiments provided below can refer to the limitations on the task management method above, and will not be repeated here.

[0133] In an exemplary embodiment, Figure 7 As shown, a task management device is provided, comprising:

[0134] The task adding module 10 is used to add each to-be-executed task to the target task queue according to the task information of each to-be-executed task and the queue information of the target task queue;

[0135] The score determination module 20 is used to determine the task score of each task to be executed according to the waiting execution time required for each task to be executed;

[0136] The task execution module 30 is used to assign each task to be executed to a corresponding thread in the thread pool for execution according to the task score of each task to be executed when the thread pool meets the execution condition.

[0137] In an exemplary embodiment, the task adding module 10 includes:

[0138] A hash value determination unit, for performing a hash operation on the task information of each task to be executed, to obtain a hash value corresponding to the task to be executed;

[0139] A hash value search unit, used to search the hash value corresponding to the task to be executed from the queue information of the target task queue;

[0140] The task adding unit is used to determine that the task to be executed is not the first time to be executed when the hash value corresponding to the task to be executed is found, and directly add the task to be executed to the target task queue; and when the hash value corresponding to the task to be executed is not found, determine that the task to be executed is the first time to be executed, initialize the task information of the task to be executed, start the Redis transaction, and add the task to be executed to the target task queue.

[0141] In an exemplary embodiment, the score determination module 20 is specifically used to determine the task score of each task to be executed according to the waiting execution time and the queue entry time of the task to be executed; wherein the task score is directly proportional to the waiting execution time.

[0142] In an exemplary embodiment, the task execution module 30 includes:

[0143] The task selection unit is used to select the task to be executed with the smallest task score among the tasks to be executed as the target task when the thread pool meets the execution condition;

[0144] The task execution unit is used to assign the target task to the corresponding thread in the thread pool for execution, and return to execute the operation of taking the task with the smallest task score among the tasks to be executed as the target task when the thread pool meets the execution conditions.

[0145] In an exemplary embodiment, the task execution unit includes:

[0146] An event generation subunit, used to encapsulate the target task into a target event;

[0147] The processing area determination subunit is used to determine the target event processor from the candidate event processors corresponding to the thread pool according to the event type of the target event;

[0148] The event execution subunit is used to call the corresponding thread in the thread pool through the target event processor to execute the target event.

[0149] In an exemplary embodiment, the task management device further includes:

[0150] The task adding module is used to add the new task to be executed to the target task queue according to the task information of the target task and the queue information of the target task queue when it is determined that a new task to be executed is generated after the target task is completed according to the task information of the target task; wherein the target task and the new task to be executed belong to the same pipeline.

[0151] In an exemplary embodiment, the task information of the task to be executed includes the number of times the task is executed, and the task management device further includes:

[0152] The number determination module is used to compare the task execution number of the task to be executed with the preset allowed execution number for each task to be executed in the failed task queue;

[0153] The task update module is used to transfer the task to be executed from the failed task queue to the target task queue according to the task information of the task to be executed and the queue information of the target task queue when the number of task execution times is less than the allowed number of execution times, and to delete the task to be executed from the failed task queue when the number of task execution times is not less than the allowed number of execution times.

[0154] Each module in the above task management device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0155] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as task information. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a task management method is implemented.

[0156] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned task management method when executing the computer program.

[0158] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned task management method are implemented.

[0159] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned task management method when executed by a processor.

[0160] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0162] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A task management method, characterized in that: The method comprises: According to the task information of each task to be executed and the queue information of the target task queue, each task to be executed is added to the target task queue; Determine the task score of each task to be executed according to the waiting execution time required for each task to be executed; When the thread pool meets the execution conditions, each task to be executed is assigned to a corresponding thread in the thread pool for execution according to the task score of each task to be executed.

2. The method according to claim 1, characterized in that The adding each to-be-executed task to the target task queue according to the task information of each to-be-executed task and the queue information of the target task queue comprises: For each task to be executed, a hash operation is performed on the task information of the task to be executed to obtain a hash value corresponding to the task to be executed; Find the hash value corresponding to the task to be executed from the queue information of the target task queue; If the hash value corresponding to the task to be executed is found, it is determined that the task to be executed is not waiting to be executed for the first time, and the task to be executed is directly added to the target task queue; If the hash value corresponding to the task to be executed is not found, it is determined that the task to be executed is waiting for execution for the first time, the task information of the task to be executed is initialized, and a Redis transaction is started to add the task to be executed to the target task queue.

3. The method according to claim 1, characterized in that: Determining the task score of each task to be executed according to the waiting execution time required for each task to be executed includes: For each task to be executed, a task score of the task to be executed is determined according to the waiting execution time and the queue entry time of the task to be executed; wherein the task score is in direct proportion to the waiting execution time.

4. The method according to claim 1, characterized in that When the thread pool meets the execution condition, each task to be executed is assigned to a corresponding thread in the thread pool for execution according to the task score of each task to be executed, including: When the thread pool meets the execution conditions, the task with the smallest task score among all tasks to be executed is taken as the target task; The target task is assigned to a corresponding thread in the thread pool for execution, and when the thread pool meets the execution condition, the task to be executed with the smallest task score among the tasks to be executed is returned as the target task.

5. The method according to claim 4, characterized in that The step of allocating the target task to a corresponding thread in the thread pool for execution includes: Encapsulating the target task into a target event; Determine a target event handler from candidate event handlers corresponding to the thread pool according to the event type of the target event; The target event handler calls a corresponding thread in the thread pool to execute the target event.

6. The method according to claim 4, characterized in that The method further comprises: When it is determined based on the task information of the target task that a new task to be executed is generated after the target task is executed, the new task to be executed is added to the target task queue based on the task information of the new task to be executed and the queue information of the target task queue; wherein the target task and the new task to be executed belong to the same pipeline.

7. The method according to claim 1, characterized in that The task information of the task to be executed includes the number of times the task is executed, and the method further includes: For each to-be-executed task in the failed task queue, comparing the task execution count of the to-be-executed task with a preset allowed execution count; If the task execution times are less than the allowed execution times, transferring the task to be executed from the failed task queue to the target task queue according to the task information of the task to be executed and the queue information of the target task queue; If the task execution times are not less than the allowed execution times, the to-be-executed task is deleted from the failed task queue.

8. A task management device, characterized in that: The device comprises: A task adding module, used for adding each to-be-executed task to the target task queue according to the task information of each to-be-executed task and the queue information of the target task queue; A score determination module, used to determine the task score of each task to be executed according to the waiting execution time required for each task to be executed; The task execution module is used to assign each task to be executed to a corresponding thread in the thread pool for execution according to the task score of each task to be executed when the thread pool meets the execution conditions.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images