High-concurrency Multitask Scheduling and Management Method Based on Real-time Task Queue

Abstract

The present invention discloses a high-concurrency multi-task scheduling management method based on a real-time task queue. The subtasks requested by the task are placed in a first task queue of a queue buffer module; the parent task is placed in a second task queue; the parent task entering the second task queue is judged. If the parent task has a corresponding associated subtask, the parent task is set to a pause state and waits for the processing results of its corresponding associated subtasks; when all subtasks under the parent task corresponding to the subtask have been processed, the parent task corresponding to the subtask in the second task queue is set to an active state and the parent task is sent to a fourth task queue. If the parent task does not have any corresponding associated subtask, the parent task directly enters the fourth task queue; the processing result of the parent task entering the fourth task queue is judged. If the judgment result is that the processing is successful, the parent task is marked as a processing success state.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a high-concurrency multi-task scheduling management method based on a real-time task queue. Background Art

[0002] With the advancement of computer technology, the amount of data generated by humans continues to increase. This has led to various application systems experiencing the impact of high-concurrency data processing processes on system performance during operation. High concurrency specifically refers to the fact that application systems need to support a large number of data processing requests at the same time. High concurrency often leads to blocked system processes, delayed request processing, reduced system performance, and even outright system failure in severe cases. Under high concurrency, the large number of processing requests places tremendous pressure on the system. The most fundamental way to alleviate high concurrency is to fully optimize the data processing process to enable it to handle more requests per unit time. To cope with high concurrency requests, improving hardware conditions is often the solution. However, simply piling up hardware cannot fully meet the performance requirements of high concurrency. The application system's architectural design must also be fully optimized to achieve optimal performance within the current hardware conditions.

[0003] At present, the task scheduling frameworks in the existing technology include: a task scheduling framework based on Oozie technology, but when using the task scheduling framework based on Oozie technology, it is necessary to write a workflow.XML file, but the writing of this file is very complicated and not easy for secondary development. A task scheduling framework based on java and jvm technology, but this task scheduling framework has high memory consumption, high hardware requirements, and is oriented towards hadoop ecosystem components, and has poor scalability. A task scheduling framework based on azkaban technology, the design goal of this task scheduling framework is also only aimed at solving hadoop job dependencies. A task scheduling framework based on airflow technology, which is a distributed task scheduling framework that can handle workflows with superior and subordinate dependencies; but it requires manual python development of DAG, and pausing and resuming can only delete the workflow and then rerun it, which will cause task loss and chain exceptions.

[0004] It can be seen that the prior art requires a faster and lighter task scheduling framework, and a task scheduling management method based on the task scheduling framework. Summary of the Invention

[0005] The technical problems to be solved by this invention include: improving the resilience of the task scheduling framework, enabling task processing requests to automatically recover from process crashes; making the task scheduling framework lightweight and high-speed, thereby increasing task processing throughput; and implementing task status monitoring and task recycling mechanisms by constructing parent-child tasks.

[0006] In order to solve the above technical problems, the present invention provides a high-concurrency multi-task scheduling management method based on a real-time task queue, the method comprising:

[0007] When a task request is sent from the task generation module to the queue buffer module, the subtask of the task request is placed in the first task queue of the queue buffer module;

[0008] The parent task of the task request is placed in the second task queue of the queue buffer module;

[0009] The parent task entering the second task queue is judged. If the parent task has a corresponding associated child task, the parent task is set to a pause state and waits for the processing result of the corresponding associated child task;

[0010] The subtasks in the first task queue are set to active state in sequence, and then sent to the third task queue for processing by the task processing module;

[0011] The processing result of the subtask entering the third task queue is judged to determine whether the processing is successful. If the judgment result is successful, the subtask is marked as being in a successful processing state.

[0012] If the processing result of the subtask is processing failure, the processing times of the subtask are further judged. If the processing times of the subtask do not exceed the processing times threshold, the subtask is processed again; if the processing times of the subtask have exceeded the processing times threshold, the subtask is marked as processing failure;

[0013] When the processing of the subtask is completed, that is, the processing result of the subtask is a processing success state or a processing failure state; further query the processing status of all subtasks under the parent task corresponding to the subtask;

[0014] If all subtasks under the parent task corresponding to the subtask have not been fully processed, the subtask is set to a delayed state and is sent back to the beginning of the first task queue;

[0015] When all subtasks under the parent task corresponding to the subtask have been processed, the parent task corresponding to the subtask in the second task queue will be set to active state and sent to the fourth task queue for processing by the task processing module. If the parent task does not have any corresponding subtasks, the parent task will be directly sent to the fourth task queue;

[0016] The processing result of the parent task entering the fourth task queue is judged to determine whether the processing is successful. If the judgment result is successful, the subtask is marked as a successful processing state;

[0017] If the processing result of the parent task is a processing failure, the number of processing times of the parent task is further judged. If the number of processing times of the parent task does not exceed the processing times threshold, the parent task is processed again; if the number of processing times of the parent task has exceeded the processing times threshold, the parent task is marked as a processing failure state;

[0018] When the processing of the parent task is completed, that is, the processing result of the parent task is a success or failure state, the task request is processed;

[0019] In one embodiment, the tasks in the third task queue and the fourth task queue that are in an activated state are assigned to multiple task processing processes in parallel by the task processing module for data processing.

[0020] In one embodiment, a task queue is generated by an asynchronous strategy to separate the task generation module and the task processing module.

[0021] In one embodiment, the first task queue, the second task queue, the third task queue and the fourth task queue are real-time high-speed message queues built based on Redis technology.

[0022] In one embodiment, the first task queue, the second task queue, the third task queue, and the fourth task queue each include a two-layer structure, including a Vector Cache layer and a Row Cache layer; the Vector Cache layer mainly stores task description information in task requests; and the Row Cache layer is used for task processing data.

[0023] In one embodiment, the high-concurrency multi-task scheduling management method further includes a release operation for task requests, wherein the purpose of the release operation is to clear subtasks or parent tasks in each task queue.

[0024] In one embodiment, the high-concurrency multi-task scheduling management method further includes a task blocking processing operation.

[0025] In one embodiment, the high-concurrency multi-task scheduling management method further includes a pause queue operation.

[0026] In one embodiment, the high-concurrency multi-task scheduling management method also includes the queue buffer module moving the subtasks in the delayed state to the first task queue, and automatically checking the blocked tasks, that is, the subtasks in the activated state in the second task queue, but whose processing process has crashed or stopped processing.

[0027] The core invention and innovation points of the present invention are:

[0028] 1. The high-concurrency multi-task scheduling management method of the present invention has high availability. The present invention can set the concurrency of the task processing process. By setting multiple concurrent task processing processes, single point failure is avoided. It also avoids setting only one task processing process and causing task blocking of the entire system due to the crash of this task processing process.

[0029] 2. The present invention provides an independent task processing process based on a sandbox isolation mechanism. Tasks run on separate processes, isolated from other processing processes. Since tasks run in different process environments, task blocking will not occur.

[0030] 3. In the present invention, when a task is not scheduled for execution, it can ensure that the process context is not polluted and various resource deadlocks will not occur.

[0031] 4. The present invention is provided with an error retry mechanism, when the task execution fails, the task status can be changed to a delayed state.

[0032] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0034] Figure 1 Schematic diagram of the high-concurrency multi-task scheduling management system according to the present invention;

[0035] Figure 2 This is a flowchart of the high-concurrency multi-task scheduling management method according to the present invention;

[0036] Figure 3 2. It is a schematic diagram of the structure of an electronic device according to the high-concurrency multi-task scheduling management method of the present invention;

[0037] Figure 4 It is a structural diagram of a computer system according to the high-concurrency multi-task scheduling management method of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices.

[0040] Example 1

[0041] like Figure 1 As shown, the high-concurrency multi-task scheduling and management system based on a real-time task queue of the present invention includes a task generation module, i.e., a task producer; a queue buffer module, i.e., a buffer; and a task processing module, i.e., a task consumer. In this embodiment, the task producer generates pending task requests, which are input into the queue buffer module. The task consumer extracts the pending task requests from the queue buffer module and processes them accordingly.

[0042] like Figure 2 As shown, the high-concurrency multi-task scheduling management method based on real-time task queue of the present invention includes the following steps:

[0043] S100: When a task request is sent from a task generating module to a queue buffer module, a subtask of the task request is placed into a first task queue of the queue buffer module.

[0044] S101: The parent task of the task request is placed into the second task queue of the queue buffer module.

[0045] S102: The parent task entering the second task queue is judged. If the parent task has a corresponding associated child task, the parent task is set to a pause state and waits for the processing result of the corresponding associated child task.

[0046] S103, setting the subtasks in the first task queue to an active state in sequence, and then sending them to a third task queue to be processed by a task processing module.

[0047] S104: judging the processing result of the subtask entering the third task queue to determine whether the processing is successful. If the judgment result is that the processing is successful, marking the subtask as being in a processing success state.

[0048] S105, if the processing result of the subtask is processing failure, the processing times of the subtask are further judged. If the processing times of the subtask does not exceed the processing times threshold, the subtask is processed again; if the processing times of the subtask has exceeded the processing times threshold, the subtask is marked as processing failure status.

[0049] S106 , when the processing of the subtask is completed, that is, the processing result of the subtask is a processing success state or a processing failure state; further query the processing status of all subtasks under the parent task corresponding to the subtask.

[0050] S107: If all subtasks under the parent task corresponding to the subtask have not been completely processed, the subtask is set to a delayed state and is sent back to the start position of the first task queue.

[0051] S108, when all subtasks under the parent task corresponding to the subtask have been processed, the parent task corresponding to the subtask in the second task queue is set to an activated state, and the parent task is sent to the fourth task queue for processing by the task processing module. If the parent task does not have any corresponding subtasks, the parent task directly enters the fourth task queue.

[0052] S109: judging the processing result of the parent task entering the fourth task queue to determine whether the processing is successful. If the judgment result is that the processing is successful, marking the subtask as a processing success state.

[0053] S110, if the processing result of the parent task is processing failure, the number of processing times of the parent task is further judged. If the number of processing times of the parent task does not exceed the processing times threshold, the parent task is processed again; if the number of processing times of the parent task has exceeded the processing times threshold, the parent task is marked as processing failure status.

[0054] S111, when the processing of the parent task is completed, that is, the processing result of the parent task is a processing success state or a processing failure state, the task request processing is completed.

[0055] In this embodiment, the tasks in the third task queue and the fourth task queue that are in an activated state are assigned to multiple task processing processes in parallel by the task processing module for data processing.

[0056] The task queue in this implementation uses an asynchronous method to separate the producer and consumer sides when processing high-concurrency application scenarios. The specific advantages of using task queues are as follows: (1) Asynchronous transmission under high concurrency: In high-concurrency scenarios, a large number of task requests need to be processed. If the application system's task processing limit is reached, data will accumulate, and in severe cases, data loss will occur. At the same time, if some task requests take a lot of time to process, other task requests will be blocked, affecting the overall performance of the system. The task queue provides an asynchronous processing mechanism that separates the operation of entering the queue and reading out of the queue. By providing a buffer module to ensure that the speed of writing to the queue is not affected by the processing results read from the queue, asynchronous processing ensures that task requests can be processed efficiently. (2) Ensure the security and reliability of data transmission: When applications in the data platform process data, data loss often occurs due to abnormal situations. The task queue will persist the data until the task request data is completely processed. When deleting a task request from the queue, the system will confirm whether the task request has been processed. On the other hand, if some components in the system fail and the process of processing task requests becomes abnormal, the task request data in the task queue can still be processed again after the system is restored, ensuring that each task request can be processed safely and reliably. (3) Guarantee the order of task requests: In many application scenarios, it is necessary to ensure the sequential processing of data. The task queue itself can well ensure that task requests are processed in the order specified by the user. (4) Decoupling between application modules: In the early stages of the system, it is difficult to predict the situations that the system may encounter in the later stages. The task queue can provide a data-based interface layer, requiring the applications on both ends of the queue to implement this interface.

[0057] By ensuring unified standards, the processing on both sides of the queue can be scaled independently. Therefore, as data volume increases, only the processing needs to be expanded, without requiring code modifications. Given the role of task queues in high concurrency, various design options exist to mitigate this problem. The throughput of a single task queue is fixed in a specific hardware environment and varies based on the concurrency of task requests. When the application generates a low concurrency of task requests, even during peak periods, the concurrency will not exceed the processing capacity of a single task queue instance. In this case, deploying a single task queue can meet application requirements. However, when the application generates a high concurrency of task requests, a single task queue cannot support the transmission of these requests. In this case, a task queue cluster is required for the application. Multiple nodes in the cluster work together to ensure efficient transmission of task requests. The number of nodes deployed is also based on the overall concurrency of task requests. During operation, task request servers inevitably experience downtime due to unexpected failures. Without a high-availability task queue solution, server downtime requires manual restarts to resume task request delivery. With a master-slave task queue, if the master queue stops functioning, the slave task queue can immediately take over and continue delivering task requests. Therefore, task queue solutions can be categorized as single-node task queues or high-availability master-slave node task queues, depending on the varying task request concurrency requirements of different applications. Task queue architectures can be categorized as either single-node task queues or clusters of multiple task queues, depending on the concurrency requirements of different applications. However, when task request concurrency is high and application requirements become diverse, traditional task queue architectures cannot meet the high concurrency demands of diverse applications. Because task queues are typically deployed in a centralized manner, it's difficult to configure the appropriate task queue service for each specific task request.

[0058] In this embodiment, the first task queue, the second task queue, the third task queue, and the fourth task queue are real-time high-speed message queues built based on Redis technology. First, the Redis real-time high-speed message queue has very high read and write performance. Secondly, the Redis real-time high-speed message queue supports richer data types than Memcache. In addition to simple Key / Value, it also provides list, set, hash and other data structures, and has powerful operation functions for these data structures. Therefore, Redis can make the design of message queues more flexible. Content that needs to be sorted in chronological order is stored in a list, and unordered content is stored in a set.

[0059] In this implementation, when using the Redis message queue to transmit task requests, the task requests are received and stored in the MySQL database. In this implementation, you can write scripts in Lua and submit them to Redis for execution. Redis will execute the entire Lua script as a whole, and other commands will not be interrupted during execution (Redis is single-threaded). The corresponding Lua script is as follows:

[0060] String luaScript="ifredis.call('sismember','STU-NO-'..KEYS[1],ARGV[1])==1then\n"+"\t return 0\n"+ / /

[0061] "end\n"+

[0062] "local maxNum=redis.call('hget', KEYS[1], 'maxNum')\n"+ "local hasNum=redis.call('hget', KEYS[1], 'hasNum')\n"+

[0063] “iftonumber(maxNum)>tonumber(hasNum)

[0064] then\n”+ / /

[0065] "\t redis.call('hincrby', KEYS, 'has-Num', 1)\n"+ / /

[0066] "\t redis.call('lpush', 'taskSelect', KEYS[1]..' / '..ARGV[1])\n"+ / /

[0067] "\t redis.call('sadd', 'STUNO-'..KEYS[1], ARGV[1])\n"+ / /

[0068] “\t return 1\n”+ / /

[0069] "else\n"+"\t return 0\n"+ / /

[0070] "end\n";

[0071] Object result=jedis.evalsha(jedis.scriptLoad(luaS-script),Arrays.asList(csno),Arrays.asList(stuno));

[0072] In this embodiment, a hierarchical cache design is also employed when using a real-time, high-speed message queue built based on Redis technology. Since the system's cache capacity is limited, when designing the cache, the most frequently requested data is generally cached. In a practical system, the core of all data processing logic is data and results. Caching all of these core database collections would require a large amount of memory space and significantly reduce the cache update speed. To improve cache efficiency, the database collection can first be split to implement a hierarchical cache design.

[0073] In this embodiment, the first task queue, the second task queue, the third task queue, and the fourth task queue all comprise a two-layer structure, comprising a Vector Cache layer and a Row Cache layer. The Vector Cache layer primarily stores task description information in task requests, such as the correspondence between parent tasks and child tasks in the task request, and various status flags of the parent tasks and child tasks. The Row Cache layer, on the other hand, is primarily used to store entity content, such as task processing data. When designing a cache message queue, priority should be given to ensuring the resources required by the more frequently accessed Vector Cache layer. Only after achieving a high-performance Vector Cache layer can the Row Cache layer be established. Accessing the cache generally involves first calculating the name of the key in the Vector Cache layer based on the content to be obtained, searching the VectorCache based on the key, finding a task, and then indexing the content cache based on these tasks to obtain the entity content. Regardless of any step that misses a target, the database must be accessed directly and the cache updated.

[0074] The task scheduling management method of this embodiment also includes a release operation for task requests. The purpose of the release operation is to clear subtasks or parent tasks from each task queue. However, subtasks or parent tasks under special circumstances will be retained in their respective task queues. The subtasks and parent tasks under special circumstances are: 1. Activated subtasks in the third task queue; 2. Activated parent tasks in the fourth task queue whose subtasks have all been processed; 3. Subtasks in the first task queue that belong to the same parent task as subtasks in the third task queue.

[0075] The task scheduling management method of the present embodiment is also provided with task blocking processing operation.When a task arrives at the task processing module and distributes processing process and starts to process, the system will lock the task interior, to protect this task from being modified by any other processing process inside.Simultaneously, processing process needs regular notification system and it is still at work.If because processing process takes up a large amount of CPU resources and causes CPU to be very busy, then processing process may not have time to update lock and tell queue that it is still processing this task, so this task may be marked as stopped.The blocked task will be moved back to the first task queue or the second task queue and become waiting state again, and will be processed again by another processing process, or if it has reached its maximum concurrent number, then move to failure queue.Thus, in this implementation, by guaranteeing that processing process is sufficiently frequently returned to the task processing module with control to avoid task blocking.

[0076] The task scheduling management method of this embodiment also provides a pause queue operation. The pause queue operation includes global or local pause queues. When the task processing module does not extract any tasks from the queue, the queue will be globally paused. When the pause queue is activated, the processing process currently busy processing a task will continue to process the task until the task is completed. The task processing module will then remain idle until the pause queue operation is canceled. In this embodiment, the queue is paused by calling the pause method.

[0077] This embodiment also allows for the suspension of specific tasks. Pausing tasks works similarly to a global pause: after a process finishes its current task, it will not process any new tasks. The task processing module will wait for the process to complete all tasks it is currently processing. If you do not want to wait for a process to complete its current task, you can also control the running process to stop processing tasks by issuing a forced pause command:

[0078] In this embodiment, parent-child task management involves the queue buffer module moving delayed subtasks to the first task queue and automatically checking for blocked tasks, i.e., subtasks in the second task queue that are active but whose processing has crashed or stopped. Crashed or stopped tasks are moved back or failed. When executing delayed tasks, fallback retries, and rate limiting functions, queue scheduling is required to at least set the execution position of a given queue.

[0079] This embodiment enables smooth shutdown of a process. When shutting down a process, it is essential to minimize the number of stalled tasks. Therefore, even in the case of an "abnormal shutdown," the task blocking handling operation of the present invention allows a new process to take over the blocked task and continue processing. At the same time, the original process is marked as closed. At this point, the original process will not accept new tasks and will wait until all current tasks have been processed.

[0080] The embodiments of the present disclosure further disclose a computer program product, which includes a computer program / instruction, and implements any of the above method steps when the computer program / instruction is executed by a processor.

[0081] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the diagram or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, as well as the combination of boxes in the block diagram and / or flow chart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0082] The units or modules described in the embodiments of the present disclosure may be implemented in software or hardware. The units or modules described may also be provided in a processor, and the names of these units or modules do not, in certain circumstances, limit the units or modules themselves.

[0083] As another aspect, embodiments of the present disclosure further provide a computer-readable storage medium. This computer-readable storage medium may be included in the apparatus described in the above embodiments, or may be a standalone computer-readable storage medium not incorporated into the apparatus. The computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the methods described in the embodiments of the present disclosure.

[0084] Example 2

[0085] This embodiment discloses an electronic device, Figure 3A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 3 As shown, the electronic device 300 includes a memory 301 and a processor 302; wherein,

[0086] The memory 301 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 302 to implement the above method steps.

[0087] Figure 4 It is a structural diagram of a computer system suitable for implementing the data reading and writing method according to the present invention.

[0088] like Figure 4 As shown, the computer system 400 includes a processing unit 401, which can execute various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the computer system 400 are also stored in the RAM 403. The processing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0089] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom can be installed into the storage section 408 as needed. Among them, the processing unit 401 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0090] In particular, according to embodiments of the present disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a computer-readable medium, the computer program comprising program code for executing the data reading and writing methods. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 409 and / or installed from the removable medium 411.

[0091] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A high-concurrency multi-task scheduling management method based on real-time task queue, characterized in that: The method comprises: When a task request is sent from the task generation module to the queue buffer module, the subtask of the task request is placed in the first task queue of the queue buffer module; The parent task of the task request is placed in the second task queue of the queue buffer module; The parent task entering the second task queue is judged. If the parent task has a corresponding associated child task, the parent task is set to a pause state and waits for the processing result of the corresponding associated child task; The subtasks in the first task queue are set to active state in sequence, and then sent to the third task queue for processing by the task processing module; The processing result of the subtask entering the third task queue is judged to determine whether the processing is successful. If the judgment result is successful, the subtask is marked as being in a successful processing state. If the processing result of the subtask is processing failure, the processing times of the subtask are further judged. If the processing times of the subtask do not exceed the processing times threshold, the subtask is processed again; if the processing times of the subtask have exceeded the processing times threshold, the subtask is marked as processing failure; When the processing of the subtask is completed, that is, the processing result of the subtask is a processing success state or a processing failure state; further query the processing status of all subtasks under the parent task corresponding to the subtask; If all subtasks under the parent task corresponding to the subtask have not been fully processed, the subtask is set to a delayed state and is sent back to the beginning of the first task queue; When all subtasks under the parent task corresponding to the subtask have been processed, the parent task corresponding to the subtask in the second task queue is set to the activated state, and the parent task is sent to the fourth task queue for processing by the task processing module; if the parent task does not have any corresponding subtasks, the parent task is directly sent to the fourth task queue; The processing result of the parent task entering the fourth task queue is judged to determine whether the processing is successful. If the judgment result is successful, the parent task is marked as a successful processing state; If the processing result of the parent task is a processing failure, the number of processing times of the parent task is further judged. If the number of processing times of the parent task does not exceed the processing times threshold, the parent task is processed again; if the number of processing times of the parent task has exceeded the processing times threshold, the parent task is marked as a processing failure state; When the processing of the parent task is completed, that is, the processing result of the parent task is a processing success state or a processing failure state, the task request processing is completed.

2. The high-concurrency multi-task scheduling management method according to claim 1, characterized in that: The tasks in the third task queue and the fourth task queue that are in an activated state are assigned to multiple task processing processes in parallel by the task processing module for data processing.

3. The high-concurrency multi-task scheduling management method according to claim 1, characterized in that: The task queue is generated through an asynchronous strategy to separate the task generation module and the task processing module.

4. The high-concurrency multi-task scheduling management method according to claim 1, characterized in that: The first task queue, the second task queue, the third task queue and the fourth task queue are real-time high-speed message queues built based on Redis technology.

5. The high-concurrency multi-task scheduling management method according to claim 1, characterized in that: The first task queue, the second task queue, the third task queue and the fourth task queue all include a two-layer structure, which includes a Vector Cache layer and a Row Cache layer; the Vector Cache layer is used to store task description information in the task request; and the Row Cache layer is used for task processing data.

6. The high-concurrency multi-task scheduling management method according to claim 1, characterized in that: The high-concurrency multi-task scheduling management method further includes a release operation for task requests, wherein the purpose of the release operation is to clear subtasks or parent tasks in each task queue.

7. The high-concurrency multi-task scheduling management method according to claim 1, characterized in that: The high-concurrency multi-task scheduling management method also includes task blocking processing operations and pause queue operations.

8. The high-concurrency multi-task scheduling management method according to claim 1, characterized in that: The high-concurrency multi-task scheduling management method also includes the queue buffer module moving the subtask in the delayed state to the first task queue, and automatically checking the blocked task, which is a subtask in the second task queue, and the subtask is in the active state, but its processing process has crashed or stopped processing.

9. A computer device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the high-concurrency multi-task scheduling management method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute any one of the high-concurrency multi-task scheduling management methods described in 1-8.

Images