Task processing method and device, equipment and medium

Abstract

The invention discloses a task processing method and device, equipment and a medium. The method comprises the following steps: acquiring batch online tasks; splitting the batch online task into a plurality of sub-tasks according to a preset priority rule to obtain a corresponding sub-task list; wherein each subtask in the subtask list comprises a priority level; storing the sub-task list to a first-level priority task queue set according to the priority level; wherein the first-level priority task queue set comprises multiple types of priority queues; based on the priority sequence of the multi-class priority queues, sequentially storing the sub-tasks in the sub-task list to a second-level task queue; and pulling each sub-task from the second-level task queue for concurrent execution to obtain a processing result. According to the technical scheme, splitting forwarding, priority management and concurrent execution are carried out on the batch online large tasks based on the two-stage message queues, so that server resources can be fully utilized, and the scheduling capability and the overall performance of a distributed system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a task processing method and device, equipment and medium. BACKGROUND

[0002] At present, with the development of the Internet, the demand for rapid processing of a large amount of information becomes urgent, so the distribution and processing of data tasks become very important.

[0003] The traditional task distribution distributes tasks to each server of a distributed system for execution through a load balancing device. This method relies on a single server to process tasks, which can balance the system load for small tasks, but lacks effective task scheduling and distribution mechanism for large tasks, and the server resources cannot be fully utilized, resulting in resource waste and single point failure affecting a large area. SUMMARY

[0004] The present application provides a task processing method, device, equipment and medium, based on two-level message queue for batch online large task splitting and forwarding, priority management and concurrent execution to make full use of server resources, avoid resource waste, and improve the scheduling capability and overall performance of the distributed system.

[0005] According to an aspect of the present application, a task processing method is provided, comprising:

[0006] Obtaining a batch online task;

[0007] Splitting the batch online task according to a preset priority rule to obtain a plurality of subtasks and form a corresponding subtask list; wherein each subtask in the subtask list includes a priority level;

[0008] Storing the subtask list to a first-level priority task queue set according to the priority level; wherein the first-level priority task queue set includes a plurality of priority queues;

[0009] Storing each subtask to a second-level task queue in turn based on the priority order of the plurality of priority queues;

[0010] Pulling a subtask from the second-level task queue for processing to obtain a processing result.

[0011] According to another aspect of the present application, a task processing device is provided, comprising:

[0012] A task acquisition module for acquiring a batch online task;

[0013] The task splitting module is configured to split the batch online task according to a preset priority rule to obtain a plurality of subtasks and form a corresponding subtask list, wherein each subtask in the subtask list comprises a priority level.

[0014] The first storage module is configured to store the subtask list into a first-level priority task queue set according to the priority level, wherein the first-level priority task queue set comprises a plurality of priority queues.

[0015] The second storage module is configured to sequentially store each subtask into a second-level task queue based on a priority order of the plurality of priority queues.

[0016] The task execution module is configured to pull a subtask from the second-level task queue for processing to obtain a processing result.

[0017] According to another aspect of the present application, an electronic device is provided, which comprises:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein

[0020] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the task processing method according to any one of the embodiments of the present application.

[0021] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the task processing method according to any one of the embodiments of the present application when executed by the processor.

[0022] The technical solution of the embodiments of the present application comprises the following steps: obtaining a batch online task; splitting the batch online task according to a preset priority rule to obtain a plurality of subtasks and form a corresponding subtask list, wherein each subtask in the subtask list comprises a priority level; storing the subtask list into a first-level priority task queue set according to the priority level, wherein the first-level priority task queue set comprises a plurality of priority queues; sequentially storing each subtask into a second-level task queue based on a priority order of the plurality of priority queues; and pulling a subtask from the second-level task queue for processing to obtain a processing result. The technical solution is based on two-level message queues to split, forward, manage the priority of and concurrently execute a batch online large task, so that the server resources can be fully utilized, resource waste is avoided, and the scheduling capability and overall performance of the distributed system are improved.

[0023] It is to be understood that the embodiments described herein are merely exemplary of the application and that a person skilled in the art can devise other embodiments without departing from the scope of the present application. It is also to be understood that not all of the features and / or benefits described and / or illustrated herein need be present in every embodiment of the application. The scope of the application should therefore not be limited to the features and / or benefits described and / or illustrated herein, but should be given the full scope that the claims afford based on the entirety of the specification. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0025] Figure 1 is a flow chart of a task processing method according to an embodiment of the present application;

[0026] Figure 2 is a flow chart of a task processing method according to an embodiment of the present application;

[0027] Figure 3 is an architectural schematic diagram of a whole scheme of a task processing method according to an embodiment of the present application;

[0028] Figure 4 is a structural schematic diagram of a task processing device according to an embodiment of the present application;

[0029] Figure 5 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the technical personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should be within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to such a process, method, product or device.

[0032] Embodiment one

[0033] Figure 1 is a flowchart of a task processing method according to an embodiment one of the present application. The embodiment can be applicable to the case of processing batch online large tasks. The method can be executed by a task processing device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device with data processing capability. As shown in Figure 1 , the method comprises:

[0034] The technical solution of the embodiment can be executed by a batch online large task processing system, which can include a task producer module, a two-level message queue module and a task consumer module. In a distributed environment, the task producer and the task consumer are composed of multiple servers, and each server can independently run a specific task or function. The task producer module is responsible for task splitting and forwarding. The two-level message queue module is the task scheduling center of the present technical solution, and realizes priority-based task scheduling and decoupling of the task parallel consumption process. Through the task consumer module, each consumer acquires sub-tasks from the second-level waiting task queue for execution, realizing concurrent execution of tasks in a distributed system.

[0035] S110, acquiring a batch online task.

[0036] The batch online task can refer to a single task composed of multiple independent same type online transactions. Such tasks often need to execute a large number of same type transactions in a short time, bringing traffic peaks to the application system, and have the characteristics of large scale and long task execution time. Exemplarily, the batch online task can be tasks such as enterprise payroll, mail / sms mass sending and consumer subsidy distribution. The task producer module in the batch online large task processing system in the embodiment can acquire the batch online task.

[0037] S120, split the batch online task according to the preset priority rule to obtain a plurality of subtasks, and form a corresponding subtask list.

[0038] In the embodiment, the priority level can refer to the priority level divided according to the rule in the static priority. In the embodiment, the priority divided according to the preset priority rule belongs to the static type priority. In addition, the embodiment can also include a dynamic type priority and a self-defined priority, and the corresponding priority is sequentially increased. The priority level can refer to a low priority, a medium priority, and a high priority level. The subtask list can refer to a task list formed by the plurality of subtasks divided. In the embodiment, the number of the subtask list is determined by the specific division rule. The preset priority rule can be a rule set in advance. In the embodiment, the specific priority rule can be set according to the specific business scenario and the business attribute.

[0039] For example, in the embodiment, when the enterprise issues a marketing red packet, the specific priority rule can be set according to the red packet amount interval, the user's belonging region and other factors. The business attribute can include the practicality of the business; in the embodiment, the priority can be determined according to the high and low of the practicality. For example, some businesses with high real-time performance can be given a high priority, and the dimension attribute can be divided according to the actual business attribute to determine the corresponding priority. For example, different priority levels can be divided for different regional tags according to the marketing effect of the business.

[0040] In the embodiment, after each producer in the batch online task processing system receives the batch online task, the batch online task is split into a plurality of subtasks according to the preset priority rule, thereby obtaining a corresponding subtask list, and each subtask is given corresponding identification information. The identification information can include a subtask number, a belonging task identification, and a static priority, etc. It can be understood that the producers in the embodiment can be considered as servers. In the embodiment, the task is split and scheduled according to the priority based on the task division idea, which can better adapt to the task demand in different scenarios, and finally improves the task processing efficiency and user satisfaction of the whole system.

[0041] S130, store the subtask list to a first priority task queue set according to the priority level.

[0042] The first-level priority task queue set includes multiple priority queues. The multiple priority queues include a first priority sub-queue, a second priority queue, and a third priority queue. The first priority queue can be a static priority queue. The second priority queue can be a dynamic priority queue. The third priority queue can be a custom priority queue. In addition, the static priority queue in this embodiment can further include a low priority sub-queue, a medium priority sub-queue, and a high priority sub-queue.

[0043] In this embodiment, the sub-task list can be stored in the first-level priority task queue set of the two-level message queue module, and each sub-task in the sub-task list can be stored in each sub-queue of the corresponding static priority queue in the first-level priority task queue set according to the priority level of each sub-task. Then, the dynamic priority queue and the custom priority queue can be determined based on system resource dynamic adjustment and business settings.

[0044] S140, each sub-task is sequentially stored in the second-level task queue based on the priority order of the multiple priority queues.

[0045] The priority order can be an order from high to low priority. The second-level task queue can be used to store a queue of tasks waiting for execution. In this embodiment, a task refreshing thread can be set, and the system can refresh the sub-tasks from the first-level queue to the second-level waiting task queue in order of priority from high to low at a fixed time. Specifically, when the system performs task refreshing in this embodiment, it is first determined whether the number of sub-tasks in the second-level waiting task queue is less than the length L of the waiting task queue. If so, enough sub-tasks are taken from each task queue in the first level in order of priority from high to low and cached into the second-level waiting task queue to complete a priority-based task refreshing.

[0046] S150, a sub-task is pulled from the second-level task queue for processing to obtain a processing result.

[0047] The processing result can be the result obtained after each sub-task is processed. In this embodiment, a specified number of sub-tasks can be pulled from the second-level task queue for concurrent processing by multiple servers, so as to obtain the execution result corresponding to each sub-task. The specified number can be determined based on the task processing capacity of the multiple servers, which is not limited in this embodiment.

[0048] In this embodiment, the task pulling thread of the task consumer module obtains sub-tasks from the second-level task queue, and each consumer concurrently executes each sub-task and feeds back the processing progress and the processing result.

[0049] Further, the two-stage message queue module of the embodiment is the task scheduling center of the scheme, realizes the priority-based task scheduling and the task parallel consumption process decoupling, the first-stage queue realizes the priority ordering of the task and the task transfer between the queues, supports the convenient and flexible adjustment of the task execution order. The second-stage queue realizes the parallel execution of the task, fully utilizes the cluster resources to process the task in parallel, and improves the execution efficiency. Redis is the commonly used middleware of the system, is based on the memory operation, is stable and high in efficiency, and can also realize the function of the message queue. Therefore, the two-stage message queue module in the embodiment can be realized based on Redis.

[0050] In addition, the two-stage message queue module in the embodiment is a key link of the scheme, and if the module is abnormal, the business will be interrupted, and the high availability of the system will be affected. The scheme adopts the complementary bottom-up strategy of Redis and the database, realizes the mechanism that the two-stage message queue module service is automatically switched to the database service when Redis is abnormal, and improves the high availability of the scheme. The mechanism is composed of three parts of the Redis service cluster, the database service and the Redis health check probe. The Redis health check probe real-time sniffs the health status of the current Redis service, if the service state is normal, the two-stage message queue module based on Redis is realized, the data update operation in the message queue is real-time synchronized to the database table; if the Redis server state is abnormal, the system is automatically switched to the database service, the two-stage queue table is defined in the database, and the task data consistent with the Redis message queue is stored, the system reads the corresponding task queue from the database table, and realizes the two-stage message queue module service based on the database. After the Redis service is monitored to be normal, the system restores the incremental unexecuted task data generated by the database table to the corresponding Redis message queue, and after the data migration is completed, the system is switched to the Redis service again. The embodiment realizes the double protection of the task scheduling center-the two-stage message queue module service, when Redis is abnormal, the service can be automatically degraded to the database, and the high availability of the scheme is ensured.

[0051] The technical scheme of the embodiment of the application comprises the following steps: obtaining a batch online task; splitting the batch online task according to a preset priority rule to obtain a plurality of subtasks and forming a corresponding subtask list; wherein each subtask in the subtask list comprises a priority level; storing the subtask list into a first-level priority task queue set according to the priority level; wherein the first-level priority task queue set comprises a plurality of priority queues; storing each subtask into a second-level task queue according to the priority order of the plurality of priority queues; pulling the subtask from the second-level task queue for processing to obtain a processing result; the technical scheme can fully utilize the server resources by splitting and forwarding the batch online task, priority management and concurrent execution based on the two-level message queue, thereby avoiding resource waste and improving the scheduling capability and overall performance of the distributed system.

[0052] Embodiment two

[0053] Figure 2 is a flowchart of a task processing method according to the embodiment two of the application, which is optimized based on the above-mentioned embodiment. The specific optimization is that the plurality of priority subqueues comprises a first priority queue, a second priority queue and a third priority queue; accordingly, storing the subtask list into the first-level priority task queue set according to the priority level comprises: storing the subtask list into the first priority queue according to the priority level and determining the entry time of each subtask in the subtask list; adjusting the first priority of each subtask to the second priority based on the entry time and moving to the corresponding second priority queue; in response to the business setting, adjusting the priority of the subtask of the first priority task queue and / or the second priority task queue to the third priority and moving to the corresponding third priority queue. As shown in Figure 2 , the method comprises:

[0054] S210, obtaining a batch online task.

[0055] S220, splitting the batch online task according to a preset priority rule to obtain a plurality of subtasks and forming a corresponding subtask list.

[0056] Each subtask in the subtask list comprises a priority level. The first-level priority task queue set comprises a plurality of priority queues; the plurality of priority queues comprises a first priority queue, a second priority queue and a third priority queue. In the embodiment, the first priority queue can be a static priority queue. The second priority queue can be a dynamic priority queue. The third priority queue can be a self-defined priority queue. It can be understood that in the embodiment, the priority in the dynamic priority queue is higher than that in the static priority queue; the priority in the self-defined priority queue is higher than that in the dynamic priority queue.

[0057] S230, store the subtask list into the first type priority queue according to the priority level, and determine the entry time of each subtask in the subtask list.

[0058] The first type priority queue can be a static priority queue. In the embodiment, the first type priority queue includes a low priority subqueue, a medium priority subqueue, and a high priority subqueue. The entry time can be understood as the time when the corresponding priority queue exists. In the embodiment, each subtask is stored in each type of priority queue in the first level priority task queue set, and there is corresponding storage time data.

[0059] In the embodiment, each subtask in the subtask list can be stored in the first type priority queue of the first level priority task queue set according to the priority level, and the entry time of each subtask contained in the subtask list is determined.

[0060] For example, in the embodiment, the static priority queue is used to store tasks with fixed priority calculated according to a preset rule, which is determined by the actual business scenario requirement. For example, the static priority of a task is divided into three levels of high, medium, and low, and the task of sending a red packet to a VIP user is set as high priority, and the task is cached in the high priority subqueue.

[0061] Further, in the embodiment, for the static priority queue, in addition to setting the static priority rule of the task according to the specific business requirement, the priority can also be set according to the historical processing data of the task. For example, the types of tasks that have more errors or delays in the past task processing process are counted, and a higher priority is set in the custom queue to pay more attention to and process these tasks that are prone to problems in subsequent processing, thereby improving the success rate and efficiency of task processing.

[0062] S240, adjust the first type priority of each subtask to the second type priority based on the entry time and move to the corresponding second type priority queue.

[0063] The second type priority can be a dynamic priority, and the corresponding second type priority queue is a dynamic priority queue. It can be understood that in the embodiment, the first type priority is a static priority, which is obtained by dividing the priority level according to the pre-priority rule and belongs to a fixed priority. The second type priority in the embodiment is a task priority that can be dynamically adjusted according to the real-time resource state of the system or the change of the external business environment.

[0064] In this example, the time threshold can be set according to the entry time of each subtask in the first type of priority, and the static priority of each subtask is adjusted to a dynamic priority based on the comparison result of the entry time and the dynamic threshold, and is moved to the second type of priority queue corresponding to the dynamic priority, so as to improve the execution priority of the task.

[0065] In this embodiment, the first type of priority of each subtask is adjusted to the second type of priority based on the entry time and moved to the corresponding second type of priority queue, which includes: comparing the entry time with the time threshold to obtain a comparison result; and adjusting the first type of priority of the subtask to the second type of priority according to the comparison result and moving it to the corresponding second type of priority queue.

[0066] In this embodiment, the time threshold can be pre-set. For example, the time threshold in this embodiment can be 5 minutes or 10 minutes, which can be set according to business conditions. The comparison result can be the result that the entry time is greater than the time threshold.

[0067] In this embodiment, the entry time can be compared with the time threshold to obtain a comparison result, and then the first type of priority of the subtask in the first type of priority queue whose entry time is greater than the time threshold is adjusted to the second type of priority, i.e. the static priority is adjusted to the dynamic priority, and is moved to the corresponding dynamic priority queue.

[0068] In this embodiment, the dynamic priority queue can dynamically adjust the priority of the task according to the real-time resource state of the system or the change of the external business environment. For example, in order to avoid the subtask with low static priority always in the queue waiting state, a timeout task monitoring thread is set to scan each static priority queue regularly, and the subtask whose entry time exceeds the time threshold but is still in the static priority task queue is moved to the dynamic priority task queue. Further, in this embodiment, for the dynamic priority queue, in addition to the rule of improving the priority of the task to the dynamic priority queue according to the timeout of the task, a user feedback mechanism can also be introduced to adjust the priority. For example, in the process of enterprise marketing red packet distribution, if the user feedback or complaint of the red packet distribution in a certain area is more, the priority of the subsequent red packet distribution subtask in this area can be improved, so as to timely process and solve the user problem and improve the user experience.

[0069] In this embodiment, such setting can improve the execution priority of the subtask, so as to avoid the situation that the subtask with static priority cannot be processed due to the high priority task entering the queue all the time.

[0070] S250, in response to the service setting, adjusting the priority of the sub-tasks of the first priority task queue and / or the second priority task queue to a third priority, and moving to a third priority queue corresponding to the third priority.

[0071] The service setting can be an actual service requirement setting. In this embodiment, the service setting can be a setting made by manual operation through a man-machine interaction interface. The third priority can be a custom priority, which has the highest priority level. The third priority queue can be a custom priority queue corresponding to the third priority.

[0072] In this embodiment, the priority of one or more sub-tasks in the first priority task queue or the second priority task queue can be adjusted to a custom priority in response to the service setting, or the priority of one or more sub-tasks in the first priority task queue or the second priority task queue can be adjusted to a custom priority, and the sub-tasks can be moved to a custom priority queue corresponding to the custom priority.

[0073] Further, the custom queue can also set a corresponding priority queue according to actual service requirements. For example, two custom priority queues are set, namely, an urgent task queue and a revoked task queue, to achieve flexible control during task execution. The urgent task queue is the highest priority queue in the first queue, and the purpose is to achieve urgent processing of urgent tasks. The tasks in the revoked task queue will not be flushed to the second level waiting task queue, and therefore will not be executed. This embodiment also provides a revoked task recovery function, which can recover the tasks from the revoked queue to the corresponding static queue for re-execution when the recovery business condition is met.

[0074] S260, based on the priority order of the multiple priority queues, storing each sub-task to the second level task queue in turn.

[0075] In this embodiment, the multiple priority queues can be stored in the second level task queue in turn based on the priority order of the multiple priority queues, including: obtaining the current queue length and the total queue length of the second level task queue; in the case that the current queue length is less than the total queue length, polling the multiple priority queues according to the priority order, and based on the list length of the sub-task list and the current queue length, taking out the sub-tasks in turn and storing them to the second level task queue.

[0076] The current queue length can be understood as the real-time queue length in the second-level task queue. The total queue length can be understood as the total queue length in the second-level task queue. It can be understood that the queue storage capacity of the second-level task queue in the embodiment is limited, and the used storage space of the current second-level task queue can be determined according to the current queue length, and the total storage space of the second-level task queue is determined according to the total queue length, so that it can be determined whether the subtasks can continue to be stored in the second-level task queue. The list length of the subtask list can be understood as the list length corresponding to the subtask list, that is, how many subtasks are actually contained in the subtask list.

[0077] In the present example, the current queue length and the total queue length of the second-level task queue can be obtained, and then it is determined whether the current queue length is less than the total queue length. In the case where the current queue length is less than the total queue length, the multiple priority queues in the first-level queue set are polled in priority order, and it is determined whether the current first-level queue set is empty. If the multiple priority queues in the current first-level queue set are not empty, the online task numbers of the multiple priority queues in the first-level queue set are obtained, the corresponding subtask lists of the tasks are queried, and the number of subtasks that can be taken out is determined according to the list length, the current list length and the total queue length of the subtask list in the multiple priority queues, so that the corresponding subtasks are taken out according to the number of subtasks and stored in the second-level task queue.

[0078] Further, in the embodiment, the second-level waiting task queue is used as a task distribution queue, and faces the consumer module, that is, the consumer directly pulls the task from the queue for execution. The length of the queue is set considering two aspects, one is not too short, there should be enough subtask data for all consumers to consume in the queue to ensure system resource utilization, and the other is not too long, otherwise it will not be able to guarantee the sensitivity of the first-level queue to task priority adjustment. Assuming that the period of each consumer to pull the task is f, each consumer can execute an average of m subtasks in one pulling period, and the number of consumers is n, the specific determination method of the total queue length L of the waiting queue can be as follows:

[0079] L = n*m + b;

[0080] Wherein, n is the number of consumers, and b is an integer greater than 0, to avoid the task production speed from not keeping up with the consumption speed.

[0081] In the embodiment, through such setting, the efficient acquisition and execution of the second-level task queue can be ensured, the cluster resources are fully utilized for parallel processing of global tasks, and the system performance is effectively improved.

[0082] In the embodiment, optionally, the sub-tasks are sequentially taken out based on the list length of the sub-task list and the current queue length and stored into the second-level task queue, including: determining the number of sub-tasks to be taken out based on the list length of the sub-task list and the current queue length; taking out the corresponding sub-tasks from the sub-task list according to the number of sub-tasks to be taken out and storing them into the second-level task queue.

[0083] The number of sub-tasks to be taken out can be understood as the number of sub-tasks that can be taken out from the first-level task queue and stored into the second-level task queue. In the embodiment, the length sum can be obtained by adding the list length of the sub-task list and the current queue length, and then the length sum is compared with the total list length of the second-level task queue, the number of sub-tasks to be taken out is determined according to the comparison result, and then the corresponding sub-tasks are taken out from the sub-task list according to the number of sub-tasks to be taken out and stored into the second-level task queue.

[0084] In the embodiment, by such a setting, the current storable space can be determined according to the length limit of the second-level task queue, so that the number of sub-tasks to be taken out is determined based on the storable space, thereby realizing efficient acquisition and execution of the second-level task queue and improving the efficiency of task processing.

[0085] In the embodiment, optionally, the number of sub-tasks to be taken out is determined based on the list length of the sub-task list and the current queue length, including: determining a target length based on the list length of the sub-task list and the current queue length; determining whether the target length is less than or equal to the total queue length; if the target length is less than or equal to the total queue length, determining the sub-task list as the number of sub-tasks to be taken out; and if the target length is greater than the total queue length, determining the number of sub-tasks to be taken out based on the total queue length and the current queue length.

[0086] The target length can be length data determined by fusing the list length of the sub-task list and the current queue length. In the embodiment, the specific way of determining the target length based on the list length of the sub-task list and the current queue length can be to add the list length of the sub-task list and the current queue length of the second-level task queue to obtain the corresponding target length.

[0087] In this embodiment, the data obtained by adding the list length of the subtask list and the current queue length can be taken as the target length, and then it is determined whether the target length is less than or equal to the total queue length of the second-level task queue. If the target length is less than or equal to the total queue length, it can be considered that the list length of the subtask list can be completely put into the second-level task queue, that is, all the subtasks in the subtask list of the batch online task can be determined as the number of subtasks to be taken out and all the subtasks are taken out and added to the second-level task queue. If the target length is greater than the total queue length, the data obtained by subtracting the current queue length from the total queue length can be taken as the number of subtasks to be taken out, so that the corresponding subtasks are taken out from the subtask list based on the number of subtasks to be taken out and added to the second-level task queue.

[0088] In this embodiment, by such a setting, the current number of subtasks to be taken out can be determined based on the queue length of the actual second-level task queue and the number of subtask lists, so that the corresponding number of subtasks can be accurately taken out and stored in the second-level task queue, so as to provide the accuracy and reliability of the number determination for subsequent processing operations.

[0089] S270, pulling the subtask from the second-level task queue for processing to obtain a processing result.

[0090] In this embodiment, the subtask is pulled from the second-level task queue for processing to obtain a processing result, including: determining the number of subtasks executed in a pulling period; and pulling the corresponding subtasks from the second-level task queue for concurrent execution according to the number of subtasks executed to obtain an execution result.

[0091] The pulling period can be the period of pulling the task by each consumer, and can be pre-set. In this embodiment, the number of subtasks that can be executed by each consumer in a pulling period, that is, the number of subtasks executed, can be determined. Specifically, in this embodiment, the system resources such as CPU / memory can be obtained, and it is determined whether the system resources exceed the set resource threshold. If the set resource threshold is not exceeded, the resource processing capability can be evaluated based on the system resources, the number of subtasks executable in the pulling period is calculated, and the task pulling logic is executed by the system timing cycle. The corresponding subtasks are pulled from the second-level task queue according to the number of subtasks executed for concurrent execution, and the task execution result is obtained after the task execution is completed.

[0092] Further, in this embodiment, the subtask execution state can be updated after the task execution is completed, and then it is determined whether all the subtasks under the batch online large task are executed. If all the subtasks under the batch online large task are executed, the corresponding subtasks of the batch online large task can be deleted from the first-level queue.

[0093] Through the arrangement in the embodiment, each task consumer can obtain a task from the second queue for execution through task pulling, and the system cyclically executes the task pulling logic to provide efficiency of task execution.

[0094] For example, the architecture diagram of the overall scheme in the embodiment is shown in Figure 3 To reasonably control the scheduling and distribution of subtasks, the scheme is based on the producer / consumer model and proposes a priority-sensitive Redis two-level message queue as a buffer for task forwarding and scheduling. The first level is a priority queue set, and the second level is a waiting task queue. The scheme decouples the task splitting and forwarding of the producer, the priority-based task scheduling of the task scheduling center, and the task consumption and execution process of the consumer, realizes flexible scheduling and efficient processing of tasks in a distributed environment, and considers high-availability design. The task scheduling center realizes automatic switching of the two-level scheduling queue based on Redis and a database. The overall process of the scheme is as follows: the task producer module receives batch online large tasks; the tasks are split into subtasks according to preset priority rules and are assigned priority attributes and other identification information to obtain a subtask set; the subtasks are placed in the corresponding first-level priority task queue set in priority order, which includes low-priority subqueues, medium-priority subqueues, and high-priority subqueues in the static priority queue; and includes the urgent priority queue and the revoked priority queue in the dynamic priority queue and the custom priority queue; the system task refreshing thread refreshes the subtasks in the second-level waiting task queue in priority order from high to low; each server in the task consumer module obtains the subtasks from the second queue through the task pulling thread; each consumer concurrently executes the subtasks and feeds back the processing progress and results; the system monitors the load and task state and dynamically adjusts the task priority; the task is completed, the task processing results are summarized, and the state is updated. Further, the data in the two-level message queue module can be persisted to the database in the embodiment, and the health status of the current Redis service can be detected in real time through the Redis health check process. The resources of each server can be monitored in the embodiment, so that the task consumption can be adjusted in real time according to the resource status of the server node through the resource monitoring mechanism, effectively realizing load balancing while ensuring system stability.

[0095] In the embodiment, the task priority scheduling and parallel consumption process are decoupled through the design of the two-level message queue. The first queue realizes priority sorting of tasks and task transfer between queues, supports convenient and flexible adjustment of task execution order, and provides the function of intervening in ongoing tasks in emergency through the on-demand custom urgent and revoked task queues, and realizes controllability in the task execution process.

[0096] The technical scheme of the embodiment of the application comprises the following steps: obtaining a batch online task; splitting the batch online task according to a preset priority rule to obtain a plurality of subtasks and forming a corresponding subtask list; wherein each subtask in the subtask list comprises a priority level; a first-level priority task queue set comprises a plurality of priority queues; the plurality of priority queues comprise a first priority queue, a second priority queue and a third priority queue; storing the subtask list in the first priority queue according to the priority level and determining the entry time of each subtask in the subtask list; adjusting the first priority of each subtask to the second priority and moving to the corresponding second priority queue based on the entry time; in response to a service setting, adjusting the priority of the subtask of the first priority task queue and / or the second priority task queue to the third priority and moving to the corresponding third priority queue; storing each subtask in the second-level task queue in turn based on the priority order of the plurality of priority queues; pulling the subtask from the second-level task queue for processing to obtain a processing result; the technical scheme can fully utilize the server resources by splitting and forwarding the batch online task, priority management and concurrent execution based on the two-level message queue, avoids resource waste and improves the scheduling capability and overall performance of the distributed system.

[0097] Embodiment three

[0098] Figure 4 is a structural schematic diagram of a task processing device provided according to the embodiment three of the application.

[0099] As Figure 4 shown, the device comprises:

[0100] The task acquisition module 410 is configured to acquire a batch online task.

[0101] The task splitting module 420 is configured to split the batch online task according to a preset priority rule to obtain a plurality of subtasks and form a corresponding subtask list; wherein each subtask in the subtask list comprises a priority level.

[0102] The first storage module 430 is configured to store the subtask list in a first-level priority task queue set according to the priority level; wherein the first-level priority task queue set comprises a plurality of priority queues.

[0103] The second storage module 440 is configured to store each subtask in a second-level task queue in turn based on the priority order of the plurality of priority queues.

[0104] The task execution module 450 is configured to pull the subtask from the second-level task queue for processing to obtain a processing result.

[0105] Optionally, the plurality of priority queues include a first priority queue, a second priority queue, and a third priority queue.

[0106] Correspondingly, the first storage module 430 includes:

[0107] The time determination unit is configured to store the subtask list into the first priority queue according to the priority level, and determine the enqueue time of each subtask in the subtask list;

[0108] The first adjustment unit is configured to adjust the first priority of each subtask to the second priority and move to the corresponding second priority queue based on the enqueue time;

[0109] The second adjustment unit is configured to adjust the priority of the subtask of the first priority task queue and / or the second priority task queue to the third priority and move to the corresponding third priority queue in response to the service setting.

[0110] Optionally, the first adjustment unit is specifically configured to:

[0111] compare the enqueue time with the time threshold to obtain a comparison result;

[0112] adjust the first priority of the subtask to the second priority and move to the corresponding second priority queue according to the comparison result.

[0113] Optionally, the second storage module 440 includes:

[0114] The length acquisition unit is configured to acquire the current queue length and the total queue length of the second-level task queue;

[0115] The subtask extraction unit is configured to poll the plurality of priority queues in the priority order in the case that the current queue length is less than the total queue length, extract the subtasks in turn based on the list length of the subtask list and the current queue length, and store the subtasks into the second-level task queue.

[0116] Optionally, the subtask extraction unit includes:

[0117] The quantity determination subunit is configured to determine the number of subtasks to be extracted based on the list length of the subtask list and the current queue length;

[0118] The task extraction subunit is configured to extract the corresponding subtasks from the subtask list according to the number of subtasks to be extracted, and store the subtasks into the second-level task queue.

[0119] Optionally, the quantity determination subunit is specifically configured to:

[0120] determine a target length based on the list length of the subtask list and the current queue length;

[0121] determining whether the target length is less than or equal to the total queue length;

[0122] if the target length is less than or equal to the total queue length, determining the subtask list as the number of subtasks to be fetched;

[0123] if the target length is greater than the total queue length, determining the number of subtasks to be fetched based on the total queue length and the current queue length.

[0124] Optionally, the task execution module 450 is specifically configured to:

[0125] determine the number of subtask executions in the pulling period;

[0126] pull corresponding subtasks from the second-level task queue according to the number of subtask executions for concurrent execution to obtain an execution result.

[0127] The task processing device provided in the embodiments of the present application can execute the task processing method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0128] Embodiment Four

[0129] Figure 5 is a structural schematic diagram of an electronic device according to the embodiment four of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0130] As Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0131] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0132] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the task processing method.

[0133] In some embodiments, the task processing method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the task processing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the task processing method by any other appropriate means, such as by means of firmware.

[0134] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0135] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0136] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0138] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0139] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0140] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0141] The above detailed description does not limit the scope of the present disclosure. It is understood that various modifications, combinations, sub-combinations, and alternatives can be made to the detailed disclosure without departing from the spirit and principles of the present disclosure. Any modifications, equivalent substitutions, improvements, and the like that are made within the spirit and principles of the present disclosure are included in the scope of the present disclosure.

Claims

1. A task processing method characterized by, The method comprises: obtaining a batch online task; splitting the batch online task according to a preset priority rule to obtain a plurality of subtasks and forming a corresponding subtask list, wherein each subtask in the subtask list comprises a priority level; storing the subtask list into a first-level priority task queue set according to the priority level, wherein the first-level priority task queue set comprises a plurality of priority queues; storing each subtask into a second-level task queue in turn based on the priority order of the plurality of priority queues; pulling a subtask from the second-level task queue for processing to obtain a processing result.

2. The method of claim 1, wherein, The plurality of priority queues comprises a first priority queue, a second priority queue and a third priority queue; Accordingly, storing the subtask list into the first-level priority task queue set according to the priority level comprises: storing the subtask list into the first priority queue according to the priority level, and determining the entry time of each subtask in the subtask list; adjusting the first priority of each subtask to the second priority based on the entry time and moving to the corresponding second priority queue; in response to a business setting, adjusting the priority of the subtask of the first priority task queue and / or the second priority task queue to the third priority and moving to the corresponding third priority queue.

3. The method of claim 2, wherein, Adjusting the first priority of each subtask to the second priority based on the entry time and moving to the corresponding second priority queue comprises: comparing the entry time with a time threshold to obtain a comparison result; adjusting the first priority of the subtask to the second priority according to the comparison result and moving to the corresponding second priority queue.

4. The method of claim 1, wherein, Storing each subtask into the second-level task queue in turn based on the priority order of the plurality of priority queues comprises: obtaining the current queue length and the total queue length of the second-level task queue; in the case where the current queue length is less than the total queue length, polling the plurality of priority queues according to the priority order, and taking out the subtasks from the subtask list in turn and storing them into the second-level task queue based on the list length of the subtask list and the current queue length.

5. The method of claim 4, wherein, Taking out the subtasks from the subtask list in turn and storing them into the second-level task queue based on the list length of the subtask list and the current queue length comprises: determining the number of subtasks to be taken out based on the list length of the subtask list and the current queue length; taking out the corresponding subtask from the subtask list according to the number of subtasks to be taken out and storing it into the second-level task queue.

6. The method of claim 5, wherein, Determining the number of subtasks to be taken out based on the list length of the subtask list and the current queue length comprises: determining a target length based on the list length of the subtask list and the current queue length; determining whether the target length is less than or equal to the total queue length; if the target length is less than or equal to the total queue length, determining the subtask list as the number of subtasks to be taken out; if the target length is greater than the total queue length, determining the number of subtasks to be taken out based on the total queue length and the current queue length.

7. The method of claim 1, wherein, Pulling a sub-task from the second-level task queue for processing, obtaining a processing result, comprising: Determining the number of sub-task executions in a pulling period; According to the number of sub-task executions, the corresponding sub-tasks are pulled from the second-level task queue for concurrent execution to obtain the execution result.

8. A task processing apparatus characterized by comprising: Comprising: Task acquisition module, used for acquiring batch online tasks; Task splitting module, used for splitting the batch online tasks according to a preset priority rule to obtain a plurality of sub-tasks and form a corresponding sub-task list; wherein each sub-task in the sub-task list includes a priority level; First storage module, used for storing the sub-task list to a first-level priority task queue set according to the priority level; wherein the first-level priority task queue set includes a plurality of priority queues; Second storage module, used for sequentially storing each sub-task to a second-level task queue based on the priority order of the plurality of priority queues; Task execution module, used for pulling a sub-task from the second-level task queue for processing to obtain a processing result.

9. An electronic device, comprising: The electronic device comprises: At least one processor; and The memory is in communication connection with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the task processing method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to enable the processor to execute the task processing method of any one of claims 1-7 when executed.

Images