Delay task management method and system, computing device and computer storage medium

Abstract

The embodiment of the invention discloses a delayed task management method and system, computing equipment and a computer storage medium, and the method comprises the steps: determining a target sorting bucket from a plurality of sorting buckets corresponding to a plurality of service nodes according to the task information of a to-be-managed delayed task, and storing the task information into the target sorting bucket; for each sorting bucket, pre-loading the task information of which the execution time is within the first time interval in the sorting bucket into a delay message queue, and compensating and loading the task information of which the execution time is within the second time interval in the sorting bucket into the delay message queue; and when the current time reaches the execution time of any task information in the delay message queue, the consumer pulls the task information from the delay message queue and executes the task information. The task information is managed in a mode of combining the sorting bucket and the delay message queue, so that the delay task storage capacity can be transversely expanded according to service storage requirements, and different task delay time requirements can be met.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of Internet technologies, and in particular, to a method, a system, a computing device, and a computer storage medium for managing delayed tasks. Background Art

[0002] In many business scenarios, it is necessary to execute delayed tasks. For example, in the order processing scenario, if an order is not paid within a certain period of time after being placed, the order needs to be cancelled; in another example, in the policy business scenario, different incentive tasks are issued at different times after a new user registers; and in yet another example, in the timed scheduling scenario, when a certain task is executed, it is necessary to query whether the upstream data on which it depends is ready. If it is not ready, the task is delayed and then re-executed after waiting for a period of time.

[0003] In the prior art, the delayed tasks are usually managed by means of a message queue. Among them, the task information of the delayed tasks is pushed by a producer into the message queue. After a period of delay, when it arrives at a specified future time point, a consumer pulls the task information from the message queue and executes it. The existing method of managing delayed tasks by using a message queue has limitations in terms of storage capacity and task delay time. Specifically, the task information of the stored delayed tasks cannot exceed the node storage capacity, and the task delay time cannot exceed a certain time threshold (for example, 7 days), which cannot meet the requirements of storing a large number of delayed tasks and executing them after a long delay in Internet services. Therefore, there is an urgent need for a solution that can support the storage of a large number of delayed tasks and whose task delay time is not restricted. Summary of the Invention

[0004] In view of the above problems, the present application provides a method, a system, a computing device, and a computer storage medium for managing delayed tasks, which are used to solve the following problems: The existing method of managing delayed tasks has limitations in terms of storage capacity and task delay time, and cannot meet the requirements of storing a large number of delayed tasks and executing them after a long delay in Internet services.

[0005] According to one aspect of the embodiments of the present application, a method for managing delayed tasks is provided, including:

[0006] Receiving the task information of the delayed tasks to be managed, determining a target sorting bucket from multiple sorting buckets corresponding to multiple service nodes according to the task information, and storing the task information in the target sorting bucket;

[0007] For each sorting bucket, preloading the task information in the sorting bucket whose execution time is within the first time interval into the delayed message queue, and compensating and loading the task information in the sorting bucket whose execution time is within the second time interval into the delayed message queue;

[0008] When the execution time of any task information in the delay message queue arrives at the current time, the consumer pulls any task information from the delay message queue and executes it.

[0009] Further, before receiving the task information of the delay task to be managed, the method further includes:

[0010] Construct a cluster containing multiple service nodes;

[0011] For each service node in the cluster, construct a sorting bucket corresponding to the service node in the disk of the service node.

[0012] Further, determining the target sorting bucket from the multiple sorting buckets corresponding to the multiple service nodes according to the task information further includes:

[0013] Take the delay task identifier and execution time in the task information as the information to be processed;

[0014] Use a preset hash algorithm to perform hash mapping on the information to be processed, and map it to the target sorting bucket among the multiple sorting buckets corresponding to the multiple service nodes.

[0015] Further, storing the task information in the target sorting bucket further includes:

[0016] Store the task information in the form of data key-value pairs in the target sorting bucket, and establish an index for the target sorting bucket according to the data key in the data key-value pairs.

[0017] Further, storing the task information in the form of data key-value pairs in the target sorting bucket further includes:

[0018] Combine the sorting bucket number of the target sorting bucket and the execution time of the task information as the data key in the data key-value pair, and take all or part of the information of the task information as the data value in the data key-value pair;

[0019] Store the data key-value pairs in the target sorting bucket in an index-ordered manner.

[0020] Further, preloading the task information with execution times within the first time interval in the sorting bucket into the delay message queue further includes:

[0021] Determine the target unit time interval where the current time is located, and take the next unit time interval of the target unit time interval as the first time interval;

[0022] Loop to execute the preloading process, use the preloading process to preload the task information with execution times within the first time interval in the sorting bucket into the delay message queue, and after the loading is completed, remove the task information with execution times within the first time interval from the sorting bucket.

[0023] Further, compensating and loading the task information in the sorting bucket whose execution time is within the second time interval into the delayed message queue further includes:

[0024] Taking the current time as the start time of the second time interval, and taking the end time of the target unit time interval where the current time is located as the end time of the second time interval, to construct the second time interval;

[0025] Circularly executing the compensation loading process, using the compensation loading process to compensate and load the task information in the sorting bucket whose execution time is within the second time interval into the delayed message queue, and after the loading is completed, removing the task information in the sorting bucket whose execution time is within the second time interval from the sorting bucket.

[0026] Further, the method further includes:

[0027] Dynamically adjusting the number of service nodes in the cluster according to the resource usage of each service node in the cluster.

[0028] According to another aspect of the embodiments of the present application, a delayed task management system is provided, including:

[0029] A storage module, adapted to receive the task information of the delayed task to be managed, determine a target sorting bucket from multiple sorting buckets corresponding to multiple service nodes according to the task information, and store the task information into the target sorting bucket;

[0030] A loading module, adapted to, for each sorting bucket, pre-load the task information in the sorting bucket whose execution time is within the first time interval into the delayed message queue, and compensate and load the task information in the sorting bucket whose execution time is within the second time interval into the delayed message queue;

[0031] An execution module, adapted to, when the current time reaches the execution time of any task information in the delayed message queue, have a consumer pull any task information from the delayed message queue and execute it.

[0032] Further, the system further includes: a construction module;

[0033] The construction module is adapted to: construct a cluster including multiple service nodes; for each service node in the cluster, construct a sorting bucket corresponding to the service node in the disk of the service node.

[0034] Further, the storage module is further adapted to:

[0035] Taking the delayed task identifier and the execution time in the task information as the information to be processed;

[0036] Using a preset hash algorithm to perform hash mapping on the information to be processed, and mapping it to the target sorting bucket among multiple sorting buckets corresponding to multiple service nodes.

[0037] Further, the storage module is further adapted to:

[0038] Store the task information in the form of data key-value pairs into the target sorting bucket, and establish an index for the target sorting bucket according to the data key in the data key-value pairs.

[0039] Further, the storage module is further adapted to:

[0040] Combine the sorting bucket number of the target sorting bucket and the execution time of the task information as the data key in the data key-value pair, and use all or part of the information of the task information as the data value in the data key-value pair;

[0041] Store the data key-value pairs in the target sorting bucket in an index-ordered manner.

[0042] Further, the loading module is further adapted to:

[0043] Determine the target unit time interval in which the current time is located, and use the next unit time interval of the target unit time interval as the first time interval;

[0044] Loop to execute the preloading process, and use the preloading process to preload the task information in the sorting bucket whose execution time is within the first time interval into the delayed message queue, and after the loading is completed, remove the task information in the sorting bucket whose execution time is within the first time interval.

[0045] Further, the loading module is further adapted to:

[0046] Use the current time as the start time of the second time interval, and use the end time of the target unit time interval in which the current time is located as the end time of the second time interval to construct the second time interval;

[0047] Loop to execute the compensation loading process, and use the compensation loading process to compensate and load the task information in the sorting bucket whose execution time is within the second time interval into the delayed message queue, and after the loading is completed, remove the task information in the sorting bucket whose execution time is within the second time interval.

[0048] Further, the construction module is further adapted to:

[0049] Dynamically adjust the number of service nodes in the cluster according to the resource usage of each service node in the cluster.

[0050] According to another aspect of the embodiments of the present application, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete communication with each other through the communication bus;

[0051] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the above-mentioned delayed task management method.

[0052] According to another aspect of the embodiments of the present application, there is provided a computer storage medium storing at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the above-mentioned delayed task management method.

[0053] According to still another aspect of the embodiments of the present application, there is provided a computer program product including at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the above-mentioned delayed task management method.

[0054] According to the delayed task management method, system, computing device and computer storage medium provided by the embodiments of the present application, corresponding sorting buckets are set in the service nodes. The sorting buckets are used to store the task information of the delayed tasks. According to the received task information of the delayed tasks to be managed, a target sorting bucket is determined from the multiple sorting buckets corresponding to the multiple service nodes, and the task information is stored in the target sorting bucket. The multiple service nodes work together in a cluster form, so that the storage capacity of the delayed tasks can be horizontally expanded according to the business storage requirements, and the storage of a large number of delayed tasks can be conveniently realized, breaking the limitation of the existing delayed task management method in terms of storage capacity; the task information of the delayed tasks is uniformly managed in a manner combining the sorting buckets and the delayed message queue. The task information of the delayed tasks close to execution is managed through the delayed message queue, which can meet different task delay time requirements and break the limitation of the existing delayed task management method in terms of task delay time; in addition, for each sorting bucket, a preloading mechanism and a compensation loading mechanism are set. According to the compensation loading mechanism, the task information that has missed the preloading opportunity can be conveniently compensated and loaded into the delayed message queue, effectively avoiding the omission of such task information; this solution realizes high-performance and horizontally scalable delayed task management and optimizes the delayed task management method.

[0055] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to be able to understand the technical means of the embodiments of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and understandable, the following specifically describes the specific embodiments of the present application. Description of the Drawings

[0056] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the embodiments of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0057] Figure 1 Shows a schematic flow chart of a delayed task management method according to an embodiment of the present application;

[0058] Figure 2a Shows a schematic flow chart of a delayed task management method according to another embodiment of the present application;

[0059] Figure 2b Shows a schematic diagram of the principle architecture of the delayed task management method;

[0060] Figure 3 Shows a block diagram of the structure of a delayed task management system according to an embodiment of the present application;

[0061] Figure 4 Shows a schematic diagram of the structure of a computing device according to an embodiment of the present application. Detailed implementation manners

[0062] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0063] First, the noun terms involved in one or more embodiments of the present application are explained.

[0064] Delayed task: refers to those tasks that are arranged to be executed at a future time point instead of immediately. This mechanism allows the program to automatically execute the specified operations after a specific time or when certain conditions are met without manual intervention. Delayed tasks are widely used in various scenarios, such as timing scheduling, batch processing jobs, timing reminders, etc.

[0065] Horizontal expansion: refers to adding more service nodes and enabling the service nodes to work together in a cluster to jointly share the system load and improve the overall performance and capacity of the system.

[0066] Figure 1 Shows a schematic flow chart of a delayed task management method according to an embodiment of the present application, as Figure 1 shown, the method includes the following steps:

[0067] Step S101, receiving the task information of the delayed task to be managed, and according to the task information, determining a target sorting bucket from multiple sorting buckets corresponding to multiple service nodes, and storing the task information into the target sorting bucket.

[0068] In an embodiment of the present application, a sorting bucket is set in a service node. The sorting bucket is used to store task information of delayed tasks. Multiple service nodes work together in a cluster form, so that the storage capacity of delayed tasks can be horizontally expanded according to business storage requirements, enabling convenient storage of a large number of delayed tasks and breaking the limitation of the existing delayed task management method in terms of storage capacity.

[0069] Among them, a sorting bucket corresponding to the service node is constructed in the disk of each service node. When receiving the task information of the delayed task to be managed, according to the task information, through load balancing processing methods such as polling, hash mapping, and random allocation, a target sorting bucket for storing the task information is allocated for the task information from multiple sorting buckets corresponding to multiple service nodes, and then the task information is stored in the target sorting bucket. Specifically, the task information may include a delayed task identifier, an execution time, task content information, etc.

[0070] Step S102, for each sorting bucket, pre-load the task information in the sorting bucket whose execution time is within the first time interval into the delayed message queue, and compensatorily load the task information in the sorting bucket whose execution time is within the second time interval into the delayed message queue.

[0071] The embodiment of the present application proposes to uniformly manage the task information of delayed tasks in a manner combining a sorting bucket and a delayed message queue. The task information of each delayed task is stored in multiple sorting buckets corresponding to multiple service nodes, and before the delayed task is about to be executed, the task information about to be executed is added to the delayed message queue. That is to say, the delayed message queue is used to manage the task information about to be executed. Using the method of combining a sorting bucket and a delayed message queue to manage the task information of delayed tasks can meet different task delay time requirements and break the limitation of the existing delayed task management method in terms of task delay time.

[0072] For each sorting bucket, a pre-loading mechanism and a compensating loading mechanism are set. Specifically, according to the pre-loading mechanism, the task information in the sorting bucket whose execution time is within the first time interval is pre-loaded into the delayed message queue; considering that in Internet services, the delayed tasks are numerous, and new task information of delayed tasks needs to be managed continuously, and the execution time of the new delayed tasks may miss the pre-loading opportunity because it is before the first time interval. To avoid omission of such task information, the embodiment of the present application not only sets a pre-loading mechanism but also sets a compensating loading mechanism. According to the compensating loading mechanism, the task information in the sorting bucket whose execution time is within the second time interval is compensatorily loaded into the delayed message queue. Among them, the second time interval is different from the first time interval. For example, the second time interval is before the first time interval.

[0073] Step S103, when the current time reaches the execution time of any task information in the delay message queue, the consumer pulls any task information from the delay message queue and executes it.

[0074] In the delay message queue, when the current time reaches the execution time of any task information therein, the consumer pulls the any task information from the delay message queue and executes according to the any task information. After successful execution, the task information is deleted from the delay message queue.

[0075] According to the delay task management method provided by the embodiments of the present application, a corresponding sorting bucket is set in the service node. The sorting bucket is used to store the task information of the delay task. According to the received task information of the delay task to be managed, the target sorting bucket is determined from the multiple sorting buckets corresponding to the multiple service nodes, and the task information is stored in the target sorting bucket. The multiple service nodes work together in the form of a cluster, so that the storage capacity of the delay task can be horizontally expanded according to the business storage requirements, and the storage of a large number of delay tasks can be conveniently realized, breaking the limitation of the existing delay task management method in terms of storage capacity; the task information of the delay task is uniformly managed in a manner combining the sorting bucket and the delay message queue. By managing the task information of the delay task approaching execution through the delay message queue, different task delay time requirements can be met, breaking the limitation of the existing delay task management method in terms of task delay time; in addition, for each sorting bucket, a preloading mechanism and a compensation loading mechanism are set. According to the compensation loading mechanism, the task information that misses the preloading opportunity can be conveniently compensated and loaded into the delay message queue, effectively avoiding the omission of such task information; this solution realizes high-performance and horizontally scalable delay task management and optimizes the delay task management method.

[0076] Figure 2a The flowchart showing the delay task management method according to another embodiment of the present application is as Figure 2a shown, and the method includes the following steps:

[0077] Step S201, construct a cluster including multiple service nodes. For each service node in the cluster, construct a sorting bucket corresponding to the service node in the disk of the service node.

[0078] In the embodiments of the present application, the task information of delayed tasks is managed in the form of a cluster. In step S201, a cluster is constructed. The cluster includes multiple service nodes. And for each service node in the cluster, a sorting bucket corresponding to the service node is constructed in the disk of the service node. The corresponding relationship between the service node and the sorting bucket can be a one-to-one relationship or a one-to-many relationship, that is, one service node corresponds to one sorting bucket, or one service node corresponds to multiple sorting buckets. Those skilled in the art can set according to actual needs and no specific limitation is made here. The multiple service nodes work together in the form of a cluster, so that the storage capacity of delayed tasks can be horizontally expanded according to the business storage requirements, and the storage of a large number of delayed tasks can be conveniently realized without being limited by the storage capacity of a single service node instance.

[0079] Among them, the sorting bucket can be a storage structure of data key-value pairs. The records stored in the sorting bucket include two parts: a data key (key) and a data value (value). The strings of the data keys of all the records in the sorting bucket are indexed in lexicographical order, and each record within the data key range can be quickly queried according to the index.

[0080] Step S202, receive the task information of the delayed task to be managed, use the delayed task identifier and the execution time in the task information as the information to be processed, and perform a hash mapping on the information to be processed by using a preset hash algorithm, and map it to the target sorting bucket among the multiple sorting buckets corresponding to the multiple service nodes.

[0081] Multiple sorting buckets are constructed through step S201, and the sorting buckets are used to store the task information of delayed tasks. When the task information of the delayed task to be managed is received, a hash mapping can be performed according to the task information, and the sorting bucket for storing the task information is determined from the multiple sorting buckets corresponding to the multiple service nodes through the mapping. The determined sorting bucket is called the target sorting bucket.

[0082] The task information may include a delayed task identifier, an execution time, task content information, etc. Among them, the delayed task identifier is the information used to uniquely identify the delayed task, such as the delayed task id; the execution time records the time point when the delayed task needs to be executed in the future, and specifically can be recorded in the form of a time stamp. Among them, the time stamp of the execution time is greater than the time stamp of the current time; the task content information refers to the specific task content of the delayed task.

[0083] In a specific application, a combined hash mapping can be performed on the delay task identifier and execution time in the task information to determine the target sorting bucket. Specifically, the delay task identifier and execution time in the task information are used as the information to be processed, and a preset hash algorithm is used to perform a hash mapping on the information to be processed, and it is mapped to the target sorting bucket among the multiple sorting buckets corresponding to multiple service nodes.

[0084] For example, the preset hash algorithm can be the consistent hash algorithm. The consistent hash algorithm can select the Murmur hash algorithm (Murmur Hash), etc. The Murmur hash algorithm is a non-cryptographic hash algorithm and is suitable for scenarios based on hash value lookup, and can be well applicable to the scenario of finding the matching target sorting bucket from multiple sorting buckets in the embodiments of the present application.

[0085] Taking the case where there is a one-to-one correspondence between service nodes and sorting buckets as an example, a preset number of virtual nodes are generated for all service nodes in the cluster. The node hash values of the virtual nodes are calculated through the consistent hash algorithm, so that the node hash values of the preset number of virtual nodes form a hash ring on the preset data domain. If the obtained node hash value is a 32-bit hash value, then the 32-bit hash values of the virtual nodes form a hash ring on the 32-bit integer domain; the information to be processed composed of the delay task identifier and execution time in the task information is also calculated through the consistent hash algorithm to obtain a hash value, and a virtual node whose node hash value matches this hash value is found from the preset number of virtual nodes. The service node corresponding to the matching virtual node is the target service node, and then the sorting bucket corresponding to the target service node is determined as the target sorting bucket.

[0086] The consistent hash algorithm virtualizes the concept of a ring. For example, a 32-bit integer domain can be formed on the ring, that is, corresponding integer values are 0 to 2 32 of virtual nodes. The node hash values of multiple virtual nodes are mapped to this ring to form a consistent hash ring. In the embodiments of the present application, the 32-bit hash values of the virtual nodes form a consistent hash ring on the 32-bit integer domain. When using the consistent hash algorithm, even if the number of service nodes in the cluster changes, the missing virtual nodes in the hash ring will evenly distribute the load pressure to other virtual nodes.

[0087] In another alternative implementation manner, a polling mechanism can also be adopted to sequentially allocate the task information of the delay tasks to be managed to the sorting buckets corresponding to each service node in a cyclic manner to achieve load balancing.

[0088] In another alternative implementation, a random allocation mechanism can also be adopted. Using a random algorithm, a service node is randomly selected from the cluster, and the task information of the delay task to be managed is assigned to the sorting bucket corresponding to the randomly selected service node, achieving load balancing through probability distribution.

[0089] Step S203: Store the task information in the form of data key-value pairs into the target sorting bucket, and establish an index for the target sorting bucket based on the data key in the data key-value pairs.

[0090] After determining the target sorting bucket, store the task information in the target sorting bucket, so as to be persistently stored on the disk of the service node. Among them, store the task information in the form of data key-value pairs into the target sorting bucket, and establish an index for the target sorting bucket according to the data key in the data key-value pairs in lexicographical order. Specifically, combine the sorting bucket number of the target sorting bucket and the execution time of the task information as the data key in the data key-value pairs, and use all or part of the information of the task information as the data value in the data key-value pairs; store the data key-value pairs in the target sorting bucket in an index-ordered manner.

[0091] In an application scenario, the data key key can be {bucket_num}execute_time, where bucket_num represents the sorting bucket number and execute_time represents the execution time in the task information; the data value value can be all the information of the task information, that is, the information of all fields in the task information.

[0092] Figure 2b Shows the schematic diagram of the principle architecture of the delay task management method, as Figure 2b As shown, sorting buckets corresponding to each service node in the cluster are constructed on the disk of each service node, thus obtaining multiple sorting buckets, which are the 0th sorting bucket to the nth sorting bucket respectively, where n is an integer greater than 0; the sorting bucket numbers of the multiple sorting buckets are b_0 to b_n respectively. When receiving the task information of the delay task to be managed, perform a hash mapping according to the task information and map it to the sorting bucket (i.e., the target sorting bucket) for storing the task information, and persistently store it on the disk.

[0093] As Figure 2bAs shown, in each sorting bucket, task information is stored in the form of key-value. Taking the nth sorting bucket as an example, in the nth sorting bucket, the key of a record of a certain task information is {b_n}time4, and the value is task14, where b_n is the sorting bucket number of the nth sorting bucket, time4 is the specific execution time of this task information, and task14 is all the information of this task information; the records of all task information in the nth sorting bucket are sorted in an orderly manner according to the key index; according to the index, each record within the key range can be quickly queried within a range.

[0094] Step S204, for each sorting bucket, preload the task information with the execution time within the first time interval in the sorting bucket into the delayed message queue, and compensatorily load the task information with the execution time within the second time interval in the sorting bucket into the delayed message queue.

[0095] In the embodiment of the present application, the task information of delayed tasks is uniformly managed in a manner combining sorting buckets and a delayed message queue, supporting unrestricted task delay time and being able to meet different task delay time requirements. Among them, the task information of each delayed task is stored in multiple sorting buckets corresponding to multiple service nodes, and before the delayed task is about to be executed, the task information about to be executed is added to the delayed message queue and managed by the delayed message queue.

[0096] In each sorting bucket, two processes are started to respectively execute the preloading and compensating loading of the task information of the delayed tasks.

[0097] (1) Preloading process: In each sorting bucket, a preloading process is started. When the preloading process is executed, the task information of the delayed tasks about to be executed in this sorting bucket is preloaded into the delayed message queue. For the convenience of preloading, time is pre-divided into multiple unit time intervals at unit time intervals. Those skilled in the art can set the unit time interval according to actual needs, and no specific limitation is made here.

[0098] Determine the target unit time interval in which the current time is located. Since it is necessary to preload the task information about to be executed, the next unit time interval of the target unit time interval can be used as the first time interval; then the preloading process is looped, and the task information with the execution time within the first time interval in this sorting bucket is preloaded into the delayed message queue by using the preloading process, and after the loading is completed, the task information with the execution time within the first time interval is removed from this sorting bucket, that is, the task information that has been completed loading is removed from this sorting bucket.

[0099] Such as Figure 2bAs shown, for the nth sorting bucket, assuming that the target unit time interval where the current time is located is the interval [time0, time2), and the next unit time interval of this target unit time interval is the interval [time2, time4), the interval [time2, time4) is used as the first time interval. The preloading process is executed in a loop, and the task information in the nth sorting bucket whose execution time is in the interval [time2, time4) is preloaded into the delayed message queue. That is to say, all the task information with the key in the interval [{b_n}time2, {b_n}time4) is loaded into the delayed message queue; and after the loading is completed, all the loaded task information is removed from the nth sorting bucket.

[0100] Specifically, assuming that the unit time interval is 2 hours, with the target unit time interval where the current time is located being the interval [4 pm, 6 pm), and the next unit time interval of this target unit time interval being the interval [6 pm, 8 pm), the interval [6 pm, 8 pm) is used as the first time interval. The preloading process is executed in a loop, and the task information in the nth sorting bucket whose execution time is in the interval [6 pm, 8 pm) is preloaded into the delayed message queue, and after the loading is completed, it is removed from the nth sorting bucket.

[0101] (2) Compensation loading process: Since there are a large number of delayed tasks in Internet services, new task information of delayed tasks will continuously be received and need to be managed. The execution time of the new delayed tasks may miss the preloading opportunity because it is before the first time interval. To avoid missing such task information, the embodiment of the present application also sets a compensation loading mechanism on the basis of the preloading mechanism, so as to compensate and load the task information in the sorting bucket whose execution time is in the second time interval into the delayed message queue. Among them, the second time interval is different from the first time interval. For example, the second time interval is adjacent to the first time interval and the second time interval is before the first time interval.

[0102] A compensation loading process is started for each sorting bucket, and the compensation loading process is executed to compensate and load the task information of the delayed tasks with near execution time in this sorting bucket into the delayed message queue. Among them, the current time is used as the start time of the second time interval, and the end time of the target unit time interval where the current time is located is used as the end time of the second time interval to construct the second time interval; the compensation loading process is executed in a loop, and the task information in the sorting bucket whose execution time is in the second time interval is compensated and loaded into the delayed message queue by using the compensation loading process, and after the loading is completed, the task information whose execution time is in the second time interval is removed from the sorting bucket, that is to say, the task information that has been completed loading is removed from this sorting bucket.

[0103] Such asFigure 2b As shown in the figure, for the nth sorting bucket, assuming that the target unit time interval where the current time cur_time is located is the [time0, time2) interval, the current time cur_time is used as the start time of the second time interval, and the end time time2 of the target unit time interval where the current time cur_time is located is used as the end time of the second time interval. The constructed second time interval is the [cur_time, time2) interval. The compensation loading process is executed in a loop, and the task information with the execution time in the [cur_time, time2) interval in the nth sorting bucket is compensated and loaded into the delay message queue. That is to say, all the task information with the key in the [{b_n}cur_time, {b_n}time2) interval is loaded into the delay message queue; and after the loading is completed, all the loaded task information is removed from the nth sorting bucket.

[0104] Specifically, assuming that the unit time interval is 2 hours, the current time is 5:30 pm, and the target unit time interval where the current time is located is the [4 pm, 6 pm) interval, the constructed second time interval is the [5:30 pm, 6 pm) interval. The compensation loading process is executed in a loop, and the task information with the execution time in the [5:30 pm, 6 pm) interval in the nth sorting bucket is compensated and loaded into the delay message queue, and is removed from the nth sorting bucket after the loading is completed.

[0105] Step S205, when the current time reaches the execution time of any task information in the delay message queue, the consumer pulls any task information from the delay message queue and executes it.

[0106] When the current time reaches the execution time of any task information in the delay message queue, the consumer pulls the any task information from the delay message queue and executes it according to the any task information. After the execution is successful, the task information is deleted from the delay message queue. As Figure 2b shown, the number of consumers can be multiple, which can effectively ensure the efficient execution of a large number of delayed tasks when their execution times arrive.

[0107] Among them, the delayed message queue can be implemented using a multi-level time wheel, Redis sorted set (such as zset), etc., to construct an efficient and reliable delayed message queue. The multi-level time wheel divides time into multiple levels, such as hours, minutes, seconds, etc. Each level corresponds to a time wheel. By combining multiple time wheels, the task information of the delayed task gradually descends from the high-level time wheel to the low-level time wheel until it is triggered for execution. Through the multi-level time wheel, delayed tasks with longer task delay times can be managed more efficiently without occupying too much memory. For the Redis sorted set, the timestamp of the execution time in the task information is used as the score of the sorted set. Consumers can use a polling mechanism to periodically scan the scores of each element in the sorted set to determine and execute the delayed tasks that need to be executed.

[0108] Optionally, for a delayed task that fails to execute, its execution time can be updated and a preset number of retry attempts can be set. When the current time reaches its updated execution time, the consumer pulls it from the delayed message queue and retries. If the number of retry attempts reaches the preset number of retry attempts and the delayed task still fails to execute successfully, the task information of the delayed task can be added to the execution failure list so that the operation and maintenance personnel can know which delayed tasks have failed to execute and analyze the reasons for their failure.

[0109] In addition, the embodiment of the present application also provides the ability to dynamically adjust the number of service nodes in the cluster. According to the resource usage of each service node in the cluster, the number of service nodes in the cluster is dynamically adjusted. The resource usage of the service node can include the usage of hardware resources such as the CPU, GPU, memory, and disk of the service node. When the resource usage of each service node indicates that the hardware resources of the service nodes in the cluster are insufficient, the number of service nodes in the cluster can be increased to achieve horizontal expansion; when the resource usage of each service node indicates that the idle rate of the hardware resources of the service nodes in the cluster exceeds the preset threshold, the number of service nodes in the cluster can also be reduced. For the service node that needs to be destroyed, it is necessary to wait until the task information in the sorting bucket corresponding to the service node is loaded into the delayed message queue before destroying it, so as to ensure that the reduction of the number of service nodes will not affect the normal management of the delayed tasks.

[0110] According to the delay task management method provided by the embodiments of the present application, the task information of delay tasks is managed in the form of a cluster. Sorting buckets corresponding to each service node are constructed on the disk of each service node. The sorting buckets are used to store the task information of delay tasks. Multiple service nodes work together in the form of a cluster, so that the storage capacity of delay tasks can be horizontally expanded according to the business storage requirements, and the storage of a large number of delay tasks can be conveniently realized without being limited by the storage capacity of a single service node instance. When receiving the task information of a delay task to be managed, hash mapping can be performed according to the task information, and the sorting bucket for storing the task information can be determined from multiple sorting buckets corresponding to multiple service nodes through mapping, realizing the load balancing of multiple service nodes. The task information is stored in the sorting bucket in the form of data key-value pairs, and an index of the sorting bucket is established according to the data key in the data key-value pair, which not only realizes the efficient storage of task information, but also can quickly query each record within the data key range according to the index, realizing the quick query of task information. The task information of delay tasks is uniformly managed in the way of combining sorting buckets and delay message queues, supporting unrestricted task delay time and being able to meet different task delay time requirements. In addition, a preloading process is started for each sorting bucket, and the task information with the execution time within the first time interval in the sorting bucket is preloaded into the delay message queue by using the preloading process, and the task information with the execution time within the second time interval in the sorting bucket is compensated and loaded into the delay message queue by using the compensation loading process, which not only realizes the efficient loading of task information, but also effectively avoids the omission of task information. When a delay task needs to be executed, it is pulled and executed by a consumer from the delay message queue, which can effectively ensure the efficient execution of a large number of delay tasks.

[0111] Figure 3 FIG. shows a structural block diagram of a delay task management system according to an embodiment of the present application, as Figure 3 shown, the system includes: a storage module 310, a loading module 320, and an execution module 330.

[0112] The storage module 310 is adapted to: receive the task information of a delay task to be managed, determine a target sorting bucket from multiple sorting buckets corresponding to multiple service nodes according to the task information, and store the task information in the target sorting bucket.

[0113] The loading module 320 is adapted to: for each sorting bucket, preload the task information with the execution time within the first time interval in the sorting bucket into the delay message queue, and compensate and load the task information with the execution time within the second time interval in the sorting bucket into the delay message queue.

[0114] The execution module 330 is adapted to: when the current time reaches the execution time of any task information in the delay message queue, pull any task information from the delay message queue by the consumer and execute it.

[0115] Further, the system further includes: a construction module 340; the construction module 340 is adapted to: construct a cluster including multiple service nodes; for each service node in the cluster, construct a sorting bucket corresponding to the service node in the disk of the service node.

[0116] Further, the storage module 310 is further adapted to: use the delay task identifier and the execution time in the task information as the information to be processed; perform hash mapping on the information to be processed by using a preset hash algorithm, and map it to the target sorting bucket among the multiple sorting buckets corresponding to the multiple service nodes.

[0117] Further, the storage module 310 is further adapted to: store the task information in the form of data key-value pairs into the target sorting bucket, and establish an index of the target sorting bucket according to the data key in the data key-value pairs.

[0118] Further, the storage module 310 is further adapted to: combine the sorting bucket number of the target sorting bucket and the execution time of the task information as the data key in the data key-value pairs, and use all or part of the information of the task information as the data value in the data key-value pairs; store the data key-value pairs in the target sorting bucket in an index-ordered manner.

[0119] Further, the loading module 320 is further adapted to: determine the target unit time interval in which the current time is located, and use the next unit time interval of the target unit time interval as the first time interval; loop and execute the preloading process, and use the preloading process to preload the task information in the sorting bucket whose execution time is within the first time interval into the delay message queue, and after the loading is completed, remove the task information in the sorting bucket whose execution time is within the first time interval.

[0120] Further, the loading module 320 is further adapted to: use the current time as the start time of the second time interval, and use the end time of the target unit time interval in which the current time is located as the end time of the second time interval, and construct the second time interval; loop and execute the compensation loading process, and use the compensation loading process to compensate and load the task information in the sorting bucket whose execution time is within the second time interval into the delay message queue, and after the loading is completed, remove the task information in the sorting bucket whose execution time is within the second time interval.

[0121] Further, the construction module 340 is further adapted to: dynamically adjust the number of service nodes in the cluster according to the resource usage conditions of each service node in the cluster.

[0122] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments and will not be repeated here.

[0123] According to the delay task management system provided by the embodiments of the present application, the task information of delay tasks is managed in the form of a cluster. A sorting bucket corresponding to each service node is constructed in the disk of each service node. The sorting bucket is used to store the task information of delay tasks. Multiple service nodes work together in the form of a cluster, so that the storage capacity of delay tasks can be horizontally expanded according to the business storage requirements, and the storage of a large number of delay tasks can be conveniently realized without being limited by the storage capacity of a single service node instance. When receiving the task information of a delay task to be managed, hash mapping can be performed according to the task information, and the sorting bucket for storing the task information is determined from multiple sorting buckets corresponding to multiple service nodes through mapping, realizing the load balancing of multiple service nodes. The task information is stored in the sorting bucket in the form of data key-value pairs, and an index of the sorting bucket is established according to the data key in the data key-value pair, which not only realizes the efficient storage of task information, but also can quickly query each record within the data key range according to the index, realizing the quick query of task information. The task information of delay tasks is uniformly managed in the way of combining sorting buckets and delay message queues, supporting unlimited task delay time and meeting different task delay time requirements. In addition, a preloading process is started for each sorting bucket, and the task information in the sorting bucket whose execution time is within the first time interval is preloaded into the delay message queue by using the preloading process, and the task information in the sorting bucket whose execution time is within the second time interval is compensated and loaded into the delay message queue by using the compensation loading process, which not only realizes the efficient loading of task information, but also effectively avoids the omission of task information. When a delay task needs to be executed, it is pulled and executed by a consumer from the delay message queue, which can effectively ensure the efficient execution of a large number of delay tasks.

[0124] The embodiments of the present application provide a non-volatile computer storage medium, and the computer storage medium stores at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform the operations corresponding to the delay task management method in any of the above method embodiments.

[0125] The embodiments of the present application provide a computer program product, and the computer program product includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform the operations corresponding to the delay task management method in any of the above method embodiments.

[0126] Figure 4 The structural schematic diagram of a computing device according to an embodiment of the present application is shown. The specific implementation of the computing device is not limited in the specific embodiments of the present application.

[0127] AsFigure 4 As shown, the computing device may include: a processor 402, a communications interface 404, a memory 406, and a communication bus 408.

[0128] Among them: The processor 402, the communications interface 404, and the memory 406 communicate with each other through the communication bus 408. The communications interface 404 is used to communicate with network elements of other devices such as clients or other servers. The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the foregoing method embodiments for managing delay tasks of the computing device.

[0129] Specifically, the program 410 may include program code, and the program code includes computer operation instructions.

[0130] The processor 402 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the computing device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0131] The memory 406 is used to store the program 410. The memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0132] The program 410 is specifically used to cause the processor 402 to execute the delay task management method in any of the foregoing method embodiments. For the specific implementation of each step in the program 410, reference may be made to the corresponding steps and descriptions in the corresponding units in the foregoing delay task management embodiments, which will not be elaborated here. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the devices and modules described above can refer to the corresponding process descriptions in the foregoing method embodiments, which will not be elaborated here.

[0133] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used in conjunction with the teachings based hereon. The structure required to construct such systems will be apparent from the above description. In addition, embodiments of the present application are not directed to any particular programming language. It should be appreciated that the content of the embodiments of the present application described herein can be implemented using various programming languages, and the description of a particular language above is to disclose the best mode of the embodiments of the present application.

[0134] In the specification provided herein, numerous specific details are set forth. However, it can be understood that embodiments of the present application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0135] Similarly, it should be understood that, in order to streamline the present disclosure and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together in a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that the claimed embodiments of the present application require more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all the features of the single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate embodiment of the present application.

[0136] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and disposed in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be adopted for all the features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) can be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0137] In addition, those skilled in the art can understand that although some embodiments described herein include certain features included in other embodiments rather than other features, the combination of features of different embodiments means that it is within the scope of the embodiments of the present application and forms different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0138] Each component embodiment of the embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present application. The embodiments of the present application can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the embodiments of the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0139] It should be noted that the above embodiments illustrate the embodiments of the present application rather than limit the embodiments of the present application, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The embodiments of the present application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim listing several devices, several of these devices can be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names.

Claims

1. A method for managing delayed tasks, comprising: Receiving task information of a delayed task to be managed, and determining a target sorting bucket from multiple sorting buckets corresponding to multiple service nodes according to the task information, and storing the task information into the target sorting bucket; For each sorting bucket, preloading task information in the sorting bucket whose execution time is within a first time interval into a delayed message queue, and compensating and loading task information in the sorting bucket whose execution time is within a second time interval into the delayed message queue; When the current time reaches the execution time of any task information in the delayed message queue, a consumer pulls the any task information from the delayed message queue and executes it.

2. The method according to claim 1, before receiving the task information of the delayed task to be managed, the method further comprises: Constructing a cluster including multiple service nodes; For each service node in the cluster, constructing a sorting bucket corresponding to the service node in the disk of the service node.

3. The method according to claim 1, the determining a target sorting bucket from multiple sorting buckets corresponding to multiple service nodes according to the task information further comprises: Taking the delayed task identifier and execution time in the task information as information to be processed; Performing hash mapping on the information to be processed by using a preset hash algorithm, and mapping it to a target sorting bucket among multiple sorting buckets corresponding to multiple service nodes.

4. The method according to any one of claims 1-3, the storing the task information into the target sorting bucket further comprises: Storing the task information in the form of a data key-value pair into the target sorting bucket, and establishing an index of the target sorting bucket according to the data key in the data key-value pair.

5. The method according to claim 4, the storing the task information in the form of a data key-value pair into the target sorting bucket further comprises: Combining the sorting bucket number of the target sorting bucket and the execution time of the task information as the data key in the data key-value pair, and taking all or part of the information of the task information as the data value in the data key-value pair; Storing the data key-value pairs in the target sorting bucket in an index-ordered manner.

6. The method according to any one of claims 1-5, the preloading task information in the sorting bucket whose execution time is within a first time interval into a delayed message queue further comprises: Determining a target unit time interval in which the current time is located, and taking the next unit time interval of the target unit time interval as the first time interval; Circularly executing a preloading process, and using the preloading process to preload task information in the sorting bucket whose execution time is within the first time interval into the delayed message queue, and after the loading is completed, removing the task information whose execution time is within the first time interval from the sorting bucket.

7. The method according to any one of claims 1-6, the compensating and loading task information in the sorting bucket whose execution time is within a second time interval into a delayed message queue further comprises: Construct the second time interval by using the current time as the start time of the second time interval and the end time of the target unit time interval in which the current time is located as the end time of the second time interval; Loop to execute the compensation loading process, use the compensation loading process to compensate and load the task information with the execution time within the second time interval in the sorting bucket into the delay message queue, and after the loading is completed, remove the task information with the execution time within the second time interval from the sorting bucket.

8. The method according to any one of claims 1-7, the method further comprising: Dynamically adjust the number of service nodes in the cluster according to the resource usage of each service node in the cluster.

9. A delay task management system, comprising: A storage module, adapted to receive the task information of the delay task to be managed, determine a target sorting bucket from a plurality of sorting buckets corresponding to a plurality of service nodes according to the task information, and store the task information into the target sorting bucket; A loading module, adapted to preload the task information with the execution time within the first time interval in the sorting bucket into the delay message queue for each sorting bucket, and compensate and load the task information with the execution time within the second time interval in the sorting bucket into the delay message queue; An execution module, adapted to when the current time reaches the execution time of any task information in the delay message queue, pull the any task information from the delay message queue by a consumer and execute it.

10. A computing device, comprising: A processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the delay task management method according to any one of claims 1-8.

11. A computer storage medium, in which at least one executable instruction is stored, and the executable instruction causes the processor to perform the operations corresponding to the delay task management method according to any one of claims 1-8.

12. A computer program product, comprising at least one executable instruction, and the executable instruction causes the processor to perform the operations corresponding to the delay task management method according to any one of claims 1-8.

Images