Task scheduling method, electronic device, storage medium and program product

Abstract

The present disclosure provides a task scheduling method, an electronic device, a readable storage medium and a computer program product. The task scheduling method of the present disclosure includes: obtaining access rights to a task queue; in response to obtaining access rights to the task queue, obtaining tasks to be executed in the task queue; dividing the tasks to be executed into multiple subtasks to be executed; distributing each subtask to be executed to an execution node for execution, and obtaining the subtask execution results of each execution node; aggregating the subtask execution results of each execution node into a final result, and releasing the access rights to the task queue.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a task scheduling method, electronic equipment, storage medium, and program product. Background Art

[0002] Currently, many application scenarios use distributed task queues to dispatch tasks. In the distributed task queue architecture, a task can often only be consumed by one consumer. If the task has a large amount of data and the consumer's machine performance is weak, it is easy to cause memory overflow when executing the task, resulting in abnormal or failed task execution. Summary of the invention

[0003] The present disclosure provides a task scheduling method, electronic equipment, storage medium and program product.

[0004] According to one aspect of the present disclosure, a task scheduling method is provided, including: obtaining access rights to a task queue; in response to obtaining access rights to the task queue, obtaining to-be-executed tasks in the task queue; dividing the to-be-executed tasks into multiple to-be-executed subtasks; distributing each to-be-executed subtask to an execution node for execution, and obtaining subtask execution results of each of the execution nodes; aggregating the subtask execution results of each of the execution nodes into a final result, and releasing access rights to the task queue.

[0005] According to at least one embodiment of the task scheduling method of the present disclosure, obtaining access rights to a task queue includes: requesting to set a target key in a target database; and in response to successful setting of the target key, determining that access rights to the task queue have been obtained.

[0006] According to the task scheduling method of at least one embodiment of the present disclosure, the task queue is configured to allow access by a node holding the target key.

[0007] According to the task scheduling method of at least one embodiment of the present disclosure, the target database is configured to: in response to a request to set the target key, query the target key in the target database; and in response to the target key not existing in the target database, save the target key in the target database and return a message indicating that the target key is set successfully.

[0008] According to the task scheduling method of at least one embodiment of the present disclosure, releasing the access right of the task queue includes: requesting deletion of the target key in the target database; and determining that the access right of the task queue has been released in response to the target key being deleted.

[0009] According to at least one embodiment of the task scheduling method disclosed herein, it is characterized in that the task data set is divided into multiple sub-tasks to be executed, including: obtaining the data volume of the task to be executed, the number of the execution nodes and the number of the task queues; determining the single-node data volume according to the data volume of the task to be executed, the number of the execution nodes and the number of the task queues; and dividing the data volume of the task to be executed into multiple portions of the single-node data volume, and creating a sub-task to be executed based on each portion of the single-node data volume.

[0010] According to the task scheduling method of at least one embodiment of the present disclosure, each subtask to be executed is distributed to an execution node for execution, including: obtaining the queue identifier of the task queue and determining the specific identifier corresponding to the queue identifier; and searching for the execution node with the specific identifier, and distributing the subtask to be executed to the execution node with the specific identifier for execution.

[0011] According to the task scheduling method of at least one embodiment of the present disclosure, the to-be-executed task is configured to be added to the task queue with the least number of tasks when generated.

[0012] According to the task scheduling method of at least one embodiment of the present disclosure, the task queue is configured as follows: when the consumption rate of the task to be executed is less than a consumption rate threshold, a new task queue is added; and when the peak number of tasks to be executed is less than a peak threshold, the added task queue is removed.

[0013] According to at least one embodiment of the task scheduling method of the present disclosure, it also includes: obtaining intermediate data generated by the execution node executing the subtask to be executed; and storing the intermediate data in a cache queue; wherein the cache queue is configured to allow any node to access it.

[0014] According to the task scheduling method of at least one embodiment of the present disclosure, the task to be executed includes a data verification task, the subtask to be executed includes a data verification subtask, and the subtask execution result includes a verification subresult; the subtask execution results of each of the execution nodes are summarized as a final result, including: obtaining normal results and abnormal results in the verification subresult; determining normal data based on the normal results, and counting each of the normal data as the final result; or determining abnormal data based on the abnormal results, and counting each of the abnormal data as the final result; or determining normal data based on the normal results, and determining abnormal data based on the abnormal results, and integrating the normal and abnormal data into the final result.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the task scheduling method of any embodiment of the present disclosure.

[0016] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the task scheduling method of any embodiment of the present disclosure is implemented.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the task scheduling method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0019] Figure 1 It is a flowchart of a task scheduling method according to an embodiment of the present invention.

[0020] Figure 2 It is a flowchart of a task scheduling method according to an embodiment of the present invention.

[0021] Figure 3 It is a flowchart of a task scheduling method according to an embodiment of the present invention.

[0022] Figure 4 It is a flowchart of a task scheduling method according to an embodiment of the present invention.

[0023] Figure 5 It is a flowchart of a task scheduling method according to an embodiment of the present invention.

[0024] Figure 6 It is a flowchart of a task scheduling method according to an embodiment of the present invention.

[0025] Figure 7 It is a flowchart of a task scheduling method according to an embodiment of the present invention.

[0026] Figure 8 It is a flowchart of a task scheduling method according to an embodiment of the present invention.

[0027] Fig. 9 It is a schematic block diagram of the structure of a task scheduling device according to an embodiment of the present invention.

[0028] Fig.10 It is a schematic block diagram of the structure of an electronic device equipped with a task scheduling device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0030] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] At present, related fields usually use distributed queues to process task scheduling, such as a large number of tasks that are executed asynchronously for a long time, large-scale real-time tasks, scheduled tasks, etc. However, the architecture of distributed queues can often only dispatch one task to one consumer for execution. In cases where the task data volume is large and the consumer's machine performance is weak, it is easy to cause memory overflow, resulting in abnormal or failed task execution.

[0032] The present disclosure provides a task scheduling method M100, which can schedule a task in a distributed queue to multiple consumers for execution, so as to distribute the memory pressure to the multiple consumers and reduce the risk of memory overflow during task execution.

[0033] Figure 1 FIG. 1 is a schematic diagram showing the overall process of a task scheduling method M100 according to an embodiment of the present disclosure. Figure 1 The method shown includes steps S110 to S190. The method can be executed by electronic devices such as mobile phones and tablet computers.

[0034] Specifically, Figure 1 The methods shown include: S110: Obtain access rights to the task queue.

[0035] The task queue may refer to a distributed queue, such as a distributed queue based on the Celery architecture. In addition to distributed queues, it may also be applicable to other types of asynchronous task queues, such as the Asynq queue based on the Go library, which are not listed here.

[0036] The execution subject of step S110 is a consumer / worker in the distributed system. This article takes the consumer as an example for introduction. The processes and steps that can be executed by the consumer can also be replaced by workers, without limitation. The consumer / worker in the distributed system refers to specific electronic devices, such as mobile phones, computers, etc., and a consumer in the distributed system is referred to as a node below.

[0037] S130 . In response to obtaining the access right to the task queue, obtaining the to-be-executed tasks in the task queue.

[0038] Among them, the pending tasks are tasks generated by producers in the distributed system and waiting to be executed. After being generated, the pending tasks are placed in the task queue, waiting for the nodes in the distributed system to obtain the tasks and consume them. The task queue architecture of the distributed system is configured to allow only one node to access at the same time. Based on this, in the embodiment of the present disclosure, the task queue is configured with access rights, and the idle online nodes in the distributed system can all try to obtain access rights to the task queue. The nodes that successfully compete for the access rights can access the task queue and obtain the pending tasks in the task queue.

[0039] When a node obtains the right to access a task queue, it can initiate an access request to obtain the right to access the task queue. In one example, a pre-configured rights management service (or program, system, etc., which is described below using the service as an example) processes the access request and determines whether the node initiating the request is eligible to obtain the right to access, and returns a corresponding determination result, which includes eligible access and unqualified access. If the returned determination result indicates that the node is eligible to obtain the right to access the task queue, a response to obtain the right to access the task queue is triggered at the node, so that the node executes the process of accessing the task queue and obtaining the tasks to be executed in the task queue.

[0040] In some embodiments of the present disclosure, the pre-configured rights management service may be a service based on a Redis library. There is a key in the Redis library that is used to indicate the state of the queue access right, and this key is recorded as the target key. When a node wants to obtain access rights to a task queue, the node initiates an access request to set the target key in the Redis library.

[0041] If the target key already exists in the Redis library, indicating that other nodes have already competed for the target key, the Redis library returns a definite result indicating that the node is not eligible to access the task queue. At this point, if there are other task queues, this node can try to initiate a request to obtain access rights to other task queues. If there are no other task queues, or this node also fails to compete for access rights to other task queues, you can wait for a period of time and then try to initiate a request to obtain access rights to the task queue again, until you can obtain access rights to the task queue, triggering the response of step S130.

[0042] If the target key can be successfully set in the Redis library, it means that no other node currently occupies the access right to the task queue. At this time, the current node successfully obtains the access right to the task queue by successfully setting the target key in the Redis library. Correspondingly, if the target key can be successfully set in the Redis library, the Redis library returns a determination result indicating that the node is eligible to access the task queue. The node triggers a response of successfully obtaining the access right to the task queue based on this determination result. For the sake of distinction, this article refers to the node that obtains the access right to the task queue in the distributed system as the master node.

[0043] In some embodiments of the present disclosure, the master node is configured to obtain only one task to be executed in the task queue when the access right to the task queue is obtained each time. In this way, the risk of repeated execution of the task to be executed can be avoided. In addition, in the subsequent steps, the task to be executed is divided into multiple subtasks to be executed. If the master node obtains multiple tasks to be executed at one time, for example, the task to be executed includes task A and task B, task A is divided into multiple subtasks a, and task B is divided into multiple subtasks b, then when summarizing the execution results of the subtasks, it is possible to mistakenly summarize the execution results of subtask a as the execution results of subtask b of task B, resulting in errors in the obtained execution results. Therefore, the master node only obtains one task to be executed in the task queue each time, which is also conducive to ensuring the accuracy of the task execution results. In addition, only one task to be executed is obtained each time for processing, which has less burden pressure on the memory and can avoid causing memory overflow. When multiple tasks to be executed are included in the task queue, the task to be executed that is first put into the task queue can be preferentially obtained for processing according to the first-in-first-out processing logic.

[0044] S150: Split the task to be executed into multiple subtasks to be executed.

[0045] In some embodiments of the present disclosure, the task to be executed involves multiple entities that need to be processed, and each entity is processed according to various corresponding processing rules. In this case, the processing of an entity can be divided as the smallest segmentation unit, and each segmented subtask to be executed includes a processing task of at least one entity. For example, if the task to be executed includes a data inspection task for 1,000 entities, the task to be executed can be divided into 1,000 subtasks to be executed, and each subtask to be executed involves a data inspection of 1 entity; or, the task to be executed can be divided into 500 subtasks to be executed, and each subtask to be executed involves a data inspection of 2 entities; or each subtask to be executed can also involve data inspections of 3, 4, 5 or more entities, which are not listed here one by one.

[0046] In some embodiments of the present disclosure, the task to be executed involves multiple different processing rules. In this case, the processing task based on one processing rule can be segmented as the minimum segmentation unit, and each segmented subtask to be executed includes a processing task corresponding to at least one processing rule. For example, the task to be executed includes performing relevant processing on an entity according to rules 1, 2, and 3 respectively, and there is no mutual dependence between the processing results based on rules 1, 2, and 3. Based on this, the task to be executed can be segmented into three subtasks to be executed, and each subtask to be executed involves a processing task based on one processing rule.

[0047] In addition to the above-mentioned implementation, the partitioning scheme of the tasks to be executed can be configured based on the specific business rules corresponding to the tasks to be executed, which is not limited here. The purpose of the implementation of the present disclosure to partition the tasks to be executed into multiple subtasks to be executed is to convert the execution mode of processing the tasks to be executed by one node into processing multiple subtasks to be executed by multiple nodes, so that the amount of task data processed by each node is smaller than the amount of data of the tasks to be executed, avoiding memory overflow. Based on this, the partitioning of the tasks to be executed only needs to satisfy that the amount of data of the subtasks to be executed after the partitioning is less than the amount of data of the tasks to be executed, and each subtask to be executed is an independently executable task.

[0048] S170: Distribute each subtask to be executed to an execution node for execution, and obtain the subtask execution result of each execution node.

[0049] Among them, the execution node is a node in the distributed system used to execute the subtasks to be executed. The execution node can be all the remaining online nodes in the distributed system except the master node, or it can be a part of all the remaining online nodes except the master node. In other words, the nodes of the distributed system can be judged as follows: whether it is a master node; and whether it is an online node. If a node in the distributed system is not a master node and is an online node, the node can be selected as an execution node. When a node is selected as an execution node, it means that the node can undertake tasks assigned by the master node, such as executing subtasks to be executed.

[0050] In addition to the two selection criteria of whether it is a master node and whether it is an online node, the selection criteria for the execution node can also include: whether it has a specific identification, whether it has sufficient computing power, whether it has sufficient computing resources, etc. It can be configured based on specific business conditions and is not limited here.

[0051] The subtask execution result obtained by the execution node when executing the subtask to be executed can be returned to the master node that dispatched the subtask to be executed, or stored in a storage space accessible to the master node. There is no restriction here, as long as the master node can obtain the subtask execution result.

[0052] Each subtask to be executed is distributed to an execution node for execution, which means that one subtask to be executed is executed by one execution node to ensure that each subtask to be executed can be executed by the execution node. At this time, one execution node can be assigned to one subtask to be executed, or it can be assigned to more than one subtask to be executed, and there is no restriction here. Among them, the number of execution nodes is at least 2 to ensure that at least one execution node shares the memory pressure of another execution node, that is, even if an execution node is assigned to multiple subtasks to be executed, the total data volume of multiple subtasks to be executed is still less than the data volume of the task to be executed, ensuring that the memory pressure of one execution node can be reduced and the risk of memory overflow can be reduced.

[0053] S190: Aggregate the subtask execution results of each execution node into a final result, and release the access right to the task queue.

[0054] Among them, if there is no error in the task execution, the final result obtained by summarizing the subtask execution results of each execution node should be the same as the task execution result obtained by only one node executing the task to be executed before the split. In this process, the method of obtaining the subtask execution results separately by multiple execution nodes and then summarizing them can not only distribute the memory pressure of one node executing the task to be executed to multiple execution nodes, reducing the risk of memory overflow during the execution of each subtask to be executed; it can also use multiple execution nodes to execute tasks synchronously to shorten the time to obtain the final result and improve the efficiency of task execution.

[0055] When the final result is obtained, it means that the current task to be executed has been completed, and the next task to be executed in the task queue can be executed at this time. In some embodiments of the present disclosure, each time a node competes for the access right to the task queue, it only processes one task to be executed in the task queue. When the task to be executed is completed, the master node releases the access right to the task queue. When the access right to the task queue is released, any online node in the distributed system can try to obtain the access right to the task queue to compete for the access right. The node that successfully competes for the access right serves as the new master node to process the next task to be executed in the task queue.

[0056] In summary, in the disclosed embodiments, the task to be executed can be divided into multiple subtasks to be executed, which are distributed to multiple nodes for execution, and then the subtask execution results of each node are summarized to obtain the final result, so that the memory pressure can be distributed to multiple nodes in the process of obtaining the final result, reducing the risk of memory overflow when a single node executes the task. In addition, multiple nodes do not obtain tasks from the task queue individually, but compete for the access right to the task queue. The master node that obtains the access right to the task queue obtains the task to be executed from the task queue, and divides the task to be executed into multiple subtasks to be executed and distributes them to multiple execution nodes for execution, so that the risk of the same task in the task queue being repeatedly executed can be avoided.

[0057] Regarding step S110, in some embodiments of the present disclosure, it may include: Figure 2 Steps S1101 to S1102 are shown.

[0058] S1101. Request to set a target key in the target database.

[0059] S1102: In response to the target key being set successfully, determining that the access right to the task queue has been acquired.

[0060] The target key is a preconfigured key that can be set in the target database using a specific command. Specifically, in some embodiments of the present disclosure, the task queue is configured to allow access by nodes holding the target key. In addition, the target database is configured to query the target key in the target database in response to a request to set the target key; and in response to the absence of the target key in the target database, save the target key in the target database, and return a successful target key setting. If a node successfully sets the target key in the target database, it is considered that the node holds the target key. Based on the above configuration of the target database and the task queue, if a node wants to obtain a task in the task queue, it must successfully set the target key in the target database. In addition, the target key can only be successfully set in the target database if there is no target key in the target database. In this way, it can be ensured that only one node that successfully sets the target key in the target database can access the task queue at the same time, so as to avoid the same task to be executed in the task queue being obtained by multiple nodes and increasing the risk of the task being repeatedly executed.

[0061] Specifically, if a node wants to access the task queue, the node needs to initiate a request to the target database, request to set the target key in the target database, and trigger a response based on the result returned by the target database. If the result returned by the target database indicates that the target key is set successfully, a response of successful target key setting is triggered, and it is determined that the access right to the task queue has been obtained. If the result returned by the target database indicates that the target key setting fails, a response of target key setting failure is triggered. In one example, in response to the target key setting failure, a request to set the target key is retried after waiting for a predetermined time.

[0062] Take the target database as Redis library as an example. Assuming the target key is "my_lock", when a node tries to obtain access to the task queue, it can request to set the "my_lock" key in the Redis library. If the Redis library returns "ok", it means that the node has successfully set and holds the "my_lock" key; if the Redis library returns "nil", it means that the "my_lock" key has been held by other nodes.

[0063] In one example, when a node successfully sets the target key, the node can identify itself as the master node and share the master node information to the router so that other nodes can identify the node as the master node. In this way, the master node can be identified by other master nodes, preventing the master node from being assigned to a subtask to be executed by other master nodes.

[0064] In one example, the target key has a corresponding target value, and the target key and the corresponding target value are a pair of key-value pairs, recorded as a target key-value pair. Among them, the target value can be based on the value specified in the content of the target key setting request sent by the node, and can be a value automatically generated by the target database according to the configured rules when generating the target key, and there is no limitation here. In one example, the target value is used to identify the holder of the target key. For example, when a node sets the target key, the target value is set to the identity ID of this node. In this way, when this node accesses the task queue, it can be determined whether this node holds the target key by judging whether the identity ID of this node is the same as the target value.

[0065] Regarding releasing the access right to the task queue in step S190, in some embodiments of the present disclosure, it may include: Figure 3 Steps S1930 to S1950 are shown.

[0066] S1930. Request to delete the target key in the target database.

[0067] S1950: In response to the target key being deleted, determining that the access right to the task queue has been released.

[0068] After the node completes the processing of the task to be executed, it will request to delete the target key in the target database. When the target key is deleted in the target database, it means that the node has released the access right to the task queue. At this time, the target key does not exist in the target database, so that other nodes can compete for the access right to the task queue by requesting to set the target key in the target database. In some embodiments of the present disclosure, the subtask results are summarized as the final result as a signal that the processing of the task to be executed is completed. For example, when the final result is generated, the above step S1930 is triggered. In other embodiments, the acquisition of each subtask result can also be used as a signal that the processing of the task to be executed is completed. For example, when the results of each subtask are obtained, the above step S1930 is triggered, and the process of summarizing the results of each subtask into the final result is triggered at the same time.

[0069] In one example, the target key is set with an expiration time. When the current time reaches the expiration time, the target database will delete the target key even if it has not received a request to delete the target key. In this way, even if the master node crashes while holding the target key, it can ensure that the target key can be released.

[0070] In one example, based on the expiration time set for the target key, it is possible to periodically check whether the master node crashes. If the master node does not crash, a renewal operation is triggered before the expiration time is about to arrive, thereby extending the expiration time of the target key.

[0071] For example, the master node is configured to periodically feedback its own health status when holding the target key. The health status of the master node is used to indicate whether the master node has crashed or gone offline. If the health status reported by the master node indicates that the master node has not crashed or gone offline, a renewal operation is triggered to extend the expiration time; if the health status reported by the master node indicates that the master node has crashed or gone offline, or the target database has not received the health status feedback from the master node within the specified time, the renewal operation will not be triggered, and the target key will be deleted when the expiration time is reached, so that the access rights to the task queue can be released immediately.

[0072] Regarding step S150, in some embodiments of the present disclosure, it may include: Figure 4 Steps S1501 to S1503 are shown.

[0073] S1501. Obtain the data volume of the task to be executed, the number of execution nodes, and the number of task queues.

[0074] S1502: Determine the single node data volume based on the data volume of the task to be executed, the number of execution nodes, and the number of task queues. S1503: Divide the data volume of the task to be executed into multiple single-node data volumes, and create a subtask to be executed based on each single-node data volume.

[0075] The above-mentioned scheme of step S1501 to step S1503 is applicable to the case where each execution node executes one subtask to be executed. It should be noted that the implementation method of the present disclosure does not limit one execution node to be assigned only one subtask to be executed, and the scheme of assigning multiple subtasks to be executed to one execution node is a scheme that can be obtained based on the inspiration of the specific embodiment provided by the present disclosure.

[0076] When each execution node executes one subtask to be executed, the strategy of evenly allocating the task volume is used to configure the amount of data allocated to each execution node. Specifically, assuming that the current number of execution nodes is P and the current number of task queues is Q, then based on the evenly allocating strategy, each task queue is allocated Based on this, assuming that the amount of data in a task to be executed in one of the task queues is M, by assigning execution nodes for processing, then based on the average allocation strategy, the average amount of data allocated to each execution node is , recorded as the single node data volume. In this way, each execution node should process In the case of data volume, the amount of data contained in each subtask to be executed is determined as , that is, the single-node data volume is used as the data volume of each subtask to be executed. In this way, when the data volume of each subtask to be executed is determined, the data volume of the task to be executed can be divided into multiple single-node data volumes, and a subtask to be executed is created based on each single-node data volume. In this way, the memory pressure in the process of executing the tasks to be executed can be evenly distributed to multiple execution nodes, and the time required for each execution node to execute its own subtask to be executed can be relatively close, shortening the time taken for multiple execution nodes to execute tasks synchronously.

[0077] Regarding step S170, in some embodiments of the present disclosure, it may include the following steps: Figure 5 Steps S1701 to S1702 are shown.

[0078] S1701. Obtain a queue identifier of a task queue, and determine a specific identifier corresponding to the queue identifier.

[0079] S1702: Search for an execution node with a specific identifier, and distribute the subtask to be executed to the execution node with the specific identifier for execution.

[0080] Combined with the average allocation strategy mentioned above, when the number of task queues is 2 or more, allocate An execution node processes the tasks in the queue. In this case, an execution node is specifically used to process the tasks in a task queue. Based on this, step S1701 to step S1702 utilizes the queue identification of the task queue and the specific identification of the execution node to set up the corresponding relationship between the task queue and the execution node, so that when the master node distributes the subtasks to be performed of a task queue, only the subtasks to be performed are distributed to the execution node dedicated to this task queue.

[0081] For example, suppose the task queue includes queue A and queue B, queue A has queue ID "A", queue ID "A" corresponds to special ID "a", and queue B has queue ID "B", queue ID "B" corresponds to special ID "b". When there are execution nodes, the execution nodes in the online The execution node is given a special identifier "a", and the other An execution node is assigned a special identifier "b". If the task to be executed is obtained from queue A in the master node and divided into multiple sub-tasks to be executed, then when the master node distributes the multiple sub-tasks to be executed in queue A, it will only be distributed to the execution node with the special identifier "a" for execution. In this way, the execution node with the special identifier "b" will not be distributed to the sub-tasks to be executed in queue A, which can avoid the execution node with the special identifier "b" being used to execute the sub-tasks to be executed in queue A, causing the sub-tasks to be executed in queue B to be not processed in time, resulting in a backlog of tasks in queue B. Similarly, the execution node with the special identifier "a" will not be distributed to the sub-tasks to be executed in queue B, which can avoid the execution node with the special identifier "a" being used to execute the sub-tasks to be executed in queue B, causing the sub-tasks to be executed in queue A to be not processed in time, resulting in a backlog of tasks in queue A.

[0082] In summary, in some embodiments of the present disclosure, the queue identifier of the task queue and the specific identifier of the execution node establish a correspondence between the task queue and the execution node, and an execution node specifically for processing the tasks therein is assigned to the task queue, so as to avoid the problem of task backlog in a task queue due to insufficient allocation of execution nodes.

[0083] Based on the fact that the number of task queues can be 2 or more than 2, in some embodiments of the present disclosure, the tasks to be executed are configured to be added to the task queue with the least number of tasks when they are generated. In this way, the tasks to be executed can be placed in a relatively idle task queue, reducing the situation where the task queue appears to be backlogged.

[0084] In one example, a task quantity monitoring service is configured for a distributed system to monitor the number of tasks in each task queue. When a producer in the distributed system generates a task to be executed, the number of tasks in each task queue is obtained from the task quantity monitoring service, and the task queue with the least number of tasks is determined, and the newly generated task to be executed is added to the task queue with the least number of tasks.

[0085] In some embodiments of the present disclosure, the number of task queues can be adjusted dynamically. When there is a backlog of tasks in the task queue, additional task queues are added to increase the efficiency of task consumption. When the task queue is idle, a portion of the task queue is removed to reduce the resources consumed by maintaining the task queue.

[0086] Among them, whether there is a backlog in the task queue can be determined by monitoring the consumption rate of tasks in the task queue. The consumption rate of tasks refers to the ratio of the number of tasks reduced in the task queue within a certain period of time to the time. In an example, according to the historical records of changes in the number of tasks in the task queue recorded by the task quantity monitoring service, the number of tasks reduced in the task queue within a certain period of time can be obtained, and then the consumption rate of the task can be calculated. If the consumption of the task is less than the consumption rate threshold, it means that the progress of task consumption is slow, that is, it is determined that there is a backlog of tasks, so a new task queue can be added.

[0087] Whether a task queue is idle can be determined by monitoring the peak number of tasks in the task queue. The peak number of tasks refers to the maximum number of tasks in the task queue within a certain period of time. In one example, the peak number of tasks can be calculated based on the historical records of changes in the number of tasks in the task queue recorded by the task number monitoring service. If the peak number of tasks is less than the peak threshold, it means that the task queue is idle, so a part of the task queue can be removed.

[0088] In some embodiments of the present disclosure, one task queue that cannot be removed is configured as a basic queue. In addition to the basic queue, other task queues can be added or removed when conditions are met.

[0089] In one example, the task queue is configured such that when the consumption rate of the tasks to be executed is less than the consumption rate threshold, a new task queue is added. If the number of current task queues is greater than 1, a new queue can be added when the consumption rate of the tasks to be executed in any task queue is less than the consumption rate threshold.

[0090] In one example, the additional task queue is configured to be removed when the peak value of the number of tasks to be executed is less than the peak value threshold. Specifically, when the peak value of the number of tasks to be executed in any task queue is less than the peak value threshold, a newly added task queue is randomly selected to be removed. When a task queue is selected as the queue to be removed, new tasks to be executed will not be added to this task queue, and while waiting for all tasks to be executed in the task queue to be completed, the task queue will be removed, including canceling the processing thread of the task queue, removing the system resources of the task queue, etc.

[0091] In some embodiments of the present disclosure, the task to be executed may be a data verification task. The data verification task is to verify the correctness or consistency of each entity data in the entity data set that needs to be verified according to a pre-set verification rule. Based on this, the subtasks to be executed obtained by dividing the task to be executed are recorded as data verification subtasks, and each data verification subtask is used to compare whether the data of an entity in different systems is consistent or correct. The subtask execution result obtained by executing the data verification subtask is recorded as a verification subresult, and a verification subresult is used to indicate whether an entity data is normal or abnormal. In this way, the normality or abnormality of the overall data in the entity data set can be obtained by summarizing the various verification subresults.

[0092] Specifically, for ease of explanation, the step S190 in which the subtask execution results of each execution node are aggregated into a final result is recorded as step S1910 (see Figure 3 ). Regarding step S1910, in some embodiments of the present disclosure, it may include: Figure 6 Steps S1911 to S1914 are shown.

[0093] S1911. Obtain normal results and abnormal results in the syndrome results.

[0094] If the syndrome result indicates that the entity data is correct or consistent, the syndrome result is a normal result; otherwise, if the syndrome result indicates that the entity data is not correct or consistent, the syndrome result is an abnormal result.

[0095] S1912. Determine normal data based on the normal result, and count each normal data as the final result.

[0096] The solution of step S1912 is to count only the normal data obtained by verification as the final result. When the verification result corresponding to an entity data is a normal result, the entity data is determined as normal data. Based on this, the corresponding normal data can be obtained according to each normal result, and finally all the normal data are counted to obtain the final result.

[0097] S1913. Determine abnormal data based on the abnormal result, and count each abnormal data as the final result.

[0098] The solution of step S1913 is to count only the abnormal data obtained by verification as the final result. When the verification result corresponding to an entity data is an abnormal result, the entity data is determined as abnormal data. Based on this, the corresponding abnormal data can be obtained according to each abnormal result, and finally all the abnormal data are counted to obtain the final result.

[0099] S1914. Determine normal data based on the normal results, and determine abnormal data based on the abnormal results, and integrate the normal and abnormal data into a final result.

[0100] The solution of step S1914 is to statistically check the normal data and abnormal data as the final result, so that the final result can simultaneously reflect the correct data and abnormal data in the entity data set.

[0101] After step S1911, any one of step S1901, step S1902 and step S1903 may be selectively executed without limitation herein.

[0102] In some embodiments of the present disclosure, the task scheduling method M100 may also include: Figure 7 Step S180 is shown.

[0103] S180, caching intermediate data generated when executing the subtask to be executed.

[0104] Specifically, regarding step S180, in some embodiments of the present disclosure, it may include: Figure 8 Steps S1801 to S1802 are shown.

[0105] S1801. Obtain the intermediate data generated by the execution node when executing the subtask to be executed.

[0106] Among them, intermediate data refers to temporary data generated when the execution node executes the subtask to be executed. Intermediate data often needs to be shared between multiple steps or tasks.

[0107] S1802: Store the intermediate data in a cache queue.

[0108] The cache queue is configured to allow any node to access it. That is, the master node, execution node, and producer can all access the cache queue and perform access operations on the intermediate data therein. In this way, the execution efficiency of tasks can be improved and memory loss can be prevented when intermediate data needs to be shared.

[0109] Based on any of the above implementations, the present disclosure also provides a task scheduling device. Fig. 9 It is a schematic block diagram of the structure of a task scheduling device according to an embodiment of the present invention.

[0110] like Fig. 9 As shown, the task scheduling device includes: The request module 110 is used to obtain the access right to the task queue.

[0111] The acquisition module 120 is used to acquire the to-be-executed tasks in the task queue in response to acquiring the access right to the task queue.

[0112] The task division module 130 is used to divide the task to be executed into multiple subtasks to be executed.

[0113] The task distribution module 140 is used to distribute each subtask to be executed to an execution node for execution, and obtain the subtask execution result of each execution node.

[0114] The termination module 150 is used to aggregate the subtask execution results of each execution node into a final result and release the access right to the task queue.

[0115] In some embodiments of the present disclosure, the task scheduling device further includes a slicing module (not shown), which is used to split the knowledge package to obtain multiple slicing packages. Specifically, the slicing module can be used to obtain the knowledge package and identify the topological structure of the knowledge package; and in response to the topological structure of the knowledge package being a decision flow forest structure, split the knowledge package into multiple slicing packages, each slicing package corresponding to a decision flow in the knowledge package.

[0116] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0117] The executor of the task scheduling method in the specific implementation manner of the present disclosure may be a mobile phone, a tablet computer, a server or other electronic device.

[0118] Therefore, based on any of the above embodiments, the present disclosure further provides an electronic device, which can execute the task scheduling method of any of the above embodiments of the present disclosure, and the task scheduling device of any of the above embodiments can be configured on the electronic device.

[0119] Fig.10 It is a schematic block diagram of the structure of an electronic device 1000 equipped with a task scheduling device according to an embodiment of the present disclosure.

[0120] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0121] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0122] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0123] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0124] A computer program or instruction may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, a computer program or instruction may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0125] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable task scheduling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable task scheduling device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable task scheduling device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable task scheduling device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0129] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0130] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0131] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A task scheduling method, characterized in that: include: Get access to the task queue; In response to obtaining access rights to the task queue, obtaining tasks to be executed in the task queue; Splitting the task to be executed into a plurality of subtasks to be executed; Distribute each subtask to be executed to an execution node for execution, and obtain the subtask execution result of each execution node; The subtask execution results of each of the execution nodes are aggregated into a final result, and the access right to the task queue is released.

2. The task scheduling method according to claim 1, characterized in that: Get access to the task queue, including: Requesting to set the target key in the target database; and In response to the target key being successfully set, determining that access to the task queue has been acquired; Optionally, the task queue is configured as: Allow access to the node holding the target key; The target database is configured as: In response to a request to set the target key, querying the target key in the target database; and In response to the target key not existing in the target database, saving the target key in the target database, and returning a message indicating that the target key is set successfully; Optionally, releasing the access right to the task queue includes: Requesting deletion of the target key in the target database; and In response to the target key being deleted, it is determined that access to the task queue has been released.

3. The task scheduling method according to claim 1, characterized in that: The task data set is divided into a plurality of subtasks to be performed, including: Obtaining the data volume of the task to be executed, the number of the execution nodes and the number of the task queues; Determine the single-node data volume according to the data volume of the task to be executed, the number of the execution nodes and the number of the task queues; and The data volume of the task to be executed is divided into multiple portions of the single-node data volume, and one subtask to be executed is created based on each portion of the single-node data volume.

4. The task scheduling method according to claim 1, characterized in that: Distribute each subtask to be executed to an execution node for execution, including: Obtaining a queue identifier of the task queue and determining a specific identifier corresponding to the queue identifier; and The execution node with the specific identifier is searched, and the subtask to be executed is distributed to the execution node with the specific identifier for execution.

5. The task scheduling method according to claim 1, characterized in that: The tasks to be executed are configured to: be added to the task queue with the least number of tasks when generated; Optionally, the task queue is configured as: When the consumption rate of the task to be executed is less than the consumption rate threshold, a new task queue is added; and When the peak value of the number of tasks to be executed is less than the peak value threshold, the added task queue is removed.

6. The task scheduling method according to claim 1, characterized in that: Also includes: Obtaining intermediate data generated by the execution node executing the subtask to be executed; as well as Storing the intermediate data in a cache queue; The cache queue is configured to allow any node to access it.

7. The task scheduling method according to claim 1, characterized in that: The task to be executed includes a data verification task, the subtask to be executed includes a data verification subtask, and the subtask execution result includes a verification subresult; The subtask execution results of each of the execution nodes are summarized into a final result, including: Obtaining normal results and abnormal results from the syndrome results; Determine normal data based on the normal result, and count each of the normal data as the final result; or Determine abnormal data based on the abnormal result, and count each abnormal data as the final result; or Normal data is determined based on the normal result, and abnormal data is determined based on the abnormal result, and the normal and abnormal data are integrated into the final result.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the task scheduling method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the task scheduling method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the task scheduling method according to any one of claims 1 to 7 is implemented.

Images