A task processing method, apparatus, device, and storage medium

Abstract

The present disclosure provides a task processing method and device, equipment and a storage medium, relates to the technical field of autonomous driving, in particular to the technical field of high-definition map, and can be used in the production scene of electronic map. The specific implementation scheme is as follows: determining resource requirement information of a to-be-processed task, for example, a high-definition map data processing task; selecting a target physical machine from candidate physical machines according to the resource requirement information, resource remaining information reported by the candidate physical machines and the number of tasks in processing; and controlling the target physical machine to execute the to-be-processed task. The rationality of physical machine scheduling can be improved in the process of executing a task, thereby ensuring the rational use of physical machine resources.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and more particularly to the field of high-precision map technology, which can be used in electronic map production scenarios. Background Technology

[0002] High-precision maps, also known as high-resolution maps, are used by autonomous vehicles. They possess precise vehicle location information and rich road element data, helping cars anticipate complex road conditions such as slope, curvature, and heading, thus better avoiding potential risks. The production process of high-precision maps involves various tasks. Since different tasks require different amounts of data and execution speeds, the specifications of the physical machines providing these tasks also vary. Therefore, how to rationally utilize the resources provided by the physical machines to execute the tasks is crucial. Summary of the Invention

[0003] This disclosure provides a task processing method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, a task processing method is provided, comprising:

[0005] Determine the resource requirements of the tasks to be processed;

[0006] Based on the resource demand information, the remaining resource information reported by the candidate physical machines, and the number of tasks being processed, a target physical machine is selected from the candidate physical machines;

[0007] Control the target physical machine to execute the task to be processed.

[0008] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0009] At least one processor; and

[0010] A memory that is communicatively connected to at least one processor; wherein,

[0011] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the task processing method of any embodiment of the present disclosure.

[0012] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform a task processing method according to any embodiment of this disclosure.

[0013] According to the technology disclosed herein, the rationality of physical machine scheduling can be improved during task execution, thereby ensuring the rational utilization of physical machine resources.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0016] Figure 1A This is a flowchart of a task processing method provided according to an embodiment of the present disclosure;

[0017] Figure 1B This is an architecture diagram of a task processing system provided according to an embodiment of the present disclosure;

[0018] Figure 2 This is a flowchart of a task processing method provided according to an embodiment of the present disclosure;

[0019] Figure 3A This is a flowchart of a task processing method provided according to an embodiment of the present disclosure;

[0020] Figure 3B It is a directed acyclic graph of the original task provided according to the embodiments of this disclosure;

[0021] Figure 4 This is a schematic diagram of the structure of a task processing device according to an embodiment of the present disclosure;

[0022] Figure 5 This is a block diagram of an electronic device used to implement the task processing method of the embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] Figure 1A This is a flowchart of a task processing method provided according to an embodiment of the present disclosure. Figure 1BThis is an architecture diagram of a task processing system provided according to an embodiment of the present disclosure. This embodiment is applicable to situations where physical machines in a physical machine cluster are scheduled to execute tasks. The task processing method can be completed through interaction between a scheduling device and each physical machine in the physical machine cluster. The scheduling device can be a device that provides physical machine resource scheduling services for task execution. The physical machine cluster can include multiple physical machines capable of providing resources to execute tasks. For any given physical machine, it may not execute any tasks, may execute only one task, or may execute multiple tasks simultaneously; specifically, the execution of tasks can be based on the control of the scheduling device.

[0025] Specifically, this method can be executed by a task processing unit integrated into the scheduling device, which can be implemented in software and / or hardware. For example... Figure 1A-1B As shown, the task processing method provided in this embodiment may include:

[0026] S101, Determine the resource requirements of the task to be processed.

[0027] The task to be processed can be any task that needs to be executed currently. In this embodiment, the task to be processed can be any task that requires the resources provided by the physical machine to run. For example, it can be a high-precision map data processing task involved in the electronic map production process, such as classification feature extraction task, feature fusion modeling task, and map rendering task.

[0028] Resources can refer to the hardware environment resources required to execute a task, such as, but not limited to, disk resources, memory resources, and processor resources. Processor resources can further include central processing unit (CPU) resources and graphics processing unit (GPU) resources. Disk resources are used to store data during task execution, and memory resources are used to provide runtime memory. Resource requirement information can be related to the resources required by the task to be processed, such as, but not limited to, the types and sizes of resources required.

[0029] Optional, such as Figure 1B As shown in this embodiment, when a task needs to be processed, the task platform sends the task (such as a pending task) to the task management module of the scheduling device. Upon receiving the pending task, the task management module can transmit it to the resource requirement determination module. The resource requirement determination module then determines the resource requirement information of the pending task, specifically the type and size of the physical machine's memory, disk space, and processor required for its execution. The resource requirement determination module then feeds back the determined resource requirement information of the pending task to the task management module.

[0030] Optionally, there are many ways to determine the resource requirements of the task to be processed in this embodiment, and no limitation is imposed on this method.

[0031] One possible implementation is to pre-set the resource requirement information corresponding to each type of task. In this case, the resource requirement information corresponding to the task type to which the task to be processed belongs can be used as the resource requirement information of the task to be processed.

[0032] Another possible implementation method is to conduct a comprehensive analysis of the resources used during the historical operation of the task to be processed (such as average value analysis) to determine the resource requirements of the task in the current operation.

[0033] Another possible implementation is that when the task platform sends a task to be processed to the scheduling device, it also sends the resource requirement information of that task. For example, the task platform sends a task processing request to the scheduling device containing the task to be processed and its resource requirement information. In this case, the resource requirement information of the task to be processed can be directly obtained from the task processing request sent by the task platform.

[0034] S102, select the target physical machine from the candidate physical machines based on the resource demand information, the remaining resource information reported by the candidate physical machines, and the number of tasks being processed.

[0035] In this embodiment, each physical machine in the physical machine cluster providing resources for task execution can serve as a candidate physical machine. The number of candidate physical machines in this embodiment is typically multiple. The remaining resource information of a candidate physical machine can be the currently remaining resource information of that candidate physical machine; specifically, it can be the remaining disk, memory, and processor resources after deducting the resources occupied by the currently processing task. The "processing task" refers to the task that the candidate physical machine is currently processing. The target physical machine can be a candidate physical machine selected from the candidate physical machines to execute the task to be processed issued by the platform in this task.

[0036] Optional, such as Figure 1B As shown, in this embodiment, each candidate physical machine will report its remaining resource information and the number of tasks being processed to the resource management module of the scheduling device in real time. At this time, the physical machine scheduling module in the scheduling device can obtain the resource requirement information of the tasks to be processed from the task management module, and obtain the remaining resource information and the number of tasks being processed reported in real time by each candidate physical machine in the physical machine cluster from the resource management module. Then, based on the resource requirement information, the remaining resource information reported by the candidate physical machines, and the number of tasks being processed, the target physical machine is selected from the candidate physical machines.

[0037] Optionally, this embodiment selects a target physical machine from the candidate physical machines based on resource demand information, the remaining resource information reported by the candidate physical machines, and the number of tasks being processed, according to a preset scheduling strategy. This preset scheduling strategy can be numerous and is not limited here. For example, one scheduling strategy might be to allocate tasks to candidate physical machines with more tasks being processed, while ensuring that remaining resources meet resource demands; another scheduling strategy might be to ensure a balance in the number of tasks being processed across candidate physical machines, based on the number of tasks being processed, while ensuring that remaining resources meet resource demands.

[0038] Specifically, there are many ways to select a target physical machine from candidate physical machines based on a preset scheduling strategy, and this is not limited to any one method. One possible implementation is to input resource requirement information, remaining resource information reported by candidate physical machines, and the number of tasks being processed into a pre-trained resource scheduling model. This model can then analyze the input information according to a preset scheduling strategy and output the most suitable target physical machine for executing the task to be processed. Another possible implementation is to first select initial physical machines that can meet the resource requirement information from among the candidate physical machines based on the resource requirement information and the remaining resource information reported by the candidate physical machines; then, based on the number of tasks being processed by each initially selected physical machine, and according to a preset scheduling strategy, select a target physical machine from among the candidate physical machines.

[0039] S103 controls the target physical machine to execute the task to be processed.

[0040] Optional, such as Figure 1B As shown, after determining the target physical machine for the current task, the physical machine scheduling module in the scheduling device can control the target physical machine to execute the task through the task execution control module. Specifically, the task execution control module can transmit the task to be processed and its corresponding resource requirement information to the target physical machine, so that the target physical machine can apply for the corresponding resources locally based on the resource requirement information to execute the task.

[0041] Optionally, during the execution of the task to be processed, the target physical machine can provide real-time or periodic feedback on the task execution status to the scheduling device (such as the task execution control module of the scheduling device). This includes feedback on whether the task execution is abnormal during the process, and if so, the feedback on the abnormal stage and the cause of the abnormality. Finally, after the task is completed, the feedback on the final task execution result can be provided.

[0042] Optionally, during the execution of pending tasks, the target physical machine also needs to update its remaining resource information and the number of tasks being processed, and report this information to the resource management module of the scheduling device.

[0043] It should be noted that if this embodiment receives multiple pending tasks from the task platform, the operations of S101-S103 described above can be executed sequentially based on each pending task to complete the processing of multiple pending tasks.

[0044] The solution of this embodiment, upon receiving a task to be processed, first determines the resource requirement information of the task, and then, in conjunction with the remaining resource information reported by candidate physical machines and the number of tasks being processed by them, schedules a target physical machine from among the candidate physical machines to execute the task. This solution, when scheduling physical machines for a task, considers not only the remaining resource information of the physical machine but also the number of tasks being processed by the physical machine, thereby improving the rationality of physical machine scheduling and ensuring the reasonable utilization of physical machine resources.

[0045] Optionally, in this embodiment, another possible method for determining the resource requirements of the task to be processed is to determine the resource requirements of the task to be processed through a resource estimation model. The resource estimation model can be a deep learning model trained using a regression algorithm based on the resource usage during the historical operation of various types of tasks. Specifically, the task to be processed can be input into the resource estimation model, which can then parse the type of the task and output its corresponding resource requirements. Using a deep learning model to predict the resource requirements of the task to be processed improves the accuracy of the resource requirements information, providing a guarantee for the subsequent rational scheduling of the target physical machine based on this resource requirements information.

[0046] Optionally, some critical tasks may require an internal network physical machine cluster for execution. However, these internal physical machines cannot use existing mainstream container services (such as Kubernetes) to isolate resources for executing pending tasks. To address this issue, the scheduling device in this embodiment, when controlling the target physical machine to execute pending tasks, can allocate execution resources and execution processes to the pending tasks based on resource requirement information, and then process the pending tasks within the execution resources based on the execution processes. Specifically, the scheduling device can control the target physical machine to perform the following operations: first, based on the resource requirement information of the pending tasks, request the necessary resources from the remaining local resources; then, use a process creation algorithm (such as the vfork algorithm) to create an execution process for the pending tasks; and finally, use a resource limiting algorithm (such as the rlimit algorithm) to isolate the requested resources as execution resources, and then use the execution process within these execution resources to process the pending tasks. This achieves the goal of isolating resources to execute tasks even when the internal network physical machine cluster cannot use container services. This provides a new solution for internal network physical machines to fully utilize their resources to execute multiple tasks in parallel.

[0047] Figure 2This is a flowchart of a task processing method provided according to an embodiment of the present disclosure. Based on the above embodiments, this disclosure further explains in detail how to select a target physical machine from candidate physical machines based on resource demand information, resource remaining information reported by candidate physical machines, and the number of tasks being processed. Figure 2 As shown, the task processing method provided in this embodiment may include:

[0048] S201, Determine the resource requirements of the task to be processed.

[0049] S202, group the candidate physical machines according to the resource remaining information reported by the candidate physical machines.

[0050] Optionally, this embodiment may employ various strategies to group candidate physical machines based on the resource remaining information reported by the candidate physical machines, and this embodiment does not limit such strategies.

[0051] One possible implementation is as follows: Resources of different specifications and sizes are pre-divided into multiple levels based on resource specifications and resource size ranges. Then, for each candidate physical machine, based on its reported remaining resource information, it can be determined which level of resource specifications and size it matches, and that level is taken as the level to which the candidate physical machine belongs. Subsequently, candidate physical machines belonging to the same level are grouped together.

[0052] Another possible implementation is to statistically analyze the remaining resource information reported by each candidate physical machine to determine the operating capacity of each physical machine. For example, physical machines with larger memory and disk capacities and higher processor specifications have stronger operating capabilities. Based on the operating capabilities of each physical machine, physical machines with similar operating capabilities are grouped together.

[0053] S203, based on resource requirement information, select at least one set of candidate physical machines as available physical machines.

[0054] The available physical machines can be a subset of physical machines that are initially selected from the candidate physical machines based on the remaining resource information of the candidate physical machines and are deemed suitable for executing the task to be processed.

[0055] Optionally, this embodiment can select an available physical machine from the grouped candidate physical machines in many ways, and is not limited to this.

[0056] One possible implementation is to select at least one group of candidate physical machines from each group of candidate physical machines whose remaining resource information is closest to the resource requirement information of the task to be processed, and use them as available physical machines.

[0057] Another possible implementation is as follows: From each group of candidate physical machines, select the optional group whose remaining resource information satisfies the resource requirements of the task to be processed; then analyze the number of physical machines contained in each optional group, and select the candidate physical machines contained in at least one optional group with a larger number as the available physical machines.

[0058] S204. Select the target physical machine from the available physical machines based on the number of tasks being processed reported by the available physical machines.

[0059] Optionally, in this embodiment, the target physical machine for executing the task can be selected from the available physical machines based on the number of tasks being processed reported by each of the initially selected available physical machines. There are many ways to select the target physical machine, and this embodiment does not limit this method.

[0060] One possible implementation is to select the physical machine with the most currently processing tasks from among the available physical machines as the target physical machine. This implementation strategy aims to allocate tasks to physical machines with the most currently processing tasks, thereby maximizing resource utilization of the physical machines executing tasks within the cluster. Simultaneously, it avoids requiring all physical machines in the cluster to operate simultaneously, reducing cluster power consumption while reserving resource-efficient physical machines to execute resource-intensive tasks, thus preventing long-tail effects in task processing.

[0061] Another possible implementation is to select the available physical machine with the fewest tasks being processed from among the available physical machines as the target physical machine. This implementation strategy considers the load balancing of physical machines when allocating tasks, preventing physical machines from crashing due to excessive task execution. Furthermore, this strategy avoids tasks being concentrated on certain physical machines, which could lead to a large number of tasks failing due to physical machine malfunctions, thus improving the stability of task execution.

[0062] Another implementation method involves selecting a target physical machine from the available physical machines based on the number of tasks being processed on the available physical machines and the task execution status of the available physical machines. Specifically, the stability of each candidate physical machine can be determined by considering its task execution status. For example, the stability of a physical machine can be determined based on the number of times the physical machine has experienced anomalies in its historical and / or currently executing tasks, and / or the number of times the executed tasks have experienced anomalies. After eliminating candidate physical machines with poor stability, the target physical machine is selected from the remaining available physical machines by combining the selection strategies of the two implementation methods described above. This implementation method incorporates the task execution status of the physical machines, improving the stability of the selected target physical machine and thus ensuring the stability of the execution of the tasks to be processed.

[0063] S205, control the target physical machine to execute the task to be processed.

[0064] The solution in this embodiment, upon receiving a task to be processed, first determines the resource requirements of the task, then groups the candidate physical machines based on the remaining resource information reported by the candidate physical machines, and further selects at least one physical machine as an available physical machine based on the resource requirements. Taking into account the number of tasks currently being processed by the available physical machines, a target physical machine is scheduled from the available physical machines to execute the task to be processed. This solution introduces a grouping concept when determining the target physical machine from the candidate physical machines, enabling rapid location of available physical machines that meet the resource requirements based on the resource requirements. Then, based on the number of tasks currently being processed by the available physical machines, the target physical machine is determined from a small number of available physical machines, improving the efficiency and accuracy of target physical machine determination and providing a guarantee for subsequent execution of the task to be processed based on the target physical machine.

[0065] Figure 3A This is a flowchart of a task processing method provided according to an embodiment of the present disclosure; Figure 3B This is a directed acyclic graph of the original task provided according to the embodiments of this disclosure. Based on the above embodiments, this disclosure further provides a preferred example of a task processing method. This method is particularly suitable for handling tasks with a large workload or long processing time, such as... Figures 3A-3B As shown, the task processing method provided in this embodiment may include:

[0066] S301, Obtain the original task.

[0067] The original task can be a task that needs to be executed by each candidate physical machine in the physical machine cluster. The scheduling device in this embodiment can obtain the original task issued by the task platform.

[0068] S302, determine whether the original task meets the splitting conditions. If it does, execute S303; otherwise, execute S305.

[0069] The splitting criteria can be used to determine whether the original task needs to be split. Optionally, this embodiment can measure whether the original task meets the splitting criteria based on the task volume and / or duration of the original task.

[0070] Optionally, this embodiment can determine the estimated execution time of the original task based on the historical execution time of the original task. When determining the workload of the original task, the task type to which the original task belongs can be analyzed to determine the workload; the code size of the original task can be analyzed to determine the workload; or the number of operation steps corresponding to the original task can be parsed to determine the workload, etc.

[0071] After determining the workload and estimated time of the original task, it can be determined whether the workload of the original task reaches the task workload splitting threshold, and / or whether the estimated time of the original task reaches the time splitting threshold. If they do, it means that the original task meets the splitting conditions. At this time, S303 is triggered to split the original task. Otherwise, the original task can be considered as a task to be processed, that is, S305 is executed to treat the original task as a task to be processed.

[0072] S303 If the obtained original task meets the splitting conditions, then the original task is split into at least two subtasks according to the task integrity and / or subtask specifications.

[0073] The subtask specification can be a pre-defined criterion for dividing subtasks, such as the amount of work involved in the subtask.

[0074] Optionally, if the original task meets the splitting conditions, it means that the original task is too large. An excessively large original task will inevitably have high resource requirements. In this case, in order to avoid the original task being unable to be executed or being delayed due to insufficient remaining resources in the physical machine cluster, the original task can be split into multiple sub-tasks with smaller task sizes to improve the execution efficiency of the original task.

[0075] One possible implementation is to break down the original task based on its completeness. Specifically, based on task completeness, operations with high correlation are split into sub-tasks. For example, if the original task is to generate an electronic map of a certain area based on point cloud data and image data, then based on task completeness, the operation of extracting regional features from point cloud data and image data can be divided into a sub-task; the operation of fusing features from the extracted point cloud features and image features can be divided into a sub-task; and the operation of drawing the electronic map based on the fused features can be divided into a sub-task.

[0076] Another possible implementation is to split the original task based on subtask specifications. Specifically, the original task is split into equal parts according to the preset subtask specifications and the amount of the original task. For example, if the amount of the original task is 1G and the preset subtask specification is 200M, then the original task can be split into 5 subtasks with a size of 200M each.

[0077] Another possible implementation is to split the original task based on the completeness of the task and the specifications of the subtasks. Specifically, based on the completeness of the task, operations with high correlation can be split into subtasks. The remaining operations with relatively weak correlation can then be divided equally according to preset subtask specifications. For example, if the original task is to generate an electronic map of a certain area based on point cloud data and image data, the feature fusion process is highly correlated, and splitting it would severely affect the fusion effect. However, the correlation between feature extraction and map drawing operations is relatively weak. Therefore, in this embodiment, the operation of feature fusion based on extracted point cloud features and image features can be divided into a subtask. The operations of regional feature extraction based on point cloud data and image data, and the operation of drawing an electronic map based on the fused features, are divided equally according to preset subtask specifications.

[0078] It should be noted that the advantage of dividing subtasks based on subtask specifications is that the amount of work in the resulting subtasks is smaller, which can improve the execution efficiency of the task; the advantage of dividing subtasks based on task completeness is that the execution results of the divided subtasks are closer to the results of directly executing the original task, which can improve the accuracy of task execution. Therefore, this embodiment can select a suitable division method from the above three methods according to the actual situation of the original task.

[0079] Optionally, after splitting the original task into at least two subtasks, this embodiment can generate a directed acyclic graph (DAG) for the split tasks based on the dependencies between the subtasks. For example, this embodiment can split the original task into five subtasks (subtask 1 to subtask 5) based on task completeness and / or subtask specifications, and generate a directed acyclic graph based on the dependencies between the subtasks. Figure 3B The diagram shows a directed acyclic graph.

[0080] S304. Based on the dependency relationship between at least two tasks, determine the original execution order of at least two subtasks, and in accordance with the original execution order, treat at least two subtasks as tasks to be processed in sequence.

[0081] The original execution order can be the execution order between the subtasks after the original task has been split.

[0082] Optionally, in this embodiment, the process of determining the original execution order of at least two subtasks based on the dependency relationship between at least two tasks can be as follows: for two subtasks with a sequential relationship, determine their original execution order according to the order in which they are sequentially connected; for two subtasks with a parallel relationship, set the same original execution order for both. Then, for each subtask under the original execution order, treat it as a task to be processed and execute the subsequent operations S306-S308. After all subtasks in the original execution order have been executed, treat the subtasks of the next original execution order as tasks to be processed and execute the subsequent operations S306-S308 again. This continues until all subtasks corresponding to all original execution orders have been executed.

[0083] For example, such as Figure 3B As shown, based on the dependencies between subtasks, the original execution order of subtasks 1 to 5 is determined as follows: first, subtask 1 is executed; then subtasks 2 and 3 are executed simultaneously; next, subtask 4 is executed; and finally, subtask 5 is executed. That is, subtask 1 is first treated as a pending task. After completing operations S306-S308 on subtask 1, subtasks 2 and 3 are treated as pending tasks, and operations S306-S308 are executed in parallel on subtasks 2 and 3. Then, subtask 4 is treated as a pending task, and operations S306-S308 are executed on subtask 4. Finally, subtask 5 is treated as a pending task, and operations S306-S308 are executed on subtask 5. After these steps, the original task processing is complete.

[0084] S305, treat the original task as a pending task.

[0085] S306, Determine the resource requirements of the task to be processed.

[0086] S307: Select the target physical machine from the candidate physical machines based on the resource demand information, the remaining resource information reported by the candidate physical machines, and the number of tasks being processed.

[0087] S308 controls the target physical machine to execute the task to be processed.

[0088] It should be noted that, in this embodiment, when the target physical machine processes each subtask in the original execution order, it may use the processing results of the subtasks executed before it. Therefore, for each subtask in the original execution order, this embodiment must wait until all the subtasks before it is processed before it can process the subtask.

[0089] The solution of this embodiment, when the received original task meets the splitting conditions, can split the original task into multiple subtasks. Based on the dependencies between the subtasks, the original execution order of each subtask is determined. According to the original execution order, at least two subtasks are sequentially designated as pending tasks. The resource requirements of the pending tasks are then determined. Combined with the remaining resource information reported by the candidate physical machines and the number of tasks in processing, a target physical machine is scheduled from the candidate physical machines to execute the pending tasks. This embodiment's solution, when dealing with large tasks issued by the task platform, splits the original task into multiple subtasks for processing. This avoids situations where the original task is too large and the remaining resources of the candidate physical machines in the physical machine cluster are insufficient to execute the original task, thus improving the processing efficiency for large volumes of original tasks.

[0090] Based on the above embodiments, this disclosure may further include: if a re-execution event of the original task is detected, then, based on the latest execution status of the subtasks of the original task, determining the subtasks to be rolled back from the subtasks of the original task, and the rollback execution order of the subtasks to be rolled back; and, according to the rollback execution order, sequentially designating the subtasks to be rolled back as tasks to be processed.

[0091] The re-execution event of the original task can be an event that initiates the re-execution of the original task when an exception occurred during the most recent execution of the original task, causing the most recent execution to fail. Optionally, one way to detect the re-execution event of the original task in this embodiment is to receive a task re-execution request containing a re-execution task identifier issued by the task platform. Another way to detect it is to detect a processing exception notification reported by the target physical machine during the execution of the pending task, and the reason for the exception is insufficient resource allocation. It should be noted that, for the second possible method, the original task to which the pending task with the processing exception belongs can be used as the original task to be re-executed.

[0092] The latest execution status of the subtasks of the original task can refer to the execution status of each subtask after the original task was split in the most recent execution process. For example, it can refer to whether each subtask of the original task was executed successfully in the most recent execution process.

[0093] Optionally, after detecting a re-execution event of the original task through any of the above methods, this embodiment will search for the execution results of the subtasks of the original task reported by each target physical machine during the most recent execution of the original task, and regard the subtasks with abnormal execution results as abnormal subtasks. Then, the abnormal subtasks, as well as all subtasks whose original execution order is after the abnormal subtasks, will be regarded as subtasks to be rolled back. If there are multiple abnormal subtasks, the subtask with the earliest original execution order can be selected as the abnormal subtask.

[0094] After identifying the subtasks to be rolled back, the rollback execution order of each subtask to be rolled back can be determined according to the dependency relationship between the subtasks to be rolled back, similar to the above S304. Then, according to the rollback execution order, the subtasks to be rolled back are processed as tasks to be processed in sequence.

[0095] In this embodiment, when re-executing a failed original task, the process does not start from the first subtask of the original task. Instead, it determines a portion of the subtasks to be rolled back based on the latest execution status of the subtasks, i.e., the most recent execution status. Only this portion of the subtasks to be rolled back is executed, which greatly improves the efficiency of re-executing tasks.

[0096] Figure 4 This is a schematic diagram of a task processing apparatus according to an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to situations where physical machines in a physical machine cluster are scheduled to execute tasks. This apparatus can be implemented using software and / or hardware, and can implement the task processing methods of any embodiment of the present disclosure. Figure 4 As shown, the task processing device 400 includes:

[0097] Resource requirement determination module 401 is used to determine the resource requirement information of the task to be processed;

[0098] The physical machine scheduling module 402 is used to select a target physical machine from the candidate physical machines based on the resource demand information, the resource remaining information reported by the candidate physical machines, and the number of tasks being processed.

[0099] The task execution control module 403 is used to control the target physical machine to execute the task to be processed.

[0100] The solution of this embodiment, upon receiving a task to be processed, first determines the resource requirement information of the task, and then, in conjunction with the remaining resource information reported by candidate physical machines and the number of tasks being processed by them, schedules a target physical machine from among the candidate physical machines to execute the task. This solution, when scheduling physical machines for a task, considers not only the remaining resource information of the physical machine but also the number of tasks being processed by the physical machine, thereby improving the rationality of physical machine scheduling and ensuring the reasonable utilization of physical machine resources.

[0101] Furthermore, the physical machine scheduling module 402 includes:

[0102] The physical machine grouping unit is used to group the candidate physical machines according to the resource remaining information reported by the candidate physical machines;

[0103] The available physical machine selection unit is used to select at least one set of candidate physical machines as available physical machines based on the resource requirement information.

[0104] The target physical machine selection unit is used to select a target physical machine from the available physical machines based on the number of tasks being processed reported by the available physical machines.

[0105] Furthermore, the target physics machine selection unit is specifically used for:

[0106] From the available physical machines, select the physical machine with the largest number of tasks being processed as the target physical machine.

[0107] Furthermore, the target physics machine selection unit is also specifically used for:

[0108] Based on the number of tasks being processed in the available physical machines and the task execution status of the available physical machines, a target physical machine is selected from the available physical machines.

[0109] Furthermore, the task execution control module 403 is specifically used for:

[0110] The target physical machine is controlled to allocate execution resources and execution processes to the task to be processed based on the resource requirement information, and the task to be processed is processed within the execution resources based on the execution processes.

[0111] Furthermore, the resource requirement determination module 401 is specifically used for:

[0112] Resource estimation models are used to determine the resource requirements of tasks to be processed.

[0113] Furthermore, the device also includes:

[0114] The task splitting module is used to split the original task into at least two subtasks based on task integrity and / or subtask specifications if the obtained original task meets the splitting conditions.

[0115] The pending task determination module is used to determine the original execution order of the at least two sub-tasks based on the dependency relationship between the at least two tasks, and to sequentially treat the at least two sub-tasks as pending tasks according to the original execution order.

[0116] Furthermore, the device also includes:

[0117] The rollback information determination module is used to determine, based on the latest execution status of the subtasks of the original task, the subtasks to be rolled back from the subtasks of the original task, and the rollback execution order of the subtasks to be rolled back, if a re-execution event of the original task is detected.

[0118] The pending task determination module is used to sequentially identify the subtasks to be rolled back as pending tasks according to the rollback execution order.

[0119] The above-described products can perform the methods provided in any embodiment of this disclosure, and have the corresponding functional modules and beneficial effects for performing the methods.

[0120] In the technical solutions disclosed herein, the acquisition, storage, and application of any task, resource, or other information involved comply with relevant laws and regulations and do not violate public order and good morals.

[0121] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0122] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0123] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0124] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0125] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as task processing methods. For example, in some embodiments, the task processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the task processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform task processing methods by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0131] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem that addresses the management difficulties and weak business scalability inherent in traditional physical hosting and VPS services. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0132] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0133] Cloud computing refers to a technology system that enables access to a shared pool of physical or virtual resources via a network. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for applications such as artificial intelligence and blockchain, as well as for model training.

[0134] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0135] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A task processing method, comprising: Determine the resource requirements of the tasks to be processed; The candidate physical machines are grouped according to the resource remaining information reported by the candidate physical machines; Based on the resource requirement information, at least one set of candidate physical machines is selected as available physical machines; Select a target physical machine from the available physical machines based on the number of tasks being processed reported by the available physical machines; Control the target physical machine to execute the task to be processed; The step of grouping the candidate physical machines based on the resource remaining information reported by the candidate physical machines includes: Resources of different specifications and sizes are pre-classified into at least two levels; For each candidate physical machine, the remaining resource information reported by the candidate physical machine is matched with the resource specifications and resource size of at least two levels to obtain the level to which the candidate physical machine belongs; Candidate physics machines belonging to the same level are grouped together.

2. The method according to claim 1, wherein, The step of selecting a target physical machine from the available physical machines based on the number of tasks being processed reported by the available physical machines includes: From the available physical machines, select the physical machine with the largest number of tasks being processed as the target physical machine.

3. The method according to claim 1, wherein, The step of selecting a target physical machine from the available physical machines based on the number of tasks being processed reported by the available physical machines includes: Based on the number of tasks being processed in the available physical machines and the task execution status of the available physical machines, a target physical machine is selected from the available physical machines.

4. The method according to claim 1, wherein, The control of the target physical machine to execute the task to be processed includes: The target physical machine is controlled to allocate execution resources and execution processes to the task to be processed based on the resource requirement information, and the task to be processed is processed within the execution resources based on the execution processes.

5. The method according to claim 1, wherein, The determination of the resource requirements information for the task to be processed includes: Resource estimation models are used to determine the resource requirements of tasks to be processed.

6. The method according to claim 1, further comprising: If the obtained original task meets the splitting conditions, then the original task is split into at least two subtasks according to the task integrity and / or subtask specifications. Based on the dependencies between the at least two tasks, determine the original execution order of the at least two subtasks, and then, based on the original execution order, sequentially treat the at least two subtasks as tasks to be processed.

7. The method according to claim 6, further comprising: If a re-execution event of the original task is detected, then based on the latest execution status of the subtasks of the original task, the subtasks to be rolled back and the rollback execution order of the subtasks to be rolled back are determined from the subtasks of the original task. According to the rollback execution order, the subtasks to be rolled back are sequentially designated as tasks to be processed.

8. A task processing apparatus, comprising: The resource requirement determination module is used to determine the resource requirement information of the task to be processed; The physical machine scheduling module is used to select a target physical machine from the candidate physical machines based on the resource demand information, the resource remaining information reported by the candidate physical machines, and the number of tasks being processed. The task execution control module is used to control the target physical machine to execute the task to be processed; The physical machine scheduling module includes: The physical machine grouping unit is used to group the candidate physical machines according to the resource remaining information reported by the candidate physical machines; The available physical machine selection unit is used to select at least one set of candidate physical machines as available physical machines based on the resource requirement information. The target physical machine selection unit is used to select a target physical machine from the available physical machines based on the number of tasks being processed reported by the available physical machines. The physical machine grouping unit is specifically used for: Resources of different specifications and sizes are pre-classified into at least two levels; For each candidate physical machine, the remaining resource information reported by the candidate physical machine is matched with the resource specifications and resource size of at least two levels to obtain the level to which the candidate physical machine belongs; Candidate physics machines belonging to the same level are grouped together.

9. The apparatus according to claim 8, wherein, The target physics machine selection unit is specifically used for: From the available physical machines, select the physical machine with the largest number of tasks being processed as the target physical machine.

10. The apparatus according to claim 8, wherein, The target physics machine selection unit is also specifically used for: Based on the number of tasks being processed in the available physical machines and the task execution status of the available physical machines, a target physical machine is selected from the available physical machines.

11. The apparatus according to claim 8, wherein, The task execution control module is specifically used for: The target physical machine is controlled to allocate execution resources and execution processes to the task to be processed based on the resource requirement information, and the task to be processed is processed within the execution resources based on the execution processes.

12. The apparatus according to claim 8, wherein, The resource requirement determination module is specifically used for: Resource estimation models are used to determine the resource requirements of tasks to be processed.

13. The apparatus according to claim 8, further comprising: The task splitting module is used to split the original task into at least two subtasks based on task integrity and / or subtask specifications if the obtained original task meets the splitting conditions. The pending task determination module is used to determine the original execution order of the at least two sub-tasks based on the dependency relationship between the at least two tasks, and to sequentially treat the at least two sub-tasks as pending tasks according to the original execution order.

14. The apparatus of claim 13, further comprising: The rollback information determination module is used to determine, based on the latest execution status of the subtasks of the original task, the subtasks to be rolled back from the subtasks of the original task, and the rollback execution order of the subtasks to be rolled back, if a re-execution event of the original task is detected. The pending task determination module is used to sequentially identify the subtasks to be rolled back as pending tasks according to the rollback execution order.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the task processing method according to any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the task processing method according to any one of claims 1-7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the task processing method according to any one of claims 1-7.

Images