Task processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a task processing method and device, electronic equipment and a storage medium. The method comprises the steps that a to-be-processed task and a node idle period are obtained, and the to-be-processed task carries task starting time and task allowed delay time; performing idle period matching on the to-be-processed task based on the task starting time, the task allowable delay time and the node idle period, and determining an idle period matching result of the to-be-processed task; when the idle period matching result is successful matching, acquiring an idle node corresponding to the node idle period, and determining a target node based on the idle node; and scheduling the to-be-processed task to the target node, and executing the to-be-processed task on the target node. According to the scheme, the node resources in the idle time period are determined for the to-be-processed task, the node processing task in the idle time period is fully utilized, independent application and resource occupation for the to-be-processed task are not needed, the task processing efficiency is improved, node resource waste is avoided, and the utilization rate of the node resources in the idle time period is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a task processing method, apparatus, electronic device, and storage medium. Background Art

[0002] With the adoption of containerization and Kubernetes technologies in private clouds, service deployment efficiency has been greatly improved. However, in private clouds, services are often concentrated in certain time periods, such as during working hours, leading to significant resource waste during nighttime and holidays. During cluster sharing, when idle resources in the cluster cannot be accessed, tasks may request more resource quotas, preventing full utilization of node resources during off-peak hours and resulting in low cluster resource utilization. Summary of the Invention

[0003] This invention provides a task processing method, apparatus, electronic device, and storage medium to solve the problems of insufficient utilization of cluster resources during idle periods and low utilization of cluster resources.

[0004] According to one aspect of the present invention, a task processing method is provided, comprising:

[0005] Obtain the pending tasks and node idle time periods, where the pending tasks carry the task start time and the allowed delay time;

[0006] Based on the task start time, the allowed delay time of the task, and the idle time period of the node, the idle time period of the task to be processed is matched to determine the idle time period matching result of the task to be processed.

[0007] If the matching result during the idle period is successful, obtain the idle node corresponding to the idle period of the node, and determine the target node based on the idle node;

[0008] The pending tasks are scheduled to the target node and executed on the target node.

[0009] Optionally, the process of obtaining pending tasks includes: in response to the creation operation of pending tasks, displaying corresponding task configuration options, wherein the task configuration options include a task start time option and a task allowed delay time option; in response to the selection operation of the task start time option and the task allowed delay time option, determining the task start time and the task allowed delay time, and generating pending tasks based on the task start time, the task allowed delay time and the basic information of pending tasks, wherein the basic information includes the task name, task type and project group information.

[0010] Optionally, obtaining node idle time periods includes: obtaining historical node operation data, performing prediction processing based on the historical node operation data and a pre-trained idle time period prediction model to obtain node idle prediction information for each node; and determining node idle time periods based on the node idle prediction information for each node.

[0011] Optionally, the idle nodes corresponding to the idle time period of the node are obtained, and the target node is determined based on the idle nodes, including: obtaining the task status of each node during the idle time period of the node, selecting the node whose task status is idle, and obtaining at least one idle node, wherein the node information of the idle node includes the node level identifier; obtaining the task priority identifier in the task to be processed; traversing the idle nodes, and determining any idle node corresponding to the task to be processed based on the pre-built mapping relationship between the task priority identifier and the node level identifier, and taking any idle node as the target node.

[0012] Optionally, the node information of the idle node also includes the remaining resource quantity and the resource utilization rate during the idle period; scheduling the task to be processed to the target node includes: determining the resource configuration information of the task to be processed based on the remaining resource quantity and the resource utilization rate during the idle period of the target node; creating a task instance on the target node based on the resource configuration information, and scheduling the task to be processed to the task instance of the target node.

[0013] Optionally, the resource configuration information includes minimum resource information and resource limit information, wherein the minimum resource information is less than the resource limit information; the resource configuration information of the task to be processed is determined based on the remaining resources of the target node and the resource utilization rate during idle periods, including: determining the minimum resource information of the task to be processed based on the remaining resources of the target node; and determining the resource limit information of the task to be processed based on the resource utilization rate during idle periods and the total resources of the target node.

[0014] Optionally, after scheduling the pending task to the target node, the method further includes: during the execution of the pending task, if it is detected that the target node has received a new pending task, obtaining the task priority identifier corresponding to the new pending task; if it is determined based on the task priority identifier corresponding to the new pending task and the task priority identifier corresponding to the pending task that the task priority of the new pending task is higher than the task priority of the pending task, pausing the execution of the pending task, storing the pending task in the pending task list, and scheduling the pending task to the new target node if a new target node is detected; and continuing to execute the pending task if the task priority of the new pending task is lower than the task priority of the pending task.

[0015] According to another aspect of the present invention, a task processing apparatus is provided, comprising:

[0016] The information acquisition module is used to acquire tasks to be processed and idle time periods of nodes. The tasks to be processed contain the task start time and the allowed delay time.

[0017] The idle time period matching result determination module is used to match the idle time period of the task to be processed based on the task start time, the allowed delay time of the task, and the idle time period of the node, and determine the idle time period matching result of the task to be processed.

[0018] The target node determination module is used to obtain the idle node corresponding to the idle time period when the matching result is successful during the idle time period, and determine the target node based on the idle node;

[0019] The pending task processing module is used to schedule pending tasks to target nodes and execute them on the target nodes.

[0020] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0021] At least one processor; and

[0022] A memory that is communicatively connected to at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the task processing method of any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the task processing method of any embodiment of the present invention.

[0025] The technical solution of this invention involves acquiring a task to be processed and a node's idle time slot. The task to be processed carries a task start time and an allowed delay time. Based on the task start time, allowed delay time, and node idle time slots, the idle time slots of the task to be processed are matched to determine the matching result. If the matching result is successful, the idle node corresponding to the node's idle time slot is acquired, and a target node is determined based on the idle node. The task to be processed is then scheduled to the target node and executed there. This solution allocates corresponding idle nodes to tasks to be processed during node idle time slots for scheduling, eliminating the need for separate resource requests and additional resource allocation for tasks. It automates the allocation of resource nodes and the determination of processing time for tasks to be processed, allowing tasks to utilize the idle resources of corresponding nodes during idle time slots. This fully utilizes the resources of idle nodes during idle time slots, avoids wasting node resources, and improves the resource utilization of the entire target cluster.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. Attached Figure Description

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 This is a flowchart of a task processing method provided in Embodiment 1 of the present invention;

[0029] Figure 2 This is a flowchart of a task processing method provided in Embodiment 2 of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of a task processing device provided in Embodiment 3 of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the task processing method of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] Example 1

[0035] Figure 1 This is a flowchart of a task processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where tasks to be processed are being processed. This method can be executed by a task processing device, which can be implemented in hardware and / or software, and can be configured in electronic devices such as computers. Figure 1 As shown, the method includes:

[0036] S110. Obtain the pending tasks and node idle time periods, wherein the pending tasks carry the task start time and the allowed delay time.

[0037] Specifically, "tasks to be processed" can be understood as tasks awaiting scheduling and execution. Tasks to be processed can be created through a task management platform, by executing a task creation command, or read from a list of tasks to be processed. For example, tasks to be processed can be batch processing tasks. "Node idle time period" represents the period during which a node is idle. It can be determined based on the node operation data of each node in the target cluster. A corresponding node idle time period algorithm can be pre-constructed, and the node idle time period can be determined based on the node's historical operation data and the algorithm. For example, the node idle time period algorithm can process the node's historical operation data using a constructed time period prediction model to obtain the predicted idle time period for each node, and then determine the node idle time period based on the predicted idle time period. For instance, the predicted idle time periods of each node can be intersected or joined, and the resulting time period can be used as the node idle time period. Alternatively, statistical methods can be used to statistically analyze the node's historical operation data, and the time period with the most idle nodes can be used as the node idle time period; however, no specific limitations are imposed here.

[0038] Specifically, users can create pending tasks through the task creation function on the task management platform, or retrieve pending tasks from the pending task list. Each pending task carries a start time and an allowed delay time. The start time and allowed delay time determine the available time period for the pending task. Historical running data of each node in the target cluster can be obtained from the monitoring system. A node idle time period algorithm is then called to process the historical running data of the nodes to obtain the node idle time periods.

[0039] Optionally, the process of obtaining pending tasks includes: in response to the creation operation of pending tasks, displaying the corresponding task configuration options, wherein the task configuration options include a task start time option and a task allowed delay time option; in response to the selection operation of the task start time option and the task allowed delay time option, determining the task start time and the task allowed delay time, and generating pending tasks based on the task start time, the task allowed delay time and the basic information of pending tasks, wherein the basic information includes the task name, task type and project group information.

[0040] Specifically, users can click the "Create Task" button on the task management platform to create a task to be processed. In response to this creation, the corresponding visualization page displays task configuration options, including, but not limited to, a task start time and a allowed delay time. Users can select the task start time and allowed delay time in the task configuration options. It should be noted that the available task start time and allowed delay time can be predicted based on historical execution data. Users can also edit the task start time and allowed delay time to meet the requirements of the task to be processed. Once the task start time and allowed delay time are determined, the basic information of the task to be processed can be edited on the visualization page. This basic information can be understood as information representing the attributes of the task, including, but not limited to, task name, task type, and project group information. The system will generate corresponding pending tasks based on the task start time, the allowed delay time, and the basic information of the pending tasks. Preferably, the successfully created pending tasks and their associated information can be stored in the pending task list. The pending task list is continuously polled by a scheduled task. Once a pending task is detected in the pending task list, it will be executed automatically.

[0041] Optionally, obtaining node idle time periods includes: obtaining historical node operation data, performing prediction processing based on the historical node operation data and a pre-trained idle time period prediction model to obtain node idle prediction information for each node; and determining node idle time periods based on the node idle prediction information for each node.

[0042] The idle time prediction model can be understood as a deep learning network model used to predict the idle information of each node and obtain the node idle prediction information. For the idle time prediction model, the historical operation data of each node can be sampled to obtain a sample training set. The idle time prediction model is trained using the sample training set to obtain a pre-trained idle time prediction model.

[0043] Specifically, by sending a data query request to the monitoring system, historical operational data of each node in the target cluster (which can be a Kubernetes cluster) is obtained. A pre-trained idle period prediction model is then invoked, inputting the historical operational data of each node into the model to predict the idle period for each node. This predicted idle period includes, but is not limited to, the predicted idle period and the predicted available resources. Statistical analysis is performed based on the predicted idle period to determine the time period with the most idle nodes. This time period is then designated as the node idle period. Alternatively, the time information corresponding to nodes whose predicted available resources meet a certain resource threshold can be selected. The obtained time information can be used to construct the node idle period. Intersection or union operations can be performed on the obtained time information, and the resulting time period can be used as the node idle period; this is not limited here.

[0044] S120. Based on the task start time, the allowed delay time of the task, and the idle time period of the node, perform idle time period matching on the task to be processed to determine the idle time period matching result of the task to be processed.

[0045] The allowed delay time for a task can be understood as the maximum delay time for a task to be processed to start. In other words, if a task fails to start within the allowed delay time, it indicates that the task has failed to start. The failed task can be deleted and a new task can be created. Alternatively, the failed task can be added to the list of tasks to be processed and its status can be changed to pending status.

[0046] Specifically, the startable time period of the task to be processed is calculated based on the task start time and the allowed delay time. The startable time period is compared with the node's idle time period to determine if there is a common time point, or the startable time period and the node's idle time period are intersected to determine if the intersection result is empty. If there is a common time point or the intersection result is not empty, the idle time period matching result of the task to be processed is determined to be a successful match. If there is neither a common time point nor an empty intersection result, the idle time period matching result of the task to be processed is determined to be a failed match.

[0047] In this embodiment, the task time information of the task to be processed is checked for idle time periods to ensure that the task to be processed can meet the node idle time period, so that the task to be processed can successfully call the node in the node idle time period without applying for more node resources for the task to be processed. Idle node resources can be quickly allocated to the node in the node idle time period, which not only enables the task to be processed quickly, but also improves the utilization rate of the node in the node idle time period.

[0048] S130. If the matching result during the idle period is successful, obtain the idle node corresponding to the idle period of the node, and determine the target node based on the idle node.

[0049] Specifically, if the matching result for the idle period is successful, node operation data matching the node's idle time is obtained from the node operation data in the monitoring system based on the node's idle period. Nodes with an idle status are extracted from the obtained node operation data to obtain idle nodes that meet the node's idle period requirements. One node is selected from the idle nodes as the target node. For example, any node can be randomly selected from the idle nodes as the target node, or the idle nodes can be sorted according to the amount of available resources, and the idle node with the highest amount of available resources can be selected as the target node. Alternatively, the target node can be selected based on the node priority of each idle node. The corresponding method can be selected to determine the target node according to the actual processing task requirements, and no limitation is made here.

[0050] S140. Schedule the task to be processed to the target node and execute the task on the target node.

[0051] Specifically, after the target node is determined, the operation of scheduling the task to be processed to the target node is performed. If the task to be processed is successfully scheduled to the target node, the task to be processed is executed on the target node.

[0052] Optionally, the node information of the idle node also includes the remaining resource quantity and the resource utilization rate during the idle period; scheduling the task to be processed to the target node includes: determining the resource configuration information of the task to be processed based on the remaining resource quantity and the resource utilization rate during the idle period of the target node; creating a task instance on the target node based on the resource configuration information, and scheduling the task to be processed to the task instance of the target node.

[0053] It should be noted that when scheduling tasks to be processed, calling the Kubernetes cluster's application interface to create custom resources requires the Kubernetes admission controller to intercept the resource creation request and set the corresponding resource configuration information for the task instance to be processed. The resource configuration information includes minimum resource information and resource quota information. The minimum resource information is the minimum amount of resources required for the task instance to start, and the resource quota information is the maximum amount of resources that the task instance can use. This is to prevent the task instance from using resources without restriction, which could lead to a host machine avalanche and affect the operation of other task instances.

[0054] Specifically, resource configuration information for tasks to be processed can be set based on the remaining resources of the selected target node and the resource utilization rate during idle periods. A minimum resource quantity can be set based on the remaining resources of the target node. If the resources required by the task to be processed are lower than the remaining resources of the target node, the required resources can be set to the minimum; if the resources required are higher than the remaining resources, the remaining resources can be set to the minimum. The maximum available resource utilization rate is calculated based on the resource utilization rate during idle periods. The resource limit information is determined by multiplying the maximum available resource utilization rate by the total resources of the node. The maximum available resource utilization rate can be determined according to a preset formula: (100% - resource utilization rate during idle periods) × proportional coefficient. The proportional coefficient can be set according to actual conditions; preferably, it is 80%. After determining the resource configuration information, a resource creation request is executed. Based on the determined resource configuration information, a task instance is created on the target node to schedule the task to be processed to the task instance on the target node.

[0055] Optionally, the resource configuration information includes minimum resource information and resource limit information, wherein the minimum resource information is less than the resource limit information; the resource configuration information of the task to be processed is determined based on the remaining resources of the target node and the resource utilization rate during idle periods, including: determining the minimum resource information of the task to be processed based on the remaining resources of the target node; and determining the resource limit information of the task to be processed based on the resource utilization rate during idle periods and the total resources of the target node.

[0056] It's understandable that the minimum resource information refers to the minimum amount of resources required to start a task instance, while the resource quota information refers to the maximum amount of resources allowed for the task instance. Therefore, the resource amount corresponding to the set minimum resource information should be less than the maximum resource amount corresponding to the resource quota information. If the minimum resource amount corresponding to the determined resource configuration information is not less than the maximum resource amount, then a new target node needs to be selected.

[0057] Specifically, the remaining resources of the target node can be set as the minimum resource information for the pending tasks to ensure that the target node can start the task instance corresponding to the pending tasks; the maximum available resource utilization rate of the node is determined according to the preset calculation formula based on the resource utilization rate during idle periods, and the product of the maximum available resource utilization rate and the total resources of the target node is calculated, and the result is used as the resource limit resource.

[0058] Based on the above embodiments, after scheduling the task to be processed to the target node, the method further includes: during the execution of the task to be processed, if it is detected that the target node has received a new task to be processed, obtaining the task priority identifier corresponding to the new task to be processed; if it is determined that the task priority of the new task to be processed is higher than the task priority of the task to be processed based on the task priority identifier corresponding to the new task to be processed and the task priority identifier corresponding to the task to be processed, pausing the execution of the task to be processed, storing the task to be processed in the task to be processed list, and scheduling the task to be processed to the new target node if a new target node is detected; if the task priority of the new task to be processed is lower than the task priority of the task to be processed, continuing to execute the task to be processed.

[0059] Specifically, after scheduling the pending tasks to the target node, during the execution of the pending tasks, if the target node detects that a new pending task has been received, it identifies the task information of the new pending task, determines the task priority of the new pending task, compares the task priority of the new pending task with the task priority of the pending task, and if the task priority of the new pending task is higher than that of the pending task, the execution of the pending task needs to be paused, the corresponding task instance is deleted, and the new pending task can be scheduled to this target node. Unfinished pending tasks are stored in the pending task list, and their corresponding task status is set to incomplete. When the pending task is revisited, a new target node is determined for it. Once a new target node is determined, the scheduling operation is re-executed, and the pending task is scheduled to the new target node. In some embodiments, if the pending task is a batch task, given that batch tasks usually have the lowest priority, the pending task can be directly deleted, generating a corresponding prompt message to remind the user that the pending task execution failed, allowing the user to decide whether to recreate the pending task. If the priority of a newly added pending task is lower than the priority of the pending task, the pending task will continue to be executed, and other available idle nodes will be selected for the newly added pending task.

[0060] The technical solution of this embodiment obtains the task to be processed and the idle time period of the node. The task to be processed carries the task start time and the allowed delay time. Based on the task start time, the allowed delay time and the idle time period of the node, the idle time period of the task to be processed is matched to determine the idle time period matching result of the task to be processed. If the idle time period matching result is successful, the idle node corresponding to the node idle time period is obtained, and the target node is determined based on the idle node. The task to be processed is scheduled to the target node and executed on the target node. This solution allocates the corresponding idle node to the task to be processed during the node idle time period for scheduling the task to be processed. It does not require the task to apply for and occupy additional resources separately. It automatically allocates resource nodes and determines the processing time of the task to be processed, so that the task to be processed can use the idle resources of the corresponding node to process the task during the idle time period, making full use of the resources of the idle node during the idle time period, avoiding the waste of node resources, and improving the resource utilization of the entire target cluster.

[0061] Example 2

[0062] Figure 2 This is a flowchart of a task processing method provided in Embodiment 2 of the present invention. This embodiment is a further optimization of the method in the above embodiment. Optionally, when the matching result during the idle period is successful, the task status of each node during the idle period is obtained, and nodes with an idle task status are selected to obtain at least one idle node. The node information of the idle node includes a node level identifier. The task priority identifier in the task to be processed is obtained. The idle nodes are traversed, and based on the pre-built mapping relationship between the task priority identifier and the node level identifier, any idle node corresponding to the task to be processed is determined, and any idle node is taken as the target node. Figure 2 As shown, the method includes:

[0063] S210. Obtain the pending tasks and node idle time periods, wherein the pending tasks carry the task start time and the allowed delay time.

[0064] S220. Based on the task start time, the allowed delay time of the task, and the idle time period of the node, perform idle time period matching for the task to be processed, and determine the idle time period matching result of the task to be processed.

[0065] S230. If the matching result is successful during the idle period, obtain the task status of each node during the idle period, select the node whose task status is idle, and obtain at least one idle node. The node information of the idle node includes the node level identifier.

[0066] Specifically, if the matching result is successful during the idle period, the query interface is called to query the task status of each node during the node's idle period. Nodes with an idle task status are filtered out, and the filtering result includes at least one idle node. The node information of each idle node includes a node level identifier. It should be noted that the node level identifier is a level identifier that is pre-set for each node, and the corresponding executable task can be assigned to the node according to the node level identifier.

[0067] S240. Obtain the task priority identifier in the pending tasks.

[0068] Specifically, task priority identifiers can be extracted from the task information corresponding to the pending task. If no task priority identifier is identified in the task information corresponding to the pending task, the corresponding task priority identifier can be matched according to the task type of the pending task. The task priority identifiers corresponding to different task types will be different. The task type is specifically set according to the corresponding business type, including but not limited to batch processing task types and audit task types. Generally, batch processing task types have the lowest priority, while audit tasks have a relatively higher priority than batch processing tasks.

[0069] S250. Traverse the idle nodes. Based on the pre-built mapping relationship between task priority identifiers and node level identifiers, determine any idle node corresponding to the task to be processed, and take any idle node as the target node.

[0070] Specifically, it iterates through the idle nodes and reads the node level identifier of each idle node. It then calls the constructed mapping relationship between task priority identifiers and node level identifiers, matching the task priority identifiers in the tasks to be processed with the constructed mapping relationship to determine the node level identifier that matches the task priority identifier. Based on the matched node level identifier, it matches it with the node level identifiers of each node to determine the matching idle nodes. Finally, it selects any idle node from the matched idle nodes as the target node.

[0071] S260. Schedule the task to be processed to the target node and execute the task on the target node.

[0072] The technical solution of this embodiment involves obtaining the task to be processed and the idle time period of the node, wherein the task to be processed carries the task start time and the allowed delay time; matching the idle time period of the task to be processed with the idle time period based on the task start time, the allowed delay time and the idle time period of the node to determine the idle time period matching result of the task to be processed; if the idle time period matching result is successful, obtaining the task status of each node within the node's idle time period, selecting the node whose task status is idle to obtain at least one idle node, wherein the node information of the idle node includes a node level identifier; obtaining the task priority identifier in the task to be processed; traversing the idle nodes, based on the pre-built mapping relationship between the task priority identifier and the node level identifier, determining any idle node corresponding to the task to be processed, and taking any idle node as the target node; scheduling the task to be processed to the target node, and executing the task to be processed on the target node. This solution allocates corresponding idle nodes to pending tasks during their idle periods for scheduling. This eliminates the need for separate application and additional resource allocation for pending tasks. It automatically assigns resource nodes and determines the processing time for pending tasks, allowing them to utilize the idle resources of corresponding nodes during their idle periods. This fully utilizes the resources of idle nodes during idle periods, avoids wasting node resources, and improves the resource utilization of the entire target cluster.

[0073] Example 3

[0074] Figure 3 This is a schematic diagram of the structure of a task processing device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes:

[0075] The information acquisition module 310 is used to acquire the tasks to be processed and the idle time periods of the nodes. The tasks to be processed carry the task start time and the allowed delay time of the task.

[0076] The idle time period matching result determination module 320 is used to match the idle time period of the task to be processed based on the task start time, the allowed delay time of the task, and the idle time period of the node, and determine the idle time period matching result of the task to be processed.

[0077] The target node determination module 330 is used to obtain the idle node corresponding to the idle time period when the matching result is successful during the idle time period, and determine the target node based on the idle node.

[0078] The pending task processing module 340 is used to schedule pending tasks to target nodes and execute the pending tasks on the target nodes.

[0079] The technical solution of this embodiment obtains the tasks to be processed and the idle time periods of nodes through an information acquisition module. The tasks to be processed carry the task start time and the allowed delay time. An idle time period matching result determination module matches the tasks to be processed with the idle time periods based on the task start time, the allowed delay time, and the node idle time periods to determine the matching result. A target node determination module, when the idle time period matching result is successful, obtains the idle node corresponding to the node's idle time period and determines the target node based on the idle node. The task processing module schedules the tasks to be processed to the target node and executes them on the target node. This solution allocates corresponding idle nodes to tasks to be processed during the node's idle time periods for scheduling the tasks. This eliminates the need for separate application and additional resource allocation for tasks, automatically allocating resource nodes and determining the processing time for tasks. This allows tasks to utilize the idle resources of corresponding nodes during idle time periods, fully utilizing the resources of idle nodes during idle time periods, avoiding node resource waste, and improving the resource utilization of the entire target cluster.

[0080] Based on the above embodiments, optionally, the information acquisition module 310 is specifically used to display the corresponding task configuration options in response to the creation operation of the task to be processed, wherein the task configuration options include a task start time option and a task allowable delay time option; in response to the selection operation of the task start time option and the task allowable delay time option, the task start time and the task allowable delay time are determined, and the task to be processed is generated based on the task start time, the task allowable delay time and the basic information of the task to be processed, wherein the basic information includes the task name, the task type and the project group information.

[0081] Optionally, the information acquisition module 310 is specifically used to acquire historical operating data of nodes, perform prediction processing based on the historical operating data of nodes and a pre-trained idle time prediction model to obtain the node idle prediction information of each node; and determine the node idle time based on the node idle prediction information of each node.

[0082] Optionally, the target node determination module 330 is specifically used to obtain the task status of each node during the idle period, select the node whose task status is idle, and obtain at least one idle node. The node information of the idle node includes the node level identifier; obtain the task priority identifier in the task to be processed; traverse the idle nodes, and determine any idle node corresponding to the task to be processed based on the pre-built mapping relationship between the task priority identifier and the node level identifier, and take any idle node as the target node.

[0083] Optionally, the node information of the idle node also includes the remaining resource quantity and the resource utilization rate during the idle period; the pending task processing module 340 is specifically used to determine the resource configuration information of the pending task based on the remaining resource quantity and the resource utilization rate during the idle period of the target node; create a task instance on the target node based on the resource configuration information, and schedule the pending task to the task instance on the target node.

[0084] Optionally, the resource configuration information includes minimum resource information and resource limit information, wherein the minimum resource information is less than the resource limit information; the pending task processing module 340 is specifically used to determine the minimum resource information of the pending task based on the remaining resources of the target node; and to determine the resource limit information of the pending task based on the resource utilization rate during idle periods and the total resources of the target node.

[0085] Optionally, after scheduling the pending task to the target node, the device is specifically used to, during the execution of the pending task, if it detects that the target node has received a new pending task, obtain the task priority identifier corresponding to the new pending task; if it is determined based on the task priority identifier corresponding to the new pending task and the task priority identifier corresponding to the pending task that the task priority of the new pending task is higher than the task priority of the pending task, suspend the execution of the pending task, store the pending task in the pending task list, and schedule the pending task to the new target node if a new target node is detected; if the task priority of the new pending task is lower than the task priority of the pending task, continue to execute the pending task.

[0086] The task processing device provided in the embodiments of the present invention can execute the task processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0087] Example 4

[0088] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0089] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0090] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0091] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as task processing methods.

[0092] In some embodiments, the task processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the task processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the task processing method by any other suitable means (e.g., by means of firmware).

[0093] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0094] Computer programs for implementing the task processing methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0095] Example 5

[0096] Embodiment 5 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a task processing method, the method comprising:

[0097] Obtain the pending tasks and node idle time periods, where the pending tasks carry the task start time and the allowed delay time;

[0098] Based on the task start time, the allowed delay time of the task, and the idle time period of the node, the idle time period of the task to be processed is matched to determine the idle time period matching result of the task to be processed.

[0099] If the matching result during the idle period is successful, obtain the idle node corresponding to the idle period of the node, and determine the target node based on the idle node;

[0100] The pending tasks are scheduled to the target node and executed on the target node.

[0101] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0103] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0104] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0105] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0106] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A task processing method, characterized in that: include: Obtaining pending tasks and node idle periods, wherein the pending tasks carry the task start time and the task allowed delay time; Matching the idle period of the task to be processed based on the task start time, the task allowed delay time and the node idle period, and determining the idle period matching result of the task to be processed; When the idle period matching result is a successful match, obtaining an idle node corresponding to the node idle period, and determining a target node based on the idle node; The to-be-processed task is dispatched to the target node, and the to-be-processed task is executed on the target node.

2. The method according to claim 1, characterized in that The obtaining of tasks to be processed includes: In response to the creation operation of the task to be processed, display the corresponding task configuration options, wherein the task configuration options include a task start time option and a task allowable delay time option; In response to the selection operation of the task start time option and the task allowed delay time option, the task start time and the task allowed delay time are determined, and the to-be-processed task is generated based on the task start time, the task allowed delay time and the basic information of the to-be-processed task, wherein the basic information includes the task name, task type and project group information.

3. The method according to claim 1, characterized in that Obtaining the idle period of the node, including: Acquire historical operation data of the nodes, perform prediction processing based on the historical operation data of the nodes and a pre-trained idle period prediction model, and obtain node idle period prediction information of each of the nodes; The node idle period is determined based on the node idle prediction information of each of the nodes.

4. The method according to claim 1, characterized in that: The acquiring of idle nodes corresponding to the node idle period, and determining a target node based on the idle nodes, comprises: Acquire the task status of each node in the node idle period, select a node whose task status is an idle state, and obtain at least one idle node, wherein the node information of the idle node includes a node level identifier; Obtaining a task priority identifier in the pending task; The idle nodes are traversed, and based on the pre-constructed mapping relationship between the task priority identifier and the node level identifier, any idle node corresponding to the task to be processed is determined, and the any idle node is used as the target node.

5. The method according to claim 1, characterized in that The node information of the idle node also includes the remaining resource amount and the resource utilization rate during the idle period; The step of scheduling the to-be-processed task to the target node comprises: Determine resource configuration information of the task to be processed based on the remaining resource amount of the target node and the resource utilization rate in the idle period; A task instance is created at the target node based on the resource configuration information, and the to-be-processed task is scheduled to the task instance of the target node.

6. The method according to claim 5, characterized in that The resource configuration information includes minimum resource information and resource limit information, wherein the minimum resource information is smaller than the resource limit information; The determining the resource configuration information of the task to be processed based on the remaining resource amount of the target node and the resource usage rate in the idle period includes: Determine the minimum resource information of the task to be processed based on the remaining resource amount of the target node; The resource quota information of the task to be processed is determined based on the resource usage rate of the idle period and the total amount of resources of the target node.

7. The method according to claim 1, characterized in that After scheduling the to-be-processed task to the target node, the method further includes: During the execution of the task to be processed, if it is detected that the target node receives a newly added task to be processed, obtaining a task priority identifier corresponding to the newly added task to be processed; In the case where it is determined based on the task priority identifier corresponding to the newly added task to be processed and the task priority identifier corresponding to the task to be processed that the task priority of the newly added task to be processed is higher than the task priority of the task to be processed, suspending the execution of the task to be processed, storing the task to be processed in the list of tasks to be processed, and scheduling the task to be processed to the new target node when a new target node is detected; When the task priority of the newly added tasks to be processed is lower than the task priority of the tasks to be processed, the tasks to be processed continue to be executed.

8. A task processing device, characterized in that: include: An information acquisition module is used to acquire pending tasks and node idle periods, wherein the pending tasks carry the task start time and the task allowed delay time; An idle period matching result determination module is used to match the idle period of the task to be processed based on the task start time, the task allowed delay time and the node idle period, and determine the idle period matching result of the task to be processed; A target node determination module, configured to obtain an idle node corresponding to the node idle period if the idle period matching result is a successful match, and determine a target node based on the idle node; The pending task processing module is used to schedule the pending task to the target node and execute the pending task on the target node.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the task processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the task processing method according to any one of claims 1 to 7 when executed.

Images