Task scheduling method and device, medium and equipment

Abstract

The invention discloses a task scheduling method and device, a medium and equipment. And obtaining task description information of the calculation task. And inputting the task description information into a pre-trained first prediction model to obtain first execution time of the calculation task output by the first prediction model. And determining a resource scheduling strategy of the calculation task according to the task description information and the first execution time. And executing the calculation task according to the resource scheduling strategy and the task description information. Determining resource utilization information in the execution process of the calculation task, and determining second execution time of the calculation task according to the resource utilization information. According to the second execution time, a resource scheduling strategy of the calculation task is adjusted, and the calculation task continues to be executed according to the adjusted resource scheduling strategy. By dynamically adjusting the resource scheduling strategy of the calculation task, the utilization efficiency of calculation resources and the execution efficiency of the calculation task are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task scheduling method, device, medium and equipment. Background Art

[0002] At present, with the rapid development of technologies such as big data, cloud computing and artificial intelligence, large-scale intelligent computing clusters (hereinafter referred to as clusters) composed of computing resources such as thousands of central processing units (CPUs), graphic processing units (GPUs) and other dedicated hardware accelerators can provide powerful computing capabilities for data centers or cloud computing platforms to handle various computing tasks.

[0003] However, with the expansion of cluster size and the diversification of computing task types, how to effectively manage and schedule the huge computing resources in the cluster to ensure that computing tasks can be completed within a reasonable time while maximizing computing resource utilization has become a challenge.

[0004] Existing methods for cluster computing resource management and computing task scheduling cannot accurately predict the execution time of computing tasks. If the predicted execution time is too long, it may lead to the allocation of too many computing resources, resulting in resource waste and subsequent delays in the execution of computing tasks. If the predicted execution time is too short, it may lead to the failure of computing task execution or intensified competition for computing resources.

[0005] To this end, this specification provides a task scheduling method, apparatus, medium and device. Summary of the invention

[0006] This specification provides a task scheduling method, device, medium and equipment to partially solve the above-mentioned problems existing in the prior art.

[0007] This manual adopts the following technical solutions:

[0008] This specification provides a task scheduling method, including:

[0009] Get the task description information of the computing task;

[0010] Inputting the task description information into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model;

[0011] Determining a resource scheduling strategy for the computing task according to the task description information and the first execution time;

[0012] Execute the computing task according to the resource scheduling strategy and the task description information;

[0013] Determining resource utilization information during the execution of the computing task, and determining a second execution time of the computing task based on the resource utilization information;

[0014] The resource scheduling policy of the computing task is adjusted according to the second execution time, so as to continue to execute the computing task according to the adjusted resource scheduling policy.

[0015] Optionally, inputting the task description information into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model specifically includes:

[0016] Extracting features from the task description information to determine features of the task description information;

[0017] Determining a first pre-trained prediction model that matches the computing task according to the characteristics of the task description information;

[0018] The features of the task description information are input into the first prediction model to obtain a first execution time of the computing task output by the first prediction model.

[0019] Optionally, training a first prediction model includes:

[0020] Obtaining task description information of each historical computing task, and the actual execution time of each historical computing task;

[0021] Extracting features from the task description information of each of the historical computing tasks to determine features of each of the historical computing tasks;

[0022] Clustering the features of each of the historical computing tasks to determine each clustering result;

[0023] For each clustering result, a target feature is selected from the clustering result, and the target feature is input into a first prediction model to be trained to obtain a prediction result output by the first prediction model to be trained;

[0024] A loss value is determined based on a difference between the prediction result and an actual execution time of a historical computing task corresponding to the target feature, and the first prediction model to be trained is trained based on the loss value, wherein the difference is positively correlated with the loss value.

[0025] Optionally, determining the second execution time of the computing task according to the resource utilization information specifically includes:

[0026] Extracting features from the resource utilization information and the task description information to obtain features of the computing task;

[0027] The features of the computing task are input into a pre-trained second prediction model to obtain a second execution time output by the second prediction model.

[0028] Optionally, training a second prediction model includes:

[0029] Obtaining description information of historical computing tasks and resource utilization information of the historical computing tasks;

[0030] Extracting features from the description information and resource utilization information of the historical computing tasks to determine features of the historical computing tasks;

[0031] Inputting the characteristics of the historical computing task into a second prediction model to be trained, and obtaining a prediction result output by the second prediction model to be trained;

[0032] A loss value is determined based on the difference between the prediction result and the actual execution time of the historical computing task, and the second prediction model to be trained is trained based on the loss value, wherein the difference is positively correlated with the loss value.

[0033] Optionally, adjusting the resource scheduling policy of the computing task according to the second execution time to continue executing the computing task according to the adjusted resource scheduling policy specifically includes:

[0034] In the case where the second execution time is greater than the first execution time, determining the computing resources that need to be continuously allocated to the computing task and the usage time of the computing resources that need to be continuously allocated as the increased resource information of the computing task;

[0035] If it is determined that there are no idle computing resources that meet the requirements of the resource increase information, then determine the second execution time of the other computing tasks according to the resource utilization information collected during the execution of the other computing tasks, and determine the computing tasks that need to reduce computing resources according to the second execution time of the other computing tasks as the resource-sufficient tasks;

[0036] From the various resource-sufficient tasks, determine a resource-sufficient task whose reduced computing resources and usage time of the computing resources satisfy the increased resource information of the computing task as a target task;

[0037] Using the description information of the part of the tasks required to add the resource information in the computing task as the first description information, and using the description information of the remaining part of the tasks in the computing task except the part of the tasks as the second description information;

[0038] From the computing resources required to execute the target task, determine the computing resources that match the computing resources in the added resource information and execute the part of the task corresponding to the first description information, and continue to execute the remaining tasks according to the resource scheduling strategy and the second description information.

[0039] Optionally, determining a resource scheduling strategy for the computing task according to the task description information and the first execution time specifically includes:

[0040] Determining an initial resource scheduling strategy for the computing task according to the task description information and the first execution time;

[0041] Determining, according to the computing resources included in the initial resource scheduling strategy and the usage time of the computing resources, an execution efficiency value of executing the computing task according to the initial resource scheduling strategy;

[0042] Taking maximizing the execution efficiency value as an optimization goal, the computing resources included in the initial resource scheduling strategy and the usage time of the computing resources are adjusted to determine the resource scheduling strategy when the computing task is executed.

[0043] This specification provides a task scheduling device, including:

[0044] An acquisition module is used to obtain task description information of a computing task;

[0045] An input module, used for inputting the task description information into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model;

[0046] A first determining module, configured to determine a resource scheduling strategy for the computing task according to the task description information and the first execution time;

[0047] An execution module, used for executing the computing task according to the resource scheduling strategy and the task description information;

[0048] a second determining module, configured to determine resource utilization information during the execution of the computing task, and determine a second execution time of the computing task according to the resource utilization information;

[0049] An adjustment module is used to adjust the resource scheduling policy of the computing task according to the second execution time, so as to continue to execute the computing task according to the adjusted resource scheduling policy.

[0050] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned task scheduling method is implemented.

[0051] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a task scheduling method when executing the program.

[0052] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0053] The task scheduling method provided in this specification first obtains the task description information of the computing task. The task description information is input into a pre-trained first prediction model to obtain the first execution time of the computing task output by the first prediction model. The resource scheduling strategy of the computing task is determined based on the task description information and the first execution time. The computing task is executed based on the resource scheduling strategy and the task description information. The resource utilization information during the execution of the computing task is determined, and the second execution time of the computing task is determined based on the resource utilization information. The resource scheduling strategy of the computing task is adjusted based on the second execution time to continue to execute the computing task based on the adjusted resource scheduling strategy.

[0054] Through the first prediction model, the execution time of the computing task is predicted before the computing task is executed, so as to allocate appropriate computing resources to the computing task and perform task scheduling. During the execution of the computing task, the execution time of the computing task is determined again according to the resource utilization information of the computing task execution, and the resource scheduling strategy of the computing task is dynamically adjusted to improve the utilization efficiency of computing resources and the execution efficiency of computing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0056] Figure 1 A flowchart of a task scheduling method provided in an embodiment of this specification;

[0057] Figure 2 A schematic diagram of large-scale intelligent computing cluster resource scheduling provided in this manual;

[0058] Figure 3 A schematic diagram of a large-scale intelligent computing cluster resource scheduling process provided in this specification;

[0059] Figure 4 A schematic diagram of a task scheduling device provided in this specification;

[0060] Figure 5 A method corresponding to the Figure 1Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0063] Figure 1 A flowchart of a task scheduling method provided in an embodiment of this specification includes the following steps:

[0064] S100: Obtain task description information of a computing task.

[0065] In this specification, the process of performing execution time prediction and resource scheduling can be performed by a server in the embodiments of this specification. Of course, this specification does not limit which device performs the process of execution time prediction and resource scheduling, and devices such as personal computers and mobile terminals can also be used to perform execution time prediction and resource scheduling. For the convenience of description, the following description is based on the server that controls a large-scale intelligent computing cluster (hereinafter referred to as the server) as the execution subject.

[0066] In one or more embodiments of the present specification, the server obtains task description information of the computing task to be executed. The task description information includes the computing task number (ID), the number of GPUs required for the computing task, the number of CPUs required for the computing task, the task creation time, the task start command, the task parallel computing method, and other contents.

[0067] S102: Input the task description information into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model.

[0068] In one or more embodiments of the present specification, the server inputs the acquired description information of the computing task into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model.

[0069] Specifically, the server extracts features from the task description information, determines features of the task description information, and determines a pre-trained first prediction model that matches the computing task based on the features of the task description information. The server can then input the features of the task description information into the first prediction model to obtain a first execution time of the computing task output by the first prediction model.

[0070] Among them, the method for training the first prediction model is that the server obtains the task description information of each historical computing task and the actual execution time of each historical computing task. Then, the task description information of each historical computing task is extracted to determine the characteristics of each historical computing task. The server can cluster the characteristics of each historical computing task to determine each clustering result. Then, for each clustering result, the server selects the target feature from the clustering result, and inputs the target feature into the first prediction model to be trained to obtain the prediction result output by the first prediction model to be trained. Afterwards, the server can determine the loss value based on the difference between the actual execution time of the historical computing task corresponding to the prediction result and the target feature, and train the first prediction model to be trained based on the loss value, taking the minimum loss value as the optimization goal, wherein the difference is positively correlated with the determined loss value.

[0071] It is worth noting that, in this specification, the form of expression of the task description information obtained by the server is not limited, and the task description information can be in a variety of forms. Therefore, when the task description information is subjected to feature extraction, if the task description information is a segment of audio information, the audio information can be converted into the task description information in text form by voice recognition technology. If the task description information is image data, the task description information in the image data can be extracted by image recognition technology, and the image data can be converted into the task description information in text form. Of course, the task description information obtained by the server can also be the task description information in text form, and no additional text conversion steps are required. Afterwards, the server extracts the vectors from the text in the task description information, such as the task ID, the number of GPUs, and the number of CPUs, through a word vector model (such as Word2Vec), and obtains the feature vector matrix of the task description information. The server then clusters the feature vector matrix through a clustering algorithm, and the clustering algorithm is not limited in this specification, such as the K-means (KMEANS) algorithm or the density-based spatial clustering (Density-Based Spatial Clustering of Applications withNoise, DBSCAN) algorithm.

[0072] S104: Determine a resource scheduling strategy for the computing task according to the task description information and the first execution time.

[0073] S106: Execute the computing task according to the resource scheduling strategy and the task description information.

[0074] In one or more embodiments of the present specification, the server determines the resource scheduling strategy for the computing task based on the determined first execution time and the task description information of the computing task. For example, the first execution time of the computing task is relatively long, and based on the information such as the number of GPUs or the number of CPUs required for the computing task described in the task description information, in order to ensure the efficient execution of the computing task and maximize the utilization of computing resources, the resource scheduling strategy determined by the server can allocate more computing resources and the use time of computing resources to the computing task to meet the longer first execution time and ensure the smooth execution of the computing task. Alternatively, the server determines that the first execution time of the computing task is 6 hours in the early morning of the same day. Since the network bandwidth and other computing resources are less occupied by humans during the night execution, more computing resources can be allocated to the computing task to speed up the completion of the computing task, without having to delay until daytime to complete the computing task, thereby avoiding the competition for computing resources of large-scale intelligent computing clusters during the daytime.

[0075] After that, computing resources can be allocated to computing tasks according to resource scheduling strategies, and computing tasks can be executed according to task description information. It is worth noting that computing resources are computing devices included in large-scale intelligent computing clusters such as GPUs, CPUs, and other dedicated hardware accelerators.

[0076] S108: Determine resource utilization information during the execution of the computing task, and determine a second execution time of the computing task based on the resource utilization information.

[0077] S110: According to the second execution time, the resource scheduling policy of the computing task is adjusted to continue executing the computing task according to the adjusted resource scheduling policy.

[0078] In one or more embodiments of the present specification, in order to maximize the computing resource utilization in a large-scale intelligent computing cluster and dynamically adjust the usage of computing resources of computing tasks to better execute computing tasks, during the execution of computing tasks, the server collects resource utilization information during the execution of computing tasks, and determines the second execution time of the computing tasks based on the resource utilization information.

[0079] Specifically, the server extracts features from the resource utilization information and the task description information to obtain features of the computing task, and then inputs the features of the computing task into a pre-trained second prediction model to obtain a second execution time output by the second prediction model.

[0080] Among them, the resource utilization information can be information such as GPU utilization, CPU utilization, and memory utilization, which is mainly used to indicate the utilization status of computing resources. The second execution time at this time is the execution time of the computing task evaluated according to the computing resource utilization status of the computing task, that is, the second execution time is the duration obtained after the server actually executes the computing task for a period of time according to the resource scheduling strategy, and then predicts the execution time of the computing task again according to the utilization status of the computing resources during the execution process. It is worth noting that the resource utilization information determined by the server can also be in a variety of forms. When extracting features from the resource utilization information, if the resource utilization information is a piece of audio information, the audio information can be converted into resource utilization information in text form through speech recognition technology. If the resource utilization information is image data, the resource utilization information in the image data can be extracted through image recognition technology to realize the conversion of the image data into resource utilization information in text form. Of course, the resource utilization information determined by the server can also be resource utilization information in text form, and no additional text conversion steps are required. Therefore, for the feature extraction of resource utilization information, reference can be made to the above-mentioned method of feature extraction of task description information.

[0081] Afterwards, the server can adjust the resource scheduling policy of the computing task according to the second execution time of the computing task. Among them, when the second execution time is the time required for the predicted completion of the computing task as a whole, if the second execution time is greater than the first execution time, it indicates that the computing resources allocated to the computing task in the resource scheduling policy determined according to the first execution time are not sufficient, so it is necessary to adjust the resource scheduling policy to allocate more computing resources to the computing task to meet its needs. If the second execution time is less than or equal to the first execution time, it indicates that the computing resources allocated to the computing task in the resource scheduling policy determined according to the first execution time are sufficient, and the resource scheduling policy is not adjusted at this time. Therefore, in this specification, not adjusting the resource scheduling policy can also be regarded as a way to adjust the resource scheduling policy, that is, zero adjustment. Of course, the second execution time can also be the remaining execution time of the computing task, which is not limited in this specification.

[0082] In this specification, the method for training the second prediction model may be that the server obtains the description information of the historical computing tasks and the resource utilization information of the historical computing tasks. Then, the server may extract features from the description information and resource utilization information of the historical computing tasks, determine the features of the historical computing tasks, and input the features of the historical computing tasks into the second prediction model to be trained to obtain the prediction results output by the second prediction model to be trained. In the subsequent process, the server may determine the loss value based on the difference between the prediction result and the actual execution time of the historical computing tasks, and train the second prediction model to be trained based on the loss value with the minimum loss value as the optimization goal, wherein the difference is positively correlated with the loss value.

[0083] In addition, in one or more embodiments of the present specification, when the second execution time is greater than the first execution time, it means that the computing resources allocated to the computing task in the resource scheduling strategy determined according to the first execution time are not sufficient. Therefore, in order to ensure that the computing task can be executed smoothly and will not stop executing the computing task due to lack of computing resources, and to improve the execution efficiency of the computing task, the server needs to further determine the computing resources that need to be continuously allocated to the computing task and the usage time of the computing resources that need to be continuously allocated as the additional resource information of the computing task.

[0084] If the server determines that in a large-scale intelligent computing cluster, it is impossible to continue to allocate computing resources for computing tasks because there are no idle computing resources that meet the requirements of increasing resource information, then it is necessary to determine the second execution time of other computing tasks based on the resource utilization information collected during the execution of other computing tasks, and determine the computing tasks that need to reduce computing resources based on the second execution time of other computing tasks as resource-sufficient tasks, that is, other computing tasks whose second execution time is less than the first execution time are regarded as resource-sufficient tasks. Therefore, the resource-sufficient tasks mentioned here refer to the computing resources included in the resource scheduling strategy allocated by the server to them within the first execution time, which are not only sufficient for their own tasks to be completed on time within the first execution time, but also have computing resources that can be called to execute the remaining computing tasks without affecting the completion of their own tasks on time.

[0085] After that, the server can determine the resource-sufficient task whose reduced computing resources and computing resource usage time meet the resource-sufficient information from each resource-sufficient task as the target task. The description information of the part of the task that needs to be executed to increase the resource information in the computing task is used as the first description information, and the description information of the remaining part of the task except the part of the task corresponding to the resource-sufficient information in the computing task is used as the second description information.

[0086] Finally, the server can determine the computing resources that match the computing resources in the added resource information from the computing resources required to execute the target task and execute the part of the task corresponding to the first description information, and continue to execute the remaining tasks according to the resource scheduling strategy and the second description information.

[0087] When there are no idle computing resources required for computing tasks in a large-scale intelligent computing cluster, the allocated computing resources can still be circulated, thereby maximizing the utilization of computing resources in the large-scale intelligent computing cluster. At the same time, it can also ensure that the computing tasks are completed within a reasonable execution time, thereby improving the execution efficiency of the computing tasks.

[0088] In one or more embodiments of the present specification, in order to further ensure that the determined resource scheduling strategy can ensure efficient execution of the computing task while maximizing the computing resource utilization, the server may also determine the initial resource scheduling strategy of the computing task based on the task description information and the first execution time. Afterwards, the server determines the execution efficiency value of the computing task executed according to the initial resource scheduling strategy based on the computing resources included in the initial resource scheduling strategy and the usage time of the computing resources. The execution efficiency value can refer to the resource utilization information of the historical computing tasks that are allocated the same computing resources as the computing task, so as to determine the execution efficiency value of the computing task.

[0089] With the maximum execution efficiency value as the optimization goal, the computing resources and the usage time of the computing resources included in the initial resource scheduling strategy are adjusted to determine the resource scheduling strategy of the computing task when it is executed. This process can be understood as determining the maximum execution efficiency value based on the resource utilization information of the historical computing tasks that are assigned the same computing resources as the computing task, and determining the resource scheduling strategy of the computing task based on the adjusted resource scheduling strategy of the historical computing tasks.

[0090] Figure 2 A schematic diagram of large-scale intelligent computing cluster resource scheduling provided in this manual. Figure 2 The medium-to-large-scale intelligent computing cluster includes a front-end task receiving and prediction module, a database and task monitoring module, and a back-end task starting and scheduling module. Among them, the server receives the task description file of the computing task submitted by the user through the front-end task receiving and static prediction module, and parses it into task description information, and then inputs the first prediction model to obtain the predicted execution time. The server then submits the predicted execution time and the task description file to form the task complete information to the database and task monitoring module, stores it in the database, and the process monitoring scans the database to detect the computing task submitted to the database. Then, after the server detects the occurrence of the computing task through the database and task monitoring module, it sends the task complete information to the back-end task starting and scheduling module. After that, the server starts the computing task through the back-end task starting and scheduling module, sends the task complete information to the scheduler and optimizer, determines the resource scheduling strategy, and executes the computing task according to the resource scheduling strategy. And during the execution of the computing task, the server detects the resource utilization information through the back-end task starting and scheduling module and stores it in the database, and predicts the new execution time through the second prediction model based on the resource utilization information and the task complete information. Finally, the server adjusts the resource scheduling strategy according to the new execution time and continues to execute the computing task.

[0091] Figure 3 This is a flow chart of large-scale intelligent computing cluster resource scheduling provided in this specification. Figure 3As shown, the server determines the task description information through the front-end task receiving and prediction module, and predicts the computing task execution time through the first prediction model to determine the prediction value. Then, the server sends the task description information and the execution time prediction value to the database and task monitoring module. When the database and task monitoring module detect the new task, the server sends the task description information and the prediction value to the back-end task startup and scheduling module to start the computing task. Through the back-end task startup and scheduling module, the server sends the task description information and the prediction value, that is, (task initial static information) to the scheduling and optimizer, determines the initial resource scheduling strategy, and executes the computing task according to the initial resource scheduling strategy. During the execution of the computing task, the server detects the resource utilization information through the back-end task startup and scheduling module, and predicts the new execution time (task execution time adjustment value) through the second prediction model according to the resource utilization information. Finally, the server determines whether to adjust the resource scheduling strategy according to the new execution time. If not, it continues to execute the computing task. If so, it performs secondary scheduling of computing resources to determine the resource scheduling strategy.

[0092] In summary, the task scheduling method provided in this specification predicts the execution time of the computing task through the first prediction model before the computing task is executed, so as to allocate suitable computing resources to the computing task and perform task scheduling. And in the process of computing task execution, the execution time of the computing task is determined again according to the resource utilization information of the computing task execution, and the resource scheduling strategy of the computing task is dynamically adjusted to improve the utilization efficiency of computing resources and the execution efficiency of computing tasks. Moreover, when there are no idle computing resources required for computing tasks in large-scale intelligent computing clusters, the already allocated computing resources can still be circulated, so that the computing resource utilization rate in the large-scale intelligent computing cluster is maximized, and the computing task can also be completed within a reasonable execution time, thereby improving the execution efficiency of the computing task. At the same time, the resource scheduling strategy of the computing task can be determined with reference to the adjusted resource scheduling strategy of the historical computing task to maximize the execution efficiency of the computing task.

[0093] The above is a task scheduling method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding task scheduling device, such as Figure 4 shown.

[0094] Figure 4 A schematic diagram of a task scheduling device provided in this specification specifically includes:

[0095] An acquisition module 400 is used to acquire task description information of a computing task;

[0096] An input module 402 is used to input the task description information into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model;

[0097] A first determination module 404, configured to determine a resource scheduling strategy for the computing task according to the task description information and the first execution time;

[0098] An execution module 406, configured to execute the computing task according to the resource scheduling strategy and the task description information;

[0099] A second determination module 408, configured to determine resource utilization information during the execution of the computing task, and determine a second execution time of the computing task according to the resource utilization information;

[0100] The adjustment module 410 is used to adjust the resource scheduling policy of the computing task according to the second execution time, so as to continue to execute the computing task according to the adjusted resource scheduling policy.

[0101] Optionally, the input module 402 is used to extract features from the task description information, determine features of the task description information, determine a pre-trained first prediction model that matches the computing task based on the features of the task description information, input the features of the task description information into the first prediction model, and obtain a first execution time of the computing task output by the first prediction model.

[0102] Optionally, the input module 402 is used to obtain task description information of each historical computing task, the actual execution time of each historical computing task, perform feature extraction on the task description information of each historical computing task, determine the characteristics of each historical computing task, cluster the characteristics of each historical computing task, determine each clustering result, for each clustering result, select a target feature from the clustering result, and input the target feature into the first prediction model to be trained to obtain a prediction result output by the first prediction model to be trained, determine a loss value based on the difference between the prediction result and the actual execution time of the historical computing task corresponding to the target feature, and train the first prediction model to be trained based on the loss value, wherein the difference is positively correlated with the loss value.

[0103] Optionally, the second determination module 408 is used to perform feature extraction on the resource utilization information and the task description information to obtain features of the computing task, input the features of the computing task into a pre-trained second prediction model, and obtain a second execution time output by the second prediction model.

[0104] Optionally, the second determination module 408 is used to obtain the description information of the historical computing task and the resource utilization information of the historical computing task, perform feature extraction on the description information and resource utilization information of the historical computing task, determine the characteristics of the historical computing task, input the characteristics of the historical computing task into the second prediction model to be trained, obtain the prediction result output by the second prediction model to be trained, determine the loss value based on the difference between the prediction result and the actual execution time of the historical computing task, and train the second prediction model to be trained based on the loss value, wherein the difference is positively correlated with the loss value.

[0105] Optionally, the adjustment module 410 is used to determine the computing resources that need to be continuously allocated to the computing task and the usage time of the computing resources that need to be continuously allocated, as the increased resource information of the computing task, if it is determined that there are no idle computing resources that meet the requirements of the increased resource information, then determine the second execution time of the other computing tasks according to the resource utilization information collected during the execution of the other computing tasks, and determine the computing tasks that need to reduce computing resources according to the second execution time of the other computing tasks as the resource-sufficient tasks, and determine the reduced computing resources from each resource-sufficient task. A resource-sufficient task whose computing resources and the usage time of the computing resources satisfy the increased resource information of the computing task is taken as the target task, the description information of the part of the tasks required to be executed by the increased resource information in the computing task is taken as the first description information, and the description information of the remaining tasks in the computing task except the part of the tasks is taken as the second description information, from the computing resources required to execute the target task, the computing resources that match the computing resources in the increased resource information are determined and the part of the tasks corresponding to the first description information is executed, and the remaining tasks are continued to be executed according to the resource scheduling strategy and the second description information.

[0106] Optionally, a first determination module 404 is used to determine an initial resource scheduling strategy for the computing task based on the task description information and the first execution time, and determine an execution efficiency value of executing the computing task according to the initial resource scheduling strategy based on the computing resources included in the initial resource scheduling strategy and the usage time of the computing resources, so as to maximize the execution efficiency value as an optimization goal, adjust the computing resources included in the initial resource scheduling strategy and the usage time of the computing resources to determine the resource scheduling strategy for the computing task during execution.

[0107] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1Provides a task scheduling method.

[0108] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The task scheduling method.

[0109] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0110] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0111] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0112] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0113] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0114] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0115] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0119] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0120] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0122] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0124] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0125] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A task scheduling method, characterized in that: include: Get the task description information of the computing task; Inputting the task description information into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model; Determining a resource scheduling strategy for the computing task according to the task description information and the first execution time; Execute the computing task according to the resource scheduling strategy and the task description information; Determining resource utilization information during the execution of the computing task, and determining a second execution time of the computing task based on the resource utilization information; The resource scheduling policy of the computing task is adjusted according to the second execution time, so as to continue to execute the computing task according to the adjusted resource scheduling policy.

2. The method according to claim 1, characterized in that Inputting the task description information into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model specifically includes: Extracting features from the task description information to determine features of the task description information; Determining a first pre-trained prediction model that matches the computing task according to the characteristics of the task description information; The features of the task description information are input into the first prediction model to obtain a first execution time of the computing task output by the first prediction model.

3. The method according to claim 2, characterized in that Training the first prediction model includes: Obtaining task description information of each historical computing task, and the actual execution time of each historical computing task; Extracting features from the task description information of each of the historical computing tasks to determine features of each of the historical computing tasks; Clustering the features of each of the historical computing tasks to determine each clustering result; For each clustering result, a target feature is selected from the clustering result, and the target feature is input into a first prediction model to be trained to obtain a prediction result output by the first prediction model to be trained; A loss value is determined based on a difference between the prediction result and an actual execution time of a historical computing task corresponding to the target feature, and the first prediction model to be trained is trained based on the loss value, wherein the difference is positively correlated with the loss value.

4. The method according to claim 1, characterized in that Determining a second execution time of the computing task according to the resource utilization information specifically includes: Extracting features from the resource utilization information and the task description information to obtain features of the computing task; The features of the computing task are input into a pre-trained second prediction model to obtain a second execution time output by the second prediction model.

5. The method according to claim 4, characterized in that Training the second prediction model includes: Obtaining description information of historical computing tasks and resource utilization information of the historical computing tasks; Extracting features from the description information and resource utilization information of the historical computing tasks to determine features of the historical computing tasks; Inputting the characteristics of the historical computing task into a second prediction model to be trained, and obtaining a prediction result output by the second prediction model to be trained; A loss value is determined based on the difference between the prediction result and the actual execution time of the historical computing task, and the second prediction model to be trained is trained based on the loss value, wherein the difference is positively correlated with the loss value.

6. The method according to claim 1, characterized in that Adjusting the resource scheduling policy of the computing task according to the second execution time to continue executing the computing task according to the adjusted resource scheduling policy specifically includes: In the case where the second execution time is greater than the first execution time, determining the computing resources that need to be continuously allocated to the computing task and the usage time of the computing resources that need to be continuously allocated as the increased resource information of the computing task; If it is determined that there are no idle computing resources that meet the requirements of the resource increase information, then determine the second execution time of the other computing tasks according to the resource utilization information collected during the execution of the other computing tasks, and determine the computing tasks that need to reduce computing resources according to the second execution time of the other computing tasks as the resource-sufficient tasks; From the various resource-sufficient tasks, determine a resource-sufficient task whose reduced computing resources and usage time of the computing resources satisfy the increased resource information of the computing task as the target task; Using the description information of the part of the tasks required to add the resource information in the computing task as the first description information, and using the description information of the remaining part of the tasks in the computing task except the part of the tasks as the second description information; From the computing resources required to execute the target task, determine the computing resources that match the computing resources in the added resource information and execute the part of the task corresponding to the first description information, and continue to execute the remaining tasks according to the resource scheduling strategy and the second description information.

7. The method according to claim 1, characterized in that Determining a resource scheduling strategy for the computing task according to the task description information and the first execution time specifically includes: Determining an initial resource scheduling strategy for the computing task according to the task description information and the first execution time; Determining, according to the computing resources included in the initial resource scheduling strategy and the usage time of the computing resources, an execution efficiency value of executing the computing task according to the initial resource scheduling strategy; Taking maximizing the execution efficiency value as an optimization goal, the computing resources included in the initial resource scheduling strategy and the usage time of the computing resources are adjusted to determine the resource scheduling strategy when the computing task is executed.

8. A task scheduling device, characterized in that: include: An acquisition module is used to obtain task description information of a computing task; An input module, used for inputting the task description information into a pre-trained first prediction model to obtain a first execution time of the computing task output by the first prediction model; A first determining module, configured to determine a resource scheduling strategy for the computing task according to the task description information and the first execution time; An execution module, used for executing the computing task according to the resource scheduling strategy and the task description information; a second determining module, configured to determine resource utilization information during the execution of the computing task, and determine a second execution time of the computing task according to the resource utilization information; An adjustment module is used to adjust the resource scheduling policy of the computing task according to the second execution time, so as to continue to execute the computing task according to the adjusted resource scheduling policy.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images