Model training method and task execution time prediction method and device

Abstract

The invention discloses a model training method and a task execution time prediction method and device, and the method specifically comprises the steps: screening similar historical model training tasks according to historical resource occupation data of historical model training tasks; and inputting the historical resource occupation data of the historical model training task and the similar historical model training task into a prediction model, so that resource occupation feature data corresponding to the historical model training task is determined, and the prediction execution duration corresponding to the historical model training task is determined. And training according to the predicted execution duration and the historical task execution duration. And the trained prediction model determines the prediction execution duration of the target model training task according to the resource occupation data of the target model training task. The method in the specification is higher in prediction efficiency and more accurate. Therefore, the utilization rate of training resources is effectively improved during subsequent resource allocation, and the training efficiency of the whole training process is improved to a great extent while resource waste and shelving are avoided.

Description

Technical Field

[0001] This specification relates to the field of model training, and in particular to a model training method, a task execution time prediction method and a device. Background Art

[0002] With the continuous development and progress of science and technology, the importance of intelligent models in today's world is self-evident. Before the model is officially put into use, it needs to go through an extremely rigorous training process to ensure the actual working ability of the model. How to reasonably control the resource cost spent in the training process and the allocation of overall resources is an extremely important issue.

[0003] By reasonably predicting and controlling the execution time of model training tasks, the overall resource utilization efficiency can be greatly improved, and resource waste and shelving can be reduced. In the current technology, the execution time prediction of model training tasks is mostly carried out through task awareness. When predicting the execution time of model training tasks, this method often requires changing the structural code of the model in training to obtain model training data in real time, such as adding interfaces to collect model internal structure data, number of parameters, batch size and other training data through code modification. This highly invasive way of collecting data can easily affect the training situation and effect of the model, and may even cause security issues such as data leakage due to the setting of additional interface codes. In addition, this method has strong limitations and may not be applicable to any type of model structure. In summary, predicting the execution time of training tasks through task awareness will greatly affect the training efficiency and security of the overall model training process, extend the execution time of model training tasks, and make the allocation and use of overall training resources more unreasonable.

[0004] Therefore, how to efficiently predict the execution time of model training tasks without affecting the model training process, and then reasonably allocate training resources, is a very important issue. Summary of the invention

[0005] This specification provides a model training method, a task execution time prediction method and a device to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a model training method, including:

[0008] Obtain historical resource usage data and historical task execution time corresponding to historical model training tasks;

[0009] According to the historical resource occupancy data, similar historical model training tasks corresponding to the historical model training tasks are screened out from a preset database, and the historical resource occupancy data corresponding to the similar historical model training tasks are obtained;

[0010] Inputting the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task into the prediction model to be trained, so that the prediction model determines the resource occupancy characteristic data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task, and determines the predicted execution time corresponding to the historical model training task according to the resource occupancy characteristic data;

[0011] According to the predicted execution time and the historical task execution time, a loss value for the prediction model is determined, and according to the loss value, the prediction model is trained, wherein the size of the loss value is negatively correlated with the similarity between the predicted execution time and the historical task execution time.

[0012] Optionally, before screening out similar historical model training tasks corresponding to the historical model training tasks from a preset database according to the historical resource occupancy data and obtaining the historical resource occupancy data corresponding to the similar historical model training tasks, the method further includes:

[0013] Performing data preprocessing on the historical resource occupancy data corresponding to the historical model training task to obtain the preprocessed historical resource occupancy data corresponding to the historical model training task, wherein the data preprocessing at least includes performing outlier deletion processing, data noise reduction processing, and vacant data filling processing on the historical resource occupancy data corresponding to the historical model training task;

[0014] According to the historical resource occupancy data, similar historical model training tasks corresponding to the historical model training tasks are screened out from a preset database, and historical resource occupancy data corresponding to the similar historical model training tasks are obtained, specifically including:

[0015] According to the preprocessed historical resource occupancy data corresponding to the historical model training task, similar historical model training tasks corresponding to the historical model training task are screened out from the database, and the historical resource occupancy data corresponding to the similar historical model training tasks are obtained.

[0016] Optionally, based on the historical resource occupancy data, similar historical model training tasks corresponding to the historical model training tasks are screened out from a preset database, specifically including:

[0017] For each historical resource occupancy data corresponding to the historical model training task to be selected in the database, determine the similarity between the historical resource occupancy data corresponding to the historical model training task to be selected and the historical resource occupancy data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task;

[0018] When the similarity is greater than a preset similarity threshold, the to-be-selected historical model training task is used as a similar historical model training task corresponding to the historical model training task.

[0019] Optionally, the historical resource occupancy data contains resource occupancy data of different resource types corresponding to the historical model training task, and the resource types include at least CPU occupancy, GPU occupancy, memory usage, and disk throughput;

[0020] For each historical model training task to be selected in the database, determining the similarity between the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task, specifically includes:

[0021] For each resource occupancy data of a resource type in the historical resource occupancy data corresponding to the historical model training task to be selected, determine the similarity between the resource occupancy data of the resource type corresponding to the historical model training task to be selected and the resource occupancy data of the resource type corresponding to the historical model training task;

[0022] The resource occupancy data of each resource type corresponding to the historical model training task to be selected are weighted in the same category, and the comprehensive similarity corresponding to the historical model training task to be selected is determined, and used as the similarity between the historical resource occupancy data corresponding to the historical model training task to be selected and the historical resource occupancy data corresponding to the historical model training task.

[0023] Optionally, the historical resource occupancy data is time series data consisting of resource occupancy data of the historical model training tasks at various time points collected at preset time intervals;

[0024] Inputting the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task into the prediction model to be trained, so that the prediction model determines the resource occupancy feature data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task, specifically including:

[0025] The historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task are input into the prediction model to be trained, so that the prediction model determines the data similarity between the resource occupancy data corresponding to each time point in each historical resource occupancy data and the resource occupancy data corresponding to other time points in the historical resource occupancy data, and determines the attention weight corresponding to the time point according to the data similarity, and performs self-attention weighted processing on the historical resource occupancy data according to the attention weights corresponding to each time point to obtain a self-attention feature sequence corresponding to the historical resource occupancy data;

[0026] The resource occupancy feature data corresponding to the historical model training task is determined according to the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task.

[0027] Optionally, determining resource occupancy feature data corresponding to the historical model training task according to the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task specifically includes:

[0028] According to the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, for the self-attention feature data at each time point in the self-attention feature sequence corresponding to the historical model training task, determine the feature similarity between the self-attention feature data at the time point and the self-attention feature data at the time point in the self-attention feature sequence corresponding to the similar historical model training task that are in the same time point arrangement order, and determine the cross-attention weight corresponding to the self-attention feature data at the time point according to the feature similarity;

[0029] According to the cross-attention weights corresponding to the self-attention feature data at each time point in the self-attention feature sequence corresponding to the historical model training task, cross-attention weighted processing is performed on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, so as to perform feature fusion on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, and the fused feature data of the historical model training task and the similar historical model training task are used as the resource occupancy feature data corresponding to the historical model training task.

[0030] This specification provides a task execution time prediction method, including:

[0031] Monitor and collect resource usage data corresponding to the target model training task within the predicted duration after it starts executing;

[0032] According to the resource occupancy data corresponding to the target model training task, similar historical model training tasks corresponding to the target model training task are screened out from a preset database, and historical resource occupancy data corresponding to the similar historical model training tasks are obtained;

[0033] The resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to the similar historical model training tasks are input into the prediction model, so that the prediction model determines the target feature data corresponding to the target model training task based on the resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to the similar historical model training tasks, and determines the execution duration of the prediction task corresponding to the target model training task based on the target feature data, wherein the prediction model is obtained through the above-mentioned model training method.

[0034] This specification provides a model training device, including:

[0035] The acquisition module is used to obtain the historical resource usage data and historical task execution time corresponding to the historical model training tasks;

[0036] A screening module, used to screen out similar historical model training tasks corresponding to the historical model training tasks from a preset database according to the historical resource occupancy data, and obtain the historical resource occupancy data corresponding to the similar historical model training tasks;

[0037] A prediction module, used for inputting the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task into the prediction model to be trained, so that the prediction model determines the resource occupancy characteristic data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task, and determines the predicted execution time corresponding to the historical model training task according to the resource occupancy characteristic data;

[0038] A training module is used to determine a loss value for the prediction model based on the predicted execution time and the historical task execution time, and to train the prediction model based on the loss value, wherein the size of the loss value is negatively correlated with the similarity between the predicted execution time and the historical task execution time.

[0039] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned model training method and task execution time prediction method are implemented.

[0040] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned model training method and task execution time prediction method when executing the program.

[0041] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0042] It can be seen from the above method that in the model training method and task execution time prediction method provided in this specification, the corresponding similar historical model training tasks can be screened out from the database based on the historical resource occupancy data corresponding to the acquired historical model training tasks. Then, the historical resource occupancy data corresponding to the historical model training tasks and the similar historical model training tasks are input into the prediction model to be trained, so that the prediction model determines the resource occupancy characteristic data corresponding to the historical model training tasks, and then determines the predicted execution duration corresponding to the historical model training tasks based on the resource occupancy characteristic data. The prediction model is trained based on the predicted execution duration and the historical task execution duration. The trained prediction model can determine the predicted execution duration of the target model training task based on the resource occupancy data of the target model training task within the preset duration.

[0043] It can be seen from the above content that the model training method and task execution time prediction method provided in this specification can match the corresponding similar historical model training tasks from the database according to the historical resource occupancy data corresponding to the historical model training tasks. Then, the prediction model is trained according to the corresponding historical resource occupancy data. The trained prediction model can accurately predict the predicted execution time required for the target model training task from the start of execution to the end of task execution based on the resource occupancy data corresponding to the target model training task within the preset duration. Compared with the task perception method in the current technology to predict the execution time of the training task, the method proposed in this specification is safer and more reliable, more efficient, and the prediction results are more accurate. The subsequent reasonable allocation of training resources can have an accurate basis, thereby effectively improving the utilization rate of training resources, avoiding resource waste and shelving, and greatly improving the training efficiency of the overall training process, thereby smoothly accelerating the execution of subsequent tasks or businesses based on the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0045] Figure 1 A flowchart of a model training method provided in this specification;

[0046] Figure 2 This is a schematic diagram of an example of a prediction model corresponding to a model structure provided in this specification;

[0047] Figure 3 A flowchart of a task execution time prediction method provided in this specification;

[0048] Figure 4 A schematic diagram of a model training device provided in this specification;

[0049] Figure 5 A schematic diagram of a task execution time prediction device provided in this specification;

[0050] Figure 6 A method provided in this specification corresponding to Figure 1 and Figure 3 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0052] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0053] Figure 1 A flow chart of a model training method provided in this specification includes the following steps:

[0054] S101: Obtain historical resource usage data and historical task execution time corresponding to historical model training tasks.

[0055] S102: According to the historical resource occupancy data, similar historical model training tasks corresponding to the historical model training tasks are screened out from a preset database, and the historical resource occupancy data corresponding to the similar historical model training tasks are acquired.

[0056] At present, the technologies related to intelligent AI models have gradually become the mainstream development direction in various fields in modern science and technology, and the corresponding training process is the top priority to successfully ensure the final model capabilities. By reasonably predicting the execution time of model training tasks, the reasonable allocation and use of training resources can be effectively controlled, thereby effectively improving model training efficiency and saving training costs.

[0057] At present, most execution prediction methods for model training tasks rely on task perception technology. This method requires obtaining the internal data of the model when the training task is executed to predict the task execution time. This method is somewhat invasive and may affect the data security and training efficiency of the model training process. At the same time, the need to collect internal data of the model may also limit or change the overall structure of the model, resulting in a lack of rationality in the allocation of overall training resources and time costs. Therefore, it is crucial to predict the execution time of model training tasks efficiently, accurately and safely.

[0058] To this end, this specification provides a model training method and a task execution time prediction method, wherein the execution subject adopted by the method provided in this specification can be a terminal device such as a desktop computer, a laptop computer, or a server. In addition, the execution subject of this specification can also be a subject in the form of software, such as a client installed in a terminal device. For the sake of ease of explanation and understanding, this specification below only uses the terminal device as the execution subject to explain a model training method and a task execution time prediction method provided.

[0059] Based on this, a terminal device that applies a model training method and a task execution time prediction method provided in this specification can determine the corresponding similar historical model training tasks from the database based on the historical resource occupancy data corresponding to the acquired historical model training tasks. Then, the terminal device can perform model training and optimization on the prediction model to be trained based on the historical resource occupancy data corresponding to the historical model training tasks and the similar historical model training tasks, so that a prediction model capable of predicting the execution time of the model training tasks based on the resource occupancy data can be trained.

[0060] Through the above training process, the terminal device can use the trained prediction model to predict the total task duration of the target model training task from the start to the end of execution based on the resource occupancy data within a preset time after the target model training task starts executing.

[0061] Among them, the actual application scenario corresponding to the prediction model that has undergone the training process can be determined according to the actual application needs. For example, in the scenario of reasonable allocation of training resources, the terminal device can use the trained prediction model to determine the predicted execution time corresponding to the target model training task according to the resource occupancy data within the preset time corresponding to the target model training task and the historical model training tasks in the database. The relevant technical personnel responsible for resource allocation can reasonably adjust the usage resources and training time occupied by the target model training task according to the determined predicted execution time, and pre-plan the training process of subsequent model training tasks. It effectively improves the utilization of training resources, ensures the effective utilization of resources to a great extent, and avoids idle or wasted resources.

[0062] For another example, in the design of automated execution of model training tasks, the terminal device can use the trained prediction model to predict the task execution time required for the successful completion of the target model training task based on the resource occupancy data within the preset duration of the target model training task and the historical model training tasks recorded in the database. Then, according to the predicted task execution time corresponding to each model training task, the actual training process of the model training task is reasonably allocated resources and divided into training durations, ensuring the reasonable implementation of the overall automated model training process, thereby effectively improving the execution efficiency and resource utilization of the overall model training process.

[0063] In this specification, the terminal device can obtain the historical resource occupancy data corresponding to the historical model training task and the historical task execution time corresponding to the historical model training task from a preset database. Then, the terminal device can filter out similar historical simulation training tasks corresponding to the historical model training task from the database based on the historical resource occupancy data corresponding to the historical model training task, and the terminal device can also obtain the historical resource occupancy data corresponding to the similar historical simulation training tasks.

[0064] Among them, the specific content of the historical resource occupancy data may include the resource occupancy data of different resource types corresponding to the historical model training task during the historical execution process, and the resource types may specifically include at least CPU occupancy, GPU occupancy, memory usage, and disk throughput, etc. The specific data form of the resource occupancy data corresponding to each resource type may be in the form of a time series, and the specific length of the time series is the historical task execution time of the corresponding historical model training task, and the data content may specifically be the resource occupancy data corresponding to different time points collected according to preset time intervals. The terminal device may use the resource occupancy data of each resource type corresponding to the historical model training task in the form of a time series as the historical resource occupancy data corresponding to the historical model training task.

[0065] Regarding how the terminal device specifically selects similar historical simulation training tasks corresponding to the historical model training task from the database based on the historical resource occupancy data corresponding to the historical model training task. In this specification, for each historical model training task to be selected recorded in the preset database, the terminal device can determine the similarity between the historical resource occupancy data corresponding to the historical model training task to be selected and the historical resource occupancy data corresponding to the historical model training task to be selected based on the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the historical model training task to be selected.

[0066] Then, the terminal device may determine whether the to-be-selected historical model training task is a similar historical model training task corresponding to the historical model training task based on the relationship between the similarity determined above and a preset similarity threshold.

[0067] Specifically, when the terminal device determines whether each historical model training task to be selected in the database is a similar historical model training task corresponding to the historical model training task, it can determine the same type of similarity between the time series data corresponding to the resource occupancy data of the resource type of the to-be-selected historical model training task and the time series data corresponding to the resource occupancy data of the resource type of the historical model training task, for each resource type in the historical resource occupancy data corresponding to the to-be-selected historical model training task.

[0068] Then, the terminal device can perform weighted summation processing on the same type of similarities of resource occupancy data of each resource type corresponding to the historical model training task to be selected, so as to determine the comprehensive similarity of the historical model training task to be selected relative to the historical model training task, as the similarity between the historical resource occupancy data corresponding to the historical model training task to be selected and the historical resource occupancy data corresponding to the historical model training task.

[0069] Among them, the specific implementation method for determining the similarity of resource occupancy data of the same resource type between the selected historical model training task and the historical model training task is not strictly limited in this specification, and can be implemented using a dynamic time warping algorithm (Dynamic Time Warping, DTW).

[0070] Specifically, the terminal device can calculate the DTW distance between the time series data corresponding to the resource occupancy data of the same resource type based on the dynamic time warping algorithm. The smaller the DTW distance, the more similar the two time series data are. By pre-setting the corresponding threshold conditions for the DTW distance, the terminal device can determine the similarity between the resource occupancy data of the same resource type in the selected historical model training task and the historical model training task through the dynamic time warping algorithm, and then determine the similar historical model training task corresponding to the historical model training task. In addition, similar historical model training tasks can also be determined by other methods of calculating the similarity of time series, such as the fast dynamic time warping algorithm Fast Dynamic Time Warping, FastDTW), cross-correlation function (Cross-CorrelationFunction), etc., which can be flexibly selected and set according to actual application scenarios and application requirements.

[0071] It should be noted separately that in order to ensure the safety and stability of the subsequent training process and the accuracy and effectiveness of the historical resource occupancy data corresponding to the historical model training tasks, in this specification, the terminal device may also perform data preprocessing on the historical resource occupancy data corresponding to the historical model training tasks before screening out similar historical model training tasks corresponding to the historical model training tasks from the database based on the historical resource occupancy data. The terminal device may use the preprocessed historical resource occupancy data corresponding to the historical model training tasks for the subsequent processing of determining similar historical model training tasks from the database. Furthermore, after determining the similar historical model training tasks corresponding to the historical model training tasks, the terminal device may also perform the same preprocessing process on the historical resource occupancy data corresponding to the obtained similar historical model training tasks, thereby ensuring the accuracy and effectiveness of the subsequent training process.

[0072] Among them, the specific processing methods for data preprocessing of historical resource occupancy data in this specification may at least include but are not limited to outlier value deletion processing for abnormal data in historical resource occupancy data, data noise reduction processing for historical resource occupancy data, and vacancy data filling processing for null value data in historical resource occupancy data, etc. The specific preprocessing methods for historical resource occupancy data are not strictly limited in this specification and can be flexibly set according to actual application scenarios and requirements.

[0073] S103: Input the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training tasks into the prediction model to be trained, so that the prediction model determines the resource occupancy characteristic data corresponding to the historical model training task based on the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training tasks, and determines the predicted execution time corresponding to the historical model training task based on the resource occupancy characteristic data.

[0074] In this specification, the terminal device can use the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task as the training data set corresponding to the historical model training task, and input them into the prediction model to be trained, so that the prediction model can determine the resource occupancy feature data corresponding to the historical model training task based on the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task. Then, the prediction model can predict the predicted execution time corresponding to the historical model training task based on the resource occupancy feature data.

[0075] Specifically, as mentioned before, the historical resource occupancy data can be time series data composed of the resource occupancy data of the corresponding model training task of the mobile phone at each time point according to a preset time interval. The terminal device can input the historical resource occupancy data in the form of time series data corresponding to the historical model training task and similar historical model training tasks into the prediction model to be trained, so that the prediction model can determine the data similarity between the resource occupancy data corresponding to each time point in each historical resource occupancy data and the resource occupancy data corresponding to other time points in the historical resource occupancy data.

[0076] Then, the terminal device can determine the attention weight corresponding to the resource occupancy data of other time points at this time point through the prediction model according to the data similarity between the resource occupancy data corresponding to this time point and the resource occupancy data corresponding to other time points in the historical resource occupancy data, and then the prediction model can perform self-attention weighted processing on the resource occupancy data of this time point according to the attention weight corresponding to this time point and the corresponding historical resource occupancy data, and obtain the self-attention feature data corresponding to this time point. Thus, according to the self-attention feature data corresponding to each time point in the historical resource occupancy data, the self-attention feature sequence corresponding to the historical resource occupancy data is determined.

[0077] Through the above method, the terminal device can determine the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task through the prediction model to be trained. Then, the terminal device can also perform cross-attention processing on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task through the prediction model to obtain the resource occupancy feature data corresponding to the historical model training task.

[0078] Specifically, the terminal device can use the prediction model to determine the feature similarity between the self-attention feature data at each time point in the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, based on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task.

[0079] Then, the prediction model can determine the cross-attention weight corresponding to the self-attention feature data at that time point, and perform cross-attention weighted processing on the self-attention feature data at that time point according to the cross-attention weight corresponding to that time point and the self-attention feature data of the time point in the same time point arrangement order corresponding to the similar historical model training task, to obtain the cross-attention feature data corresponding to that time point. Then, the terminal device can determine the cross-attention feature data corresponding to each time point in the self-attention feature sequence corresponding to the historical model training task through the prediction model, realize the feature fusion of the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the corresponding similar historical model training task, and obtain the resource occupancy feature data corresponding to the historical model training task.

[0080] After determining the resource occupancy characteristic data corresponding to the historical model training task, the terminal device can predict the predicted execution time corresponding to the historical model training task based on the resource occupancy characteristic data corresponding to the historical model training task through the prediction model to be trained.

[0081] Regarding the actual processing process of predicting the execution time, specifically, in this specification, the terminal device can input the resource occupancy feature data corresponding to the historical model training task into the fully connected layer preset in the prediction model, so that the fully connected layer can perform linear transformation processing on the resource occupancy feature data according to the preset linear regression function and the resource occupancy feature data corresponding to the historical model training task determined by the above steps, and then obtain the predicted execution time corresponding to the historical model training task.

[0082] In addition, there may be other ways to predict the predicted execution time based on the resource occupancy characteristic data corresponding to the historical model training task, such as through a decision tree regression model (Decision Tree Regression Analysis), a recurrent neural network (RNN), etc. This specification does not strictly limit the specific implementation method and process of determining the predicted execution time based on the resource occupancy characteristic data corresponding to the historical model training task, and can be flexibly set according to the actual application scenario and needs.

[0083] It should also be noted that this specification does not strictly limit the specific model category and structure of the prediction model mentioned above. It can be, for example, a Transformer model. The prediction model of the Transformer model first performs dimensionality conversion on the historical resource occupancy data corresponding to the historical model training tasks and similar historical model training tasks, and then performs position encoding for each time point, and then performs self-attention and cross-attention processing similar to the above method, so as to normalize and mean the resource occupancy feature data corresponding to the historical model training tasks, and then passes through the linear regression transformation preset in the fully connected layer to predict the predicted execution time corresponding to the historical model training task.

[0084] In addition, in this specification, the model structure of the prediction model can also be similar to a deep learning model based on a multi-layer neural network. For the specific processing process of determining the predicted execution time corresponding to the historical model training task by the prediction model whose model structure is a deep learning model based on a multi-layer neural network, please refer to Figure 2 as shown in .

[0085] Figure 2 This is a schematic diagram of an example of a prediction model corresponding to a model structure provided in this specification.

[0086] like Figure 2 As shown in , the terminal device can input the training data set corresponding to the historical model training task, that is, the historical resource occupancy data corresponding to the historical model training task and similar historical model training tasks, into a prediction model with a multi-layer neural network structure, and the prediction model can extract and learn the resource occupancy feature data corresponding to the historical model training task and at least one similar historical model training task in the training data set, and finally predict the predicted execution time corresponding to the historical model training task after multi-level neural network extraction and fusion.

[0087] In addition to the above-mentioned implementation method of the Transformer model and the example of the deep learning model based on the multi-layer neural network, the prediction model in this specification can also be a multi-layer perceptron model (Multilayer Perceptron, MLP), and the terminal device can input the historical resource occupancy data corresponding to the historical model training task and the similar historical model training task into the prediction model whose model structure is the multi-layer perceptron model, so that the prediction model can extract the dependency between the resource occupancy data and time at each time point in the historical resource occupancy data through the hidden layer structure inside the model, thereby predicting the predicted execution time corresponding to the historical model training task. In addition, the prediction model in this specification can also be a model structure such as a recursive neural network model (Recurrent Neural Network, RNN), a random forest model (Random Forest, RF), a decision tree model (Decision Tree, DT), etc., and the specific model structure of the prediction model can be selected and set according to the actual application scenario and needs.

[0088] S104: Determine a loss value for the prediction model based on the predicted execution time and the historical task execution time, and train the prediction model based on the loss value, wherein the size of the loss value is negatively correlated with the similarity between the predicted execution time and the historical task execution time.

[0089] In this specification, the terminal device can determine the loss value for the prediction model based on the predicted execution time determined by the prediction model to be trained and the historical task execution time corresponding to the historical model training task previously obtained. Then, based on the determined loss value, the terminal device can perform model training and optimization on the prediction model to be trained. Among them, the size of the determined loss value is negatively correlated with the similarity between the predicted execution time and the historical task execution time. The closer the predicted execution time and the historical task execution time are, the smaller the loss value is, and vice versa.

[0090] Regarding the training process for the prediction model, it should be noted that in this specification, the execution time of the historical task corresponding to the historical model training task can be used as the training annotation label of the training data set corresponding to the historical model training task, so as to guide the subsequent training process for the prediction model. In addition, when training a prediction model, a large amount of training data is often required to optimize the model parameters. When there are multiple historical model training tasks, each historical model training task can also become a member of the training data set corresponding to other historical model training tasks.

[0091] For example, assume that the terminal device determines the training data set A corresponding to the historical model training task A through the above steps, and the training data set A includes the historical resource occupancy data corresponding to the historical model training task A and its similar historical model training tasks, and the training data set B corresponding to the historical model training task B, and the training data set B includes the historical resource occupancy data corresponding to the historical model training task B and its similar historical model training tasks.

[0092] If the similarity between the historical resource occupancy data corresponding to historical model training task A and the historical resource occupancy data corresponding to historical model training task B is high, the terminal device can use the historical model training task B as a member of the similar historical model training tasks of the historical model training task A in the training data set A during the matching process of the above-mentioned similar historical model training tasks. At the same time, the terminal device can also use the historical model training task A as a member of the similar historical model training tasks of the historical model training task B in the training data set B.

[0093] However, the training annotation label corresponding to the training data set A corresponding to the historical model training task A is the historical task execution time a corresponding to the historical model training task A, and the training annotation label corresponding to the training data set B corresponding to the historical model training task B is the historical task execution time b corresponding to the historical model training task B. During the subsequent training process, when the terminal device inputs the training data set A into the prediction model to be trained, the prediction model can only predict the predicted execution time of the historical model training task A, and determine the loss value based on the historical task execution time a, while allowing the model to learn the subtle feature differences between the historical model training task A and the historical model training task B. Similarly, the same is true for the training data set B. The prediction model can only predict the predicted execution time of the historical model training task B, and determine the loss value based on the historical task execution time b.

[0094] From the above, it is not difficult to see that although different historical model training tasks may be similar historical model training tasks to each other, during the training process, the historical model training tasks corresponding to the training data set are still trained. Through the above training method, the trained prediction model can effectively have the ability to accurately distinguish historical model training tasks with too similar historical resource occupancy data, so that in the actual application stage, the predicted execution time corresponding to each target model training task can be predicted more accurately and efficiently, thereby enabling the overall training resources to be reasonably allocated and utilized.

[0095] The method provided in this specification is mainly divided into two stages, the model training stage and the actual application stage. The model training stage introduced above is mainly used to obtain a prediction model that has the ability to predict the execution time of the model training task after model training. Furthermore, in the actual application stage, the terminal device can predict the task execution time for the target model training task through the trained prediction model, so as to reasonably allocate the overall training resources according to the predicted execution time corresponding to the target model training task and reasonably plan other subsequent model training tasks.

[0096] In order to facilitate the introduction of the task execution method, the following will be explained with a flowchart of a task execution time prediction method, such as Figure 3 shown.

[0097] Figure 3 A flowchart of a task execution time prediction method provided in this specification includes the following steps:

[0098] S301: Monitor and collect resource usage data corresponding to the target model training task within the predicted duration after the start of execution.

[0099] S302: According to the resource occupancy data corresponding to the target model training task, similar historical model training tasks corresponding to the target model training task are screened out from a preset database, and historical resource occupancy data corresponding to the similar historical model training tasks are acquired.

[0100] With the continuous advancement of science and technology, intelligent mathematical models with high intelligence and strong convenience have been widely popularized and applied in many fields of modern science and technology. However, the resource cost and time consumption of training various models are extremely huge. It is crucial to reasonably allocate training resources to ensure the rational use of limited resources. Nowadays, the mainstream method of predicting the execution time of model training tasks is mostly achieved through task perception technology. This method requires deep data mining of the model in the training process when collecting data, which may affect the model in the current training process to a certain extent, affecting the model training ability and delaying the overall model training efficiency. Therefore, how to efficiently, accurately and safely predict the execution time of model training tasks, and then reasonably allocate training resources and formulate training plans, is an extremely important issue.

[0101] To this end, this specification provides a task execution time prediction method, wherein the execution subject used by the method provided in this specification can be a terminal device such as a desktop computer, a laptop computer, or a server. In addition, the execution subject of this specification can also be a subject in the form of software, such as a client installed in a terminal device. For the sake of ease of description and understanding, this specification below only uses the terminal device as the execution subject to illustrate a task execution time prediction method provided.

[0102] Based on this, the terminal device that applies the task execution time prediction method provided in this specification can predict the predicted execution time of the target model training task from the start to the completion of the execution based on the resource usage data corresponding to the target model training task within the preset time period, so that the terminal device or relevant developers can reasonably allocate training resources and training plans for subsequent model training tasks based on the predicted execution time of the target model training task.

[0103] Among them, the specific application scenario in which the terminal device predicts the execution time of the target model training task can be determined according to actual needs. For example, in the scenario of reasonable regulation of limited resources, the terminal device using the method in this specification can accurately predict the predicted execution time that the target model training task may take to complete according to the resource occupancy data corresponding to the target model training task within the preset time period, so that the relevant developers can accurately control the allocation and use of limited resources according to the predicted execution time corresponding to the target model training task, thereby effectively avoiding resource waste and shelving, improving resource utilization efficiency, and greatly improving the training efficiency of the overall model training task.

[0104] For another example, in a scenario where energy restrictions and management are performed on a model training process, a terminal device that applies the method in this specification can accurately determine the estimated execution time required for the target model training task from the start to the successful completion of the task based on the resource occupancy data corresponding to the target model training task within a preset time period. This allows the relevant developers and resource management personnel to reasonably allocate and restrict resources based on the amount of resources and time consumed by the target model to complete the training task. This effectively ensures that the resource requirements of the target model training task are met to the maximum extent possible without wasting resources.

[0105] In this specification, the terminal device can continuously detect the resource occupancy data of the target model training task after it starts to execute, and collect and obtain the resource occupancy data corresponding to the target model training task within the predicted duration after it starts to execute. Then, the terminal device can screen and determine the similar historical model training tasks corresponding to the target model training task from the preset database based on the resource occupancy data corresponding to the target model training task within the predicted duration, and obtain the historical resource occupancy data corresponding to the similar historical model training tasks.

[0106] The specific determination process of the similar historical model training task is basically the same as that of the similar historical model training task corresponding to the historical model training task in the above model training method, so it will not be elaborated here. The specific data form and content of the resource occupancy data corresponding to the target model training task within the prediction duration and the historical resource occupancy data corresponding to the similar historical model training task are also basically the same as those mentioned in the above model training method.

[0107] S303: Input the resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to the similar historical model training tasks into the prediction model, so that the prediction model determines the target feature data corresponding to the target model training task based on the resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to the similar historical model training tasks, and determines the execution duration of the prediction task corresponding to the target model training task based on the target feature data.

[0108] In this specification, the terminal device can input the acquired resource occupancy data corresponding to the target model training task within a preset duration and the historical resource occupancy data corresponding to similar historical model training tasks into the prediction model, so that the prediction model can determine the target feature data corresponding to the target model training task based on the resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to similar historical model training tasks, and then the terminal device can predict the prediction task execution duration corresponding to the target model training task based on the target feature data corresponding to the target model training task through the prediction model.

[0109] Among them, although the resource occupancy data corresponding to the target model training task is smaller than the historical resource occupancy data corresponding to the similar historical model training tasks, it is only the resource occupancy data corresponding to the target model training task within a preset period of time. However, its small volume of data will not seriously affect the prediction results of the final prediction model. In this specification, the prediction model trained by the above-mentioned model training method can effectively capture the mapping relationship between resource occupancy data and actual execution time. Therefore, even if the execution time is predicted based on the small volume of data corresponding to the target model training task, the missing data content and the corresponding feature data can be effectively supplemented due to the existence of the corresponding feature data of the remaining multiple similar historical model training tasks, thereby effectively ensuring the precision and accuracy of the final prediction results.

[0110] From the above content, it can be seen that the model training method and task execution time prediction method provided in this specification can match the corresponding similar historical model training tasks from the database according to the historical resource occupancy data corresponding to the historical model training tasks. Then, the prediction model to be trained is trained according to the historical resource occupancy data corresponding to the historical model training tasks and similar historical model training tasks. The trained prediction model can accurately predict the predicted task execution time required for the target model training task from the start of execution to the end of task execution based on the resource occupancy data corresponding to the target model training task within the preset duration.

[0111] Compared with the current technology that uses task perception to predict the execution time of training tasks, the method in this manual is safer, more reliable, more efficient, and more accurate in prediction, so that both terminal devices and related developers can have an accurate basis for the reasonable allocation of training resources in the future. While effectively improving the utilization rate of training resources and avoiding resource waste and shelving, it also greatly improves the training efficiency of the overall training process, thereby smoothly accelerating the execution of subsequent tasks or businesses based on the trained model.

[0112] The above are one or more implementation methods of this specification. Based on the same idea, this specification also provides a corresponding model training device, such as Figure 4 shown.

[0113] Figure 4 A schematic diagram of a model training device provided in this specification, comprising:

[0114] Acquisition module 401, used to acquire historical resource occupancy data and historical task execution time corresponding to historical model training tasks;

[0115] A screening module 402 is used to screen out similar historical model training tasks corresponding to the historical model training tasks from a preset database according to the historical resource occupancy data, and obtain the historical resource occupancy data corresponding to the similar historical model training tasks;

[0116] Prediction module 403, used for inputting the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task into the prediction model to be trained, so that the prediction model determines the resource occupancy characteristic data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task, and determines the predicted execution time corresponding to the historical model training task according to the resource occupancy characteristic data;

[0117] The training module 404 is used to determine the loss value for the prediction model based on the predicted execution time and the historical task execution time, and train the prediction model based on the loss value, wherein the size of the loss value is negatively correlated with the similarity between the predicted execution time and the historical task execution time.

[0118] Optionally, the acquisition module 401 is specifically used to perform data preprocessing on the historical resource occupancy data corresponding to the historical model training task to obtain the preprocessed historical resource occupancy data corresponding to the historical model training task, wherein the data preprocessing at least includes performing outlier deletion processing, data noise reduction processing, and vacant data filling processing on the historical resource occupancy data corresponding to the historical model training task;

[0119] The screening module 402 is specifically used to screen out similar historical model training tasks corresponding to the historical model training tasks from the database according to the preprocessed historical resource occupancy data corresponding to the historical model training tasks, and obtain the historical resource occupancy data corresponding to the similar historical model training tasks.

[0120] Optionally, the screening module 402 is specifically used to determine, for each historical resource occupancy data corresponding to the historical model training task to be selected in the database, the similarity between the historical resource occupancy data corresponding to the historical model training task to be selected and the historical resource occupancy data corresponding to the historical model training task; when the similarity is greater than a preset similarity threshold, the historical model training task to be selected is used as a similar historical model training task corresponding to the historical model training task.

[0121] Optionally, the historical resource occupancy data contains resource occupancy data of different resource types corresponding to the historical model training task, and the resource types include at least CPU occupancy, GPU occupancy, memory usage, and disk throughput;

[0122] The screening module 402 is specifically used to determine, for each resource type in the historical resource occupancy data corresponding to the historical model training task to be selected, the similarity between the resource occupancy data of the resource type corresponding to the historical model training task to be selected and the resource occupancy data of the resource type corresponding to the historical model training task to be selected; perform weighted processing on the similarities of the resource occupancy data of each resource type corresponding to the historical model training task to be selected, determine the comprehensive similarity corresponding to the historical model training task to be selected, and use it as the similarity between the historical resource occupancy data corresponding to the historical model training task to be selected and the historical resource occupancy data corresponding to the historical model training task to be selected.

[0123] Optionally, the historical resource occupancy data is time series data consisting of resource occupancy data of the historical model training tasks at various time points collected at preset time intervals;

[0124] The prediction module 403 is specifically used to input the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task into the prediction model to be trained, so that the prediction model determines the data similarity between the resource occupancy data corresponding to each time point in each historical resource occupancy data and the resource occupancy data corresponding to other time points in the historical resource occupancy data, and determines the attention weight corresponding to the time point based on the data similarity, and performs self-attention weighted processing on the historical resource occupancy data based on the attention weights corresponding to each time point to obtain the self-attention feature sequence corresponding to the historical resource occupancy data; and determines the resource occupancy feature data corresponding to the historical model training task based on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task.

[0125] Optionally, the prediction module 403 is specifically used to determine, for the self-attention feature data of each time point in the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, the feature similarity between the self-attention feature data at the time point and the self-attention feature data at the time point in the self-attention feature sequence corresponding to the similar historical model training task in the same time point arrangement order, and determine the cross-attention weight corresponding to the self-attention feature data at the time point based on the feature similarity; perform cross-attention weighted processing on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task based on the cross-attention weights corresponding to the self-attention feature data at each time point in the self-attention feature sequence corresponding to the historical model training task, so as to perform feature fusion on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, and use the fused feature data of the historical model training task and the similar historical model training task as the resource occupancy feature data corresponding to the historical model training task.

[0126] Based on the same idea, this specification also provides a corresponding task execution time prediction device, such as Figure 5 shown.

[0127] Figure 5 A schematic diagram of a task execution time prediction device provided in this specification includes:

[0128] The acquisition module 501 is used to monitor and collect the resource usage data corresponding to the target model training task within the predicted duration after the start of execution;

[0129] A screening module 502 is used to screen out similar historical model training tasks corresponding to the target model training task from a preset database according to the resource occupancy data corresponding to the target model training task, and obtain historical resource occupancy data corresponding to the similar historical model training tasks;

[0130] The prediction module 503 is used to input the resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to the similar historical model training tasks into the prediction model, so that the prediction model determines the target feature data corresponding to the target model training task based on the resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to the similar historical model training tasks, and determines the prediction task execution duration corresponding to the target model training task based on the target feature data, wherein the prediction model is obtained through the above-mentioned model training method.

[0131] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The model training method and Figure 3 The task execution time prediction method is provided.

[0132] This manual also provides Figure 6 The one shown corresponds to Figure 1 and Figure 3 A schematic diagram of the electronic device. Figure 6 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The model training method and Figure 3 The task execution time prediction method.

[0133] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0134] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0135] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0136] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0137] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0138] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0141] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0142] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0143] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0144] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0145] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0147] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A model training method, characterized in that: include: Obtain historical resource usage data and historical task execution time corresponding to historical model training tasks; According to the historical resource occupancy data, similar historical model training tasks corresponding to the historical model training tasks are screened out from a preset database, and the historical resource occupancy data corresponding to the similar historical model training tasks are obtained; Inputting the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task into the prediction model to be trained, so that the prediction model determines the resource occupancy characteristic data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task, and determines the predicted execution time corresponding to the historical model training task according to the resource occupancy characteristic data; According to the predicted execution time and the historical task execution time, a loss value for the prediction model is determined, and according to the loss value, the prediction model is trained, wherein the size of the loss value is negatively correlated with the similarity between the predicted execution time and the historical task execution time.

2. The method according to claim 1, characterized in that Before screening out similar historical model training tasks corresponding to the historical model training tasks from a preset database according to the historical resource occupancy data and obtaining the historical resource occupancy data corresponding to the similar historical model training tasks, the method further includes: Performing data preprocessing on the historical resource occupancy data corresponding to the historical model training task to obtain the preprocessed historical resource occupancy data corresponding to the historical model training task, wherein the data preprocessing at least includes performing outlier deletion processing, data noise reduction processing, and vacant data filling processing on the historical resource occupancy data corresponding to the historical model training task; According to the historical resource occupancy data, similar historical model training tasks corresponding to the historical model training tasks are screened out from a preset database, and historical resource occupancy data corresponding to the similar historical model training tasks are obtained, specifically including: According to the preprocessed historical resource occupancy data corresponding to the historical model training task, similar historical model training tasks corresponding to the historical model training task are screened out from the database, and the historical resource occupancy data corresponding to the similar historical model training tasks are obtained.

3. The method according to claim 1, characterized in that According to the historical resource occupancy data, similar historical model training tasks corresponding to the historical model training tasks are screened out from a preset database, specifically including: For each historical resource occupancy data corresponding to the historical model training task to be selected in the database, determine the similarity between the historical resource occupancy data corresponding to the historical model training task to be selected and the historical resource occupancy data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task; When the similarity is greater than a preset similarity threshold, the to-be-selected historical model training task is used as a similar historical model training task corresponding to the historical model training task.

4. The method according to claim 3, characterized in that The historical resource occupancy data contains resource occupancy data of different resource types corresponding to the historical model training task, and the resource types at least include CPU occupancy rate, GPU occupancy rate, memory usage, and disk throughput; For each historical model training task to be selected in the database, determining the similarity between the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task, specifically includes: For each resource occupancy data of a resource type in the historical resource occupancy data corresponding to the historical model training task to be selected, determine the similarity between the resource occupancy data of the resource type corresponding to the historical model training task to be selected and the resource occupancy data of the resource type corresponding to the historical model training task; The resource occupancy data of each resource type corresponding to the historical model training task to be selected are weighted in the same category, and the comprehensive similarity corresponding to the historical model training task to be selected is determined, and used as the similarity between the historical resource occupancy data corresponding to the historical model training task to be selected and the historical resource occupancy data corresponding to the historical model training task.

5. The method according to claim 1, characterized in that The historical resource occupancy data is time series data consisting of resource occupancy data of the historical model training task at each time point collected at preset time intervals; Inputting the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task into the prediction model to be trained, so that the prediction model determines the resource occupancy feature data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task, specifically including: The historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task are input into the prediction model to be trained, so that the prediction model determines the data similarity between the resource occupancy data corresponding to each time point in each historical resource occupancy data and the resource occupancy data corresponding to other time points in the historical resource occupancy data, and determines the attention weight corresponding to the time point according to the data similarity, and performs self-attention weighted processing on the historical resource occupancy data according to the attention weights corresponding to each time point to obtain a self-attention feature sequence corresponding to the historical resource occupancy data; The resource occupancy feature data corresponding to the historical model training task is determined according to the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task.

6. The method according to claim 5, characterized in that Determining resource occupancy feature data corresponding to the historical model training task according to the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, specifically comprising: According to the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, for the self-attention feature data at each time point in the self-attention feature sequence corresponding to the historical model training task, determine the feature similarity between the self-attention feature data at the time point and the self-attention feature data at the time point in the self-attention feature sequence corresponding to the similar historical model training task that are in the same time point arrangement order, and determine the cross-attention weight corresponding to the self-attention feature data at the time point according to the feature similarity; According to the cross-attention weights corresponding to the self-attention feature data at each time point in the self-attention feature sequence corresponding to the historical model training task, cross-attention weighted processing is performed on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, so as to perform feature fusion on the self-attention feature sequence corresponding to the historical model training task and the self-attention feature sequence corresponding to the similar historical model training task, and the fused feature data of the historical model training task and the similar historical model training task are used as the resource occupancy feature data corresponding to the historical model training task.

7. A task execution time prediction method, characterized in that: include: Monitor and collect resource usage data corresponding to the target model training task within the predicted duration after it starts executing; According to the resource occupancy data corresponding to the target model training task, similar historical model training tasks corresponding to the target model training task are screened out from a preset database, and historical resource occupancy data corresponding to the similar historical model training tasks are obtained; The resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to the similar historical model training tasks are input into the prediction model, so that the prediction model determines the target feature data corresponding to the target model training task based on the resource occupancy data corresponding to the target model training task and the historical resource occupancy data corresponding to the similar historical model training tasks, and determines the execution duration of the prediction task corresponding to the target model training task based on the target feature data, wherein the prediction model is obtained by the method described in any one of claims 1 to 6 above.

8. A model training device, characterized in that: include: The acquisition module is used to obtain the historical resource usage data and historical task execution time corresponding to the historical model training tasks; A screening module, used to screen out similar historical model training tasks corresponding to the historical model training tasks from a preset database according to the historical resource occupancy data, and obtain the historical resource occupancy data corresponding to the similar historical model training tasks; A prediction module, used for inputting the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task into the prediction model to be trained, so that the prediction model determines the resource occupancy characteristic data corresponding to the historical model training task according to the historical resource occupancy data corresponding to the historical model training task and the historical resource occupancy data corresponding to the similar historical model training task, and determines the predicted execution time corresponding to the historical model training task according to the resource occupancy characteristic data; A training module is used to determine a loss value for the prediction model based on the predicted execution time and the historical task execution time, and to train the prediction model based on the loss value, wherein the size of the loss value is negatively correlated with the similarity between the predicted execution time and the historical task execution time.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images