Deep learning task execution method and device based on running time length prediction

Abstract

The invention discloses a deep learning task execution method and device based on running time length prediction. The method comprises the following steps: executing a target deep learning task according to a current task allocation strategy through each computing node in an intelligent computing cluster, and obtaining a performance monitoring data sequence of computing resources on each computing node within a specified duration after the target deep learning task is executed for the specified duration, processing the performance monitoring data corresponding to each computing node, and for target monitoring data corresponding to each computing node, extracting a time domain feature of the target monitoring data in a time domain and a frequency domain feature of the target monitoring data in a frequency domain; and inputting the time domain feature and the frequency domain feature into a pre-trained time length prediction model, determining a running time length required for completing the target deep learning task, and according to the running time length, adjusting a current task allocation strategy so as to execute the target deep learning task according to the adjusted task allocation strategy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a method and device for executing a deep learning task based on runtime prediction. Background Art

[0002] With the rapid development of deep learning technology, deep learning models have an increasing demand for computing resources such as networks, memory, GPUs, and CPUs in intelligent computing clusters, which has brought more challenges to resource allocation and cost management. Accurately predicting the running time of these models when performing training and inference tasks is crucial to optimizing resource scheduling, reducing computing costs, and improving production efficiency.

[0003] However, currently, the task runtime is usually predicted by deeply analyzing the internal structure and operation mechanism of the deep learning model, including network architecture, parameter scale, intelligent computing cluster parallel computing strategy, and the configuration of intelligent computing cluster hardware resources (such as GPU, CPU, memory), etc. However, this method requires intrusive analysis of user code, which raises privacy and security issues. In addition, this method requires the forecaster to have a deep understanding of the deep learning model and training framework, which greatly increases the prediction cost and efficiency, and it is difficult to guarantee the accuracy of the prediction results.

[0004] Therefore, how to improve the accuracy and efficiency of predicting the duration of deep learning tasks, reduce the prediction cost, and further ensure the resource utilization and execution efficiency of deep learning tasks is an urgent problem to be solved. Summary of the invention

[0005] This specification provides a method and device for executing deep learning tasks based on runtime prediction.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a method for executing a deep learning task based on runtime prediction, including:

[0008] Receive an execution request for a target deep learning task;

[0009] According to the execution request, the target deep learning task is executed by each computing node in the intelligent computing cluster according to the current task allocation strategy, and after monitoring the execution of the target deep learning task for a specified time, a performance monitoring data sequence of the computing resources on each computing node within the specified time is obtained as the performance monitoring data corresponding to each computing node;

[0010] Processing the performance monitoring data corresponding to each computing node so that the performance monitoring data corresponding to different nodes are aligned on a time scale, and obtaining target monitoring data corresponding to each computing node;

[0011] For the target monitoring data corresponding to each computing node, extract the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain;

[0012] Inputting the time domain features and the frequency domain features into a pre-trained duration prediction model to determine the running time required to complete the target deep learning task through the duration prediction model;

[0013] According to the running time, the current task allocation strategy is adjusted to execute the target deep learning task according to the adjusted task allocation strategy.

[0014] Optionally, the performance monitoring data corresponding to each computing node is processed so that the performance monitoring data corresponding to different nodes are aligned on a time scale to obtain target monitoring data corresponding to each computing node, specifically including:

[0015] For each computing node, if the data collection time in the performance monitoring data corresponding to the computing node is different from the data collection time in the performance monitoring data corresponding to other computing nodes, the data collection time in the performance monitoring data corresponding to the other computing nodes is used as the target time;

[0016] Based on the performance monitoring data corresponding to the computing node, the data collected by the computing node at each target time is determined as the target monitoring data corresponding to the computing node.

[0017] Optionally, processing the performance monitoring data corresponding to each computing node specifically includes:

[0018] For the performance monitoring data corresponding to each computing node, determine the missing value in the performance monitoring data, and use the data collection time corresponding to the missing value as the missing time;

[0019] Supplementing the missing value according to the data value corresponding to the data collection time adjacent to the missing time; and / or

[0020] Identify abnormal values ​​in the performance monitoring data whose data values ​​exceed a preset threshold, and process the abnormal values.

[0021] Optionally, extracting the time domain features of the target monitoring data in the time domain specifically includes:

[0022] For each computing node, determining the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node;

[0023] For each computing resource in the computing node, determining at least one time domain feature corresponding to the computing resource according to performance monitoring data corresponding to the computing resource as an independent time domain feature;

[0024] Extracting correlation information between independent time domain features corresponding to each computing resource in the computing node as a joint time domain feature corresponding to the computing node;

[0025] The time domain features of the target monitoring data corresponding to the computing node are determined according to the independent time domain features corresponding to each computing resource in the computing node and the joint time domain features corresponding to the computing node.

[0026] Optionally, for each computing resource in the computing node, at least one time domain feature corresponding to the computing resource is determined according to the performance monitoring data corresponding to the computing resource as an independent time domain feature, specifically including:

[0027] For each computing resource in the computing node, performance monitoring data corresponding to the computing resource is intercepted based on a sliding window of a preset time length, and a feature for characterizing a continuous change trend of the performance monitoring data over time is determined as a first time domain feature according to a statistic corresponding to each segment of intercepted data;

[0028] Identify each data mutation point in the performance monitoring data corresponding to the computing resource, and determine a second time domain feature according to the time interval between each adjacent mutation point;

[0029] Determine the long-term and short-term dependencies in the performance monitoring data corresponding to the computing resource, and determine the third time domain feature according to the long-term and short-term dependencies;

[0030] An independent time domain feature corresponding to the computing resource is determined according to at least one of the first time domain feature, the second time domain feature, and the third time domain feature.

[0031] Optionally, for the target monitoring data corresponding to each computing node, extracting frequency domain features of the target monitoring data in the frequency domain specifically includes:

[0032] For each computing node, determining the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node;

[0033] For each computing resource in the computing node, determining at least one frequency domain feature corresponding to the computing resource according to performance monitoring data corresponding to the computing resource as an independent frequency domain feature;

[0034] Extracting correlation information between independent frequency domain features corresponding to each computing resource in the computing node as a joint frequency domain feature corresponding to the computing node;

[0035] The frequency domain features of the target monitoring data corresponding to the computing node are determined according to the independent frequency domain features corresponding to each computing resource in the computing node and the joint frequency domain features corresponding to the computing node.

[0036] Optionally, for each computing resource in the computing node, at least one frequency domain feature corresponding to the computing resource is determined according to the performance monitoring data corresponding to the computing resource as an independent frequency domain feature, specifically including:

[0037] For each computing resource in the computing node, Fourier transform is performed on the performance monitoring data corresponding to the computing resource to obtain a frequency domain representation of the performance monitoring data corresponding to the computing resource, and the frequency domain representation coefficient is used as a first frequency domain feature;

[0038] Determine a frequency spectrum bandwidth corresponding to the frequency domain representation, and use the frequency spectrum bandwidth as a second frequency domain feature;

[0039] Determine a spectrum entropy corresponding to the performance monitoring data corresponding to the computing resource, and use the spectrum entropy as a third frequency domain feature;

[0040] An independent frequency domain feature corresponding to the computing resource is determined according to at least one of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature.

[0041] Optionally, determining the independent frequency domain feature corresponding to the computing resource according to at least one of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature specifically includes:

[0042] The first frequency domain feature, the second frequency domain feature and the third frequency domain feature are screened to determine key features therein as the independent frequency domain feature, wherein the key features are used to characterize the frequency domain features containing the most information.

[0043] Optionally, training the duration prediction model specifically includes:

[0044] Acquire a performance monitoring data sequence corresponding to computing resources on each computing node in the intelligent computing cluster when the intelligent computing cluster executes the target deep learning task based on the historical task allocation strategy, as the historical performance monitoring data corresponding to each computing node;

[0045] Processing the historical performance monitoring data corresponding to each computing node so that the historical performance monitoring data corresponding to different nodes are aligned on a time scale, and obtaining the historical target monitoring data corresponding to each computing node;

[0046] For the historical target monitoring data corresponding to each computing node, extract the historical time domain features of the historical target monitoring data in the time domain and the historical frequency domain features of the target monitoring data in the frequency domain;

[0047] Construct training samples based on the time domain features and frequency domain features corresponding to each computing node;

[0048] Inputting the training sample into a duration prediction model to be trained, so as to determine, through the duration prediction model, a predicted running time required to complete the deep learning task under the historical task allocation strategy;

[0049] According to the predicted deviation between the predicted running time and the actual label corresponding to the training sample, a loss value corresponding to the duration prediction model is determined, and the duration prediction model is trained according to the loss value.

[0050] Optionally, determining a loss value corresponding to the duration prediction model according to a prediction deviation between the predicted running duration and an actual label corresponding to the training sample specifically includes:

[0051] Determining a weight function and a basic loss value corresponding to the duration prediction model according to a prediction deviation between the predicted running duration and the actual label;

[0052] According to the weight function value and the basic loss value, the loss value corresponding to the duration prediction model is determined, wherein the weight function value is positively correlated with the loss value corresponding to the duration prediction model, and when the prediction deviation is greater than the model parameter value of the duration prediction model, the prediction deviation and the weight function value are positively correlated, and when the prediction deviation is less than the model parameter value of the duration prediction model, the weight function value is a preset value, and the preset value is less than the minimum weight function value when the prediction deviation is greater than the model parameter value of the duration prediction model.

[0053] This specification provides a method for executing a deep learning task based on runtime prediction, including:

[0054] A receiving module, used for receiving an execution request for a target deep learning task;

[0055] An acquisition module is used to execute the target deep learning task according to the execution request through each computing node in the intelligent computing cluster according to the current task allocation strategy, and after monitoring the execution of the target deep learning task for a specified time, obtain a performance monitoring data sequence of the computing resources on each computing node within the specified time as the performance monitoring data corresponding to each computing node;

[0056] A processing module, used to process the performance monitoring data corresponding to each computing node, so that the performance monitoring data corresponding to different nodes are aligned on a time scale, and obtain target monitoring data corresponding to each computing node;

[0057] An extraction module, for extracting, for each computing node corresponding to the target monitoring data, the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain;

[0058] A prediction module, configured to input the time domain features and the frequency domain features into a pre-trained duration prediction model, so as to determine the running time required to complete the target deep learning task through the duration prediction model;

[0059] An execution module is used to adjust the current task allocation strategy according to the running time, so as to execute the target deep learning task according to the adjusted task allocation strategy.

[0060] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned deep learning task execution method based on runtime prediction.

[0061] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned deep learning task execution method based on runtime prediction is implemented.

[0062] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0063] In the deep learning task execution method based on runtime prediction provided in this specification, each computing node in the intelligent computing cluster executes the target deep learning task according to the current task allocation strategy, and after the target deep learning task is executed for a specified duration, a performance monitoring data sequence of the computing resources on each computing node within the specified duration is obtained, and the performance monitoring data corresponding to each computing node is processed, and for the target monitoring data corresponding to each computing node, the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain are extracted; the time domain features and the frequency domain features are input into a pre-trained duration prediction model to determine the runtime required to complete the target deep learning task, and the current task allocation strategy is adjusted according to the runtime to execute the target deep learning task according to the adjusted task allocation strategy.

[0064] It can be seen from the above method that this scheme can obtain a performance monitoring data sequence of a specified duration during the execution of a deep learning task, extract time domain features and frequency domain features after processing them, and input the features into a duration prediction model to predict the running time required to complete the target deep learning task, thereby adjusting the task allocation strategy according to the prediction results. Compared with the current method of analyzing the internal structure and operating mechanism of the model, this scheme does not need to obtain the internal parameters and operating logic of the model, thereby ensuring the privacy and security of the model data, and through model prediction, it avoids the large amount of cost consumed by the predictor for manual prediction, thereby achieving improved efficiency and accuracy in duration prediction, and further ensuring the resource utilization and execution efficiency of deep learning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0066] Figure 1 A flowchart of a method for executing a deep learning task based on runtime prediction provided in this specification;

[0067] Figure 2 A schematic diagram of a time domain feature extraction method provided in this specification;

[0068] Figure 3 A schematic diagram of the model structure of a duration prediction model provided in this specification;

[0069] Figure 4 A schematic diagram of a change curve of a weight function provided in this specification;

[0070] Figure 5 A schematic diagram of a deep learning task execution device based on runtime prediction provided in this specification;

[0071] Figure 6 A method corresponding to the Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0073] The current deep learning task runtime prediction methods are mainly divided into two categories: one is the white box method and the other is the black box method. The white box method deeply analyzes the internal structure and operation mechanism of the deep learning model, including the network architecture, parameter scale, intelligent computing cluster parallel computing strategy, and the configuration of the intelligent computing cluster hardware resources (such as GPU, CPU, memory). This method theoretically models the operation process of the model and can provide higher prediction accuracy. However, it requires intrusive analysis of user code, which may cause privacy and security issues. In addition, this method requires the forecaster to have an in-depth understanding of the deep learning model and training framework, which increases the complexity of use and learning cost. The black box method focuses on analyzing the external features of the model, such as the model's metadata (such as the creator, model name, dataset information) and the dynamic monitoring data of the intelligent computing cluster resources occupied during operation. This method does not invade the model, protecting the privacy and security of user data. It uses the periodic characteristics of resource usage in deep learning tasks for prediction, but faces challenges in data collection, storage, and analysis, such as the difficulty in processing large-scale operating status data, the processing of abnormal data, and the problem of feature selection, all of which may affect the accuracy of runtime prediction.

[0074] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0075] Figure 1 A flowchart of a method for executing a deep learning task based on runtime prediction provided in this specification includes the following steps:

[0076] S101: receiving an execution request for a target deep learning task;

[0077] S102: According to the execution request, the target deep learning task is executed by each computing node in the intelligent computing cluster according to the current task allocation strategy, and after monitoring the execution of the target deep learning task for a specified time, a performance monitoring data sequence of the computing resources on each computing node within the specified time is obtained as the performance monitoring data corresponding to each computing node.

[0078] In response to the shortcomings of existing deep learning task runtime prediction methods, this specification provides a deep learning task runtime prediction method based on the fusion of time-frequency domain features of dynamic monitoring data of intelligent computing cluster resources. This method collects, stores and analyzes the dynamic monitoring data of the intelligent computing cluster resources occupied when deep learning tasks are running, extracts effective features and fully integrates them, and predicts the runtime of deep learning model training and reasoning tasks, thereby providing sufficient information support for optimizing resource utilization efficiency and model training and reasoning efficiency.

[0079] In this specification, the execution subject for implementing a deep learning task execution method based on runtime prediction can be a designated device such as a server. For the sake of ease of description, the following will only take the server as the execution subject as an example to illustrate a deep learning task execution method based on runtime prediction provided in this specification.

[0080] Among them, the server can receive execution requests for target deep learning tasks. In actual applications, the deep learning tasks can be training, testing and reasoning tasks for specified deep learning models (such as image recognition, natural language processing, anomaly detection, etc.).

[0081] After receiving the task execution request, the server can execute the target deep learning task according to the task execution request through each computing node in the intelligent computing cluster according to the current task allocation strategy.

[0082] Among them, the intelligent computing cluster is a centralized computing platform, which includes several computing nodes. The computing resources on each computing node include but are not limited to: GPU resources, CPU resources, memory resources, network resources, and I / O read and write resources.

[0083] The above task scheduling strategy is used to indicate the execution order of deep learning tasks on each computing node and the corresponding computing resource allocation method on each node.

[0084] After the target deep learning task is executed for a specified period of time, the server can obtain a performance monitoring data sequence of the computing resources on each computing node in the intelligent computing cluster within the specified period of time as the performance monitoring data corresponding to each computing node, where the specified period of time can be set according to actual conditions, and this manual does not make any specific restrictions on this.

[0085] In one embodiment provided in this specification, a distributed cluster monitoring system can be pre-established, which can cover all computing nodes involved in deep learning tasks. The monitoring system consists of the following main components:

[0086] Monitoring agent: used to collect resource usage of each node in the cluster. This component needs to be deployed on each computing node, periodically query the operating system and hardware interface to collect node resource usage, and send the collected information to the central monitor. The resource information collected by this component includes but is not limited to GPU usage, CPU usage, memory usage, network bandwidth usage, and I / O read and write speed;

[0087] Central monitor: used to receive data sent by the monitoring agent on each node and perform preliminary processing on the data, including summarizing the resource usage data of each node and splicing the periodic data of a single node.

[0088] First, the central monitor compresses the data to be transmitted (performance monitoring data) to reduce the network bandwidth usage, and adopts a lightweight network transmission protocol to send data. At the same time, it uses a message queue to temporarily store data. When it is determined that the distributed database has enough space to store the data, the data in the message queue is sent to the distributed database system to ensure the integrity of the data and the reliability of the transmission.

[0089] After receiving data from the central monitor, the distributed database system stores the data in the distributed database. The system dynamically allocates data to different storage nodes according to the distributed database load and resource usage to ensure load balancing.

[0090] The server can obtain the performance monitoring data corresponding to each computing node within a specified period of time from the distributed database system, and its format can be expressed as follows:

[0091] {

[0092] Mission name: <name-1>,

[0093] Resource usage: {

[0094] Node 1: [<data 1-1>,...,<data 1-N>],

[0095] …

[0096] Node n: [<data n-1>,...,<data nN>].

[0097] }

[0098] }

[0099] Among them, n is the number of computing nodes occupied when the target deep learning task is running, N is the number of cycles of resource usage data collected during the running of the task, that is, the number of times the monitoring agent deployed on each node sends data to the central controller during the running of the task, <data ij> is the resource usage data of the monitoring agent deployed on the i-th node at the j-th data collection time, and its format can be expressed as: <timestamp, resource 1 usage, ..., resource m usage>.

[0100] Among them, timestamp is the timestamp of the time when the data is collected, and m is the number of resource types collected.

[0101] S103: Processing the performance monitoring data corresponding to each computing node to align the performance monitoring data corresponding to different nodes on a time scale, and obtaining target monitoring data corresponding to each computing node.

[0102] The server may process the performance monitoring data corresponding to each computing node so as to align the performance monitoring data corresponding to different nodes on a time scale, and obtain the target monitoring data corresponding to each computing node.

[0103] Specifically, for each computing node, if the data collection moments in the performance monitoring data corresponding to the computing node are different from the data collection moments in the performance monitoring data corresponding to other computing nodes, the server can use the data collection moments in the performance monitoring data corresponding to the other computing nodes as target moments, and then determine the data collected by the computing node at each target moment based on the performance monitoring data corresponding to the computing node, as the target monitoring data corresponding to the computing node.

[0104] For example, if the data collection time corresponding to the performance monitoring data of other computing nodes is "1, 3, 5, 7", and the data collection time corresponding to the performance monitoring data of computing node x is "2, 4, 6, 8", the server can use "1, 3, 5, 7" as the target time, and then based on the data change curve corresponding to the performance monitoring data of computing node x, determine the performance monitoring data collected by computing node x at "1, 3, 5, 7" as the performance monitoring data corresponding to computing node x, so that the performance monitoring data corresponding to different computing nodes are aligned on the time scale.

[0105] In addition to aligning timestamps, the server can also perform missing value processing, outlier processing, etc. on the performance monitoring data corresponding to each computing node.

[0106] Among them, missing value processing is to process missing data caused by collection, transmission, storage, etc. In the process of processing missing values, for the performance monitoring data corresponding to each computing node, the server can determine the missing value in the performance monitoring data, and use the data collection time corresponding to the missing value as the missing time, and then use the data before and after the timestamps to interpolate to obtain the estimated value of the missing value, so as to supplement the missing value with the estimated value.

[0107] In addition, the server may also identify abnormal values ​​in the performance monitoring data whose data values ​​exceed a preset threshold value, and process the abnormal values.

[0108] In this specification, the server can set the abnormal value to the maximum value of the performance provided by the abnormal resource, and the server can reset the value of GPU utilization exceeding 100% to 100%. Of course, the server can also set the above abnormal value to the middle value or mean value of the performance data.

[0109] S104: For the target monitoring data corresponding to each computing node, extract the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain.

[0110] In the process of extracting time domain features, since deep learning training or reasoning tasks occupy many types of resources and there is a correlation between the use of various types of resources, the server can extract the time domain features of resource usage dynamic data from two aspects.

[0111] First: For each computing node, the server can determine the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node, and for each computing resource, the server can determine the independent time domain characteristics corresponding to the computing resource based on the performance monitoring data corresponding to the computing resource.

[0112] In this specification, for each computing resource in the computing node, the performance monitoring data corresponding to the computing resource is intercepted based on a sliding window of a preset time length, and a feature used to characterize the continuous change trend of the performance monitoring data over time is determined according to the statistics corresponding to each segment of the intercepted data, as the first time domain feature. For ease of understanding, this specification provides a schematic diagram of a method for extracting time domain features, such as Figure 2 shown.

[0113] Figure 2 A schematic diagram of a method for extracting time domain features provided in this specification.

[0114] Specifically, the server may adopt a sliding window method, starting from an initial timestamp, intercept a continuous section of data in the dynamic data, perform operations such as calculating the mean, median, and variance, and extract the first time domain feature of the continuous change trend of the dynamic data;

[0115] In addition, the server may identify each data mutation point in the performance monitoring data corresponding to the computing resource, and determine the second time domain feature according to the time interval between each adjacent mutation point;

[0116] In addition, the server can identify the long-term and short-term dependencies in the performance monitoring data corresponding to the computing resource through a preset long short-term memory network, and determine the third time domain feature based on the long-term and short-term dependencies. The long-term and short-term dependencies are used to represent the relationship between data that are far apart in time in the performance data sequence.

[0117] The server may use one or more of the first time domain feature, the second time domain feature, and the third time domain feature as independent time domain features of its corresponding computing resources.

[0118] Second: For each computing node, the server can extract the correlation information between the independent time domain features corresponding to each computing resource in the computing node (such as Pearson correlation coefficient, mutual information, dynamic time warping, Granger causality test, etc.), and use the extracted correlation information as the joint time domain feature corresponding to the computing node.

[0119] The server may then determine the time domain characteristics of the computing node in the time domain according to the joint time domain characteristics corresponding to the computing node and the independent time domain characteristics corresponding to each computing resource on the computing node.

[0120] Similarly, the server can also extract frequency domain features of resource usage dynamic data from two aspects.

[0121] First, for each computing node, the server can determine the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node. For each computing resource in the computing node, the server can determine the independent frequency domain characteristics corresponding to the computing resource based on the performance monitoring data corresponding to the computing resource.

[0122] Specifically, for each computing resource in the computing node, the server may perform Fourier transform on the performance monitoring data corresponding to the computing resource to obtain a frequency domain representation of the performance monitoring data corresponding to the computing resource, and use the frequency domain representation coefficient as the first frequency domain feature;

[0123] In addition, the server may determine the spectrum bandwidth corresponding to the above frequency domain representation (i.e., the frequency range occupied by the main part of the spectrum), and use the spectrum bandwidth as the second frequency domain feature;

[0124] In addition, the server may determine the spectrum entropy (ie, a measure of uniformity and complexity of spectrum distribution) corresponding to the performance monitoring data corresponding to the computing resource, and use the spectrum entropy as the third frequency domain feature.

[0125] Afterwards, the server may use one or more of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature as independent frequency domain features corresponding to the computing resource.

[0126] Second, the server can extract the correlation information between the independent frequency domain features corresponding to each computing resource in the computing node (such as cross-power spectrum density, phase spectrum correlation, Pearson correlation coefficient, etc.) as the joint frequency domain features corresponding to the computing node.

[0127] Furthermore, in order to remove the redundant information that may exist between the features and reduce the feature dimension, it is necessary to identify the key features. The server can screen the above independent frequency domain features and joint frequency domain features to determine the key features and use the key features as independent frequency domain features, wherein the key features are used to characterize the frequency domain features containing the most information.

[0128] For example, the server can select features whose variance exceeds a certain threshold. This is because features with too small a variance cannot effectively distinguish different time series and contain less useful information;

[0129] For another example, the server may perform principal component analysis on each frequency domain feature, thereby extracting the most informative features and achieving the purpose of reducing the number of features.

[0130] S104: Input the time domain features and the frequency domain features into a pre-trained duration prediction model to determine the running time required to complete the target deep learning task through the duration prediction model.

[0131] After determining the time domain features and frequency domain features corresponding to each computing node, the server can input the time domain features and frequency domain features corresponding to each computing node into a pre-trained duration prediction model, thereby determining the running time required to complete the target deep learning task through the duration prediction model.

[0132] In this specification, the duration prediction model may be provided with a feature fusion network and a prediction network. For ease of understanding, this specification provides a schematic diagram of the model structure of the duration prediction model, such as Figure 3 shown.

[0133] Figure 3 A schematic diagram of the model structure of a duration prediction model provided in this specification.

[0134] Among them, the feature fusion network is used to fuse the time domain features and the frequency domain features to obtain an output fusion feature. In practical applications, the models used by the feature fusion network include but are not limited to the splicing model, the cross attention model, etc.

[0135] The prediction network takes the fused features as input, determines and outputs the running time required to complete the target deep learning task. In practical applications, the model structure used by the prediction network includes but is not limited to support vector machine model, neural network regression model, ridge regression model and lasso regression model.

[0136] Furthermore, before using the above-mentioned duration prediction model, it is usually necessary to train the model. Only after the duration prediction model meets the training objectives (such as reaching a preset number of training times or converging to a preset range) can it be deployed for actual prediction of the running time of deep learning tasks.

[0137] Specifically, the server can obtain the performance monitoring data sequence corresponding to the computing resources on each computing node in the intelligent computing cluster when the intelligent computing cluster executes the target deep learning task based on the historical task allocation strategy, as the historical performance monitoring data corresponding to each computing node, and obtain the actual running time of the intelligent computing cluster to complete the target deep learning task based on the historical task allocation strategy, and use the actual running time as a label to construct a training sample according to the historical performance monitoring data corresponding to each computing node. The data set of the training sample can be expressed as:

[0138]

[0139] Where N is the number of sample points in the data set, t i , f i are the time domain features and frequency domain features extracted from the dynamic monitoring data of resources occupied by the deep learning task when the i-th training sample is running, r i is the actual running time to complete the deep learning task in the i-th training sample.

[0140] The server can process the above historical performance monitoring data to obtain the historical target monitoring data corresponding to each computing node, and then extract the historical time domain features of the historical target monitoring data in the time domain and the historical frequency domain features of the target monitoring data in the frequency domain for the historical target monitoring data corresponding to each computing node (please see S103~S104 for specific processing steps and feature extraction steps).

[0141] The server can then input the historical time domain features and the historical frequency domain features into the duration prediction model to be trained, so as to determine the predicted running time required to complete the deep learning task under the historical task allocation strategy through the duration prediction model.

[0142] The server can determine the loss value corresponding to the duration prediction model based on the predicted deviation between the predicted running time and the actual label corresponding to the training sample (i.e., the actual running time to complete the deep learning task), and train the duration prediction model according to the loss value.

[0143] Specifically, the server may determine a weight function and a basic loss value corresponding to the duration prediction model according to a prediction deviation between the predicted running time and the actual label, and then determine a loss value corresponding to the duration prediction model according to the weight function and the basic loss value. The loss value may be expressed as:

[0144]

[0145] in, is the weight function, τ is the model parameter (τ>0), is the basic loss value, and The relationship between Figure 4 shown.

[0146] Figure 4 The diagram is a variation curve diagram of a weight function provided in this specification.

[0147] Depend on Figure 4 It can be seen that when the prediction error of the i-th sample is less than the parameter τ, a smaller weight is assigned to the sample, and when the prediction error exceeds the parameter τ, the weight function value increases as the prediction error increases.

[0148] It can be seen that the prediction deviation between the predicted duration and the actual duration There is a positive correlation between the prediction deviation and the weight function value. When the prediction deviation is greater than the model parameter value of the duration prediction model, there is a positive correlation between the prediction deviation and the weight function value. When the prediction deviation is less than the model parameter value of the duration prediction model, the weight function value is a preset value, and the preset value is less than the minimum weight function value when the prediction deviation is greater than the model parameter value of the duration prediction model.

[0149] S105: According to the running time, the current task allocation strategy is adjusted to execute the target deep learning task according to the adjusted task allocation strategy.

[0150] After predicting the runtime required for the target deep learning model to complete the target deep learning task according to the current task allocation strategy based on the performance monitoring data corresponding to each computing node of the specified duration, the server can adjust the current task allocation strategy based on the runtime. For example, when the predicted runtime is greater than the preset threshold, it can be considered that the current task allocation strategy is ineffective. The server can adjust the task execution order of each computing node for the target deep learning task, as well as the allocation strategy of computing resources in each computing node, so as to reduce the overall runtime of the deep learning task and further improve the utilization of computing resources and task execution efficiency.

[0151] From the above method, it can be seen that this solution can overcome the limitations of the existing deep learning task runtime prediction methods in terms of accuracy and ease of use, and achieve more efficient and accurate predictions. By comprehensively considering key factors such as data collection, storage, cleaning, and feature selection, intrusive analysis is avoided, user privacy and data security are protected, and the high requirements for user expertise are reduced, reducing the learning cost.

[0152] By accurately capturing the periodic characteristics of deep learning tasks in the use of intelligent computing cluster resources, the accuracy of running time prediction is improved, and the collection and storage of large-scale running status data, the identification and processing of abnormal data, and the screening of key features are effectively solved. This not only optimizes resource scheduling and reduces computing costs, but also improves production efficiency, which is particularly important for resource-intensive deep learning applications.

[0153] The above is one or more implementations of the deep learning task execution method based on runtime prediction of this specification. Based on the same idea, this specification also provides a corresponding deep learning task execution device based on runtime prediction, such as Figure 5 shown.

[0154] Figure 5 A schematic diagram of a deep learning task execution device based on runtime prediction provided in this specification includes:

[0155] A receiving module 501 is used to receive an execution request for a target deep learning task;

[0156] An acquisition module 502 is used to execute the target deep learning task according to the execution request through each computing node in the intelligent computing cluster according to the current task allocation strategy, and after monitoring the execution of the target deep learning task for a specified time, obtain a performance monitoring data sequence of the computing resources on each computing node within the specified time as the performance monitoring data corresponding to each computing node;

[0157] The processing module 503 is used to process the performance monitoring data corresponding to each computing node so as to align the performance monitoring data corresponding to different nodes on a time scale and obtain target monitoring data corresponding to each computing node;

[0158] An extraction module 504 is used to extract, for each computing node, the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain;

[0159] A prediction module 505 is used to input the time domain features and the frequency domain features into a pre-trained duration prediction model to determine the running time required to complete the target deep learning task through the duration prediction model;

[0160] The execution module 506 is used to adjust the current task allocation strategy according to the running time, so as to execute the target deep learning task according to the adjusted task allocation strategy.

[0161] Optionally, the processing module 503 is specifically used to, for each computing node, if the data collection moments in the performance monitoring data corresponding to the computing node are different from the data collection moments in the performance monitoring data corresponding to other computing nodes, then use the data collection moments in the performance monitoring data corresponding to the other computing nodes as target moments; based on the performance monitoring data corresponding to the computing node, determine the data collected by the computing node at each target moment as the target monitoring data corresponding to the computing node.

[0162] Optionally, the processing module 503 is specifically used to determine, for the performance monitoring data corresponding to each computing node, the missing values ​​in the performance monitoring data, and use the data collection time corresponding to the missing value as the missing time; supplement the missing value according to the data value corresponding to the data collection time adjacent to the missing time; and / or identify abnormal values ​​in the performance monitoring data whose data values ​​exceed a preset threshold, and process the abnormal values.

[0163] Optionally, the extraction module 504 is specifically used to, for each computing node, determine the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node; for each computing resource in the computing node, determine at least one time domain feature corresponding to the computing resource based on the performance monitoring data corresponding to the computing resource as an independent time domain feature; extract correlation information between the independent time domain features corresponding to each computing resource in the computing node as a joint time domain feature corresponding to the computing node; and determine the time domain features of the target monitoring data corresponding to the computing node based on the independent time domain features corresponding to each computing resource in the computing node and the joint time domain features corresponding to the computing node.

[0164] Optionally, the extraction module 504 is specifically used to, for each computing resource in the computing node, intercept the performance monitoring data corresponding to the computing resource based on a sliding window of a preset time length, and determine a feature used to characterize the continuous change trend of the performance monitoring data over time as a first time domain feature based on the statistics corresponding to each segment of the intercepted data; identify each data mutation point in the performance monitoring data corresponding to the computing resource, and determine a second time domain feature based on the time interval between each adjacent mutation point; determine the long-term and short-term dependencies in the performance monitoring data corresponding to the computing resource, and determine a third time domain feature based on the long-term and short-term dependencies; determine an independent time domain feature corresponding to the computing resource based on at least one of the first time domain feature, the second time domain feature and the third time domain feature.

[0165] Optionally, the extraction module 504 is specifically used to, for each computing node, determine the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node; for each computing resource in the computing node, determine at least one frequency domain feature corresponding to the computing resource based on the performance monitoring data corresponding to the computing resource as an independent frequency domain feature; extract correlation information between the independent frequency domain features corresponding to each computing resource in the computing node as a joint frequency domain feature corresponding to the computing node; and determine the frequency domain features of the target monitoring data corresponding to the computing node based on the independent frequency domain features corresponding to each computing resource in the computing node and the joint frequency domain features corresponding to the computing node.

[0166] Optionally, the extraction module 504 is specifically used to perform Fourier transform on the performance monitoring data corresponding to each computing resource in the computing node, obtain a frequency domain representation of the performance monitoring data corresponding to the computing resource, and use the frequency domain representation coefficient as a first frequency domain feature; determine the spectrum bandwidth corresponding to the frequency domain representation, and use the spectrum bandwidth as a second frequency domain feature; determine the spectrum entropy corresponding to the performance monitoring data corresponding to the computing resource, and use the spectrum entropy as a third frequency domain feature; and determine an independent frequency domain feature corresponding to the computing resource based on at least one of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature.

[0167] Optionally, the extraction module 504 is specifically used to screen the independent frequency domain features and the joint frequency domain features to determine the key features therein as the frequency domain features of the target monitoring data corresponding to the computing node, wherein the key features are used to characterize the frequency domain features containing the most information.

[0168] Optionally, the device also includes: a training module 507, which is used to obtain a performance monitoring data sequence corresponding to the computing resources on each computing node in the intelligent computing cluster when the intelligent computing cluster executes the target deep learning task based on the historical task allocation strategy, as the historical performance monitoring data corresponding to each computing node, and construct a training sample based on the historical performance monitoring data; process the historical performance monitoring data corresponding to each computing node so that the historical performance monitoring data corresponding to different nodes are aligned on the time scale to obtain the historical target monitoring data corresponding to each computing node; for the historical target monitoring data corresponding to each computing node, extract the historical time domain features of the historical target monitoring data in the time domain and the historical frequency domain features of the target monitoring data in the frequency domain; construct a training sample based on the time domain features and frequency domain features corresponding to each computing node; input the training sample into the duration prediction model to be trained, so as to determine the predicted running time required to complete the deep learning task under the historical task allocation strategy through the duration prediction model; determine the loss value corresponding to the duration prediction model based on the prediction deviation between the predicted running time and the actual label corresponding to the training sample, and train the duration prediction model based on the loss value.

[0169] Optionally, the training module 507 is specifically used to determine the weight function and basic loss value corresponding to the duration prediction model according to the predicted deviation between the predicted running time and the actual label; determine the loss value corresponding to the duration prediction model according to the weight function value and the basic loss value, wherein the weight function value is positively correlated with the loss value corresponding to the duration prediction model, and when the prediction deviation is greater than the model parameter value of the duration prediction model, the prediction deviation and the weight function value are positively correlated, and when the prediction deviation is less than the model parameter value of the duration prediction model, the weight function value is a preset value, and the preset value is less than the minimum weight function value when the prediction deviation is greater than the model parameter value of the duration prediction model.

[0170] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A deep learning task execution method based on runtime prediction is provided.

[0171] This manual also provides Figure 6 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 6As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The method for executing deep learning tasks based on runtime prediction. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0172] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0173] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0174] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0175] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0176] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0177] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0180] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0181] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0182] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0183] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0184] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0186] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A deep learning task execution method based on runtime prediction, characterized in that: include: Receive an execution request for a target deep learning task; According to the execution request, the target deep learning task is executed by each computing node in the intelligent computing cluster according to the current task allocation strategy, and after monitoring the execution of the target deep learning task for a specified time, a performance monitoring data sequence of the computing resources on each computing node within the specified time is obtained as the performance monitoring data corresponding to each computing node; Processing the performance monitoring data corresponding to each computing node so that the performance monitoring data corresponding to different nodes are aligned on a time scale, and obtaining target monitoring data corresponding to each computing node; For the target monitoring data corresponding to each computing node, extract the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain; Inputting the time domain features and the frequency domain features into a pre-trained duration prediction model to determine the running time required to complete the target deep learning task through the duration prediction model; According to the running time, the current task allocation strategy is adjusted to execute the target deep learning task according to the adjusted task allocation strategy.

2. The method according to claim 1, characterized in that Processing the performance monitoring data corresponding to each computing node to align the performance monitoring data corresponding to different nodes on a time scale, and obtaining target monitoring data corresponding to each computing node, specifically including: For each computing node, if the data collection time in the performance monitoring data corresponding to the computing node is different from the data collection time in the performance monitoring data corresponding to other computing nodes, the data collection time in the performance monitoring data corresponding to the other computing nodes is used as the target time; Based on the performance monitoring data corresponding to the computing node, the data collected by the computing node at each target time is determined as the target monitoring data corresponding to the computing node.

3. The method according to claim 2, characterized in that Processing the performance monitoring data corresponding to each computing node specifically includes: For the performance monitoring data corresponding to each computing node, determine the missing value in the performance monitoring data, and use the data collection time corresponding to the missing value as the missing time; Supplementing the missing value according to the data value corresponding to the data collection time adjacent to the missing time; and / or Identify abnormal values ​​in the performance monitoring data whose data values ​​exceed a preset threshold, and process the abnormal values.

4. The method according to claim 1, characterized in that Extract the time domain features of the target monitoring data in the time domain, including: For each computing node, determining the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node; For each computing resource in the computing node, determining at least one time domain feature corresponding to the computing resource according to performance monitoring data corresponding to the computing resource as an independent time domain feature; Extracting correlation information between independent time domain features corresponding to each computing resource in the computing node as a joint time domain feature corresponding to the computing node; The time domain features of the target monitoring data corresponding to the computing node are determined according to the independent time domain features corresponding to each computing resource in the computing node and the joint time domain features corresponding to the computing node.

5. The method according to claim 4, characterized in that For each computing resource in the computing node, at least one time domain feature corresponding to the computing resource is determined according to the performance monitoring data corresponding to the computing resource as an independent time domain feature, specifically including: For each computing resource in the computing node, performance monitoring data corresponding to the computing resource is intercepted based on a sliding window of a preset time length, and a feature for characterizing a continuous change trend of the performance monitoring data over time is determined as a first time domain feature according to a statistic corresponding to each segment of intercepted data; Identify each data mutation point in the performance monitoring data corresponding to the computing resource, and determine a second time domain feature according to the time interval between each adjacent mutation point; Determine the long-term and short-term dependencies in the performance monitoring data corresponding to the computing resource, and determine the third time domain feature according to the long-term and short-term dependencies; An independent time domain feature corresponding to the computing resource is determined according to at least one of the first time domain feature, the second time domain feature, and the third time domain feature.

6. The method according to claim 1, characterized in that For the target monitoring data corresponding to each computing node, extract the frequency domain features of the target monitoring data in the frequency domain, specifically including: For each computing node, determining the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node; For each computing resource in the computing node, determining at least one frequency domain feature corresponding to the computing resource according to performance monitoring data corresponding to the computing resource as an independent frequency domain feature; Extracting correlation information between independent frequency domain features corresponding to each computing resource in the computing node as a joint frequency domain feature corresponding to the computing node; The frequency domain features of the target monitoring data corresponding to the computing node are determined according to the independent frequency domain features corresponding to each computing resource in the computing node and the joint frequency domain features corresponding to the computing node.

7. The method according to claim 6, characterized in that For each computing resource in the computing node, at least one frequency domain feature corresponding to the computing resource is determined according to the performance monitoring data corresponding to the computing resource as an independent frequency domain feature, specifically including: For each computing resource in the computing node, Fourier transform is performed on the performance monitoring data corresponding to the computing resource to obtain a frequency domain representation of the performance monitoring data corresponding to the computing resource, and the frequency domain representation coefficient is used as a first frequency domain feature; Determine a frequency spectrum bandwidth corresponding to the frequency domain representation, and use the frequency spectrum bandwidth as a second frequency domain feature; Determine a spectrum entropy corresponding to the performance monitoring data corresponding to the computing resource, and use the spectrum entropy as a third frequency domain feature; An independent frequency domain feature corresponding to the computing resource is determined according to at least one of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature.

8. The method according to claim 7, characterized in that Determining the frequency domain features of the target monitoring data corresponding to the computing node according to the independent frequency domain features corresponding to each computing resource in the computing node and the joint frequency domain features corresponding to the computing node, specifically includes: The independent frequency domain features and the joint frequency domain features are screened to determine key features therein as the frequency domain features of the target monitoring data corresponding to the computing node, wherein the key features are used to characterize the frequency domain features containing the most information.

9. The method according to claim 1, characterized in that Training the duration prediction model specifically includes: Acquire a performance monitoring data sequence corresponding to computing resources on each computing node in the intelligent computing cluster when the intelligent computing cluster executes a target deep learning task based on a historical task allocation strategy, as historical performance monitoring data corresponding to each computing node, and construct a training sample based on the historical performance monitoring data; Processing the historical performance monitoring data corresponding to each computing node so that the historical performance monitoring data corresponding to different nodes are aligned on a time scale, and obtaining the historical target monitoring data corresponding to each computing node; For the historical target monitoring data corresponding to each computing node, extract the historical time domain features of the historical target monitoring data in the time domain and the historical frequency domain features of the target monitoring data in the frequency domain; Construct training samples based on the time domain features and frequency domain features corresponding to each computing node; Inputting the training sample into a duration prediction model to be trained, so as to determine, through the duration prediction model, a predicted running time required to complete the deep learning task under the historical task allocation strategy; According to the predicted deviation between the predicted running time and the actual label corresponding to the training sample, a loss value corresponding to the duration prediction model is determined, and the duration prediction model is trained according to the loss value.

10. The method according to claim 1, characterized in that Determining a loss value corresponding to the duration prediction model according to a predicted deviation between the predicted running time and an actual label corresponding to the training sample specifically includes: Determining a weight function value and a basic loss value corresponding to the duration prediction model according to a prediction deviation between the predicted running duration and the actual label; According to the weight function value and the basic loss value, the loss value corresponding to the duration prediction model is determined, wherein the weight function value is positively correlated with the loss value corresponding to the duration prediction model, and when the prediction deviation is greater than the model parameter value of the duration prediction model, the prediction deviation and the weight function value are positively correlated, and when the prediction deviation is less than the model parameter value of the duration prediction model, the weight function value is a preset value, and the preset value is less than the minimum weight function value when the prediction deviation is greater than the model parameter value of the duration prediction model.

11. A deep learning task execution method based on runtime prediction, characterized in that: include: A receiving module, used for receiving an execution request for a target deep learning task; An acquisition module is used to execute the target deep learning task according to the execution request through each computing node in the intelligent computing cluster according to the current task allocation strategy, and after monitoring the execution of the target deep learning task for a specified time, obtain a performance monitoring data sequence of the computing resources on each computing node within the specified time as the performance monitoring data corresponding to each computing node; A processing module, used to process the performance monitoring data corresponding to each computing node, so that the performance monitoring data corresponding to different nodes are aligned on a time scale, and obtain target monitoring data corresponding to each computing node; An extraction module, for extracting, for each computing node corresponding to the target monitoring data, the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain; A prediction module, configured to input the time domain features and the frequency domain features into a pre-trained duration prediction model, so as to determine the running time required to complete the target deep learning task through the duration prediction model; An execution module is used to adjust the current task allocation strategy according to the running time, so as to execute the target deep learning task according to the adjusted task allocation strategy.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 10 is implemented.

Images