A deep learning task execution method and device based on runtime length prediction

Abstract

The specification discloses a deep learning task execution method and device based on runtime length prediction. The method comprises: executing a target deep learning task by each computing node in an intelligent computing cluster according to a current task allocation strategy, and obtaining performance monitoring data sequences of computing resources on each computing node within a specified time length after the target deep learning task is executed for the specified time length; processing the performance monitoring data corresponding to each computing node; for the target monitoring data corresponding to each computing node, extracting time domain features of the target monitoring data in the time domain and frequency domain features of the target monitoring data in the frequency domain; inputting the time domain features and the frequency domain features into a pre-trained time length prediction model to determine a runtime length required for completing the target deep learning task; and adjusting the current task allocation strategy according to the runtime length, so as to execute the target deep learning task according to the adjusted task allocation strategy.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and device for executing deep learning tasks based on runtime prediction. Background Art

[0002] With the rapid development of deep learning technology, deep learning models are placing increasing demands on intelligent computing clusters for computing resources such as networks, memory, GPUs, and CPUs, posing additional challenges to resource allocation and cost management. Accurately predicting the runtime of these models during training and inference tasks is crucial for optimizing resource scheduling, reducing computing costs, and improving production efficiency.

[0003] However, current approaches typically predict task runtimes by deeply analyzing the internal structure and operational mechanisms of deep learning models, including network architecture, parameter scale, intelligent computing cluster parallel computing strategies, and the configuration of intelligent computing cluster hardware resources (such as GPUs, CPUs, and memory). However, this approach requires intrusive analysis of user code, raising privacy and security concerns. Furthermore, this approach requires a deep understanding of deep learning models and training frameworks, significantly increasing prediction costs and efficiency, and making it difficult to guarantee the accuracy of prediction results.

[0004] Therefore, how to improve the accuracy and efficiency of predicting the duration of deep learning tasks, reduce the prediction cost, and further ensure the resource utilization and execution efficiency of deep learning tasks is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a method and device for executing deep learning tasks based on runtime prediction.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a method for executing deep learning tasks based on runtime prediction, including:

[0008] Receive an execution request for a target deep learning task;

[0009] According to the execution request, each computing node in the intelligent computing cluster executes the target deep learning task according to the current task allocation policy, and after monitoring the execution of the target deep learning task for a specified duration, obtains a performance monitoring data sequence of the computing resources on each computing node within the specified duration as the performance monitoring data corresponding to each computing node;

[0010] Processing the performance monitoring data corresponding to each computing node so that the performance monitoring data corresponding to different nodes are aligned on a time scale, thereby obtaining target monitoring data corresponding to each computing node;

[0011] For the target monitoring data corresponding to each computing node, extract the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain;

[0012] Inputting the time domain features and the frequency domain features into a pre-trained duration prediction model to determine the runtime required to complete the target deep learning task through the duration prediction model;

[0013] According to the running time, the current task allocation strategy is adjusted to execute the target deep learning task according to the adjusted task allocation strategy.

[0014] Optionally, processing the performance monitoring data corresponding to each computing node to align the performance monitoring data corresponding to different nodes on a time scale to obtain target monitoring data corresponding to each computing node specifically includes:

[0015] For each computing node, if the data collection moments in the performance monitoring data corresponding to the computing node are different from the data collection moments in the performance monitoring data corresponding to other computing nodes, the data collection moments in the performance monitoring data corresponding to the other computing nodes are used as the target moments;

[0016] Based on the performance monitoring data corresponding to the computing node, the data collected by the computing node at each target time is determined as the target monitoring data corresponding to the computing node.

[0017] Optionally, processing the performance monitoring data corresponding to each computing node specifically includes:

[0018] For each computing node's corresponding performance monitoring data, determine the missing value in the performance monitoring data, and use the data collection time corresponding to the missing value as the missing time;

[0019] Supplementing the missing value based on the data value corresponding to the data collection time adjacent to the missing time; and / or

[0020] Identify abnormal values ​​in the performance monitoring data whose data values ​​exceed a preset threshold and process the abnormal values.

[0021] Optionally, extracting the time domain features of the target monitoring data in the time domain specifically includes:

[0022] For each computing node, determining the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node;

[0023] For each computing resource in the computing node, determining at least one time domain feature corresponding to the computing resource according to performance monitoring data corresponding to the computing resource as an independent time domain feature;

[0024] Extracting correlation information between independent time domain features corresponding to each computing resource in the computing node as a joint time domain feature corresponding to the computing node;

[0025] The time domain features of the target monitoring data corresponding to the computing node are determined according to the independent time domain features corresponding to each computing resource in the computing node and the joint time domain features corresponding to the computing node.

[0026] Optionally, for each computing resource in the computing node, at least one time domain feature corresponding to the computing resource is determined based on the performance monitoring data corresponding to the computing resource as an independent time domain feature, specifically including:

[0027] For each computing resource in the computing node, performance monitoring data corresponding to the computing resource is intercepted based on a sliding window of a preset time length, and a feature used to characterize a continuous temporal change trend of the performance monitoring data is determined based on a statistic corresponding to each segment of the intercepted data as a first time-domain feature;

[0028] Identifying each data mutation point in the performance monitoring data corresponding to the computing resource, and determining a second time domain feature based on the time interval between adjacent mutation points;

[0029] Determining a long-term and short-term dependency relationship in the performance monitoring data corresponding to the computing resource, and determining a third time domain feature based on the long-term and short-term dependency relationship;

[0030] An independent time domain feature corresponding to the computing resource is determined according to at least one of the first time domain feature, the second time domain feature, and the third time domain feature.

[0031] Optionally, for the target monitoring data corresponding to each computing node, extracting frequency domain features of the target monitoring data in the frequency domain specifically includes:

[0032] For each computing node, determining the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node;

[0033] For each computing resource in the computing node, determining at least one frequency domain feature corresponding to the computing resource according to performance monitoring data corresponding to the computing resource as an independent frequency domain feature;

[0034] Extracting correlation information between independent frequency domain features corresponding to each computing resource in the computing node as a joint frequency domain feature corresponding to the computing node;

[0035] The frequency domain features of the target monitoring data corresponding to the computing node are determined according to the independent frequency domain features corresponding to each computing resource in the computing node and the joint frequency domain features corresponding to the computing node.

[0036] Optionally, for each computing resource in the computing node, at least one frequency domain feature corresponding to the computing resource is determined as an independent frequency domain feature based on performance monitoring data corresponding to the computing resource, specifically including:

[0037] For each computing resource in the computing node, perform Fourier transform on the performance monitoring data corresponding to the computing resource to obtain a frequency domain representation of the performance monitoring data corresponding to the computing resource, and use the frequency domain representation coefficient as a first frequency domain feature;

[0038] determining a spectrum bandwidth corresponding to the frequency domain representation, and using the spectrum bandwidth as a second frequency domain feature;

[0039] Determine a spectrum entropy corresponding to the performance monitoring data corresponding to the computing resource, and use the spectrum entropy as a third frequency domain feature;

[0040] An independent frequency domain feature corresponding to the computing resource is determined according to at least one of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature.

[0041] Optionally, determining, according to at least one of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature, an independent frequency domain feature corresponding to the computing resource specifically includes:

[0042] The first frequency domain feature, the second frequency domain feature, and the third frequency domain feature are screened to determine a key feature therein as the independent frequency domain feature, wherein the key feature is used to characterize the frequency domain feature containing the most information.

[0043] Optionally, training the duration prediction model specifically includes:

[0044] Obtaining a performance monitoring data sequence corresponding to computing resources on each computing node in the intelligent computing cluster when the intelligent computing cluster executes the target deep learning task based on the historical task allocation strategy, as the historical performance monitoring data corresponding to each computing node;

[0045] Processing the historical performance monitoring data corresponding to each computing node to align the historical performance monitoring data corresponding to different nodes on a time scale, thereby obtaining historical target monitoring data corresponding to each computing node;

[0046] For the historical target monitoring data corresponding to each computing node, extract the historical time domain features of the historical target monitoring data in the time domain and the historical frequency domain features of the target monitoring data in the frequency domain;

[0047] Construct training samples based on the time domain features and frequency domain features corresponding to each computing node;

[0048] Inputting the training sample into a duration prediction model to be trained, so as to determine, through the duration prediction model, a predicted running time required to complete the deep learning task under the historical task allocation strategy;

[0049] According to the predicted deviation between the predicted running time and the actual label corresponding to the training sample, a loss value corresponding to the duration prediction model is determined, and the duration prediction model is trained according to the loss value.

[0050] Optionally, determining a loss value corresponding to the duration prediction model according to a predicted deviation between the predicted running duration and an actual label corresponding to the training sample specifically includes:

[0051] Determining a weight function and a basic loss value corresponding to the duration prediction model according to a predicted deviation between the predicted running time and the actual label;

[0052] According to the weight function value and the basic loss value, the loss value corresponding to the duration prediction model is determined, wherein the weight function value and the loss value corresponding to the duration prediction model are positively correlated, and when the prediction deviation is greater than the model parameter value of the duration prediction model, the prediction deviation and the weight function value are positively correlated, and when the prediction deviation is less than the model parameter value of the duration prediction model, the weight function value is a preset value, and the preset value is less than the minimum weight function value when the prediction deviation is greater than the model parameter value of the duration prediction model.

[0053] This specification provides a method for executing deep learning tasks based on runtime prediction, including:

[0054] A receiving module, configured to receive an execution request for a target deep learning task;

[0055] An acquisition module is configured to execute the target deep learning task according to the execution request through each computing node in the intelligent computing cluster according to the current task allocation policy, and after monitoring the execution of the target deep learning task for a specified duration, obtain a performance monitoring data sequence of the computing resources on each computing node within the specified duration as the performance monitoring data corresponding to each computing node;

[0056] a processing module, configured to process the performance monitoring data corresponding to each computing node so as to align the performance monitoring data corresponding to different nodes on a time scale and obtain target monitoring data corresponding to each computing node;

[0057] An extraction module is used to extract the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain for the target monitoring data corresponding to each computing node;

[0058] A prediction module, configured to input the time domain features and the frequency domain features into a pre-trained duration prediction model, so as to determine the runtime required to complete the target deep learning task through the duration prediction model;

[0059] An execution module is used to adjust the current task allocation strategy according to the runtime, so as to execute the target deep learning task according to the adjusted task allocation strategy.

[0060] This specification provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for executing deep learning tasks based on runtime prediction.

[0061] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for executing a deep learning task based on runtime prediction is implemented.

[0062] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0063] In the deep learning task execution method based on runtime prediction provided in this specification, each computing node in the intelligent computing cluster executes the target deep learning task according to the current task allocation strategy, and after the target deep learning task is executed for a specified duration, a performance monitoring data sequence of the computing resources on each computing node within the specified duration is obtained, and the performance monitoring data corresponding to each computing node is processed. For the target monitoring data corresponding to each computing node, the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain are extracted; the time domain features and the frequency domain features are input into a pre-trained duration prediction model to determine the runtime required to complete the target deep learning task, and the current task allocation strategy is adjusted according to the runtime to execute the target deep learning task according to the adjusted task allocation strategy.

[0064] It can be seen from the above method that this scheme can obtain a performance monitoring data sequence of a specified duration during the execution of a deep learning task, extract time domain features and frequency domain features after processing them, and input the features into a duration prediction model to predict the running time required to complete the target deep learning task, thereby adjusting the task allocation strategy according to the prediction results. Compared with the current method of analyzing the internal structure and operation mechanism of the model, this scheme does not need to obtain the internal parameters and operation logic of the model, thereby ensuring the privacy and security of the model data, and through model prediction, it avoids the large amount of cost consumed by the predictor for manual prediction, thereby achieving improved efficiency and accuracy in duration prediction, and further ensuring the resource utilization and execution efficiency of deep learning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0066] Figure 1 This is a flowchart of a method for executing a deep learning task based on runtime prediction provided in this specification;

[0067] Figure 2 A schematic diagram of a time domain feature extraction method provided in this specification;

[0068] Figure 3 This is a schematic diagram of the model structure of a duration prediction model provided in this specification;

[0069] Figure 4 A schematic diagram of a change curve of a weight function provided in this specification;

[0070] Figure 5 A schematic diagram of a deep learning task execution device based on runtime prediction provided in this specification;

[0071] Figure 6 This manual provides a corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0072] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0073] Current deep learning task runtime prediction methods fall into two main categories: white-box and black-box. White-box methods analyze the internal structure and operating mechanisms of deep learning models, including network architecture, parameter size, parallel computing strategies within the intelligent computing cluster, and the configuration of the intelligent computing cluster's hardware resources (such as GPUs, CPUs, and memory). This approach theoretically models the model's runtime and can provide high prediction accuracy. However, it requires intrusive analysis of user code, potentially raising privacy and security concerns. Furthermore, this method requires the predictor to have a deep understanding of deep learning models and training frameworks, increasing complexity and learning costs. Black-box methods, on the other hand, focus on analyzing external model features, such as model metadata (e.g., creator, model name, dataset information) and dynamic monitoring of intelligent computing cluster resources consumed during runtime. This approach does not intrude into the model's internals, protecting the privacy and security of user data. It uses the periodic characteristics of resource usage in deep learning tasks for prediction, but faces challenges in data collection, storage, and analysis, such as the difficulty in processing large-scale operating status data, the handling of abnormal data, and feature selection issues, all of which may affect the accuracy of runtime predictions.

[0074] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0075] Figure 1 This is a flowchart of a method for executing a deep learning task based on runtime prediction provided in this specification, including the following steps:

[0076] S101: Receive an execution request for a target deep learning task;

[0077] S102: Based on the execution request, the target deep learning task is executed by each computing node in the intelligent computing cluster according to the current task allocation strategy, and after monitoring the execution of the target deep learning task for a specified time, a performance monitoring data sequence of the computing resources on each computing node within the specified time is obtained as the performance monitoring data corresponding to each computing node.

[0078] In response to the shortcomings of existing deep learning task runtime prediction methods, this manual provides a deep learning task runtime prediction method based on the fusion of time-frequency domain features of dynamic monitoring data of intelligent computing cluster resources. This method collects, stores and analyzes the dynamic monitoring data of the intelligent computing cluster resources occupied during the execution of deep learning tasks, extracts effective features and fully integrates them, and predicts the runtime of deep learning model training and reasoning tasks, thereby providing sufficient information support for optimizing resource utilization efficiency and model training and reasoning efficiency.

[0079] In this specification, the execution subject for implementing a deep learning task execution method based on runtime prediction can be a designated device such as a server. For the sake of convenience of description, the following will only use the server as an example of the execution subject to illustrate a deep learning task execution method based on runtime prediction provided in this specification.

[0080] Among them, the server can receive execution requests for target deep learning tasks. In actual applications, the deep learning tasks can be training, testing and reasoning tasks for specified deep learning models (such as image recognition, natural language processing, anomaly detection, etc.).

[0081] After receiving the task execution request, the server can execute the target deep learning task according to the task execution request through each computing node in the intelligent computing cluster according to the current task allocation strategy.

[0082] Among them, the intelligent computing cluster is a centralized computing platform, which includes several computing nodes. The computing resources on each computing node include but are not limited to: GPU resources, CPU resources, memory resources, network resources, and I / O read and write resources.

[0083] The above task scheduling strategy is used to indicate the execution order of deep learning tasks on each computing node and the corresponding computing resource allocation method on each node.

[0084] After the target deep learning task is executed for a specified period of time, the server can obtain a sequence of performance monitoring data of the computing resources on each computing node in the intelligent computing cluster within the specified period of time as the performance monitoring data corresponding to each computing node. The specified period of time can be set according to actual conditions, and this manual does not make any specific restrictions on this.

[0085] In one embodiment provided in this specification, a distributed cluster monitoring system can be pre-established that covers all computing nodes involved in deep learning tasks. The monitoring system consists of the following main components:

[0086] Monitoring agent: Used to collect resource usage information for each node in the cluster. This component needs to be deployed on each computing node, periodically querying the operating system and hardware interfaces to collect node resource usage information, and sending the collected information to the central monitoring system. The resource information collected by this component includes but is not limited to GPU usage, CPU usage, memory usage, network bandwidth usage, and I / O read and write speeds.

[0087] Central monitor: used to receive data sent from the monitoring agent on each node and perform preliminary processing on the data, including summarizing the resource usage data of each node and splicing the periodic data of a single node.

[0088] First, the central monitor compresses the data to be transmitted (performance monitoring data) to reduce network bandwidth usage, and uses a lightweight network transmission protocol to send data. At the same time, it uses a message queue to temporarily store data. When it is determined that the distributed database has enough space to store data, the data in the message queue is sent to the distributed database system to ensure data integrity and transmission reliability.

[0089] After receiving data from the central monitor, the distributed database system stores the data in the distributed database. The system dynamically allocates data to different storage nodes based on the distributed database load and resource usage to ensure load balancing.

[0090] The server can obtain the performance monitoring data corresponding to each computing node within a specified period of time from the distributed database system. The format can be expressed as follows:

[0091] {

[0092] Mission Name: <Name-1>,

[0093] Resource usage:

[0094] Node 1: [<data 1-1>,...,<data 1-N>],

[0095] …

[0096] Node n: [<data n-1>,...,<data nN>].

[0097] }

[0098] }

[0099] Where n is the number of computing nodes occupied when the target deep learning task is running, N is the number of cycles of resource usage data collected during the task running, that is, the number of times the monitoring agent deployed on each node sends data to the central controller during the task running, <data ij> is the resource usage data of the monitoring agent deployed on the i-th node at the j-th data collection time, and its format can be expressed as: <timestamp, resource 1 usage, ..., resource m usage>.

[0100] The timestamp is the timestamp of the data collection time, and m is the number of resource types collected.

[0101] S103: Processing the performance monitoring data corresponding to each computing node to align the performance monitoring data corresponding to different nodes on a time scale, and obtaining target monitoring data corresponding to each computing node.

[0102] The server can process the performance monitoring data corresponding to each computing node to align the performance monitoring data corresponding to different nodes on a time scale, and obtain target monitoring data corresponding to each computing node.

[0103] Specifically, for each computing node, if the data collection moments in the performance monitoring data corresponding to the computing node are different from the data collection moments in the performance monitoring data corresponding to other computing nodes, the server can use the data collection moments in the performance monitoring data corresponding to the other computing nodes as the target moments, and then determine the data collected by the computing node at each target moment based on the performance monitoring data corresponding to the computing node, as the target monitoring data corresponding to the computing node.

[0104] For example, if the data collection time corresponding to the performance monitoring data of other computing nodes is "1, 3, 5, 7", and the data collection time corresponding to the performance monitoring data of computing node x is "2, 4, 6, 8", the server can use "1, 3, 5, 7" as the target time, and then based on the data change curve corresponding to the performance monitoring data of computing node x, determine the performance monitoring data collected by computing node x at "1, 3, 5, 7" as the performance monitoring data corresponding to computing node x, so that the performance monitoring data corresponding to different computing nodes are aligned on the time scale.

[0105] In addition to aligning timestamps, the server can also perform missing value processing and outlier processing on the performance monitoring data corresponding to each computing node.

[0106] Among them, missing value processing is to process missing data caused by collection, transmission, storage, etc. In the process of processing missing values, for the performance monitoring data corresponding to each computing node, the server can determine the missing value in the performance monitoring data, and use the data collection time corresponding to the missing value as the missing time, and then use the data before and after the timestamps to interpolate to obtain the estimated value of the missing value, so as to supplement the missing value through the estimated value.

[0107] In addition, the server may also identify abnormal values ​​in the performance monitoring data whose data values ​​exceed a preset threshold value and process the abnormal values.

[0108] In this specification, the server can set the abnormal value to the maximum performance provided by the abnormal resource, and the server can reset the value of GPU utilization exceeding 100% to 100%. Of course, the server can also set the above abnormal value to the median or mean value of the performance data.

[0109] S104: For the target monitoring data corresponding to each computing node, extract the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain.

[0110] In the process of extracting time domain features, since deep learning training or inference tasks occupy many types of resources and there is a correlation between the use of various types of resources, the server can extract the time domain features of resource usage dynamic data from two aspects.

[0111] First: For each computing node, the server can determine the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node, and for each computing resource, the server can determine the independent time domain characteristics corresponding to the computing resource based on the performance monitoring data corresponding to the computing resource.

[0112] In this specification, for each computing resource in the computing node, the performance monitoring data corresponding to the computing resource is intercepted based on a sliding window of a preset time length, and based on the statistics corresponding to each segment of intercepted data, a feature used to characterize the continuous change trend of the performance monitoring data over time is determined as the first time domain feature. For ease of understanding, this specification provides a schematic diagram of the method for extracting time domain features, such as Figure 2 shown.

[0113] Figure 2 This is a schematic diagram of a time domain feature extraction method provided in this specification.

[0114] Specifically, the server can use a sliding window method to intercept a continuous section of data from the dynamic data starting from the initial timestamp, perform operations such as calculating the mean, median, and variance, and extract the first time domain feature of the continuous change trend of the dynamic data;

[0115] In addition, the server may identify each data mutation point in the performance monitoring data corresponding to the computing resource, and determine a second time domain feature based on the time interval between adjacent mutation points;

[0116] In addition, the server can identify the long-term and short-term dependencies in the performance monitoring data corresponding to the computing resource through a preset long-short-term memory network, and determine the third time domain feature based on the long-short-term dependencies. The long-short-term dependencies are used to represent the relationship between data that are far apart in time in the performance data sequence.

[0117] The server may use one or more of the first time domain feature, the second time domain feature, and the third time domain feature as independent time domain features of its corresponding computing resources.

[0118] Second: For each computing node, the server can extract the correlation information between the independent time domain features corresponding to each computing resource in the computing node (such as Pearson correlation coefficient, mutual information, dynamic time warping, Granger causality test, etc.), and use the extracted correlation information as the joint time domain feature corresponding to the computing node.

[0119] The server may then determine the time domain characteristics of the computing node in the time domain based on the joint time domain characteristics corresponding to the computing node and the independent time domain characteristics corresponding to each computing resource on the computing node.

[0120] Similarly, the server can also extract frequency domain features of resource usage dynamic data from two aspects.

[0121] First, for each computing node, the server can determine the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node. For each computing resource in the computing node, the server can determine the independent frequency domain characteristics corresponding to the computing resource based on the performance monitoring data corresponding to the computing resource.

[0122] Specifically, for each computing resource in the computing node, the server may perform Fourier transform on the performance monitoring data corresponding to the computing resource to obtain a frequency domain representation of the performance monitoring data corresponding to the computing resource, and use the frequency domain representation coefficient as the first frequency domain feature;

[0123] In addition, the server may determine the spectrum bandwidth corresponding to the above frequency domain representation (i.e., the frequency range occupied by the main part of the spectrum), and use the spectrum bandwidth as the second frequency domain feature;

[0124] In addition, the server may determine the spectrum entropy (ie, a measure of uniformity and complexity of spectrum distribution) corresponding to the performance monitoring data corresponding to the computing resource, and use the spectrum entropy as the third frequency domain feature.

[0125] The server may then use one or more of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature as independent frequency domain features corresponding to the computing resource.

[0126] Second: The server can extract the correlation information between the independent frequency domain features corresponding to each computing resource in the computing node (such as cross-power spectrum density, phase spectrum correlation, Pearson correlation coefficient, etc.) as the joint frequency domain features corresponding to the computing node.

[0127] Furthermore, in order to remove the redundant information that may exist between the features and reduce the feature dimension, it is necessary to identify the key features. The server can screen the above independent frequency domain features and joint frequency domain features to determine the key features and use the key features as independent frequency domain features, wherein the key features are used to characterize the frequency domain features containing the most information.

[0128] For example, the server can select features whose variance exceeds a certain threshold. This is because features with too small a variance cannot effectively distinguish different time series and contain less useful information.

[0129] For another example, the server may perform principal component analysis on each frequency domain feature to extract the most informative features, thereby reducing the number of features.

[0130] S104: Inputting the time domain features and the frequency domain features into a pre-trained duration prediction model to determine the running time required to complete the target deep learning task through the duration prediction model.

[0131] After determining the time domain features and frequency domain features corresponding to each computing node, the server can input the time domain features and frequency domain features corresponding to each computing node into a pre-trained duration prediction model, thereby determining the running time required to complete the target deep learning task through the duration prediction model.

[0132] In this specification, the duration prediction model may be provided with a feature fusion network and a prediction network. For ease of understanding, this specification provides a schematic diagram of the model structure of the duration prediction model, such as Figure 3 shown.

[0133] Figure 3 This is a schematic diagram of the model structure of a duration prediction model provided in this specification.

[0134] Among them, the feature fusion network is used to fuse the time domain features and the frequency domain features to obtain an output fusion feature. In practical applications, the models used by the feature fusion network include but are not limited to the splicing model and the cross attention model;

[0135] The prediction network takes the fused features as input, determines and outputs the runtime required to complete the target deep learning task. In practical applications, the model structures used by the prediction network include but are not limited to support vector machine models, neural network regression models, ridge regression models, and lasso regression models.

[0136] Furthermore, before using the above-mentioned duration prediction model, it is usually necessary to train the model. Only after the duration prediction model meets the training objectives (such as reaching a preset number of training times or converging to a preset range) can it be deployed for actual prediction of the running time of deep learning tasks.

[0137] Specifically, the server can obtain the performance monitoring data sequence corresponding to the computing resources on each computing node in the intelligent computing cluster when the intelligent computing cluster executes the target deep learning task based on the historical task allocation strategy, as the historical performance monitoring data corresponding to each computing node, and obtain the actual running time of the intelligent computing cluster to complete the target deep learning task based on the historical task allocation strategy, and use the actual running time as a label to construct a training sample based on the historical performance monitoring data corresponding to each computing node. The data set of the training sample can be expressed as:

[0138]

[0139] Among them, N is the number of sample points in the data set, t i , f i are the time domain features and frequency domain features extracted from the dynamic monitoring data of resources occupied by the deep learning task during the i-th training sample, r i is the actual running time to complete the deep learning task in the i-th training sample.

[0140] The server can process the above historical performance monitoring data to obtain the historical target monitoring data corresponding to each computing node, and then extract the historical time domain features of the historical target monitoring data in the time domain and the historical frequency domain features of the target monitoring data in the frequency domain for the historical target monitoring data corresponding to each computing node (please see S103 to S104 for specific processing steps and feature extraction steps).

[0141] The server can then input the historical time domain features and the historical frequency domain features into the duration prediction model to be trained, so as to determine the predicted running time required to complete the deep learning task under the historical task allocation strategy through the duration prediction model.

[0142] The server can determine the loss value corresponding to the duration prediction model based on the predicted deviation between the predicted running time and the actual label corresponding to the training sample (i.e., the actual running time to complete the deep learning task), and train the duration prediction model based on the loss value.

[0143] Specifically, the server may determine a weight function and a basic loss value corresponding to the duration prediction model based on the predicted deviation between the predicted running time and the actual label, and then determine a loss value corresponding to the duration prediction model based on the weight function and the basic loss value. The loss value may be expressed as:

[0144]

[0145] in, is the weight function, τ is the model parameter (τ>0), is the basic loss value, and The relationship between Figure 4 shown.

[0146] Figure 4 This is a schematic diagram of a change curve of a weight function provided in this specification.

[0147] Depend on Figure 4 It can be seen that when the prediction error of the i-th sample is less than the parameter τ, a smaller weight is assigned to the sample, and when the prediction error exceeds the parameter τ, the weight function value increases as the prediction error increases.

[0148] It can be seen that the prediction deviation between the predicted duration and the actual duration There is a positive correlation between the prediction deviation and the weight function value. When the prediction deviation is greater than the model parameter value of the duration prediction model, there is a positive correlation between the prediction deviation and the weight function value. When the prediction deviation is less than the model parameter value of the duration prediction model, the weight function value is a preset value, and the preset value is less than the minimum weight function value when the prediction deviation is greater than the model parameter value of the duration prediction model.

[0149] S105: Adjust the current task allocation strategy according to the running time, so as to execute the target deep learning task according to the adjusted task allocation strategy.

[0150] After predicting the runtime required for the target deep learning model to complete the target deep learning task according to the current task allocation strategy based on the performance monitoring data corresponding to each computing node of the specified duration, the server can adjust the current task allocation strategy based on the runtime. For example, when the predicted runtime is greater than the preset threshold, it can be considered that the current task allocation strategy is ineffective. The server can adjust the task execution order of each computing node for the target deep learning task, as well as the allocation strategy of computing resources in each computing node, so as to reduce the overall runtime of the deep learning task and further improve the utilization of computing resources and task execution efficiency.

[0151] As can be seen from the above method, this solution can overcome the limitations of existing deep learning task runtime prediction methods in terms of accuracy and ease of use, achieving more efficient and accurate predictions. By comprehensively considering key factors such as data collection, storage, cleaning, and feature selection, it avoids intrusive analysis, protects user privacy and data security, and reduces the high requirements for user expertise and learning costs.

[0152] By precisely capturing the cyclical nature of deep learning tasks in their use of intelligent computing cluster resources, the accuracy of runtime predictions is improved, effectively addressing the collection and storage of large-scale operational status data, the identification and processing of abnormal data, and the screening of key features. This not only optimizes resource scheduling and reduces computing costs, but also improves production efficiency, which is particularly important for resource-intensive deep learning applications.

[0153] The above is one or more implementations of the deep learning task execution method based on runtime prediction of this specification. Based on the same idea, this specification also provides a corresponding deep learning task execution device based on runtime prediction, such as Figure 5 shown.

[0154] Figure 5 A schematic diagram of a deep learning task execution device based on runtime prediction provided in this specification includes:

[0155] Receiving module 501, configured to receive an execution request for a target deep learning task;

[0156] An acquisition module 502 is configured to execute the target deep learning task according to the execution request through each computing node in the intelligent computing cluster according to the current task allocation policy, and after monitoring the execution of the target deep learning task for a specified duration, obtain a performance monitoring data sequence of the computing resources on each computing node within the specified duration as the performance monitoring data corresponding to each computing node;

[0157] Processing module 503, configured to process the performance monitoring data corresponding to each computing node so as to align the performance monitoring data corresponding to different nodes on a time scale, thereby obtaining target monitoring data corresponding to each computing node;

[0158] An extraction module 504 is configured to extract, for each computing node, target monitoring data corresponding to the target monitoring data, a time domain feature of the target monitoring data in the time domain and a frequency domain feature of the target monitoring data in the frequency domain;

[0159] A prediction module 505 is configured to input the time domain features and the frequency domain features into a pre-trained duration prediction model to determine the runtime required to complete the target deep learning task through the duration prediction model;

[0160] The execution module 506 is configured to adjust the current task allocation strategy according to the runtime, so as to execute the target deep learning task according to the adjusted task allocation strategy.

[0161] Optionally, the processing module 503 is specifically used to, for each computing node, if the data collection moments in the performance monitoring data corresponding to the computing node are different from the data collection moments in the performance monitoring data corresponding to other computing nodes, then use the data collection moments in the performance monitoring data corresponding to the other computing nodes as the target moments; based on the performance monitoring data corresponding to the computing node, determine the data collected by the computing node at the target moments as the target monitoring data corresponding to the computing node.

[0162] Optionally, the processing module 503 is specifically used to determine the missing values ​​in the performance monitoring data corresponding to each computing node, and use the data collection time corresponding to the missing value as the missing time; supplement the missing value according to the data value corresponding to the data collection time adjacent to the missing time; and / or identify abnormal values ​​in the performance monitoring data whose data values ​​exceed a preset threshold, and process the abnormal values.

[0163] Optionally, the extraction module 504 is specifically used to, for each computing node, determine the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node; for each computing resource in the computing node, determine at least one time domain feature corresponding to the computing resource based on the performance monitoring data corresponding to the computing resource, as an independent time domain feature; extract the correlation information between the independent time domain features corresponding to each computing resource in the computing node as the joint time domain feature corresponding to the computing node; determine the time domain feature of the target monitoring data corresponding to the computing node based on the independent time domain features corresponding to each computing resource in the computing node and the joint time domain feature corresponding to the computing node.

[0164] Optionally, the extraction module 504 is specifically used to, for each computing resource in the computing node, intercept the performance monitoring data corresponding to the computing resource based on a sliding window of a preset time length, and determine the characteristics used to characterize the continuous change trend of the performance monitoring data in time as the first time domain feature based on the statistics corresponding to each segment of intercepted data; identify each data mutation point in the performance monitoring data corresponding to the computing resource, and determine the second time domain feature based on the time interval between each adjacent mutation point; determine the long-term and short-term dependencies in the performance monitoring data corresponding to the computing resource, and determine the third time domain feature based on the long-term and short-term dependencies; determine the independent time domain feature corresponding to the computing resource based on at least one of the first time domain feature, the second time domain feature and the third time domain feature.

[0165] Optionally, the extraction module 504 is specifically used to, for each computing node, determine the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node; for each computing resource in the computing node, determine at least one frequency domain feature corresponding to the computing resource based on the performance monitoring data corresponding to the computing resource, as an independent frequency domain feature; extract the correlation information between the independent frequency domain features corresponding to each computing resource in the computing node as the joint frequency domain feature corresponding to the computing node; determine the frequency domain features of the target monitoring data corresponding to the computing node based on the independent frequency domain features corresponding to each computing resource in the computing node and the joint frequency domain features corresponding to the computing node.

[0166] Optionally, the extraction module 504 is specifically used to perform Fourier transform on the performance monitoring data corresponding to each computing resource in the computing node, obtain the frequency domain representation of the performance monitoring data corresponding to the computing resource, and use the frequency domain representation coefficient as the first frequency domain feature; determine the spectrum bandwidth corresponding to the frequency domain representation, and use the spectrum bandwidth as the second frequency domain feature; determine the spectrum entropy corresponding to the performance monitoring data corresponding to the computing resource, and use the spectrum entropy as the third frequency domain feature; determine the independent frequency domain feature corresponding to the computing resource based on at least one of the first frequency domain feature, the second frequency domain feature and the third frequency domain feature.

[0167] Optionally, the extraction module 504 is specifically used to screen the independent frequency domain features and the joint frequency domain features to determine the key features therein as the frequency domain features corresponding to the target monitoring data of the computing node, wherein the key features are used to characterize the frequency domain features containing the most information.

[0168] Optionally, the device also includes: a training module 507, which is used to obtain a performance monitoring data sequence corresponding to the computing resources on each computing node in the intelligent computing cluster when the intelligent computing cluster executes the target deep learning task based on the historical task allocation strategy, as the historical performance monitoring data corresponding to each computing node, and construct a training sample based on the historical performance monitoring data; process the historical performance monitoring data corresponding to each computing node so that the historical performance monitoring data corresponding to different nodes are aligned on the time scale to obtain the historical target monitoring data corresponding to each computing node; for the historical target monitoring data corresponding to each computing node, extract the historical time domain features of the historical target monitoring data in the time domain and the historical frequency domain features of the target monitoring data in the frequency domain; construct a training sample based on the time domain features and frequency domain features corresponding to each computing node; input the training sample into the duration prediction model to be trained, so as to determine the predicted running time required to complete the deep learning task under the historical task allocation strategy through the duration prediction model; determine the loss value corresponding to the duration prediction model based on the prediction deviation between the predicted running time and the actual label corresponding to the training sample, and train the duration prediction model based on the loss value.

[0169] Optionally, the training module 507 is specifically used to determine the weight function and basic loss value corresponding to the duration prediction model based on the predicted deviation between the predicted running time and the actual label; determine the loss value corresponding to the duration prediction model based on the weight function value and the basic loss value, wherein the weight function value is positively correlated with the loss value corresponding to the duration prediction model, and when the predicted deviation is greater than the model parameter value of the duration prediction model, the predicted deviation and the weight function value are positively correlated, and when the predicted deviation is less than the model parameter value of the duration prediction model, the weight function value is a preset value, and the preset value is less than the minimum weight function value when the predicted deviation is greater than the model parameter value of the duration prediction model.

[0170] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides a method for executing deep learning tasks based on runtime prediction.

[0171] This manual also provides Figure 6 The one shown corresponds to Figure 1 Schematic diagram of the electronic equipment. Figure 6As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The method for executing a deep learning task based on runtime prediction is described. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0172] Improvements to a technology can be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always program the improved process flow into the hardware circuit to obtain the corresponding hardware circuit structure. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0173] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0174] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0175] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0176] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0180] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0181] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0182] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0183] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0184] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0186] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0187] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for executing deep learning tasks based on runtime prediction, characterized in that: include: Receive an execution request for a target deep learning task; According to the execution request, each computing node in the intelligent computing cluster executes the target deep learning task according to the current task allocation policy, and after monitoring the execution of the target deep learning task for a specified duration, obtains a performance monitoring data sequence of the computing resources on each computing node within the specified duration as the performance monitoring data corresponding to each computing node; Processing the performance monitoring data corresponding to each computing node so that the performance monitoring data corresponding to different nodes are aligned on a time scale, thereby obtaining target monitoring data corresponding to each computing node; For the target monitoring data corresponding to each computing node, extract the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain; Inputting the time domain features and the frequency domain features into a pre-trained duration prediction model to determine the runtime required to complete the target deep learning task through the duration prediction model; According to the running time, the current task allocation strategy is adjusted to execute the target deep learning task according to the adjusted task allocation strategy.

2. The method according to claim 1, wherein Processing the performance monitoring data corresponding to each computing node to align the performance monitoring data corresponding to different nodes on a time scale, and obtaining target monitoring data corresponding to each computing node, specifically including: For each computing node, if the data collection moments in the performance monitoring data corresponding to the computing node are different from the data collection moments in the performance monitoring data corresponding to other computing nodes, the data collection moments in the performance monitoring data corresponding to the other computing nodes are used as the target moments; Based on the performance monitoring data corresponding to the computing node, the data collected by the computing node at each target time is determined as the target monitoring data corresponding to the computing node.

3. The method according to claim 2, wherein Processing the performance monitoring data corresponding to each computing node includes: For each computing node's corresponding performance monitoring data, determine the missing value in the performance monitoring data, and use the data collection time corresponding to the missing value as the missing time; Supplementing the missing value based on the data value corresponding to the data collection time adjacent to the missing time; and / or Identify abnormal values ​​in the performance monitoring data whose data values ​​exceed a preset threshold and process the abnormal values.

4. The method according to claim 1, wherein Extract the time domain features of the target monitoring data in the time domain, specifically including: For each computing node, determining the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node; For each computing resource in the computing node, determining at least one time domain feature corresponding to the computing resource according to performance monitoring data corresponding to the computing resource as an independent time domain feature; Extracting correlation information between independent time domain features corresponding to each computing resource in the computing node as a joint time domain feature corresponding to the computing node; The time domain features of the target monitoring data corresponding to the computing node are determined according to the independent time domain features corresponding to each computing resource in the computing node and the joint time domain features corresponding to the computing node.

5. The method according to claim 4, wherein For each computing resource in the computing node, at least one time domain feature corresponding to the computing resource is determined based on the performance monitoring data corresponding to the computing resource as an independent time domain feature, specifically including: For each computing resource in the computing node, performance monitoring data corresponding to the computing resource is intercepted based on a sliding window of a preset time length, and a feature used to characterize a continuous temporal change trend of the performance monitoring data is determined based on a statistic corresponding to each segment of the intercepted data as a first time-domain feature; Identifying each data mutation point in the performance monitoring data corresponding to the computing resource, and determining a second time domain feature based on the time interval between adjacent mutation points; Determining a long-term and short-term dependency relationship in the performance monitoring data corresponding to the computing resource, and determining a third time domain feature based on the long-term and short-term dependency relationship; An independent time domain feature corresponding to the computing resource is determined according to at least one of the first time domain feature, the second time domain feature, and the third time domain feature.

6. The method according to claim 1, wherein For the target monitoring data corresponding to each computing node, extract the frequency domain features of the target monitoring data in the frequency domain, specifically including: For each computing node, determining the performance monitoring data corresponding to each computing resource in the target monitoring data of the computing node; For each computing resource in the computing node, determining at least one frequency domain feature corresponding to the computing resource according to performance monitoring data corresponding to the computing resource as an independent frequency domain feature; Extracting correlation information between independent frequency domain features corresponding to each computing resource in the computing node as a joint frequency domain feature corresponding to the computing node; The frequency domain features of the target monitoring data corresponding to the computing node are determined according to the independent frequency domain features corresponding to each computing resource in the computing node and the joint frequency domain features corresponding to the computing node.

7. The method according to claim 6, wherein For each computing resource in the computing node, at least one frequency domain feature corresponding to the computing resource is determined based on the performance monitoring data corresponding to the computing resource as an independent frequency domain feature, specifically including: For each computing resource in the computing node, perform Fourier transform on the performance monitoring data corresponding to the computing resource to obtain a frequency domain representation of the performance monitoring data corresponding to the computing resource, and use the frequency domain representation coefficient as a first frequency domain feature; determining a spectrum bandwidth corresponding to the frequency domain representation, and using the spectrum bandwidth as a second frequency domain feature; Determine a spectrum entropy corresponding to the performance monitoring data corresponding to the computing resource, and use the spectrum entropy as a third frequency domain feature; An independent frequency domain feature corresponding to the computing resource is determined according to at least one of the first frequency domain feature, the second frequency domain feature, and the third frequency domain feature.

8. The method according to claim 7, wherein Determining the frequency domain features of the target monitoring data corresponding to the computing node according to the independent frequency domain features corresponding to each computing resource in the computing node and the joint frequency domain features corresponding to the computing node specifically includes: The independent frequency domain features and the joint frequency domain features are screened to determine key features therein as the frequency domain features corresponding to the target monitoring data of the computing node, wherein the key features are used to characterize the frequency domain features containing the most information.

9. The method according to claim 1, wherein Training the duration prediction model specifically includes: Obtain a performance monitoring data sequence corresponding to the computing resources on each computing node in the intelligent computing cluster when the intelligent computing cluster executes the target deep learning task based on the historical task allocation strategy, as the historical performance monitoring data corresponding to each computing node, and construct a training sample based on the historical performance monitoring data; Processing the historical performance monitoring data corresponding to each computing node to align the historical performance monitoring data corresponding to different nodes on a time scale, thereby obtaining historical target monitoring data corresponding to each computing node; For the historical target monitoring data corresponding to each computing node, extract the historical time domain features of the historical target monitoring data in the time domain and the historical frequency domain features of the target monitoring data in the frequency domain; Construct training samples based on the time domain features and frequency domain features corresponding to each computing node; Inputting the training sample into a duration prediction model to be trained, so as to determine, through the duration prediction model, a predicted running time required to complete the deep learning task under the historical task allocation strategy; According to the predicted deviation between the predicted running time and the actual label corresponding to the training sample, a loss value corresponding to the duration prediction model is determined, and the duration prediction model is trained according to the loss value.

10. The method according to claim 9, wherein Determining a loss value corresponding to the duration prediction model according to a predicted deviation between the predicted running duration and an actual label corresponding to the training sample specifically includes: Determining a weight function value and a basic loss value corresponding to the duration prediction model according to a predicted deviation between the predicted running time and the actual label; According to the weight function value and the basic loss value, the loss value corresponding to the duration prediction model is determined, wherein the weight function value and the loss value corresponding to the duration prediction model are positively correlated, and when the prediction deviation is greater than the model parameter value of the duration prediction model, the prediction deviation and the weight function value are positively correlated, and when the prediction deviation is less than the model parameter value of the duration prediction model, the weight function value is a preset value, and the preset value is less than the minimum weight function value when the prediction deviation is greater than the model parameter value of the duration prediction model.

11. A method for executing deep learning tasks based on runtime prediction, characterized in that: include: A receiving module, configured to receive an execution request for a target deep learning task; An acquisition module is configured to execute the target deep learning task according to the execution request through each computing node in the intelligent computing cluster according to the current task allocation policy, and after monitoring the execution of the target deep learning task for a specified duration, obtain a performance monitoring data sequence of the computing resources on each computing node within the specified duration as the performance monitoring data corresponding to each computing node; a processing module, configured to process the performance monitoring data corresponding to each computing node so as to align the performance monitoring data corresponding to different nodes on a time scale and obtain target monitoring data corresponding to each computing node; An extraction module is used to extract the time domain features of the target monitoring data in the time domain and the frequency domain features of the target monitoring data in the frequency domain for the target monitoring data corresponding to each computing node; A prediction module, configured to input the time domain features and the frequency domain features into a pre-trained duration prediction model, so as to determine the runtime required to complete the target deep learning task through the duration prediction model; An execution module is used to adjust the current task allocation strategy according to the runtime, so as to execute the target deep learning task according to the adjusted task allocation strategy.

12. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 10 is implemented.

Images