Dynamic scheduling method and system for computing tasks in supercomputing centers based on artificial intelligence

Abstract

The present invention provides a method and system for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence. First, a set of computing task request data currently to be processed in the supercomputing center is obtained, and a set of task features of each computing task is extracted. Then, a pre-trained task scheduling strategy decision model is called to analyze the task feature set, and a set of real-time scheduling priority parameters and resource allocation constraint conditions for the computing tasks are generated. Based on the real-time scheduling priority parameters and resource allocation constraint conditions for the computing tasks, a set of dynamic resource allocation instructions for computing nodes in the supercomputing center is generated, and the computing nodes are controlled to perform task scheduling operations. By updating the resource occupancy status of the computing nodes and the execution progress of the task queues in real time, intelligent and efficient scheduling of computing tasks in the supercomputing center is achieved, and resource utilization and task execution efficiency are significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for dynamically scheduling computing tasks in a supercomputing center based on artificial intelligence. Background Art

[0002] With the rapid development of science and technology, supercomputing centers are playing an increasingly important role in scientific research, engineering computing, big data analysis and other fields. Supercomputing centers usually contain a large number of computing nodes, which need to process multiple complex computing tasks simultaneously. However, existing supercomputing center task scheduling methods are often based on static task priorities or simple resource allocation strategies, which are difficult to adapt to dynamically changing computing task requirements and resource conditions. Specifically, existing technologies are unable to accurately capture the dynamic characteristics of computing tasks, such as task submission time, resource requirements, and dependencies between tasks, resulting in inefficient task scheduling, low resource utilization, and even problems such as long task waiting times or resource conflicts. Summary of the Invention

[0003] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence, the method comprising:

[0004] Obtaining a computing task request data set currently pending at the supercomputing center, wherein the computing task request data set includes a computing task identifier, a task submission timestamp, a task resource requirement description, and a description of inter-task dependencies;

[0005] Performing task dynamic feature extraction processing on the computing task request data set to generate a task feature set for each computing task;

[0006] Calling a pre-trained task scheduling strategy decision model to perform task scheduling strategy decision on the task feature set, and generating a real-time scheduling priority parameter and a resource allocation constraint condition set for the computing task;

[0007] Generate a dynamic resource allocation instruction set for a computing node in a supercomputing center based on the real-time scheduling priority parameters and the resource allocation constraint condition set;

[0008] According to the dynamic resource allocation instruction set, the supercomputing center computing nodes are controlled to perform task scheduling operations, and the resource occupancy status of the computing nodes and the task queue execution progress are updated in real time.

[0009] On the other hand, an embodiment of the present invention also provides an artificial intelligence-based dynamic scheduling system for computing tasks in a supercomputing center, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0010] Based on the above aspects, the embodiment of the present invention can accurately evaluate the priority and resource requirements of each computing task by comprehensively acquiring and analyzing the dynamic characteristics of the computing tasks, including the task submission timestamp, resource requirement description and the dependency relationship between tasks. Furthermore, by calling the pre-trained task scheduling strategy decision model, it can dynamically generate the real-time scheduling priority parameters and resource allocation constraint condition set of the computing task according to the current resource status and task feature set of the supercomputing center, which not only realizes the intelligent scheduling of computing tasks, but also significantly improves the resource utilization and task execution efficiency of the supercomputing center. At the same time, by updating the resource occupancy status of the computing nodes and the execution progress of the task queue in real time, it ensures that the supercomputing center can flexibly respond to various dynamically changing computing task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic diagram of the execution flow of the method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence provided by an embodiment of the present invention.

[0012] Figure 2 It is a schematic diagram of exemplary hardware and software components of the artificial intelligence-based supercomputing center computing task dynamic scheduling system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flow chart of a method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence provided by an embodiment of the present invention. The method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence is introduced in detail below.

[0014] Step S110: Obtain a computing task request data set currently to be processed by the supercomputing center, wherein the computing task request data set includes a computing task identifier, a task submission timestamp, a task resource requirement description, and a description of dependencies between tasks.

[0015] During the operation of the supercomputing center, data can be acquired through the functional module responsible for data acquisition. This functional module always remains active and continuously collects data related to computing tasks waiting to be processed in the supercomputing center.

[0016] Each computing task is assigned a unique identifier by following a set of encoding rules comprised of multiple factors. These encoding rules incorporate information such as the task's initiator category, the order in which tasks are submitted to the supercomputing center, and the specific attributes of the task itself. For example, the task's initiator category is identified by a specific letter combination, the order of submissions is numbered in ascending order from the time the supercomputing center first receives the task, and task-specific attributes are represented by different symbols depending on their nature. These elements are then combined in the order of "letter combination-sequential number-symbol" to form an identifier that uniquely identifies each computing task, ensuring that each task has its own unique identity within the supercomputing center's task management system.

[0017] The task submission timestamp accurately records the moment a task enters the supercomputing center. It uses a specific starting point in the supercomputing center's operation as a benchmark and records the elapsed time from that starting point to the task's entry into the supercomputing center. To ensure the accuracy and consistency of time records, time is recorded to the second level. In other words, the task submission timestamp is calculated by accumulating the time it takes for a task to enter the supercomputing center, starting from the supercomputing center's startup. This accumulated time is the task submission timestamp, providing an accurate time basis for subsequent time-based task analysis and scheduling.

[0018] The task resource requirement description details the specific requirements of the task for various types of resources during execution. In terms of processor usage, the percentage of processor core usage of the task at different execution stages can be recorded at set time intervals. Specifically, from the start of task execution, the percentage of processor core usage is detected and recorded at fixed intervals. As time goes by, the recorded data points form a data set that reflects the changes in processor usage over time. By organizing and analyzing this data set, we can clearly understand the changes in the task's demand for processor resources at different stages.

[0019] Memory requirements are described in a similar way, recording changes in task memory requirements at set intervals. Starting from task initiation, the task's memory requirements are detected and recorded at fixed intervals. This collection of data points demonstrates the dynamic changes in task memory requirements throughout the execution process.

[0020] The storage space requirement description focuses on recording changes in storage space usage during task execution. This includes not only the actual amount of storage space occupied by the task at different stages, but also the trend of this usage over time. For example, by recording the amount of data created, modified, or deleted by the task at different points in time, as well as the time when these operations occurred, a comprehensive picture of the task's storage space usage patterns and changes can be obtained.

[0021] The description of inter-task dependencies uses a specific data structure to clearly present the relationships between tasks. If the initiation of a task depends on the completion of specific steps or the entirety of another task, then this order and dependency logic will be reflected in an intuitive manner in the data structure. For example, a tree-like data structure is used, with tasks that depend on other tasks as child nodes and the dependent tasks as parent nodes. The dependencies between tasks are clarified through the connection relationships between the nodes. In this way, the dependency path and degree of dependency between each task and other tasks can be clearly seen in the data structure.

[0022] Step S120: performing task dynamic feature extraction processing on the computing task request data set to generate a task feature set for each computing task.

[0023] After obtaining the complete computing task request data set, the task dynamic feature extraction process is immediately started. The core purpose of this process is to extract key feature information that can comprehensively and accurately reflect the unique nature and requirements of each task from the large amount of information contained in the computing task request data set.

[0024] Step S121: parsing the task resource requirement description in the computing task request data set, and extracting the processor occupancy change curve, memory capacity requirement peak and storage space fluctuation characteristics during the processing of the computing task.

[0025] Regarding the task resource requirement description, when extracting the processor occupancy change curve, we first obtain data from a dataset that records the changes in processor occupancy over time. This computing task request dataset contains information about the percentage of processor cores occupied by the task at various time points during execution. Then, the data in the dataset is sampled at a fixed time interval as the sampling period. For example, the processor occupancy percentage data at a certain time point is selected at regular intervals, and these sampled data points are arranged in sequence. Next, a suitable data fitting method, such as polynomial fitting or spline curve fitting, is used to connect these discrete data points into a smooth curve. The curve obtained in this way is the processor occupancy change curve of the computing task during processing, which intuitively shows the dynamic changes in the percentage of processor resources occupied by the task at different times.

[0026] To extract peak memory requirements, we perform a comprehensive traversal of the dataset that records memory requirements over time. During this traversal, we compare the task's memory requirements at each point in time. Whenever we encounter a value greater than the previously recorded requirement, we update the currently recorded maximum value. After traversing the entire dataset, the maximum value obtained is the task's peak memory requirement, representing the task's maximum memory requirement throughout its entire execution.

[0027] To extract storage space fluctuation characteristics, we first calculate the change in storage space within adjacent fixed time intervals. Specifically, we obtain the storage space occupancy at two adjacent time points from a dataset that records storage space usage changes over time. Then, we subtract the storage space occupancy at the previous time point from the storage space occupancy at the later time point to obtain the change within the adjacent time intervals. This yields a series of data reflecting the storage space change. Next, we perform statistical analysis on this change data, such as calculating statistics such as its variance or standard deviation. These statistics can effectively measure the degree of storage space fluctuation within different time intervals, and thus, serve as storage space fluctuation characteristics, accurately reflecting the stability and fluctuation of storage space usage during task execution.

[0028] Step S122: Analyze the task node topology structure in the inter-task dependency description, and construct a dependency path length matrix and parallel execution feasibility parameters between task nodes.

[0029] Step S1221: traverse each task node in the task node topology structure, mark task node pairs with direct dependency relationships, and generate a dependency edge set.

[0030] Each task node in the task node topology is carefully checked one by one to determine whether there is a direct dependency between it and all other task nodes. The specific inspection process is as follows: for the task node currently being checked, it is compared with each other task node in the topology. If it is found that the startup of a task node must rely on another task node to complete a specific operation or complete it completely, then it is determined that there is a direct dependency between the two task nodes and they are marked as a task node pair. All task node pairs with direct dependencies determined in this way are collected to form a dependency edge set. In this dependency edge set, each element represents a pair of directly dependent task nodes. For example, (task X, task Y) clearly indicates that task X is directly dependent on task Y.

[0031] Step S1222: construct a task node topology graph based on the dependency edge set, and calculate the shortest dependency path length and the maximum number of parallel branches from each task node to all its downstream nodes.

[0032] The task node topology graph is constructed based on the generated dependency edge set. During this construction process, each task node is treated as a vertex in the graph, and the dependency edges are the edges connecting these vertices. This way, using the elements in the dependency edge set, the task nodes can be connected according to their dependencies, forming a topology graph that intuitively displays the dependencies between tasks.

[0033] In the constructed task node topology graph, for each task node, the shortest dependent path length to all its downstream nodes must be calculated. The specific calculation method is to use a search algorithm suitable for graph structure in related technologies, such as a breadth-first search algorithm. Starting from the current task node, search the nodes in the graph layer by layer in a breadth-first manner, and record the path length from the starting task node to each downstream node. During the search process, whenever a path length to a downstream node is found to be shorter than the previously recorded path length of the node, the shortest path length record of the downstream node is updated. When the search is completed for all downstream nodes, the shortest dependent path length from the task node to all its downstream nodes is obtained.

[0034] At the same time, for each task node, the maximum number of parallel branches is calculated. Specifically, starting from the task node, traverse along the edges in the topology graph to find all task branches that can be started simultaneously without relying on the completion of other task nodes at the same level. These parallel-starting task branches are counted, and the resulting number is the maximum number of parallel branches for the task node. This maximum number of parallel branches reflects the upper limit of the number of tasks that can be executed simultaneously based on the current task node.

[0035] Step S1223: generating a dependency path length matrix including dependency depth weights between task nodes based on the shortest dependency path length, wherein the dependency depth weights are positively correlated with the shortest dependency path length.

[0036] The calculated shortest dependency path length from each task node to all its downstream nodes is used to generate a dependency path length matrix containing the dependency depth weights between task nodes.

[0037] First, determine the positive correlation between the dependency depth weight and the shortest dependency path length. For example, a simple linear positive correlation can be defined, where the dependency depth weight equals the shortest dependency path length multiplied by a pre-set positive coefficient. The choice of this positive coefficient should be determined based on the actual task characteristics and scheduling requirements of the supercomputing center. Its function is to adjust the magnitude of the dependency depth weight change with the shortest dependency path length.

[0038] Then, for each element in the dependency path length matrix, that is, each pair of task nodes, the corresponding dependency depth weight is calculated based on the shortest dependency path length between them, according to the positive correlation relationship determined above. The calculated dependency depth weight value is filled in the corresponding position in the matrix, thus constructing a dependency path length matrix that includes the dependency depth weights between task nodes. This dependency path length matrix comprehensively reflects the depth and strength of the dependency relationship between task nodes.

[0039] Step S1224: Count the number of task node pairs that have no direct dependency in the task node topology graph, and calculate the resource competition-aware parallel execution conflict probability parameter between task nodes based on the maximum number of parallel branches and the processor occupancy overlap parameter in the task resource requirement description.

[0040] In the constructed task node topology, carefully count the number of task node pairs without direct dependencies. The specific counting method is to compare each task node in the task node topology with all other task nodes. If there is no direct dependency edge connecting two task nodes, then these two task nodes are considered a pair without direct dependencies and are counted. By traversing and comparing all task nodes, the total number of task node pairs without direct dependencies is finally determined.

[0041] The processor occupancy overlap parameter is obtained from the task resource requirement description. The process of obtaining this processor occupancy overlap parameter requires a detailed analysis of the processor occupancy of different tasks within the same time interval. Specifically, a common time range is first determined, and then within this time range, the data of the processor occupancy of each task over time is sampled. The processor occupancy of different tasks at the same sampling time point is compared. If the processor occupancy of multiple tasks is high at a certain time point, it indicates that these tasks have overlapping demands for processor resources at that time. By analyzing all sampling points in the entire time range and taking into account factors such as the degree of overlap and duration, a processor occupancy overlap parameter that can reflect the degree of overlap in processor usage between tasks is calculated.

[0042] Combining the previously statistically obtained number of task node pairs without direct dependencies, the maximum number of parallel branches for each task node, and the processor occupancy overlap parameter, a specialized calculation method is designed to calculate the resource contention-aware parallel execution conflict probability parameter between task nodes. This calculation method can be to construct a mathematical model that includes these three factors, such as a multivariate function. In this multivariate function, the number of task node pairs without direct dependencies, the maximum number of parallel branches, and the processor occupancy overlap parameter serve as the function's independent variables. By performing mathematical operations on these independent variables, such as weighted summation and multiplication, a result with a value between 0 and 1 is obtained, which is the resource contention-aware parallel execution conflict probability parameter between task nodes. This parallel execution conflict probability parameter can comprehensively reflect the probability of conflict due to resource contention when tasks are executed in parallel.

[0043] Step S1225: Generate a parallel execution feasibility parameter reflecting the intensity of task parallel execution resource competition based on the parallel execution conflict probability parameter and the processor occupancy rate change curve in the task resource requirement description.

[0044] The parallel execution feasibility parameter reflecting the intensity of task parallel execution resource competition is generated by using the calculated parallel execution conflict probability parameter and the processor occupancy change curve in the task resource requirement description.

[0045] First, further analyze the processor utilization curve. In addition to extracting the curve's characteristics, we also need to consider factors such as the slope and the frequency of peaks. For example, we can calculate the average slope of the processor utilization curve over different time periods. The slope reflects the speed of processor utilization changes. We can also count the number of peaks and their distribution over the entire task execution time.

[0046] Then, in combination with the parallel execution conflict probability parameter, the parallel execution conflict probability parameter can be organically combined with the various features obtained from the analysis of the processor occupancy change curve. For example, a weighted comprehensive model can be constructed to assign different weights to factors such as the parallel execution conflict probability parameter, the average slope of the processor occupancy change curve, and the peak occurrence frequency based on the actual situation of the supercomputing center and the characteristics of the task. Then, these factors are weighted summed according to the weights or other appropriate mathematical operations are performed to obtain a parallel execution feasibility parameter. This parallel execution feasibility parameter comprehensively reflects the degree of impact of resource competition during the parallel execution of the task, thereby providing a key reference indicator for determining whether the task is suitable for parallel execution.

[0047] Step S123: Based on the task submission timestamp and the historical task execution records of the supercomputing center, generate resource competition intensity prediction parameters for the computing task within a preset time window.

[0048] After completing the feature extraction of task resource requirements and inter-task dependencies, we then use the task submission timestamps and the historical task execution records accumulated by the supercomputing center over a long period of time to predict the resource competition intensity of the computing task within the preset time window and generate corresponding prediction parameters.

[0049] First, a subset of historical task execution data matching the current task type is selected from the supercomputing center's extensive database of historical task execution records. Task type matching can be determined based on a variety of factors, such as the task's application domain, computational complexity, and resource demand patterns. Through a comprehensive analysis of these factors, task data with similar characteristics to the current task is extracted from the historical task execution records to form a subset of historical task execution data specific to the current task type. This subset includes key information such as the historical task's submission timestamp, resource utilization curve, and task execution duration.

[0050] Then, based on the submission timestamp of the current task, the corresponding historical time period is matched in the subset of historical task execution data. The specific matching method is to use the current task submission timestamp as a benchmark to find task data within the same time interval in the historical task execution records. For example, if the current task submission time is in the middle of a month, then the task data within the middle of the same month is searched in the historical task execution records. From these matched historical task data, the average processor occupancy and memory usage peak fluctuation parameters within the same time window are extracted. The average processor occupancy is obtained by calculating the average processor occupancy of all historical tasks in the time window, which reflects the average usage of processor resources in this specific time window. The memory usage peak fluctuation parameter is determined by analyzing the changes in the memory usage peak of historical tasks in the time window, such as calculating statistics such as the standard deviation or range of the memory usage peak to measure the degree of fluctuation of the memory usage peak.

[0051] Next, an in-depth analysis is conducted on the task execution records with similar dependency complexity in the historical task execution data subset. Dependency complexity can be measured by information such as the task node topology structure and the dependency path length matrix constructed previously. For tasks with similar dependency complexity, the correlation between their task start delay time and the dependency path length matrix is ​​statistically analyzed. The specific statistical method is that for each such historical task, its task start delay time is recorded, that is, the time interval from task submission to the actual start of execution, and the relevant information in the dependency path length matrix corresponding to the task is recorded at the same time. Then, through data analysis methods, such as correlation analysis or regression analysis, the functional relationship between the task start delay time and the dependency path length matrix is ​​found to obtain a correlation function. This correlation function reflects the influence of task dependency complexity on task start delay.

[0052] Based on the previously extracted average processor occupancy rate, memory usage peak fluctuation parameters, and the obtained correlation function, a resource contention intensity prediction model is constructed. This resource contention intensity prediction model can be a model based on a machine learning algorithm, such as a neural network model or a decision tree model, or a simple mathematical formula model. The submission timestamp of the current task and the dependency path length matrix are input into the resource contention intensity prediction model as input parameters. The resource contention intensity prediction model calculates and analyzes the input parameters and outputs an initial resource contention intensity prediction value. This initial resource contention intensity prediction value preliminarily reflects the resource contention intensity that the current task may face within the preset time window.

[0053] Finally, the initial resource contention intensity prediction value is dynamically revised based on the real-time resource utilization status of the supercomputing center's compute nodes. This real-time resource utilization status can be obtained through a real-time monitoring system, including information such as the current processor utilization rate, free memory, and remaining storage space of each compute node. This real-time resource utilization information is combined with the initial prediction value, and an adjustment algorithm is used to refine the initial prediction value. For example, if the processor utilization rate of the current compute nodes is generally high, indicating that overall resource competition is intense, the initial prediction value is increased accordingly; conversely, if the compute nodes' resources are relatively idle, the initial prediction value is appropriately reduced. After this dynamic revision, the resource contention intensity prediction parameter is obtained after eliminating node load deviations. This resource contention intensity prediction parameter more accurately reflects the resource contention intensity of the current task under the actual operating environment within the preset time window.

[0054] Step S130: calling a pre-trained task scheduling policy decision model to perform task scheduling policy decision on the task feature set, and generating a real-time scheduling priority parameter and resource allocation constraint condition set for the computing task.

[0055] After completing the task dynamic feature extraction and generating the task feature set, the pre-trained task scheduling strategy decision model is called next to formulate real-time scheduling priority parameters and resource allocation constraint condition sets for the computing tasks based on these task feature sets to achieve reasonable and efficient task scheduling.

[0056] First, based on the estimated execution time of the computing task, the processor utilization curve, memory capacity demand peak, and storage space fluctuation characteristics are adaptively segmented into non-uniform time segments corresponding to the task execution phase. The feature vectors of each non-uniform time segment are concatenated according to the channel dimension into a multidimensional feature tensor. Specifically, the processor utilization curve is analyzed for its changing trends and characteristics. Based on the estimated execution time, it is divided into several non-uniform time segments according to nodes with significant curve changes or based on predefined rules (such as changes in the curve slope or differences in resource requirements at different stages). For example, if the curve changes smoothly in the early stages and fluctuates significantly in the later stages, the early stages may be divided into one segment, and the later stages may be further subdivided based on the fluctuations. For the memory capacity demand peak, a similar segmentation method is used, taking into account the memory usage of the task at different stages and combining it with the estimated execution time. The same method is used for the storage space fluctuation characteristics. Then, for each non-uniform time segment, the corresponding feature representation is converted into a feature vector. For example, for a segment of the processor utilization curve, the utilization values ​​at different time points within the segment are organized into a vector; for a segment of peak memory demand, the peak-related features corresponding to the segment are organized into a vector; and for a segment of storage space fluctuation, the feature values ​​reflecting the fluctuations in the segment are combined into a vector. The feature vectors of these different time segments with different characteristics are then concatenated along the channel dimension. Assuming the feature vector for the processor utilization curve segment is vector A, the feature vector for the peak memory demand segment is vector B, and the feature vector for the storage space fluctuation segment is vector C, then the concatenated multidimensional feature tensor is formed by sequentially arranging vectors A, B, and C along the channel dimension.

[0057] Step S131: In the multi-head attention mechanism layer of the task scheduling strategy decision model, cross-segment association weights are calculated for the channel dimension of the multidimensional feature tensor to generate an attention weight distribution matrix.

[0058] The concatenated multidimensional feature tensor is input into the multi-head attention mechanism layer of the task scheduling policy decision model. In this layer, cross-segment correlation weights are calculated for the channel dimension of the multidimensional feature tensor. The specific process is as follows: The multi-head attention mechanism layer contains multiple attention heads, each of which independently processes the multidimensional feature tensor. For each attention head, the multidimensional feature tensor is first projected into a different subspace through a linear transformation to obtain query, key, and value matrices. This linear transformation is achieved by multiplying the multidimensional feature tensor with a predetermined weight matrix. For example, the multidimensional feature tensor is multiplied by the weight matrix W_Q to obtain the query matrix Q, multiplied by the weight matrix W_K to obtain the key matrix K, and multiplied by the weight matrix W_V to obtain the value matrix V. Next, the dot product of the transposed query matrix Q and the key matrix K is calculated. This dot product operation measures the correlation between features from different segments. The dot product result is then divided by a scaling factor to prevent excessively large dot product results from causing gradient instability. Next, the scaled results are normalized using the Softmax function to obtain an attention weight distribution. This attention weight distribution reflects the relative importance of features across different segments. Finally, the attention weight distribution is multiplied by the value matrix V and summed to obtain the output of each attention head. The outputs of all attention heads are concatenated to obtain the result of the multi-head attention mechanism. The resulting attention weight distribution is organized into a matrix, the attention weight distribution matrix, which reflects the association weights between different channel dimensions (i.e., different feature segments) of the multidimensional feature tensor.

[0059] Step S132: Dynamically weight the fragment vectors of the multidimensional feature tensor according to the attention weight distribution matrix to generate a resource demand intensity coding vector.

[0060] The attention weight distribution matrix generated in step S131 is used to dynamically weight and fuse the fragment vectors of the multidimensional feature tensor. Specifically, each row element in the attention weight distribution matrix is ​​multiplied by the element of the corresponding channel dimension (i.e., the corresponding fragment vector) in the multidimensional feature tensor. For example, assuming that the first row element of the attention weight distribution matrix is ​​[w1, w2, w3], and the fragment vectors corresponding to the channel dimension in the multidimensional feature tensor are vector A, vector B, and vector C, respectively, then the weighted vectors are w1*vector A, w2*vector B, and w3*vector C. Then, these weighted vectors are fused by element-wise addition (here weighted addition is used because this method can better reflect the contribution of different fragment features to the overall resource demand intensity) to obtain a resource demand intensity encoding vector. The resource demand intensity encoding vector comprehensively considers the relationship and importance between different task execution stages and different resource demand features, and encodes the intensity information of the task's resource demand.

[0061] Step S133: Input the dependency path length matrix into the graph convolutional network branch of the task scheduling strategy decision model, extract the dependency depth feature vectors between task nodes along the matrix row direction, and extract the parallel constraint strength feature vectors along the matrix column direction.

[0062] The dependency path length matrix is ​​input into the graph convolutional network branch of the task scheduling strategy decision model. In the graph convolutional network branch, first, in the adjacency matrix construction module, the non-zero elements of the dependency path length matrix are converted into weighted edge weights to generate a trainable adjacency matrix. Specifically, for each non-zero element in the dependency path length matrix, its value is converted into the weight of the edge in the graph structure according to the set conversion rule. For example, a simple linear conversion rule can be set to multiply the non-zero element value by a set coefficient to obtain the corresponding edge weight value. In this way, the dependency path length matrix is ​​converted into an adjacency matrix that can be used for graph convolution operations. The weight of the adjacency matrix reflects the strength of the dependency relationship between task nodes.

[0063] Next, the trainable adjacency matrix is ​​subjected to row-wise neighborhood feature aggregation through multiple layers of graph convolution operations to generate a deep propagation feature vector. In the graph convolution operation, each layer of graph convolution processes the adjacency matrix, aggregating the node's neighborhood information onto that node. Specifically, for each node, the feature information of its neighboring nodes is weighted and summed with the node's own feature information, with the weight determined by the edge weights in the adjacency matrix. After multiple layers of such operations, the feature information finally aggregated from each node is organized into a deep propagation feature vector along the row direction of the matrix, which reflects the deeply dependent features between task nodes.

[0064] In the column-wise parallel constraint analysis module, the dependency path length matrix is ​​transposed and then fed into a symmetric graph convolution layer to extract the backpropagation feature vector. The transposition operation analyzes the relationships between task nodes from another perspective to obtain features relevant to parallel constraints. The symmetric graph convolution layer operates similarly to the row-wise graph convolution described above, but is performed on a transposed matrix. Similarly, through multiple layers of graph convolution, node neighborhood information is aggregated, but this is performed along the rows of the transposed matrix (i.e., the columns of the original matrix). The resulting backpropagation feature vector reflects the characteristic information of the task nodes in the column direction (the direction relevant to parallel execution).

[0065] The Hadamard product operation is performed on the deep propagation feature vector and the back propagation feature vector to generate a hybrid constraint feature tensor. The Hadamard product operation is an element-wise multiplication. The elements at corresponding positions in the deep propagation feature vector and the back propagation feature vector are multiplied to obtain the hybrid constraint feature tensor. This hybrid constraint feature tensor combines the feature information of the dependency depth between task nodes and the parallel constraints.

[0066] The hybrid constraint feature tensor and the parallel execution feasibility parameter are channel-concatenated and input into a gated recurrent unit for temporal dependency modeling, generating a constraint strength encoding matrix. Channel concatenation combines the parallel execution feasibility parameter with the hybrid constraint feature tensor along the channel dimension, treating it as a new channel dimension. This combined tensor is then input into a gated recurrent unit. The gated recurrent unit, through its structure, models the temporal dependencies in the input tensor. It uses a gating mechanism to control the flow of information and update the hidden state based on the current input and the previous hidden state. After processing by the gated recurrent unit, the constraint strength encoding matrix is ​​output, which further encodes the dependencies and parallel constraints between tasks.

[0067] In the output layer of the graph convolutional network branch, the constraint strength encoding matrix is ​​subjected to dual-path aggregation using maximum pooling and average pooling to generate a dependency depth feature vector and a parallel constraint strength feature vector. The maximum pooling operation selects the maximum value in each channel dimension of the constraint strength encoding matrix as the representative value of that channel dimension; the average pooling operation calculates the average value in each channel dimension as the representative value. Through these two pooling operations, two vectors are obtained: one is a vector based on the maximum pooling result, which highlights the strongest features of each channel dimension in the constraint strength encoding matrix as the dependency depth feature vector; the other is a vector based on the average pooling result, which reflects the average features of each channel dimension as the parallel constraint strength feature vector. These two vectors reflect the dependency depth and parallel constraint strength information between task nodes from different perspectives.

[0068] Step S134: input the resource requirement intensity encoding vector, the dependency depth feature vector and the parallel constraint intensity feature vector into a cross-gating fusion unit to generate a global task feature vector.

[0069] The resource demand intensity encoding vector, the dependency depth feature vector, and the parallel constraint strength feature vector are input to the cross-gated fusion unit. First, in the feature alignment module of the cross-gated fusion unit, the resource demand intensity encoding vector is dimensionally expanded to match the dimension of the dependency depth feature vector. This dimension expansion can be achieved by repeating elements in certain dimensions of the resource demand intensity encoding vector or adding set padding values ​​to achieve the same dimension as the dependency depth feature vector.

[0070] Next, a linear transformation is performed on the dependency depth feature vector using a learnable weight matrix to generate a topology constraint projection vector. This learnable weight matrix is ​​continuously adjusted and optimized during model training. Its function is to project the dependency depth feature vector into a space that is more relevant to the resource demand intensity encoding vector. The dependency depth feature vector is multiplied by the weight matrix through matrix multiplication to obtain the topology constraint projection vector.

[0071] Calculate the element-by-element product of the topology constraint projection vector and the resource demand intensity encoding vector to generate a coupling feature vector. Multiply the topology constraint projection vector with the corresponding element in the resource demand intensity encoding vector to obtain a coupling feature vector. This coupling feature vector combines the characteristic information of resource demand and task topology dependency.

[0072] In the gating mechanism module, the parallel constraint strength feature vector is input into a Sigmoid function to generate a channel selection gating vector. The Sigmoid function maps each element in the parallel constraint strength feature vector to a value between 0 and 1, which can be viewed as a probability indicating the importance of each channel dimension. The vector composed of these values ​​is the channel selection gating vector.

[0073] The coupling feature vector is element-wise multiplied by the channel selection gating vector to generate a preliminary fused feature vector. This multiplication is then used to weight the corresponding channel dimension of the coupling feature vector according to the importance of each element in the channel selection gating vector.

[0074] The preliminary fused feature vector is residually connected to the original resource demand intensity encoding vector and fed into a multi-layer perceptron for nonlinear feature enhancement to produce the enhanced feature vector. Residual connection adds the preliminary fused feature vector to the original resource demand intensity encoding vector, preserving the original feature information while integrating new features. The result of this addition is then fed into the multi-layer perceptron, which uses multiple fully connected layers and nonlinear activation functions to perform a nonlinear transformation on the input vector, enhancing the expressive power of the features and producing the enhanced feature vector.

[0075] In the feature fusion output layer, the enhanced feature vectors undergo layer normalization to generate a global task feature vector. Layer normalization normalizes the enhanced feature vectors across the layer dimensions, ensuring that the inputs to each neuron have a similar distribution. This helps accelerate model training and improve model stability. The resulting vector is the global task feature vector, which comprehensively characterizes the overall characteristics of the task by integrating information such as resource requirements, task dependency depth, and parallel constraint strength.

[0076] Step S135: Based on the global task feature vector and the resource competition intensity prediction parameter, nonlinear mapping is performed in the fully connected network of the policy decision layer to output the original decision vector of the initial scheduling priority parameter and the resource allocation constraint condition set.

[0077] The global task feature vector and resource competition intensity prediction parameters are input into the fully connected network of the policy decision layer. First, the global task feature vector is input into the dual-path processing structure of the resource allocation constraint branch and the priority prediction branch in the fully connected network.

[0078] In the resource allocation constraint branch, three fully-connected layers are used to map the global task feature vector to a dimension that matches the number of compute node resource types, generating a resource allocation baseline vector. Each fully-connected layer linearly transforms the input vector by multiplying it by a weight matrix and adding a bias term. Nonlinearity is then introduced using a nonlinear activation function (such as the ReLU function). After these three layers, the global task feature vector is converted to a dimension that corresponds to the number of compute node resource types, resulting in a resource allocation baseline vector. This baseline vector provides a preliminary basis for assigning tasks to different resource types.

[0079] The resource contention intensity prediction parameters are fused element-by-element with the resource allocation baseline vector to generate a dynamic resource allocation primitive vector. This fusion method involves performing some operation (such as addition or multiplication, depending on the actual situation and model training) on ​​each element in the resource contention intensity prediction parameters with the corresponding element in the resource allocation baseline vector. This yields a dynamic resource allocation primitive vector that comprehensively considers the resource demand characteristics of the task and the predicted resource contention intensity, providing a more dynamic basis for resource allocation.

[0080] In the priority prediction branch, a multi-layer fully connected network performs a nonlinear transformation on the global task feature vector to generate an initial priority score. This multi-layer fully connected network also performs multiple nonlinear transformations on the global task feature vector through multiplication with a weight matrix, addition of bias terms, and nonlinear activation functions. This process mines the feature information related to task priority and ultimately outputs a scalar value, the initial priority score, which preliminarily reflects the task's scheduling priority.

[0081] Based on the parallel execution feasibility parameter and the initial priority score, an adjusted priority parameter is generated. Here, a method is needed to incorporate the parallel execution feasibility parameter into the initial priority score to obtain a more reasonable priority parameter. For example, a function can be created that takes the parallel execution feasibility parameter as one of its inputs and calculates it with the initial priority score to obtain the adjusted priority parameter. The design of this function needs to consider the impact of parallel execution on task priority. For example, the higher the parallel execution feasibility, the higher or lower the priority may be adjusted accordingly, depending on the actual situation and model training.

[0082] In the multi-objective optimization module of the policy decision layer, the original dynamic resource allocation vector and the adjusted priority parameters are jointly concatenated and input into the policy projection network to generate the original decision vector of the initial scheduling priority parameters and resource allocation constraint set. Joint vector concatenation is to connect the original dynamic resource allocation vector and the adjusted priority parameters in a set order into a longer vector. The concatenated vector is then input into the policy projection network. The policy projection network uses a series of linear and nonlinear transformations to map the input vector to the original decision vector space of the initial scheduling priority parameters and resource allocation constraint set, obtaining the original decision vector of the initial scheduling priority parameters and resource allocation constraint set, which provides the basis for the subsequent generation of the final scheduling policy.

[0083] Step S136: performing probability normalization processing and constraint compliance verification on the original decision vector to generate a real-time scheduling priority parameter and a resource allocation constraint condition set.

[0084] The original decision vector output by the policy decision layer is probabilistically normalized. Specifically, a specified normalization method (such as the Softmax function) is used to convert each element in the original decision vector into a probability value, so that the sum of these probability values ​​is 1. This is done to convert the numerical values ​​in the original decision vector into a probabilistic representation, allowing for a more reasonable determination of task scheduling priorities. For example, suppose the original decision vector contains multiple elements, each representing the scheduling priority scores of different tasks. After processing with the Softmax function, these scores are converted into the relative scheduling priority probability of each task among all tasks.

[0085] Then, the constraint compliance check is performed. For the original decision vector part of the resource allocation constraint set, check whether it meets the actual resource limitations of the computing nodes of the supercomputing center and the basic requirements for task execution. For example, check whether the amount of resource allocation is within the available resource range of the computing node, whether the task's demand for a certain resource complies with the usage rules of the resource, etc. If it is found that the constraint conditions are not met, the original decision vector is adjusted according to the pre-set correction rules. For example, if the number of processor cores allocated to a task exceeds the number of available cores of the computing node, then the processor core allocation of the task is reduced according to the set ratio, and other related parameters are adjusted accordingly to ensure that the entire resource allocation constraint set meets the compliance requirements.

[0086] After probabilistic normalization and constraint compliance verification, the real-time scheduling priority parameters and resource allocation constraint set are obtained. The real-time scheduling priority parameters accurately reflect the relative scheduling priority of each computing task under the current state of the supercomputing center. The resource allocation constraint set ensures that the task's resource allocation is consistent with the actual situation of the supercomputing center and meets the needs of task execution, providing a reliable basis for subsequent dynamic resource allocation and task scheduling operations.

[0087] Step S140: Generate a dynamic resource allocation instruction set for the computing nodes of the supercomputing center based on the real-time scheduling priority parameters and the resource allocation constraint condition set.

[0088] After obtaining the real-time scheduling priority parameters and resource allocation constraint set, the next step is to generate a dynamic resource allocation instruction set for the supercomputing center computing nodes based on this information to achieve reasonable resource allocation for computing tasks.

[0089] First, the currently pending computing tasks are globally sorted in descending order according to the real-time scheduling priority parameters to generate a priority task sequence. The specific sorting process involves associating each computing task with its corresponding real-time scheduling priority parameter, and then arranging all tasks in descending order of the real-time scheduling priority parameters. For example, for tasks A, B, and C, their corresponding real-time scheduling priority parameters are P_A, P_B, and P_C, respectively. If P_A>P_B>P_C, then the sorted priority task sequence is Task A, Task B, and Task C. This results in a task sequence arranged from high to low priority, providing a basis for the task processing order for subsequent resource allocation.

[0090] Step S141: Acquire the real-time resource status parameters of each computing node, including the number of remaining processor cores, available memory capacity, and storage bandwidth idle ratio.

[0091] The resource monitoring system within the supercomputing center obtains the resource status parameters of each computing node in real time. For each computing node, the remaining number of processor cores, that is, the number of processor cores not currently occupied by other tasks, the available memory capacity, which refers to the amount of memory currently available for allocation to new tasks on the computing node, and the idle storage bandwidth ratio, which represents the proportion of the unoccupied portion of the computing node's current storage bandwidth. For example, for computing node 1, its remaining number of processor cores is X1, its available memory capacity is Y1, and its idle storage bandwidth ratio is Z1; for computing node 2, the corresponding parameters are X2, Y2, Z2, and so on. These real-time resource status parameters are obtained for all computing nodes in the supercomputing center, providing basic data for subsequent resource allocation and matching.

[0092] Step S142: matching and verifying the resource allocation constraint condition set of each task in the priority task sequence with the real-time resource status parameters in sequence, and screening out computing nodes that meet the matching conditions to generate a candidate node set.

[0093] For the first task in the priority task sequence, extract its resource allocation constraint set. This set includes the task's specific constraints on resources such as the number of processor cores, memory capacity, and storage bandwidth. For example, the constraint on the number of processor cores might specify a range, the memory capacity might have a lower limit or specific requirements, and the storage bandwidth might also have certain conditions.

[0094] These resource allocation constraints are compared against the real-time resource status parameters of each compute node in the supercomputing center. For the number of remaining processor cores on the compute node, check whether it falls within the range required by the task. For available memory capacity, check whether it meets the minimum memory capacity requirement or specific requirements of the task. For the idle storage bandwidth ratio, confirm whether it meets the storage bandwidth requirements set by the task.

[0095] If all real-time resource status parameters of a compute node meet the resource allocation constraints of the task, then this compute node is included in the candidate node set. The above operation is repeated for each task in the priority task sequence, and its resource allocation constraint set is matched and verified with the real-time resource status parameters of all compute nodes in turn. In this way, the compute nodes that meet the matching conditions for each task are screened out to form the corresponding candidate node set.

[0096] Step S143: Calculating resource allocation benefits for the computing nodes in the candidate node set, sorting the computing nodes according to the resource allocation benefit calculation results, and selecting the node with the highest resource allocation benefit as the target node.

[0097] For each compute node in the candidate node set, a resource allocation benefit calculation must be performed. The calculation of resource allocation benefits requires a comprehensive consideration of multiple factors. First, the efficiency of computing node resource utilization when the task is executed on that compute node must be considered. For example, if the task's demand for processor cores is highly compatible with the number of remaining processor cores on the compute node, and the compute node's memory and storage bandwidth resources can be fully utilized, this indicates high resource utilization efficiency. Second, the impact of the compute node's execution of the task on the overall system resource balance of the supercomputing center must be considered. If assigning the task to that compute node can achieve a more balanced overall resource utilization rate for the supercomputing center and prevent certain resources from being excessively idle or overly busy, then this will have a positive impact on system resource balance.

[0098] Factors such as resource utilization efficiency and the impact on system resource balance are calculated using a pre-defined method to derive the resource allocation benefit. For example, a calculation formula can be established that includes a weight for resource utilization efficiency and a weight for the impact on system resource balance. Each factor is quantified and then calculated according to the formula. Assuming the quantified value of resource utilization efficiency is A, with a weight of a, and the quantified value of the impact on system resource balance is B, with a weight of b, then the resource allocation benefit C = a*A + b*B.

[0099] After calculating the resource allocation benefits for all compute nodes in the candidate node set, they are ranked from highest to lowest. The top-ranked compute node, with the highest resource allocation benefit, is selected as the target node. This target node is the most suitable compute node for allocating resources to the current task, maximizing resource utilization efficiency and maintaining a healthy balance of system resources within the supercomputing center.

[0100] Step S144: Dynamically bind the computing task to the resource slot of the target node and generate a corresponding resource allocation instruction unit.

[0101] Once the target node is determined, the current computing task needs to be dynamically bound to the resource slots of the target node. The target node has resource slots corresponding to different types of resources, such as processor core resource slots, memory resource slots, and storage bandwidth resource slots.

[0102] Based on the resource allocation constraint set of the task, the number of processor cores required by the task is mapped to the processor core resource slots of the target node to ensure that the task can obtain the corresponding number of processor core resources; the task's demand for memory capacity is matched to the memory resource slots of the target node to ensure that the supply of memory resources meets the task requirements; the task's demand for storage bandwidth is mapped to the storage bandwidth resource slots of the target node to ensure that sufficient storage bandwidth is available during the execution of the task.

[0103] In this way, computing tasks are precisely matched and bound to the various resource slots of the target node, generating a corresponding resource allocation instruction unit. This resource allocation instruction unit records the binding relationship between the task and the target node's resource slots in detail, clearly indicating which processor cores the task uses on the target node, how much memory capacity is occupied, and how much storage bandwidth is allocated. This provides accurate instructions for subsequent task scheduling operations performed by the supercomputing center's computing nodes.

[0104] Step S145: traverse all computing tasks and repeat steps S142 to S144, and finally generate a dynamic resource allocation instruction set consisting of multiple resource allocation instruction units.

[0105] Starting from the first task in the priority task sequence, each computing task is processed in sequence according to the flow from step S142 to step S144. After the resource allocation instruction unit of a task is generated, the next task is processed.

[0106] For each task, it must undergo matching verification with real-time resource status parameters, screen out a set of candidate nodes, calculate the resource allocation benefits of the candidate node set, sort and select the target node, and then dynamically bind the task to the resource slot of the target node to generate a resource allocation instruction unit.

[0107] After traversing all computing tasks in the priority task sequence, multiple resource allocation instruction units are obtained. These resource allocation instruction units are arranged in the order of the tasks in the priority task sequence and together form a dynamic resource allocation instruction set. This dynamic resource allocation instruction set comprehensively covers the resource allocation instruction information for the supercomputing center's current pending computing tasks, providing complete and orderly guidance for the supercomputing center's computing nodes to perform task scheduling operations. This enables the supercomputing center to rationally and efficiently allocate resources to each computing task based on these dynamic resource allocation instructions, thereby achieving dynamic scheduling of computing tasks.

[0108] Step S150: Control the computing nodes of the supercomputing center to perform task scheduling operations according to the dynamic resource allocation instruction set, and update the resource occupancy status of the computing nodes and the execution progress of the task queue in real time.

[0109] After receiving the dynamic resource allocation instruction set, the supercomputing center's task execution control system sends task scheduling instructions to the corresponding compute nodes based on the resource allocation instruction units within it. Each resource allocation instruction unit specifies the binding relationship between a specific computing task and a resource slot on the target compute node.

[0110] After receiving a task scheduling instruction, a compute node allocates the appropriate resources to the specified computing task and starts executing the task. For example, based on the instruction, the compute node allocates a set number of processor cores to the task, divides the memory area for the task, and allocates a set proportion of storage bandwidth resources.

[0111] During task execution, the supercomputing center's resource monitoring system monitors the resource usage of computing nodes in real time. For processor cores, it monitors which cores are being used by tasks and the extent of their usage. For memory, it tracks the actual memory usage of tasks. For storage bandwidth, it monitors the storage bandwidth usage of tasks during data transmission.

[0112] At the same time, the task execution control system updates the task queue's execution progress in real time. It records the execution status of each task on the compute node, such as whether the task has been started, is currently running, or has encountered any errors. Completed tasks are removed from the task queue. For ongoing tasks, the progress of these tasks is continuously tracked, for example, by calculating the completion percentage based on the task's execution steps or pre-set progress markers.

[0113] In this way, the supercomputing center can accurately control the computing nodes to perform task scheduling operations based on the dynamic resource allocation instruction set, and update the resource occupancy status of the computing nodes and the task queue execution progress in real time, thereby realizing dynamic and efficient management of computing tasks and ensuring the rational use of the overall resources of the supercomputing center and the smooth execution of tasks.

[0114] Furthermore, before using the task scheduling policy decision model to make task scheduling policy decisions, it needs to be trained so that it can accurately generate a reasonable set of real-time scheduling priority parameters and resource allocation constraints based on the task feature set.

[0115] First, we collected historical task data from the supercomputing center. This data contains various information about the task execution process, such as task identifiers, task submission timestamps, task resource requirements, descriptions of inter-task dependencies, actual task execution duration, actual resource allocation, and task execution results. We extracted feature data relevant to model training from this historical task data to form the training dataset.

[0116] For task resource requirements, we extract processor utilization curves, memory capacity demand peaks, and storage space fluctuations. For inter-task dependencies, we construct a task node topology and extract dependency path length matrices and parallel execution feasibility parameters. Furthermore, we combine task submission timestamps with the supercomputing center's historical task execution records to generate resource contention intensity prediction parameters. These feature data are paired with corresponding real-time scheduling priority parameters and resource allocation constraint sets, which are manually annotated based on historical task execution or generated using existing reasonable scheduling strategies, as training data.

[0117] The task scheduling strategy decision model contains multiple key modules, mainly including a multi-head attention mechanism layer, a graph convolutional network branch, a cross-gated fusion unit, and a fully connected network of the strategy decision layer.

[0118] The multi-head attention mechanism layer is used to calculate the cross-segment association weights of the channel dimension of the multi-dimensional feature tensor, capturing the association information between different task feature segments; the graph convolutional network branch processes the dependency path length matrix, extracts the dependency depth feature vectors between task nodes along the row direction of the matrix, and extracts the parallel constraint strength feature vectors along the column direction of the matrix; the cross-gated fusion unit fuses the resource demand intensity encoding vector, the dependency depth feature vector and the parallel constraint strength feature vector to generate a global task feature vector; the fully connected network of the policy decision layer outputs the original decision vector of the initial scheduling priority parameters and the resource allocation constraint set through nonlinear mapping based on the global task feature vector and the resource competition intensity prediction parameter.

[0119] Data is transferred and interacted between modules through set connection relationships to achieve step-by-step processing and analysis of task characteristics, and ultimately generate task scheduling strategy decision results.

[0120] On this basis, various parameters involved in the training process of the task scheduling strategy decision model are determined. For example, the number of attention heads in the multi-head attention mechanism layer is determined, which affects the task scheduling strategy decision model's ability to focus on different feature dimensions; the number of graph convolution layers in the graph convolution network branch is set, which affects the model's depth of extraction of task node topological structural features; the initial values ​​of the learnable weight matrix in the cross-gated fusion unit are set, which affect the starting state of feature fusion; and the number of neurons in each layer of the fully connected network of the strategy decision layer is determined, as the number of neurons determines the network's complexity and expressiveness.

[0121] At the same time, set the optimizer parameters during training, such as selecting an appropriate optimization algorithm (such as stochastic gradient descent, Adagrad, Adadelta, etc.) and determining the optimization algorithm's parameters such as the learning rate and momentum. The learning rate controls the step size of each parameter update, while momentum helps accelerate convergence and avoid falling into local optimal solutions.

[0122] The prepared training dataset can then be fed into the constructed task scheduling policy decision model. In each round of training, the task scheduling policy decision model processes the input task feature data based on the current parameter settings and generates an original decision vector for the predicted real-time scheduling priority parameters and resource allocation constraint set.

[0123] The prediction results are compared with the corresponding annotated results in the training data (i.e., the actual set of real-time scheduling priority parameters and resource allocation constraints). The difference between the predicted and annotated results is calculated using a loss function. Common loss functions include mean squared error (MSE) and cross entropy. Selecting an appropriate loss function depends on the nature and requirements of the task.

[0124] Based on the loss function, the backpropagation algorithm calculates the gradient of each parameter in the model. The gradient represents the rate of change of the loss function with respect to each parameter. Then, based on the optimizer's parameter settings, the gradient is used to update the model's parameters so that the loss function gradually decreases.

[0125] Repeat the above process for multiple rounds of training until the loss function converges to a smaller range, or until the preset number of training rounds is reached. At this point, the task scheduling policy decision model has completed training and can be used for actual task scheduling policy decisions.

[0126] Furthermore, the acquisition and processing of computing task request data sets and historical task data at the supercomputing center raises the issue of data privacy protection and leakage prevention. During the data collection phase, anonymization is employed for computing task request data that may contain privacy-sensitive information. For example, private data such as the identity of the task submitter is converted into anonymous identifiers using a pre-defined encryption algorithm, making it impossible to directly identify the task submitter's true identity from the collected data. Collected data is also categorized and labeled to clearly define privacy-sensitive data and general data. Regarding data storage, privacy-sensitive data is stored separately from general data. Encrypted storage technology is employed for privacy-sensitive data, using appropriate encryption algorithms (such as AES and RSA) to encrypt the data. This ensures that even if the data storage medium is illegally accessed, unauthorized decryption of the original data is impossible. Furthermore, strict access control is implemented to ensure that only authorized system modules or personnel can access privacy-sensitive data, and detailed records of access operations are maintained for audit and traceability. During data processing, homomorphic encryption or secure multi-party computation are employed for computational operations involving privacy-sensitive data. Homomorphic encryption allows specific computations to be performed on encrypted data. The decrypted results are identical to those of the same computations performed on the plaintext data, allowing data processing to be completed without compromising privacy. Secure multi-party computation allows multiple participants to collaboratively complete computational tasks without disclosing their private data.

[0127] For example, when generating task feature sets, if feature extraction involves privacy-sensitive data, the aforementioned techniques are employed to ensure data privacy. When training task scheduling policy decision models, if privacy-sensitive data is involved, these techniques are also employed to ensure privacy is not compromised during the training process. These data privacy protection and anti-leakage measures ensure that the supercomputing center can fully utilize data for efficient task scheduling while also ensuring data privacy and preventing data leakage risks when processing computing task-related data.

[0128] Figure 2 A schematic diagram illustrates exemplary hardware and software components of an AI-based dynamic scheduling system 100 for supercomputing center computing tasks, provided in some embodiments of the present application, that can implement the concepts of the present application. For example, the processor 120 can be used in the AI-based dynamic scheduling system 100 for supercomputing center computing tasks and perform the functions described in the present application.

[0129] The AI-based supercomputing center computing task dynamic scheduling system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the AI-based supercomputing center computing task dynamic scheduling method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0130] For example, the supercomputing center computing task dynamic scheduling system 100 based on artificial intelligence may include a network port 110 connected to the network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the supercomputing center computing task dynamic scheduling system 100 based on artificial intelligence may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The supercomputing center computing task dynamic scheduling system 100 based on artificial intelligence also includes an I / O interface 150 between the computer and other input and output devices.

[0131] For ease of explanation, only one processor is described in the dynamic scheduling system 100 for computing tasks of a supercomputing center based on artificial intelligence. However, it should be noted that the dynamic scheduling system 100 for computing tasks of a supercomputing center based on artificial intelligence in this application may also include multiple processors, so the steps performed by one processor described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the dynamic scheduling system 100 for computing tasks of a supercomputing center based on artificial intelligence executes step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0132] In addition, an embodiment of the present invention also provides a readable storage medium, in which computer-executable instructions are preset. When the processor executes the computer-executable instructions, the above-mentioned dynamic scheduling method of supercomputing center computing tasks based on artificial intelligence is implemented.

[0133] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence, characterized in that: The method comprises: Obtaining a computing task request data set currently pending at the supercomputing center, wherein the computing task request data set includes a computing task identifier, a task submission timestamp, a task resource requirement description, and a description of inter-task dependencies; Performing task dynamic feature extraction processing on the computing task request data set to generate a task feature set for each computing task; Calling a pre-trained task scheduling strategy decision model to perform task scheduling strategy decision on the task feature set, and generating a real-time scheduling priority parameter and a resource allocation constraint condition set for the computing task; Generate a dynamic resource allocation instruction set for a computing node in a supercomputing center based on the real-time scheduling priority parameters and the resource allocation constraint condition set; Controlling the supercomputing center computing nodes to perform task scheduling operations according to the dynamic resource allocation instruction set, and updating the resource occupancy status of the computing nodes and the task queue execution progress in real time; The step of performing task dynamic feature extraction processing on the computing task request data set to generate a task feature set for each computing task includes: Analyzing the task resource requirement description in the computing task request data set, and extracting the processor occupancy change curve, memory capacity requirement peak and storage space fluctuation characteristics during the processing of the computing task; Analyze the task node topology structure in the inter-task dependency description, and construct a dependency path length matrix and parallel execution feasibility parameters between the task nodes; Generate resource competition intensity prediction parameters for computing tasks within a preset time window based on the task submission timestamp and the historical task execution records of the supercomputing center; The processor occupancy change curve, memory capacity demand peak, storage space fluctuation characteristics, dependency path length matrix, parallel execution feasibility parameters and resource competition intensity prediction parameters are subjected to feature normalization processing, and the normalized feature parameters are input into a feature fusion network based on a multi-head attention mechanism. Cross-dimensional correlation analysis is achieved through cross-channel weight allocation to generate a task feature set.

2. The method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence according to claim 1, characterized in that: The analyzing the task node topology structure in the inter-task dependency description and constructing the dependency path length matrix and parallel execution feasibility parameters between the task nodes includes: Traversing each task node in the task node topology structure, marking task node pairs that have direct dependency relationships and generating a dependency edge set; Construct a task node topology graph based on the dependency edge set, and calculate the shortest dependency path length and the maximum number of parallel branches from each task node to all its downstream nodes; Generate a dependency path length matrix including dependency depth weights between task nodes based on the shortest dependency path length, wherein the dependency depth weights are positively correlated with the shortest dependency path length; Counting the number of task node pairs without direct dependencies in the task node topology graph, and calculating a resource contention-aware parallel execution conflict probability parameter between task nodes based on the maximum number of parallel branches and the processor occupancy overlap parameter in the task resource requirement description; A parallel execution feasibility parameter reflecting the intensity of competition for task parallel execution resources is generated based on the parallel execution conflict probability parameter and the processor occupancy rate change curve in the task resource requirement description.

3. The method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence according to claim 1, characterized in that: The generating of resource competition intensity prediction parameters of the computing task within a preset time window based on the task submission timestamp and the historical task execution records of the supercomputing center includes: Extracting a subset of historical task execution data that matches the current task type from the historical task execution records of the supercomputing center, wherein the subset of historical task execution data includes a historical task submission timestamp, a resource occupancy curve, and a task execution duration; Match the resource occupancy rate curve of the historical same period according to the current task submission timestamp, and extract the average processor occupancy rate and memory usage peak fluctuation parameters within the same time window; Analyze the task execution records with similar dependency complexity in the historical task execution data subset, and calculate the correlation function between the task startup delay and the dependency path length matrix; Building a resource contention intensity prediction model based on the processor average occupancy rate, memory usage peak fluctuation parameter and correlation function, and inputting the current task submission timestamp and dependency path length matrix to generate an initial resource contention intensity prediction value; The initial resource competition intensity prediction value is dynamically revised according to the real-time resource occupancy status of the computing nodes in the supercomputing center to generate a resource competition intensity prediction parameter after eliminating the node load deviation.

4. The method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence according to claim 1, characterized in that: The calling of the pre-trained task scheduling policy decision model to perform task scheduling policy decision on the task feature set, and generating a real-time scheduling priority parameter and a resource allocation constraint condition set for the computing task, includes: Based on the estimated execution time of the computing task, the processor occupancy change curve, memory capacity demand peak, and storage space fluctuation characteristics are adaptively divided into non-uniform time segments corresponding to the task execution stage, and the feature vectors of each non-uniform time segment are spliced ​​into a multi-dimensional feature tensor according to the channel dimension; In the multi-head attention mechanism layer of the task scheduling strategy decision model, cross-segment association weight calculation is performed on the channel dimension of the multidimensional feature tensor to generate an attention weight distribution matrix; Dynamically weighting and fusing the segment vectors of the multidimensional feature tensor according to the attention weight distribution matrix to generate a resource demand intensity encoding vector; Inputting the dependency path length matrix into the graph convolutional network branch of the task scheduling strategy decision model, extracting dependency depth feature vectors between task nodes along the matrix row direction, and extracting parallel constraint strength feature vectors along the matrix column direction; Input the resource requirement intensity encoding vector, the dependency depth feature vector and the parallel constraint intensity feature vector into a cross-gated fusion unit to generate a global task feature vector; Based on the global task feature vector and the resource competition intensity prediction parameter, nonlinear mapping is performed in the fully connected network of the policy decision layer to output the original decision vector of the initial scheduling priority parameter and the resource allocation constraint condition set; The original decision vector is subjected to probability normalization processing and constraint compliance verification to generate a real-time scheduling priority parameter and a resource allocation constraint condition set.

5. The method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence according to claim 4, characterized in that: Inputting the dependency path length matrix into the graph convolutional network branch of the task scheduling strategy decision model, extracting dependency depth feature vectors between task nodes along the matrix row direction, and extracting parallel constraint strength feature vectors along the matrix column direction, includes: In the adjacency matrix construction module of the graph convolutional network branch, non-zero elements of the dependency path length matrix are converted into weighted edge weights to generate a trainable adjacency matrix; Aggregating row-wise neighborhood features of the trainable adjacency matrix through multi-layer graph convolution operations to generate a deep propagation feature vector; In the column-parallel constraint analysis module, the dependent path length matrix is ​​transposed and then input into the symmetric graph convolution layer to extract the back-propagation feature vector. Performing a Hadamard product operation on the depth propagation feature vector and the back propagation feature vector to generate a hybrid constraint feature tensor; Channel-concatenate the hybrid constraint feature tensor and the parallel execution feasibility parameter, input the result into a gated recurrent unit for temporal dependency modeling, and generate a constraint strength encoding matrix; In the output layer of the graph convolutional network branch, the constraint strength encoding matrix is ​​aggregated by maximum pooling and average pooling to generate a dependent depth feature vector and a parallel constraint strength feature vector.

6. The method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence according to claim 5, characterized in that: The step of inputting the resource requirement intensity encoding vector, the dependency depth feature vector, and the parallel constraint intensity feature vector into a cross-gated fusion unit to generate a global task feature vector includes: In the feature alignment module of the cross-gated fusion unit, a dimension expansion operation is performed on the resource demand intensity encoding vector to match the dimension of the dependent depth feature vector; A linear transformation of the depth-dependent feature vector is performed through a learnable weight matrix to generate a topologically constrained projection vector; Calculating the element-by-element product of the topology constraint projection vector and the resource demand intensity encoding vector to generate a coupling feature vector; In the gating mechanism module, the parallel constraint strength feature vector is input into a Sigmoid function to generate a channel selection gating vector; Performing element-wise multiplication on the coupling feature vector and the channel selection gating vector to generate a preliminary fusion feature vector; Performing a residual connection between the preliminary fusion feature vector and the original resource demand intensity coding vector, inputting the result into a multi-layer perceptron for nonlinear feature enhancement, and obtaining an enhanced feature vector; In the feature fusion output layer, the enhanced feature vector is normalized to generate the global task feature vector.

7. The method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence according to claim 6, characterized in that: The nonlinear mapping is performed in the fully connected network of the policy decision layer based on the global task feature vector and the resource competition intensity prediction parameter, and the original decision vector of the initial scheduling priority parameter and the resource allocation constraint condition set is output, including: Inputting the global task feature vector into a dual-path processing structure of a resource allocation constraint branch and a priority prediction branch in the fully connected network respectively; In the resource allocation constraint branch, the global task feature vector is mapped to a dimension that matches the number of computing node resource types through three fully connected layers to generate a resource allocation baseline vector; fusing the resource contention intensity prediction parameter with the resource allocation baseline vector element by element to generate a dynamic resource allocation original vector; In the priority prediction branch, a nonlinear transformation is performed on the global task feature vector through a multi-layer fully connected network to generate an initial priority score; generating an adjusted priority parameter according to the parallel execution feasibility parameter and the initial priority score; In the multi-objective optimization module of the policy decision layer, the dynamic resource allocation original vector and the adjusted priority parameter are jointly vector-concatenated and input into the policy projection network to generate the original decision vector of the initial scheduling priority parameter and resource allocation constraint condition set.

8. The method for dynamic scheduling of computing tasks in a supercomputing center based on artificial intelligence according to claim 1, characterized in that: The generating of a dynamic resource allocation instruction set for a computing node in a supercomputing center based on the real-time scheduling priority parameter and the resource allocation constraint condition set includes: Sort the currently pending computing tasks in global descending order according to the real-time scheduling priority parameters to generate a priority task sequence; Obtain the real-time resource status parameters of each computing node, including the number of remaining processor cores, available memory capacity, and idle storage bandwidth ratio; Matching and verifying the resource allocation constraint condition set of each task in the priority task sequence with the real-time resource status parameters in sequence, and screening out computing nodes that meet the matching conditions to generate a candidate node set; Calculating resource allocation benefits for the computing nodes in the candidate node set, sorting the computing nodes according to the resource allocation benefit calculation results, and selecting the node with the highest resource allocation benefit as the target node; Dynamically binding the computing task to the resource slot of the target node and generating a corresponding resource allocation instruction unit; Traverse all computing tasks and repeat the above steps to finally generate a dynamic resource allocation instruction set consisting of multiple resource allocation instruction units.

9. A dynamic scheduling system for computing tasks in a supercomputing center based on artificial intelligence, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the artificial intelligence-based supercomputing center computing task dynamic scheduling method described in any one of claims 1 to 8.

Images