Hyper-computing center computing task dynamic scheduling method and system based on artificial intelligence

Abstract

The invention provides a supercomputing center computing task dynamic scheduling method and system based on artificial intelligence, and the method comprises the steps: firstly obtaining a current to-be-processed computing task request data set of a supercomputing center, and extracting a task feature set of each computing task; calling a pre-trained task scheduling strategy decision model to analyze the task feature set, generating a real-time scheduling priority parameter and a resource allocation constraint condition set of the calculation task, and calculating the task feature set based on the real-time scheduling priority parameter and the resource allocation constraint condition set of the calculation task; according to the method, the dynamic resource allocation instruction set of the computing nodes of the super-computing center is generated, the computing nodes are controlled to execute the task scheduling operation, intelligent and efficient scheduling of the computing tasks of the super-computing center is achieved by updating the resource occupancy state of the computing nodes and the task queue execution progress in real time, and the resource utilization rate and the task execution efficiency are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence. Specifically, it relates to a method and system for dynamically scheduling computing tasks in a supercomputer center based on artificial intelligence. Background Art

[0002] With the rapid development of science and technology, supercomputer centers play an increasingly important role in fields such as scientific research, engineering computing, and big data analysis. A supercomputer center usually contains a large number of computing nodes, and these computing nodes need to process multiple complex computing tasks simultaneously. However, existing task scheduling methods for supercomputer centers often rely on static task priorities or simple resource allocation strategies, and are difficult to adapt to the dynamically changing computing task requirements and resource conditions. Specifically, existing technologies cannot accurately capture the dynamic characteristics of computing tasks, such as task submission time, resource requirements, and dependencies between tasks, resulting in low task scheduling efficiency, low resource utilization, and even problems such as long task waiting or resource conflicts. Summary of the Invention

[0003] In view of the problems mentioned above, in combination with the first aspect of the present invention, embodiments of the present invention provide a method for dynamically scheduling computing tasks in a supercomputer center based on artificial intelligence. The method includes: Obtain a set of computing task request data to be processed currently in the supercomputer center. The set of computing task request data includes computing task identifiers, task submission timestamps, task resource requirement descriptions, and descriptions of dependencies between tasks; Perform task dynamic feature extraction processing on the set of computing task request data to generate a set of task features for each computing task; Call a pre-trained task scheduling policy decision model to make task scheduling policy decisions on the set of task features, and generate real-time scheduling priority parameters for the computing tasks and a set of resource allocation constraint conditions; Generate a set of dynamic resource allocation instructions for the computing nodes in the supercomputer center based on the real-time scheduling priority parameters and the set of resource allocation constraint conditions; Control the computing nodes in the supercomputer center to perform task scheduling operations according to the set of dynamic resource allocation instructions, and update the resource occupancy status of the computing nodes and the execution progress of the task queue in real time.

[0004] In another aspect, embodiments of the present invention also provide a system for dynamically scheduling computing tasks in a supercomputer center based on artificial intelligence, including a processor and a machine-readable storage medium. The machine-readable storage medium is connected to the processor. The machine-readable storage medium is used to store programs, instructions, or codes, and the processor is used to execute the programs, instructions, or codes in the machine-readable storage medium to implement the above method.

[0005] Based on the above aspects, embodiments of the present invention can accurately evaluate the priority and resource requirements of each computing task by comprehensively obtaining and analyzing the dynamic characteristics of computing tasks, including task submission timestamps, resource requirement descriptions, and dependencies between tasks. Further, by invoking a pre-trained task scheduling policy decision model, real-time scheduling priority parameters and a set of resource allocation constraint conditions for computing tasks can be dynamically generated according to the current resource status of the supercomputer center and the set of task characteristics, which not only realizes the intelligent scheduling of computing tasks but also significantly improves the resource utilization rate and task execution efficiency of the supercomputer center. At the same time, by updating the resource occupancy status of computing nodes and the execution progress of the task queue in real time, it is ensured that the supercomputer center can flexibly respond to various dynamically changing computing task requirements. Brief Description of the Drawings

[0006] Figure 1 It is a schematic flowchart of the execution process of the dynamic scheduling method for computing tasks in a supercomputer center based on artificial intelligence provided by an embodiment of the present invention.

[0007] Figure 2 It is a schematic diagram of exemplary hardware and software components of a dynamic scheduling system for computing tasks in a supercomputer center based on artificial intelligence provided by an embodiment of the present invention. Detailed Embodiments

[0008] The present invention will be specifically described below with reference to the accompanying drawings of the specification. Figure 1 It is a schematic flowchart of a dynamic scheduling method for computing tasks in a supercomputer center based on artificial intelligence provided by an embodiment of the present invention. The dynamic scheduling method for computing tasks in a supercomputer center based on artificial intelligence will be introduced in detail below.

[0009] Step S110: Obtain a set of computing task request data to be processed currently in the supercomputer center, where the set of computing task request data includes a computing task identifier, a task submission timestamp, a task resource requirement description, and a description of dependencies between tasks.

[0010] During the operation of the supercomputer center, data can be obtained through a functional module responsible for data acquisition. This functional module always remains active and continuously collects data related to computing tasks waiting to be processed in the supercomputer center.

[0011] For each computing task, a set of coding rules composed of multiple factors is required to assign a unique computing task identifier. This coding rule integrates information such as the category of the task initiator, the order of submission to the supercomputer center, and the specific attributes of the task itself. For example, the category of the task initiator is identified by a specific combination of letters, the submission order is numbered incrementally starting from the moment the supercomputer center begins to receive tasks, and the specific attributes of the task are represented by different symbols according to their different natures. Then, these elements are combined in the order of "letter combination - sequential number - symbol" to form an identifier that can uniquely identify each computing task, so that in the task management system of the entire supercomputer center, each task has a unique identity.

[0012] The task submission timestamp is used to accurately record the specific moment when the task enters the supercomputer center. It takes a certain fixed starting time point when the supercomputer center starts running as a reference, and determines the submission time of the task by recording the duration elapsed from this starting point to the moment the task enters the supercomputer center. To ensure the accuracy and consistency of time recording, the time recording is accurate to the second level. That is to say, starting from the startup of the supercomputer center, the moment when the task enters the supercomputer center is accumulated in seconds, and this accumulated duration is the task submission timestamp, providing an accurate time basis for subsequent time-based task analysis and scheduling.

[0013] The description of task resource requirements details the specific requirements of the task for various resources during execution. In terms of processor occupancy, the occupancy ratio of the processor cores by the task at different execution stages can be recorded at set time intervals. Specifically, starting from the start of task execution, the occupancy ratio of the processor cores is detected and recorded at fixed time intervals. As time goes by, the recorded data points form a dataset reflecting the change of processor occupancy over time. By sorting out and analyzing this dataset, the change of the task's occupancy requirements for processor resources at different stages can be clearly understood.

[0014] The description method of memory capacity requirements is similar. The change of the task's memory capacity requirements during execution is also recorded at set time intervals. Starting from the startup of the task, at each fixed time interval point, the memory capacity required by the task at this time is detected and recorded. The set of these data points shows the dynamic change of the task's memory requirements during the entire execution process.

[0015] The description of storage space requirements focuses on recording the changes in the use of storage space during the execution of tasks. This includes not only the actual amount of storage space occupied by tasks at different stages, but also the trend of the occupied amount over time. For example, by recording the amount of data created, modified, or deleted by a task at different time points, and the time when these operations occur, a comprehensive picture of the task's storage space usage pattern and changes can be obtained.

[0016] The description of task dependencies uses a specific data structure to clearly present the association relationships between tasks. If the start of a task must depend on another task to complete specific steps or all of them, then in this data structure, this sequence and dependency logic will be reflected in an intuitive way. For example, a tree-like data structure is adopted, where the task that depends on other tasks is used as a child node, and the task being depended on is used as a parent node. The dependency relationship between tasks is clarified through the connection relationship between nodes. In this way, in this data structure, the dependency path and degree between each task and other tasks can be clearly seen.

[0017] Step S120: Perform task dynamic feature extraction processing on the computing task request data set to generate a task feature set for each computing task.

[0018] When the complete computing task request data set is obtained, the task dynamic feature extraction process is immediately started. The core purpose of this process is to extract key feature information from the large amount of information contained in the computing task request data set, which can comprehensively and accurately reflect the unique properties and requirements of each task.

[0019] Step S121: Analyze the task resource requirement description in the computing task request data set, and extract the processor occupancy rate change curve, the peak value of memory capacity requirement, and the storage space fluctuation characteristics during the processing of the computing task.

[0020] For the task resource requirement description part, when extracting the processor occupancy rate change curve, first obtain data from the data set recording the change of processor occupancy over time. This computing task request data set contains the information about the occupancy ratio of the processor cores at each time point during the execution of the task. Then, use a fixed time interval as the sampling period to sample the data in the data set. For example, select the processor occupancy ratio data at a time point at regular intervals, and arrange these sampled data points in sequence. Next, adopt a suitable data fitting method, such as polynomial fitting or spline curve fitting, etc., to connect these discrete data points into a smooth curve. The curve obtained in this way is the processor occupancy rate change curve during the processing of this computing task, which intuitively shows the dynamic change of the occupancy ratio of the task to the processor resources at different times.

[0021] When extracting the peak value of the memory capacity requirement, a comprehensive traversal is performed on the dataset that records the memory requirement changing over time. During the traversal process, the memory capacity requirements of tasks at each time point are compared in sequence. Whenever a value larger than the previously recorded requirement is encountered, the currently recorded maximum value is updated. After traversing the entire dataset, the finally obtained maximum value is the peak value of the memory capacity requirement of the task, which represents the maximum memory capacity requirement of the task during the entire execution process.

[0022] For the extraction of the storage space fluctuation characteristics, first calculate the change amount of the storage space within adjacent fixed time intervals. The specific approach is to sequentially obtain the storage space occupancy at two adjacent time points from the dataset that records the storage space usage changing over time, and then subtract the occupancy at the previous time point from the occupancy at the latter time point to obtain the change amount within the adjacent time interval. Thus, a series of data reflecting the change amount of the storage space can be obtained. Then, statistical analysis is performed on these change amount data, such as calculating statistical quantities such as their variance or standard deviation. These statistical quantities can well measure the fluctuation degree of the storage space within different time intervals, and thus serve as the storage space fluctuation characteristics, accurately reflecting the stability and change range of the storage space usage during the task execution process.

[0023] Step S122: Analyze the task node topological structure in the task - to - task dependency description, and construct a dependency path length matrix and a parallel execution feasibility parameter between task nodes.

[0024] Step S1221: Traverse each task node in the task node topological structure, mark the pairs of task nodes with direct dependency relationships, and generate a set of dependency relationship edges.

[0025] For each task node in the task node topological structure, a detailed check is carried out one by one to determine whether there is a direct dependency relationship between it and all other task nodes. The specific check process is as follows: For the currently inspected task node, compare it with each other task node in the topological structure. If it is found that the start of a task node must depend on another task node to complete a specific operation or all operations, then it is determined that there is a direct dependency relationship between these two task nodes, and they are marked as a pair of task nodes. Collect all the pairs of task nodes with direct dependency relationships determined in this way to form a set of dependency relationship edges. In this set of dependency relationship edges, each element represents a pair of directly dependent task nodes. For example, (Task X, Task Y) clearly indicates that Task X directly depends on Task Y.

[0026] Step S1222: Construct a task node topology graph based on the set of dependency edges, and calculate the shortest dependency path length from each task node to all its downstream nodes and the maximum number of parallel branches.

[0027] Construct a task node topology graph based on the generated set of dependency edges. During the construction process, each task node is regarded as a vertex in the graph, and the dependency edges are regarded as the edges connecting these vertices. In this way, through the elements in the set of dependency edges, each task node can be connected according to their dependency relationships, forming a topology graph that intuitively shows the dependency relationships between tasks.

[0028] In the constructed task node topology graph, for each task node, calculate the shortest dependency path length from it to all its downstream nodes. The specific calculation method is to use a search algorithm suitable for graph structures in related technologies, such as the breadth-first search algorithm. Starting from the current task node, search the nodes in the graph layer by layer in a breadth-first manner, and record the path length from the starting task node to each downstream node. During the search process, whenever a path length to a certain downstream node is found to be shorter than the previously recorded path length of that node, update the shortest path length record of that downstream node. After all downstream nodes have been searched, obtain the shortest dependency path length from the task node to all its downstream nodes.

[0029] At the same time, for each task node, also count its maximum number of parallel branches. The specific approach is to start from this task node and traverse along the edges in the topology graph to find all task branches that can be started simultaneously without depending on the completion of other task nodes at the same level. Count these task branches that can be started in parallel, and the obtained quantity is the maximum number of parallel branches of this task node. This maximum number of parallel branches reflects the upper limit of the number of tasks that can be executed in parallel based on the current task node.

[0030] Step S1223: Generate a dependency path length matrix containing the dependency depth weights between task nodes based on the shortest dependency path length, and the dependency depth weight has a positive correlation with the shortest dependency path length.

[0031] Use the calculated shortest dependency path length from each task node to all its downstream nodes to generate a dependency path length matrix containing the dependency depth weights between task nodes.

[0032] First, determine the positive correlation between the dependency depth weight and the length of the shortest dependency path. For example, a simple linear positive correlation can be defined, where the dependency depth weight is equal to the length of the shortest dependency path multiplied by a preset positive coefficient. The selection of this positive coefficient needs to be determined according to the actual task characteristics and scheduling requirements of the supercomputing center, and its role is to adjust the amplitude of the change in the dependency depth weight with the length of the shortest dependency path.

[0033] Then, for each element in the dependency path length matrix, that is, for each pair of task nodes, calculate the corresponding dependency depth weight according to the positive correlation determined above based on the length of the shortest dependency path between them, and fill the calculated dependency depth weight value into the corresponding position in the matrix. Thus, a dependency path length matrix containing the dependency depth weights between task nodes is constructed. This dependency path length matrix comprehensively reflects the depth and strength of the dependency relationship between task nodes.

[0034] Step S1224: Count the number of task node pairs without direct dependency relationships in the task node topology graph, and combine the maximum number of parallel branches and the processor occupancy overlap parameter in the task resource requirement description to calculate the parallel execution conflict probability parameter for resource competition awareness between task nodes.

[0035] In the constructed task node topology graph, carefully count the number of task node pairs without direct dependency relationships. The specific counting method is to compare each task node in the task node topology graph with all other task nodes. If there is no direct dependency relationship edge connecting two task nodes, then these two task nodes are regarded as a pair of task node pairs without direct dependency relationships and are counted. By traversing and comparing all task nodes, the total number of task node pairs without direct dependency relationships is finally obtained.

[0036] Obtain the processor occupancy overlap parameter from the task resource requirement description. The process of obtaining this processor occupancy overlap parameter requires a detailed analysis of the processor occupancy of different tasks within the same time interval. Specifically, first determine a common time range, and then within this time range, sample the data of the processor occupancy rate of each task over time. Compare the processor occupancy rates of different tasks at the same sampling time point. If at a certain time point, the processor occupancy rates of multiple tasks are all relatively high, it indicates that the resource requirements of these tasks overlap at this time. Through the analysis of all sampling points within the entire time range, comprehensively considering factors such as the degree and duration of overlap, calculate a processor occupancy overlap parameter that can reflect the degree of overlap in processor usage between tasks.

[0037] Combined with the number of task node pairs without direct dependency, the maximum number of parallel branches for each task node, and the processor occupancy overlap parameter obtained from the previous statistics, a dedicated calculation method is designed to calculate the parallel execution conflict probability parameter for resource competition awareness among task nodes. This calculation method can be to construct a mathematical model that includes these three factors, such as a multivariate function. In this multivariate function, the number of task node pairs without direct dependency, the maximum number of parallel branches, and the processor occupancy overlap parameter serve as the independent variables of the function. By performing mathematical operations on these independent variables, such as weighted summation, multiplication, and other combination methods, a result ranging from 0 to 1 is obtained, which is the parallel execution conflict probability parameter for resource competition awareness among task nodes. This parallel execution conflict probability parameter can comprehensively reflect the likelihood of conflicts arising due to resource competition when tasks are executed in parallel.

[0038] Step S1225: Generate a parallel execution feasibility parameter that reflects the resource competition intensity of task parallel execution based on the parallel execution conflict probability parameter and the processor occupancy change curve in the task resource requirement description.

[0039] Utilize the already calculated parallel execution conflict probability parameter and the processor occupancy change curve in the task resource requirement description to generate a parallel execution feasibility parameter that reflects the resource competition intensity of task parallel execution.

[0040] First, conduct further analysis on the processor occupancy change curve. In addition to extracting the characteristics of the curve itself as mentioned before, factors such as the slope change of the curve and the frequency of peak occurrences also need to be considered. For example, calculate the average slope of the processor occupancy change curve in different time periods to reflect the speed of change of the processor occupancy through the magnitude of the slope. At the same time, count the number of peak occurrences and their distribution throughout the task execution time.

[0041] Then, in combination with the parallel execution conflict probability parameter, it can be to organically combine the parallel execution conflict probability parameter with the various characteristics obtained from the analysis of the processor occupancy change curve. For example, a weighted comprehensive model can be constructed. According to the actual situation of the supercomputer center and the characteristics of the task, different weights are assigned to the parallel execution conflict probability parameter, the average slope of the processor occupancy change curve, the peak occurrence frequency, and other factors. Then, these factors are weighted and summed or other appropriate mathematical operations are performed to obtain a parallel execution feasibility parameter. This parallel execution feasibility parameter comprehensively reflects the degree of influence brought about by resource competition during the parallel execution of tasks, thus providing a key reference index for determining whether a task is suitable for parallel execution.

[0042] Step S123: Generate a prediction parameter for the resource competition intensity of a computing task within a preset time window based on the task submission timestamp and the historical task execution records of the supercomputing center.

[0043] After completing the feature extraction of the task resource requirements and the dependencies between tasks as described above, next, use the task submission timestamp and the historical task execution records long accumulated by the supercomputing center to predict the resource competition intensity of a computing task within a preset time window and generate corresponding prediction parameters.

[0044] First, from the huge historical task execution record database of the supercomputing center, screen out a subset of historical task execution data that matches the current task type. The matching of task types can be determined based on multiple factors, such as the application field of the task, computational complexity, resource requirement pattern, etc. Through comprehensive analysis of these factors, extract the task data with similar characteristics to the current task from the historical task execution records to form a subset of historical task execution data for the current task type. This subset of historical task execution data contains key information such as the submission timestamp of the historical task, the resource occupancy rate curve, and the task execution duration.

[0045] Then, according to the submission timestamp of the current task, match the corresponding historical same - period time range in the subset of historical task execution data. The specific matching method is to use the submission timestamp of the current task as a benchmark and find the task data within the same time interval in the historical task execution records. For example, if the current task is submitted in the middle of a certain month, then search for the task data within the middle - month time range of the same month in the historical task execution records. From these matched historical task data, extract the average processor occupancy rate and the memory usage peak fluctuation parameter within the same time window. The average processor occupancy rate is obtained by calculating the average of the processor occupancy rates of all historical tasks within this time window, which reflects the average usage of processor resources within this specific time window. The memory usage peak fluctuation parameter is determined by analyzing the changes in the memory usage peaks of historical tasks within this time window, such as calculating statistical quantities such as the standard deviation or range of the memory usage peaks to measure the fluctuation degree of the memory usage peaks.

[0046] Next, conduct an in-depth analysis of the task execution records with similar dependency complexity in the historical task execution data subset. The dependency complexity can be measured by the task node topology structure constructed previously and information such as the dependency path length matrix. For tasks with similar dependency complexity, statistically analyze the correlation between their task startup delay durations and the dependency path length matrix. The specific statistical method is to record the task startup delay duration for each such historical task, that is, the time interval from task submission to actual execution, and at the same time record the relevant information in the dependency path length matrix corresponding to the task. Then, through data analysis methods, such as correlation analysis or regression analysis, find the functional relationship between the task startup delay duration and the dependency path length matrix to obtain an association function. This association function reflects the influence law of task dependency complexity on task startup delay.

[0047] Based on the previously extracted average processor occupancy rate, memory usage peak fluctuation parameters, and the obtained association function, construct a resource competition intensity prediction model. This resource competition intensity prediction model can be a model based on machine learning algorithms, such as a neural network model or a decision tree model, etc., or it can be a simple mathematical formula model. Take the submission timestamp of the current task and the dependency path length matrix as input parameters and input them into this resource competition intensity prediction model. The resource competition intensity prediction model calculates and analyzes the input parameters and outputs an initial resource competition intensity prediction value. This initial resource competition intensity prediction value preliminarily reflects the resource competition intensity that the current task may face within the preset time window.

[0048] Finally, dynamically correct the initial resource competition intensity prediction value according to the real-time resource occupancy status of the computing nodes in the supercomputer center. The real-time resource occupancy status of the computing nodes in the supercomputer center can be obtained through a real-time monitoring system, including information such as the processor usage rate, memory free amount, and remaining storage space of each current computing node. Combine this real-time resource occupancy status information with the initial prediction value and correct the initial prediction value through an adjustment algorithm. For example, if the processor usage rate of the current computing nodes is generally high, indicating that the overall resource competition is relatively intense, then correspondingly increase the initial prediction value; conversely, if the resources of the computing nodes are relatively idle, then appropriately reduce the initial prediction value. After such dynamic correction, obtain the resource competition intensity prediction parameter after eliminating the node load deviation. This resource competition intensity prediction parameter more accurately reflects the resource competition intensity of the current task within the preset time window in the actual operating environment.

[0049] Step S130: Invoke the pre-trained task scheduling policy decision model to make a task scheduling policy decision on the task feature set, and generate the real-time scheduling priority parameter and the resource allocation constraint condition set of the computing task.

[0050] After completing the extraction of task dynamic features and generating a task feature set, the pre-trained task scheduling policy decision model is then called to formulate real-time scheduling priority parameters and a set of resource allocation constraint conditions for the computing tasks based on these task feature sets, so as to achieve reasonable and efficient task scheduling.

[0051] First, according to the estimated execution duration of the computing task, the processor occupancy rate change curve, the peak memory capacity requirement, and the storage space fluctuation characteristics are adaptively divided into non-equal-length time segments corresponding to the task execution stages, and the feature vectors of each non-equal-length time segment are concatenated into a multi-dimensional feature tensor along the channel dimension. Specifically, for the processor occupancy rate change curve, analyze its change trend and characteristics, and combine the estimated execution duration. According to the nodes where the curve changes significantly or according to set rules (such as changes in the curve slope, differences in resource requirements in different stages, etc.), divide it into several non-equal-length time segments. For example, if the curve changes gently in the early stage and fluctuates greatly in the later stage, the early stage may be divided into one segment, and the later stage may be further subdivided according to the fluctuation situation. For the peak memory capacity requirement, considering the memory usage of the task in different stages and combining the estimated execution duration, segment it in a similar way. The same applies to the storage space fluctuation characteristics. Then, for each non-equal-length time segment, convert its corresponding feature representation into a feature vector. For example, for a segment of the processor occupancy rate change curve, organize the occupancy rate values at different time points within this segment into a vector; for the peak memory capacity requirement segment, organize the peak-related features corresponding to this segment into a vector; for the storage space fluctuation characteristics segment, form a vector from the feature values reflecting the fluctuation situation of this segment. After that, the feature vectors of these non-equal-length time segments with different features are concatenated along the channel dimension. Suppose the feature vector of the processor occupancy rate change curve segment is vector A, the feature vector of the peak memory capacity requirement segment is vector B, and the feature vector of the storage space fluctuation characteristics segment is vector C. Then the concatenated multi-dimensional feature tensor is formed by arranging vectors A, B, and C in sequence along the channel dimension.

[0052] Step S131: In the multi-head attention mechanism layer of the task scheduling policy decision model, calculate the cross-segment correlation weights for the channel dimension of the multi-dimensional feature tensor to generate an attention weight distribution matrix.

[0053] Input the multi-dimensional feature tensor obtained by the above splicing into the multi-head attention mechanism layer of the task scheduling policy decision model. In this layer, cross-segment correlation weight calculation is performed for the channel dimension of the multi-dimensional feature tensor. The specific process is as follows: The multi-head attention mechanism layer contains multiple attention heads, and each attention head processes the multi-dimensional feature tensor independently. For each attention head, first project the multi-dimensional feature tensor into different subspaces through linear transformation to obtain the query (Query), key (Key), and value (Value) matrices. Here, the linear transformation is achieved by multiplying with a set weight matrix. For example, multiply the multi-dimensional feature tensor with the weight matrix W_Q to obtain the query matrix Q, multiply with the weight matrix W_K to obtain the key matrix K, and multiply with the weight matrix W_V to obtain the value matrix V. Then, calculate the dot product of the query matrix Q and the transpose of the key matrix K. This dot product operation is to measure the correlation between different segment features. After that, divide the dot product result by a scaling factor, which is to prevent the dot product result from being too large and causing gradient instability. Then, normalize the scaled result through the Softmax function to obtain the attention weight distribution. This attention weight distribution reflects the relative importance between different segment features. Finally, multiply the attention weight distribution with the value matrix V and accumulate to obtain the output of each attention head. Concatenate the outputs of all attention heads to obtain the result after being processed by the multi-head attention mechanism. In this process, the obtained attention weight distribution, organized in matrix form, is the attention weight distribution matrix, which reflects the correlation weights between different channel dimensions (i.e., different feature segments) of the multi-dimensional feature tensor.

[0054] Step S132: Dynamically weight and fuse the segment vectors of the multi-dimensional feature tensor according to the attention weight distribution matrix to generate a resource demand intensity encoding vector.

[0055] Use the attention weight distribution matrix generated in step S131 to dynamically weight and fuse each segment vector of the multi-dimensional feature tensor. The specific approach is to multiply each row element in the attention weight distribution matrix with the elements of the corresponding channel dimension (i.e., the corresponding segment vector) in the multi-dimensional feature tensor. For example, assume the first row elements of the attention weight distribution matrix are [w1, w2, w3], and the segment vectors of the corresponding channel dimension in the multi-dimensional feature tensor are vector A, vector B, and vector C respectively. Then the weighted vectors are w1 * vector A, w2 * vector B, and w3 * vector C respectively. Then, fuse these weighted vectors by adding them element-wise (here it is weighted addition because this method can better reflect the contribution of different segment features to the overall resource demand intensity) to obtain the resource demand intensity encoding vector. This resource demand intensity encoding vector comprehensively considers the correlation and importance between different task execution stages and different resource demand features, and encodes the intensity information of the task's resource demand.

[0056] Step S133: Input the dependency path length matrix into the graph convolutional network branch of the task scheduling policy decision model, extract the dependency depth feature vectors between task nodes along the matrix row direction, and extract the parallel constraint strength feature vectors along the matrix column direction.

[0057] Input the dependency path length matrix into the graph convolutional network branch of the task scheduling policy decision model. In the graph convolutional network branch, first in the adjacency matrix construction module, convert the non-zero elements of the dependency path length matrix into weighted edge weights to generate a trainable adjacency matrix. Specifically, for each non-zero element in the dependency path length matrix, convert its value into the weight of an edge in the graph structure according to the set conversion rule. For example, a simple linear conversion rule can be set to multiply the non-zero element value by a set coefficient to obtain the corresponding edge weight value. In this way, the dependency path length matrix is converted into an adjacency matrix that can be used for graph convolution operations, and the weights of this adjacency matrix reflect the strength of the dependency relationship between task nodes.

[0058] Next, perform neighborhood feature aggregation in the row direction on the trainable adjacency matrix through multi-layer graph convolution operations to generate depth propagation feature vectors. In the graph convolution operation, each layer of graph convolution processes the adjacency matrix and aggregates the neighborhood information of the nodes to the nodes. The specific operation is to perform a weighted sum of the feature information of the neighborhood nodes of each node and the feature information of the node itself, and the weights are determined by the edge weights in the adjacency matrix. After multiple such operations, along the matrix row direction, organize the finally aggregated feature information of each node into depth propagation feature vectors, which reflect the features related to the dependency depth between task nodes.

[0059] In the column direction parallel constraint analysis module, after transposing the dependency path length matrix, input it into the symmetric graph convolutional layer to extract the reverse propagation feature vectors. The transpose operation is to analyze the relationship between task nodes from another angle to obtain features related to parallel constraints. The operation of the symmetric graph convolutional layer is similar to the graph convolution operation in the previous row direction, but is performed on the transposed matrix. Similarly, through multi-layer graph convolution operations, aggregate the neighborhood information of the nodes, but here it is along the row direction of the transposed matrix (i.e., the column direction of the original matrix). The finally obtained reverse propagation feature vectors reflect the feature information of task nodes in the column direction (the direction related to parallel execution).

[0060] Perform the Hadamard product operation on the deep propagation feature vector and the backpropagation feature vector to generate a hybrid constraint feature tensor. The Hadamard product operation is element-wise multiplication, multiplying the elements at the corresponding positions in the deep propagation feature vector and the backpropagation feature vector to obtain the hybrid constraint feature tensor. This hybrid constraint feature tensor fuses the feature information of both the depth of dependence between task nodes and the parallel constraint.

[0061] Perform channel concatenation on the hybrid constraint feature tensor and the parallel execution feasibility parameter, and input it into a gated recurrent unit for temporal dependence relationship modeling to generate a constraint strength encoding matrix. Channel concatenation combines the parallel execution feasibility parameter as a new channel dimension with the hybrid constraint feature tensor in the channel dimension. Then, input the combined tensor into the gated recurrent unit. The gated recurrent unit models the temporal dependence relationship in the input tensor through its own structure. It controls the flow of information and updates the hidden state through a gating mechanism based on the current input and the hidden state at the previous moment. After being processed by the gated recurrent unit, a constraint strength encoding matrix is output, which further encodes the dependence and parallel constraint relationships between tasks.

[0062] In the output layer of the graph convolutional network branch, perform dual-path aggregation of max pooling and average pooling on the constraint strength encoding matrix to generate a dependence depth feature vector and a parallel constraint strength feature vector. The max pooling operation selects the maximum value as the representative value for each channel dimension in the constraint strength encoding matrix; the average pooling operation calculates the average value as the representative value for each channel dimension. Through these two pooling operations, two vectors are obtained respectively. One is the vector based on the max pooling result, which highlights the strongest features of each channel dimension in the constraint strength encoding matrix and serves as the dependence depth feature vector; the other is the vector based on the average pooling result, which reflects the average features of each channel dimension and serves as the parallel constraint strength feature vector. These two vectors respectively reflect the dependence depth and parallel constraint strength information between task nodes from different perspectives.

[0063] Step S134: Input the resource demand intensity encoding vector, the dependence depth feature vector, and the parallel constraint strength feature vector into a cross-gated fusion unit to generate a global task feature vector.

[0064] Input the resource demand intensity encoding vector, the dependence depth feature vector, and the parallel constraint strength feature vector into the cross-gated fusion unit. First, in the feature alignment module of the cross-gated fusion unit, perform a dimension expansion operation on the resource demand intensity encoding vector to make its dimension match that of the dependence depth feature vector. The way of dimension expansion can be to repeat elements or add set padding values on certain dimensions of the resource demand intensity encoding vector to achieve the same dimension as the dependence depth feature vector.

[0065] Next, a linear transformation is performed on the dependent depth feature vector through a learnable weight matrix to generate a topological constraint projection vector. This learnable weight matrix is continuously adjusted and optimized during the model training process, and its role is to project the dependent depth feature vector into a space more relevant to the resource demand intensity encoding vector. Through matrix multiplication, the dependent depth feature vector is multiplied by the weight matrix to obtain the topological constraint projection vector.

[0066] Calculate the element-wise product of the topological constraint projection vector and the resource demand intensity encoding vector to generate a coupled feature vector. Multiply the elements at the corresponding positions in the topological constraint projection vector and the resource demand intensity encoding vector to obtain the coupled feature vector, which integrates the feature information of resource demand and task topological dependence.

[0067] In the gating mechanism module, the parallel constraint intensity feature vector is input into the Sigmoid function to generate a channel selection gating vector. The Sigmoid function maps each element in the parallel constraint intensity feature vector to a value between 0 and 1, which can be regarded as a probability representing the importance degree of each channel dimension. The vector composed of these values is the channel selection gating vector.

[0068] Perform an element-wise multiplication operation on the coupled feature vector and the channel selection gating vector to generate a preliminary fusion feature vector. Through this multiplication operation, according to the importance degree represented by each element in the channel selection gating vector, the corresponding channel dimensions of the coupled feature vector are weighted to obtain the preliminary fusion feature vector.

[0069] Perform a residual connection on the preliminary fusion feature vector and the original resource demand intensity encoding vector, and input it into a multi-layer perceptron for non-linear feature enhancement to obtain an enhanced feature vector. The residual connection is to add the preliminary fusion feature vector and the original resource demand intensity encoding vector, retaining the original feature information while integrating new features. Then, the added result is input into the multi-layer perceptron. The multi-layer perceptron performs non-linear transformation on the input vector through multiple fully connected layers and non-linear activation functions to enhance the expression ability of the features and obtain the enhanced feature vector.

[0070] In the feature fusion output layer, perform layer normalization on the enhanced feature vector to generate a global task feature vector. The layer normalization operation normalizes the enhanced feature vector in the layer dimension, making the input of each neuron have a similar distribution, which helps to accelerate model training and improve the stability of the model. After the layer normalization process, the obtained vector is the global task feature vector, which integrates information such as resource demand, task dependence depth, and parallel constraint intensity, comprehensively characterizing the overall features of the task.

[0071] Step S135: Perform a non-linear mapping in the fully connected network of the policy decision layer based on the global task feature vector and the resource competition intensity prediction parameter, and output the initial scheduling priority parameter and the original decision vector of the resource allocation constraint condition set.

[0072] Input the global task feature vector and the resource competition intensity prediction parameter into the fully connected network of the policy decision layer. First, input the global task feature vector into the dual-channel processing structure of the resource allocation constraint branch and the priority prediction branch in the fully connected network.

[0073] In the resource allocation constraint branch, map the global task feature vector to a dimension matching the number of resource types of the computing node through three fully connected layers to generate a resource allocation baseline vector. Each fully connected layer performs a linear transformation on the input vector by multiplying it with a weight matrix and adding a bias term, and then introduces non-linearity through a non-linear activation function (such as the ReLU function). After three such processes, the global task feature vector is transformed to the dimension corresponding to the number of resource types of the computing node to obtain the resource allocation baseline vector, which preliminarily determines the allocation basis of the task for different types of resources.

[0074] Fuse the resource competition intensity prediction parameter and the resource allocation baseline vector element by element to generate a dynamic resource allocation original vector. The specific fusion method is to perform a certain operation (such as addition, multiplication, etc., and the specific operation method is determined according to the actual situation and model training) on each element in the resource competition intensity prediction parameter and the corresponding element in the resource allocation baseline vector to obtain the dynamic resource allocation original vector, which comprehensively considers the resource demand characteristics of the task and the predicted resource competition intensity and provides a more dynamic basis for resource allocation.

[0075] In the priority prediction branch, perform a non-linear transformation on the global task feature vector through a multi-layer fully connected network to generate an initial priority score. The multi-layer fully connected network also performs multiple non-linear transformations on the global task feature vector through operations of multiplying with a weight matrix, adding a bias term, and a non-linear activation function, mines the feature information related to the task priority therein, and finally outputs a scalar value, that is, the initial priority score, which preliminarily reflects the scheduling priority of the task.

[0076] Generate the adjusted priority parameter based on the parallel execution feasibility parameter and the initial priority score. Here, a method needs to be designed to incorporate the parallel execution feasibility parameter into the initial priority score to obtain a more reasonable priority parameter. For example, through a function, the parallel execution feasibility parameter can be used as one of the inputs and operated on with the initial priority score to obtain the adjusted priority parameter. Among them, the design of this function needs to consider the impact of parallel execution on task priority. For example, the higher the parallel execution feasibility, the priority may be adjusted higher or lower accordingly, which is specifically determined according to the actual situation and model training.

[0077] In the multi-objective optimization module of the policy decision layer, concatenate the dynamic resource allocation original vector and the adjusted priority parameter into a joint vector, and input it into the policy projection network to generate the original decision vector of the initial scheduling priority parameter and the set of resource allocation constraint conditions. The joint vector concatenation is to connect the dynamic resource allocation original vector and the adjusted priority parameter into a longer vector in a set order. Then, input the concatenated vector into the policy projection network. The policy projection network maps the input vector to the original decision vector space of the initial scheduling priority parameter and the set of resource allocation constraint conditions through a series of linear and non-linear transformations, obtaining the original decision vector of the initial scheduling priority parameter and the set of resource allocation constraint conditions, providing a basis for generating the final scheduling policy.

[0078] Step S136: Perform probability normalization processing and compliance verification of constraint conditions on the original decision vector to generate the real-time scheduling priority parameter and the set of resource allocation constraint conditions.

[0079] Perform probability normalization processing on the original decision vector output by the policy decision layer. Specifically, through a set normalization method (such as the Softmax function), convert each element in the original decision vector into a probability value so that the sum of these probability values is 1. The purpose of this is to convert the numerical values in the original decision vector into a representation with probability significance to more reasonably determine the scheduling priority of tasks. For example, assume that there are multiple elements in the original decision vector, representing the scheduling priority-related scores of different tasks. After being processed by the Softmax function, these scores are converted into the relative scheduling priority probabilities of each task among all tasks.

[0080] Next, perform constraint compliance verification. For the original decision vector part of the resource allocation constraint set, check whether it meets the actual resource limitations of the computing nodes in the supercomputer center and the basic requirements for task execution. For example, check whether the quantity of resource allocation is within the available resources of the computing node, and whether the demand of a task for a certain resource complies with the usage rules of that resource. If a situation that does not meet the constraints is found, adjust the original decision vector according to the preset correction rules. For example, if the number of processor cores allocated to a certain task exceeds the available cores of the computing node, then reduce the allocated number of processor cores for this task according to the set ratio, and at the same time adjust other relevant parameters accordingly to ensure that the entire resource allocation constraint set meets the compliance requirements.

[0081] After probability normalization processing and constraint compliance verification, the real-time scheduling priority parameters and the resource allocation constraint set are obtained. The real-time scheduling priority parameters accurately reflect the relative scheduling priorities of each computing task under the current state of the supercomputer center, and the resource allocation constraint set ensures that the resource allocation of tasks not only conforms to the actual situation of the supercomputer center but also meets the task execution requirements, providing a reliable basis for subsequent dynamic resource allocation and task scheduling operations.

[0082] Step S140: Generate a set of dynamic resource allocation instructions for the computing nodes of the supercomputer center based on the real-time scheduling priority parameters and the resource allocation constraint set.

[0083] After obtaining the real-time scheduling priority parameters and the resource allocation constraint set, the next step is to generate a set of dynamic resource allocation instructions for the computing nodes of the supercomputer center based on this information to achieve reasonable resource allocation for computing tasks.

[0084] First, perform a global descending order sorting on the current computing tasks to be processed according to the real-time scheduling priority parameters to generate a priority task sequence. The specific sorting process is to associate each computing task with its corresponding real-time scheduling priority parameter, and then arrange all tasks in descending order of the real-time scheduling priority parameter. For example, for tasks A, B, and C, their corresponding real-time scheduling priority parameters are P_A, P_B, and P_C respectively. If P_A > P_B > P_C, the sorted priority task sequence is task A, task B, and task C. Thus, a task sequence sorted from high to low in priority is obtained, providing a basis for the task processing order for subsequent resource allocation.

[0085] Step S141: Obtain the real-time resource status parameters of each computing node, including the remaining number of processor cores, the available memory capacity, and the free ratio of storage bandwidth.

[0086] Through the resource monitoring system within the supercomputing center, the resource status parameters of each computing node are obtained in real time. For each computing node, the remaining number of processor cores, that is, the number of processor cores not currently occupied by other tasks, is obtained respectively; the available memory capacity, which refers to the size of the memory currently available for allocation to new tasks on the computing node; and the free ratio of the storage bandwidth, which represents the proportion of the unoccupied part in the current storage bandwidth of the computing node. For example, for computing node 1, the remaining number of processor cores is X1, the available memory capacity is Y1, and the free ratio of the storage bandwidth is Z1; for computing node 2, the corresponding parameters are X2, Y2, Z2, and so on. These real-time resource status parameters of all computing nodes in the supercomputing center are obtained to provide basic data for subsequent resource allocation matching.

[0087] Step S142: Sequentially match and verify the set of resource allocation constraint conditions for each task in the priority task sequence with the real-time resource status parameters, and filter out the computing nodes that meet the matching conditions to generate a candidate node set.

[0088] For the first task in the priority task sequence, extract the set of resource allocation constraint conditions. This set includes the specific constraint requirements of this task for resources such as the number of processor cores, memory capacity, and storage bandwidth. For example, the constraint on the number of processor cores may specify a range, there are corresponding lower limits or specific requirements for the memory capacity, and certain conditions are also set for the storage bandwidth.

[0089] Compare these resource allocation constraint conditions with the real-time resource status parameters of each computing node in the supercomputing center one by one. For the remaining number of processor cores of the computing node, check whether it is within the range of the number of processor cores required by the task; for the available memory capacity, check whether it meets the lower limit requirement or specific requirements of the task for the memory capacity; for the free ratio of the storage bandwidth, confirm whether it meets the conditions set by the task for the storage bandwidth.

[0090] If all the real-time resource status parameters of a certain computing node meet the resource allocation constraint conditions of this task, then this computing node is included in the candidate node set. For each task in the priority task sequence, repeat the above operation, and sequentially match and verify the set of its resource allocation constraint conditions with the real-time resource status parameters of all computing nodes, so as to filter out the computing nodes that meet the matching conditions for each task and form the corresponding candidate node set.

[0091] Step S143: Calculate the resource allocation benefits of the computing nodes in the candidate node set, sort the computing nodes according to the calculation results of the resource allocation benefits, and select the node with the highest resource allocation benefit as the target node.

[0092] For each computing node in the candidate node set, resource allocation benefits need to be calculated. The calculation of resource allocation benefits needs to consider multiple factors. On the one hand, consider the utilization efficiency of the computing node's resources when the task is executed on this computing node. For example, if the task's demand for processor cores matches well with the remaining number of processor cores in the computing node and can make full use of the computing node's memory and storage bandwidth resources, this indicates a high resource utilization efficiency. On the other hand, consider the impact of the computing node's execution of this task on the overall resource balance of the supercomputer center system. If allocating the task to this computing node can make the overall resource utilization rate of the supercomputer center more balanced, avoiding over-idle or over-busy of some resources, then this has a positive effect on the system resource balance.

[0093] Factors such as resource utilization efficiency and the impact on system resource balance are used to obtain the resource allocation benefit value through a preset calculation method. For example, set a calculation formula that includes the weight of resource utilization efficiency and the weight of the impact on system resource balance, and calculate according to the formula after quantifying each factor. Suppose the quantified value of resource utilization efficiency is A, its weight is a, the quantified value of the impact on system resource balance is B, and its weight is b, then the resource allocation benefit value C = a * A + b * B.

[0094] After calculating the resource allocation benefit values for all computing nodes in the candidate node set, sort these computing nodes in descending order of the benefit values. The computing node ranked first, that is, the computing node with the highest resource allocation benefit, is selected as the target node. This target node is the computing node most suitable for allocating resources for the current task, which can maximize the resource utilization efficiency and maintain a good balance of the supercomputer center system resources.

[0095] Step S144: Dynamically bind the computing task to the resource slots of the target node to generate a corresponding resource allocation instruction unit.

[0096] After determining the target node, it is necessary to dynamically bind the current computing task to the resource slots of the target node. The target node has resource slots corresponding to different types of resources, such as processor core resource slots, memory resource slots, and storage bandwidth resource slots, etc.

[0097] According to the set of resource allocation constraint conditions of the task, map the number of processor cores required by the task to the processor core resource slots of the target node to ensure that the task can obtain the corresponding number of processor core resources; match the task's demand for memory capacity to the memory resource slots of the target node to ensure that the supply of memory resources meets the task requirements; correspond the task's demand for storage bandwidth to the storage bandwidth resource slots of the target node to make sure there is sufficient storage bandwidth available during the task execution.

[0098] In this way, the computing tasks are precisely matched and bound to various resource slots of the target node, generating corresponding resource allocation instruction units. These resource allocation instruction units record in detail the binding relationships between tasks and the resource slots of the target node. For example, they clearly indicate which processor cores of the target node are used by the task, how much memory capacity is occupied, and how much storage bandwidth is allocated, etc., providing an accurate instruction basis for the subsequent task scheduling operations of the computing nodes in the supercomputer center.

[0099] Step S145: Traverse all computing tasks and repeat steps S142 to S144 to finally generate a dynamic resource allocation instruction set composed of multiple resource allocation instruction units.

[0100] Starting from the first task in the priority task sequence, each computing task is processed in turn according to the procedures of steps S142 to S144. After generating the resource allocation instruction unit for one task, the next task is then processed.

[0101] For each task, it is necessary to go through the matching and verification with the real-time resource status parameters, filter out the candidate node set, calculate and sort the resource allocation benefits of the candidate node set to select the target node, and then dynamically bind the task to the resource slots of the target node to generate a resource allocation instruction unit.

[0102] After traversing all the computing tasks in the priority task sequence, multiple resource allocation instruction units will be obtained. These resource allocation instruction units are arranged in sequence according to the order of the tasks in the priority task sequence, jointly forming a dynamic resource allocation instruction set. This dynamic resource allocation instruction set comprehensively covers the resource allocation instruction information of the computing tasks to be processed currently in the supercomputer center, providing a complete and orderly guidance for the task scheduling operations of the computing nodes in the supercomputer center, enabling the supercomputer center to reasonably and efficiently allocate resources for each computing task based on these dynamic resource allocation instructions, and thus realizing the dynamic scheduling of computing tasks.

[0103] Step S150: Control the computing nodes in the supercomputer center to execute task scheduling operations according to the dynamic resource allocation instruction set, and update the resource occupancy status and the execution progress of the task queue in real time.

[0104] After the task execution control system of the supercomputer center obtains the dynamic resource allocation instruction set, it sends task scheduling instructions to the corresponding computing nodes according to the resource allocation instruction units therein. Each resource allocation instruction unit details the binding relationship between a specific computing task and the resource slots of the target computing node.

[0105] After receiving a task scheduling instruction, the computing node allocates corresponding resources to the specified computing task according to the instruction requirements and starts task execution. For example, the computing node allocates a set number of processor cores for the task according to the instruction, divides the corresponding memory area for the task to use, and allocates a set proportion of storage bandwidth resources.

[0106] During the task execution process, the resource monitoring system of the supercomputer center monitors the resource occupancy status of the computing node in real time. For processor cores, it monitors which cores are being used by the task and the degree of usage; for memory, it tracks the actual memory occupancy of the task; for storage bandwidth, it monitors the usage of storage bandwidth by the task during data transmission.

[0107] At the same time, the task execution control system updates the execution progress of the task queue in real time. It records the execution status of each task on the computing node, such as whether the task has been started, whether it is running, whether it has encountered an error, etc. For completed tasks, it removes them from the task queue; for tasks being executed, it continuously tracks their execution progress. For example, according to the execution steps of the task or preset progress marking points, it calculates the proportion of the task that has been completed.

[0108] In this way, the supercomputer center can accurately control the computing node to perform task scheduling operations according to the dynamic resource allocation instruction set, and update the resource occupancy status of the computing node and the execution progress of the task queue in real time, realizing dynamic and efficient management of computing tasks, and ensuring the reasonable utilization of the overall resources of the supercomputer center and the smooth execution of tasks.

[0109] Furthermore, before using the task scheduling strategy decision model to make task scheduling strategy decisions, it needs to be trained so that it can accurately generate reasonable real-time scheduling priority parameters and resource allocation constraint condition sets according to the task feature set.

[0110] First, collect the historical task data of the supercomputer center. These historical task data of the supercomputer center contain various information during the task execution process, such as task identifiers, task submission timestamps, task resource requirement descriptions, task interdependency descriptions, actual task execution durations, actual resource allocation situations, and task execution results, etc. Extract the feature data related to model training from these historical task data to form a training dataset.

[0111] For the description of task resource requirements, extract the curve of the processor occupancy rate change, the peak value of the memory capacity requirement, the storage space fluctuation characteristics, etc.; for the description of the dependencies between tasks, construct the topological structure of task nodes, and extract the dependency path length matrix, the parallel execution feasibility parameters, etc.; at the same time, combine the task submission timestamp and the historical task execution records of the supercomputer center to generate the resource competition intensity prediction parameters. Use these feature data and the corresponding real-time scheduling priority parameters and the set of resource allocation constraint conditions as training data pairs, where the real-time scheduling priority parameters and the set of resource allocation constraint conditions are manually labeled according to the actual execution of historical tasks or generated by existing reasonable scheduling strategies.

[0112] The task scheduling strategy decision model includes multiple key modules, mainly including the multi-head attention mechanism layer, the graph convolutional network branch, the cross-gating fusion unit, and the fully connected network in the strategy decision layer, etc.

[0113] The multi-head attention mechanism layer is used to calculate the cross-segment correlation weights for the channel dimension of the multi-dimensional feature tensor, and capture the correlation information between different task feature segments; the graph convolutional network branch processes the dependency path length matrix, extracts the dependency depth feature vectors between task nodes along the row direction of the matrix, and extracts the parallel constraint intensity feature vectors along the column direction of the matrix; the cross-gating fusion unit fuses the resource requirement intensity encoding vector, the dependency depth feature vector, and the parallel constraint intensity feature vector to generate the global task feature vector; the fully connected network in the strategy decision layer, based on the global task feature vector and the resource competition intensity prediction parameters, outputs the original decision vectors of the initial scheduling priority parameters and the set of resource allocation constraint conditions through non-linear mapping.

[0114] Data is transmitted and interacted between each module through the set connection relationship to realize the gradual processing and analysis of task features, and finally generate the task scheduling strategy decision result.

[0115] On this basis, determine various parameters involved in the training process of the task scheduling strategy decision model. For example, determine the number of attention heads in the multi-head attention mechanism layer, which will affect the attention ability of the task scheduling strategy decision model to different feature dimensions; set the number of graph convolutional layers in the graph convolutional network branch, and the number of layers will affect the extraction depth of the task node topological structure features by the model; set the initial values of the learnable weight matrices in the cross-gating fusion unit, and these initial values will affect the starting state of feature fusion; determine the number of neurons in each layer of the fully connected network in the strategy decision layer, and the number of neurons determines the complexity and expressive ability of the network.

[0116] Meanwhile, set the optimizer parameters during the training process, such as selecting a suitable optimization algorithm (such as Stochastic Gradient Descent, Adagrad, Adadelta, etc.), and determining parameters such as the learning rate and momentum of the optimization algorithm. The learning rate controls the step size of each parameter update, and momentum helps to accelerate convergence and avoid getting stuck in local optimal solutions.

[0117] Thus, the prepared training dataset can be input into the constructed task scheduling policy decision model. In each round of training, the task scheduling policy decision model processes the input task feature data according to the current parameter settings, and generates an original decision vector of the predicted real-time scheduling priority parameters and the set of resource allocation constraint conditions.

[0118] Compare the prediction result with the corresponding annotation result in the training data (i.e., the actual real-time scheduling priority parameters and the set of resource allocation constraint conditions), and calculate the difference between the prediction result and the annotation result through a loss function. Commonly used loss functions include mean squared error loss function, cross-entropy loss function, etc., and select a suitable loss function according to the nature and requirements of the task.

[0119] Based on the calculation result of the loss function, calculate the gradients of each parameter in the model through the backpropagation algorithm. The gradient represents the rate of change of the loss function with respect to each parameter. Then, according to the parameter settings of the optimizer, use the gradients to update the parameters of the model, so that the value of the loss function gradually decreases.

[0120] Repeat the above process for multiple rounds of training until the value of the loss function converges to a small range or reaches the preset number of training rounds. At this time, the task scheduling policy decision model has completed training and can be used for actual task scheduling policy decision-making.

[0121] Furthermore, in the process of the supercomputing center obtaining and processing the computing task request data set and historical task data, issues related to data privacy protection and anti-disclosure are involved. In the data collection phase, for computing task request data that may contain privacy-sensitive information, anonymization processing methods are adopted. For example, for privacy data such as the identity information of task submitters, it is converted into an anonymous identifier through a set encryption algorithm, so that the true identity of the task submitter cannot be directly identified from the collected data. At the same time, the collected data is classified and marked to clarify which data belongs to privacy-sensitive data and which belongs to general data. In terms of data storage, privacy-sensitive data and general data are stored separately. For privacy-sensitive data, encryption storage technology is adopted, and appropriate encryption algorithms (such as AES, RSA, etc.) are selected to encrypt the data to ensure that even if the data storage medium is illegally obtained, the original data cannot be decrypted and obtained without authorization. At the same time, strict access permission control is set, and only specific system modules or personnel authorized can access privacy-sensitive data, and detailed records of access operations are made for auditing and traceability. In the data processing process, for computing operations involving privacy-sensitive data, homomorphic encryption technology or secure multi-party computing technology is adopted. Homomorphic encryption technology allows specific computing operations to be performed on encrypted data, and the calculation results are the same as those obtained by performing the same calculations on plaintext data after decryption, so as to complete data processing without revealing data privacy. Secure multi-party computing technology allows multiple participating parties to jointly complete computing tasks without revealing their respective privacy data.

[0122] For example, in the process of generating the task feature set, if the feature extraction of privacy-sensitive data is involved, the above technologies are used to ensure data privacy and security. When training the task scheduling strategy decision model, if privacy-sensitive data is used, these technologies are also adopted to ensure that the privacy of the data is not leaked during the training process. Through these data privacy protection and anti-disclosure measures, it is ensured that when the supercomputing center processes data related to computing tasks, it can not only make full use of the data for efficient task scheduling, but also ensure data privacy and security and avoid the risk of data leakage.

[0123] Figure 2 FIG. shows a schematic diagram of exemplary hardware and software components of an artificial intelligence-based supercomputing center computing task dynamic scheduling system 100 that can implement the ideas of the present application. For example, the processor 120 can be used on the artificial intelligence-based supercomputing center computing task dynamic scheduling system 100 and is used to execute the functions in the present application.

[0124] The AI-based dynamic scheduling system 100 for computing tasks in a supercomputing center can be a general-purpose server or a special-purpose server, both of which can be used to implement the AI-based dynamic scheduling method for computing tasks in a supercomputing center of the present application. Although only one server is shown in the present application, for convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0125] For example, the AI-based dynamic scheduling system 100 for computing tasks in a supercomputing center may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as disks, ROMs, or RAMs, or any combination thereof. Exemplarily, the AI-based dynamic scheduling system 100 for computing tasks in a supercomputing center may also include program instructions stored in a ROM, a RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The AI-based dynamic scheduling system 100 for computing tasks in a supercomputing center further includes an I / O interface 150 between the computer and other input / output devices.

[0126] For ease of explanation, only one processor is described in the AI-based dynamic scheduling system 100 for computing tasks in a supercomputing center. However, it should be noted that the AI-based dynamic scheduling system 100 for computing tasks in a supercomputing center in the present application may also include multiple processors. Therefore, the steps executed by one processor described in the present application can also be jointly executed or separately executed by multiple processors. For example, if the processor of the AI-based dynamic scheduling system 100 for computing tasks in a supercomputing center executes steps A and B, it should be understood that steps A and B can also be jointly executed by two different processors or separately executed in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor jointly execute steps A and B.

[0127] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When the processor executes the computer-executable instructions, the above-mentioned AI-based dynamic scheduling method for computing tasks in a supercomputing center is implemented.

[0128] It should be noted that, in order to simplify the presentation of the disclosure of the present invention and thus help the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, sometimes multiple features are merged into one embodiment, drawing, or description thereof.

Claims

1. A dynamic scheduling method for computing tasks in a supercomputing center based on artificial intelligence, characterized in that, The method includes: Obtaining a set of computing task request data to be processed currently in the supercomputing center, where the set of computing task request data includes a computing task identifier, a task submission timestamp, a task resource requirement description, and a task interdependency relationship description; Performing task dynamic feature extraction processing on the set of computing task request data to generate a set of task features for each computing task; Invoking a pre-trained task scheduling policy decision model to make a task scheduling policy decision on the set of task features, generating real-time scheduling priority parameters for the computing tasks and a set of resource allocation constraint conditions; Generating a set of dynamic resource allocation instructions for the computing nodes in the supercomputing center based on the real-time scheduling priority parameters and the set of resource allocation constraint conditions; Controlling the computing nodes in the supercomputing center to perform task scheduling operations according to the set of dynamic resource allocation instructions, and updating the resource occupancy status of the computing nodes and the execution progress of the task queue in real time.

2. The dynamic scheduling method for computing tasks in a supercomputing center based on artificial intelligence according to claim 1, characterized in that, The performing task dynamic feature extraction processing on the set of computing task request data to generate a set of task features for each computing task includes: Parsing the task resource requirement description in the set of computing task request data, and extracting the processor occupancy rate change curve, the peak memory capacity requirement, and the storage space fluctuation characteristics during the processing of the computing task; Analyzing the task node topology structure in the task interdependency relationship description, and constructing a dependency path length matrix and a parallel execution feasibility parameter between task nodes; Generating a resource competition intensity prediction parameter for the computing task within a preset time window based on the task submission timestamp and the historical task execution records of the supercomputing center; Performing feature normalization processing on the processor occupancy rate change curve, the peak memory capacity requirement, the storage space fluctuation characteristics, the dependency path length matrix, the parallel execution feasibility parameter, and the resource competition intensity prediction parameter, and inputting the normalized feature parameters into a feature fusion network based on the multi-head attention mechanism to achieve cross-dimensional correlation analysis through cross-channel weight allocation, and generating a set of task features.

3. The dynamic scheduling method for computing tasks of a supercomputing center based on artificial intelligence according to claim 2, wherein The analyzing the task node topology structure in the task interdependency relationship description and constructing a dependency path length matrix and a parallel execution feasibility parameter between task nodes includes: Traversing each task node in the task node topology structure, marking task node pairs with direct dependency relationships and generating a set of dependency edges; Constructing a task node topology graph according to the set of dependency edges, and calculating the shortest dependency path length from each task node to all its downstream nodes and the maximum number of parallel branches; Generating a dependency path length matrix containing the dependency depth weights between task nodes based on the shortest dependency path length, where the dependency depth weights are positively correlated with the shortest dependency path length; Counting the number of task node pairs without direct dependency relationships in the task node topology graph, and combining the maximum number of parallel branches and the processor occupancy overlap parameter in the task resource requirement description to calculate the parallel execution conflict probability parameter of resource competition perception between task nodes; Generate a parallel execution feasibility parameter that reflects the resource competition intensity of task parallel execution according to the processor occupancy change curve in the parallel execution conflict probability parameter and task resource requirement description.

4. The dynamic scheduling method for computing tasks in a supercomputing center based on artificial intelligence according to claim 2, wherein, Generate a resource competition intensity prediction parameter for the computing task within a preset time window based on the task submission timestamp and the historical task execution record of the supercomputer center, including: Extract a subset of historical task execution data that matches the current task type from the historical task execution record of the supercomputer center. The subset of historical task execution data includes the historical task submission timestamp, the resource occupancy curve, and the task execution duration. Match the resource occupancy curve of the historical same period according to the current task submission timestamp, and extract the average processor occupancy and the peak memory usage fluctuation parameter within the same time window. Analyze the task execution records of tasks with similar dependency relationship complexity in the subset of historical task execution data, and statistically calculate the correlation function between the task startup delay duration and the dependency path length matrix. Construct a resource competition intensity prediction model based on the average processor occupancy, the peak memory usage fluctuation parameter, and the correlation function, and input the current task submission timestamp and the dependency path length matrix to generate an initial resource competition intensity prediction value. Dynamically correct the initial resource competition intensity prediction value according to the real-time resource occupancy status of the computing nodes in the supercomputer center, and generate a resource competition intensity prediction parameter after eliminating the node load deviation.

5. The dynamic scheduling method for computing tasks in a supercomputer center based on artificial intelligence according to claim 2, wherein Call the pre-trained task scheduling policy decision model to make a task scheduling policy decision on the task feature set, and generate the real-time scheduling priority parameter and the resource allocation constraint condition set for the computing task, including: According to the estimated execution duration of the computing task, adaptively divide the processor occupancy change curve, the peak memory capacity requirement, and the storage space fluctuation characteristics into non-equal-length time segments corresponding to the task execution stage, and splice the feature vectors of each non-equal-length time segment along the channel dimension into a multi-dimensional feature tensor. In the multi-head attention mechanism layer of the task scheduling policy decision model, calculate the cross-segment correlation weight for the channel dimension of the multi-dimensional feature tensor to generate an attention weight distribution matrix. Dynamically weight and fuse the segment vectors of the multi-dimensional feature tensor according to the attention weight distribution matrix to generate a resource requirement intensity encoding vector. Input the dependency path length matrix into the graph convolutional network branch of the task scheduling policy decision model, extract the dependency depth feature vector between task nodes along the row direction of the matrix, and extract the parallel constraint intensity feature vector along the column direction of the matrix. Input the resource requirement intensity encoding vector, the dependency depth feature vector, and the parallel constraint intensity feature vector into the cross-gating fusion unit to generate a global task feature vector. Perform a non-linear mapping in the fully connected network of the policy decision layer based on the global task feature vector and the resource competition intensity prediction parameter, and output the original decision vector of the initial scheduling priority parameter and the resource allocation constraint condition set. Perform probability normalization processing and compliance verification of the constraint conditions on the original decision vector to generate the real-time scheduling priority parameter and the resource allocation constraint condition set.

6. The dynamic scheduling method for computing tasks of a supercomputing center based on artificial intelligence according to claim 5, wherein, Inputting the dependency path length matrix into the graph convolutional network branch of the task scheduling policy decision model, and extracting the dependency depth feature vector among task nodes along the matrix row direction and the parallel constraint strength feature vector along the matrix column direction includes: In the adjacency matrix construction module of the graph convolutional network branch, converting the non-zero elements of the dependency path length matrix into weighted edge weights to generate a trainable adjacency matrix; Performing neighborhood feature aggregation in the row direction on the trainable adjacency matrix through multi-layer graph convolutional operations to generate a depth propagation feature vector; In the column direction parallel constraint analysis module, transposing the dependency path length matrix and inputting it into a symmetric graph convolutional layer to extract a backward propagation feature vector; Performing a Hadamard product operation on the depth propagation feature vector and the backward propagation feature vector to generate a mixed constraint feature tensor; Concatenating the mixed constraint feature tensor and the parallel execution feasibility parameter in the channel dimension, and inputting it into a gated recurrent unit for temporal dependency relationship modeling to generate a constraint strength encoding matrix; In the output layer of the graph convolutional network branch, performing dual-path aggregation of max pooling and average pooling on the constraint strength encoding matrix to generate a dependency depth feature vector and a parallel constraint strength feature vector.

7. The dynamic scheduling method for computing tasks in a supercomputing center based on artificial intelligence according to claim 6, characterized in that, Inputting the resource demand intensity encoding vector, the dependency depth feature vector, and the parallel constraint strength feature vector into a cross-gated fusion unit to generate a global task feature vector includes: In the feature alignment module of the cross-gated fusion unit, performing a dimension expansion operation on the resource demand intensity encoding vector to match its dimension with that of the dependency depth feature vector; Performing a linear transformation on the dependency depth feature vector through a learnable weight matrix to generate a topological constraint projection vector; Calculating the element-wise product of the topological constraint projection vector and the resource demand intensity encoding vector to generate a coupled feature vector; In the gating mechanism module, inputting the parallel constraint strength feature vector into a Sigmoid function to generate a channel selection gating vector; Performing an element-wise multiplication operation on the coupled feature vector and the channel selection gating vector to generate a preliminary fusion feature vector; Performing a residual connection on the preliminary fusion feature vector and the original resource demand intensity encoding vector, and inputting it into a multi-layer perceptron for non-linear feature enhancement to obtain an enhanced feature vector; In the feature fusion output layer, performing layer normalization on the enhanced feature vector to generate a global task feature vector.

8. The method for dynamically scheduling computing tasks in a supercomputer center based on artificial intelligence according to claim 7, wherein Performing a non-linear mapping in the fully connected network of the policy decision layer based on the global task feature vector and the resource competition intensity prediction parameter, and outputting an initial scheduling priority parameter and an original decision vector of the resource allocation constraint condition set, including: Inputting the global task feature vector into a dual-path processing structure of a resource allocation constraint branch and a priority prediction branch in the fully connected network respectively; In the resource allocation constraint branch, mapping the global task feature vector to a dimension matching the number of resource types of the computing nodes through three layers of fully connected layers to generate a resource allocation baseline vector; Fuse the resource competition intensity prediction parameter and the resource allocation baseline vector element by element to generate a dynamic resource allocation original vector; In the priority prediction branch, perform a non-linear transformation on the global task feature vector through a multi-layer fully connected network to generate an initial priority score; Generate an adjusted priority parameter according to the parallel execution feasibility parameter and the initial priority score; In the multi-objective optimization module of the policy decision layer, perform a joint vector splicing on the dynamic resource allocation original vector and the adjusted priority parameter, and input the policy projection network to generate the original decision vector of the initial scheduling priority parameter and the resource allocation constraint condition set.

9. The dynamic scheduling method for computing tasks in a supercomputing center based on artificial intelligence according to claim 1, characterized in that Generating a dynamic resource allocation instruction set for the computing nodes of the supercomputer center based on the real-time scheduling priority parameter and the resource allocation constraint condition set, including: Perform a global descending order sorting on the currently to-be-processed computing tasks according to the real-time scheduling priority parameter to generate a priority task sequence; Obtain the real-time resource status parameters of each computing node, including the remaining number of processor cores, the available memory capacity, and the storage bandwidth idle ratio; Match and verify the resource allocation constraint condition set of each task in the priority task sequence with the real-time resource status parameters in turn, and filter out the computing nodes that meet the matching conditions to generate a candidate node set; Calculate the resource allocation benefit of the computing nodes in the candidate node set, and sort the computing nodes according to the resource allocation benefit calculation result, and select the node with the highest resource allocation benefit as the target node; Dynamically bind the computing task to the resource slot of the target node to generate a corresponding resource allocation instruction unit; Traverse all computing tasks and repeat the above steps to finally generate a dynamic resource allocation instruction set composed of multiple resource allocation instruction units.

10. A dynamic scheduling system for computing tasks in a supercomputing center based on artificial intelligence, characterized in that, Comprising a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the dynamic scheduling method for computing tasks of the supercomputer center based on artificial intelligence according to any one of claims 1-9 above.

Images