A GPU cluster-oriented computing power resource dynamic scheduling method

Abstract

The application provides a GPU cluster-oriented computing power resource dynamic scheduling method, comprising the following steps: collecting current power quotas and memory occupation scales of each GPU computing card of a GPU cluster, and extracting running features of a non-sensitive task to obtain first resource situation data; inputting the first resource situation data into a random forest regression model previously established on a GPU cluster management node to output an estimated remaining duration of the non-sensitive task; adding the estimated remaining duration to a current system time, and combining historical running records of the non-sensitive task in the GPU cluster to determine a memory release time of the non-sensitive task; and calculating a time interval between the memory release time of the non-sensitive task and an expected arrival time of a to-be-scheduled sensitive task to obtain a first time delay result.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a method for dynamic scheduling of computing resources for GPU clusters. Background Technology

[0002] Dynamic scheduling of computing resources for graphics processing unit (GPU) clusters is the core of ensuring the efficient operation of large-scale computing tasks. In conventional resource allocation strategies, administrators often focus on the immediate throughput of computing units, habitually allocating limited power quotas to time-sensitive tasks, attempting to complete computations quickly and release the entire GPU resource by running these tasks at near full power. This approach ignores the misalignment of resource usage cycles among different types of tasks, leading to a stalemate in the overall cluster scheduling. When a non-sensitive task is compressed to a power level far below its steady-state operation requirements under a power allocation strategy that prioritizes sensitive tasks, it is almost in a semi-dormant state in terms of computing power. The time taken for a single iteration is multiplied, but it still occupies all the originally allocated GPU memory space. The GPU memory will not automatically shrink or be returned early because the task is running slowly. At this time, the sensitive task has already completed all computations at full power, freeing up a large number of idle computing cores, while the power-suppressed non-sensitive task is still slowly progressing step by step, still holding onto the GPU memory it originally allocated, which has not yet been returned to the allocable pool. In a hybrid cluster simultaneously handling emergency weather forecasting and routine image recognition, the system drastically reduced the power allocation for image recognition tasks to ensure the weather forecasting task could complete calculations and output valid forecast conclusions before the weather forecast window closed. The weather forecasting task finished instantly, freeing up a large number of computing cores. However, the GPU core operating frequency for the image recognition task decreased accordingly after the power allocation was drastically reduced, and the iteration time per batch increased exponentially, essentially becoming slow-motion, with the large amount of GPU memory it required constantly occupied. At this point, as the cluster continuously received real-time computing requests from various business units, a new emergency task arrived, such as a temporarily inserted real-time disaster image analysis request. The scheduling system found that although there were sufficient computing cores idle in the cluster, the available GPU memory was insufficient to support the new task's model and batch data, preventing the new task from starting and forcing it to wait in the queue. This resulted in a misallocation of GPU cluster computing resources. Summary of the Invention

[0003] This invention provides a method for dynamic scheduling of computing resources for GPU clusters, mainly including: Collect the current power quota and memory usage of each GPU computing card in the GPU cluster, and extract the running characteristics of non-sensitive tasks to obtain the first resource status data; The first resource status data is input into a random forest regression model pre-built on the GPU cluster management node, and the estimated remaining time of non-sensitive tasks is output. Add the estimated remaining time to the current system time, and combine this with the historical running records of non-sensitive tasks in the GPU cluster to determine the time when the non-sensitive tasks release their GPU memory. Calculate the time interval between the memory release time of the non-sensitive task and the expected arrival time of the sensitive task to be scheduled, and obtain the first delay result; By analyzing the first latency result and the preset threshold using the latency evaluation model pre-set in the GPU cluster scheduler, if the first latency result exceeds the preset threshold, it is identified that non-sensitive tasks are excessively suppressing and occupying cards. Based on this, the tilt of the power quota of non-sensitive tasks towards sensitive tasks is narrowed to obtain the second power quota. The power of the computing cards in the GPU cluster is reallocated according to the second power quota, and the memory reclamation of the computing card is triggered based on the memory release time to obtain a set of available computing cards. The number of computing cards and the available memory in the set of available computing cards are counted to determine the actual scheduling capacity of the current GPU cluster. Computing cards are matched from the set of available computing cards according to the memory requirements of the sensitive tasks to be scheduled, and the scheduling of sensitive tasks on the cards is completed according to the expected arrival time.

[0004] Furthermore, the process of collecting the current power quota and memory usage of each GPU computing card in the GPU cluster, and extracting the operational characteristics of non-sensitive tasks to obtain first resource status data includes: By using indicator acquisition probes deployed on the GPU cluster management node to read the power quota value and memory usage of each GPU computing card at a fixed sampling period, and adding card-level identifiers to the power quota value and memory usage, a resource usage record with card-level identifiers is obtained. Offline training jobs and data preprocessing jobs with batch processing tags are selected from the job scheduling log, and the task type, historical average computing density, and current memory usage are extracted as running features. The running features and the resource usage records are merged and aligned according to the card-level identifiers to obtain the first resource status data.

[0005] Furthermore, the step of inputting the first resource situation data into a random forest regression model pre-established on the GPU cluster management node and outputting the estimated remaining duration of the non-sensitive task includes: After preprocessing the first resource situation data, a situation feature vector corresponding to the card level identifier is established. Completely executed non-sensitive job records are retrieved from the historical job archive of the GPU cluster management node. The situation feature vectors of historical jobs under different frequency reduction amplitudes are used as training sample features, and the actual time consumed from the time the historical job was suppressed to the time all iterations were completed are used as training sample labels. Multiple sample subsets are generated using a self-sampling method. A regression decision tree is independently constructed for each sample subset, and splitting features and splitting thresholds are selected according to the principle of minimizing the feature mean square error. Multiple regression decision trees are integrated and aggregated to obtain the random forest regression model. The situation feature vectors are input into the random forest regression model one by one, and the arithmetic mean of the output values ​​of all leaf nodes to which the regression decision trees fall is aggregated to obtain the estimated remaining time.

[0006] Furthermore, the step of establishing a situation feature vector corresponding to the card-level identifier after preprocessing the first resource situation data includes: normalizing the power quota value, memory usage scale, task type, historical average computing density, and current memory usage field by field; mapping the task type to a numerical vector using one-hot encoding; and appending the ratio of the current power quota value to the rated power as the frequency reduction magnitude dimension to the end of the numerical vector to form a situation feature vector corresponding to the card-level identifier.

[0007] Furthermore, the situation feature vectors are input into the random forest regression model one by one, and the arithmetic mean of the output values ​​of the leaf nodes to which all regression decision trees fall is aggregated. This includes: after obtaining the situation feature vectors, they are input into the random forest regression model one by one. Each regression decision tree inside the random forest regression model recursively routes along the tree structure towards the leaf direction according to its node splitting rules. During the recursive routing process, at each internal node, the left or right subtree is determined according to the splitting features and splitting threshold stored in the node. The corresponding leaf node is landed on each regression decision tree, and the output value of the leaf node is read. The arithmetic mean of the output values ​​of the leaf nodes given by all regression decision trees is aggregated.

[0008] Furthermore, the step of adding the estimated remaining time to the current system time and combining it with the historical execution records of non-sensitive tasks in the GPU cluster to determine the memory release time of non-sensitive tasks includes: The estimated remaining time is summed with the current system time of the GPU cluster management node to obtain the expected completion time of the non-sensitive task under the current frequency reduction. This is then recorded according to the card-level identifier to form an expected completion time table. Based on the task type of the non-sensitive task running according to the card-level identifier in the expected completion time table, the time deviation between the expected completion time and the actual completion time, and the release delay between the actual completion time and the memory return trigger time, are retrieved from the historical execution records for each task type in previous executions. The arithmetic mean of the time deviation between the expected completion time and the actual completion time, and the release delay between the actual completion time and the memory return trigger time, are respectively used to obtain the historical deviation mean and the historical release delay mean. The corrected actual completion time is obtained by adding the expected completion time to the historical deviation mean, and then adding the historical release delay mean to obtain the memory release time.

[0009] Furthermore, the step of calculating the time interval between the memory release time of the non-sensitive task and the expected arrival time of the sensitive task to be scheduled, to obtain the first delay result, includes: Extract the expected arrival time of the sensitive task to be scheduled from the sensitive task queue maintained by the GPU cluster scheduler in order of the first position of the queue, and subtract the memory release time from the expected arrival time to obtain the time interval under the corresponding card level identifier, which is determined as the first latency result.

[0010] Furthermore, by analyzing the first latency result against a preset threshold using a latency evaluation model pre-set in the GPU cluster scheduler, if the first latency result exceeds the preset threshold, it is identified that non-sensitive tasks are excessively suppressing and occupying power. Based on this, the power allocation for non-sensitive tasks is narrowed to favor sensitive tasks, resulting in a second power allocation, including: The latency evaluation model is a discriminant function established by threshold comparison. The input of the discriminant function is the time interval corresponding to each card-level identifier in the first latency result, and the output is the over-limit indication of the card-level identifier. Each time interval is compared with the preset threshold by card according to the numerical value, and card-level identifiers whose time intervals exceed the preset threshold are marked to form an over-limit card-level identifier set. If the over-limit card-level identifier set is not empty, it is identified that the non-sensitive task corresponding to the card-level identifier in the over-limit card-level identifier set has the over-suppression card occupation situation under the current power quota. The over-suppression card occupation situation refers to the slow iteration progress of the non-sensitive task at the low power level, causing the memory return time to lag behind the arrival time of the sensitive task. The tilt coefficient is used as a numerical representation of the tilt amplitude. The tilt coefficient is adjusted proportionally according to the number of elements in the over-limit card-level identifier set, so that the current power quota of the non-sensitive task is redistributed according to the adjusted tilt coefficient to obtain the second power quota.

[0011] Furthermore, the step of reallocating power to the computing cards in the GPU cluster according to the second power quota, and triggering memory reclamation of the computing card based on the memory release time to obtain a set of available computing cards includes: The second power quota is distributed to each compute card in the GPU cluster. The corresponding second power quota value is written to the power limit register of the connected power control unit. The operating voltage and GPU core frequency of the compute card are adjusted according to the written register value to obtain compute cards that have completed power redistribution. A timed recycling trigger is set on the compute cards that have completed power redistribution according to the card level identifier. When the system time reaches the memory release time, the timed recycling trigger sends a recycling command to the memory allocation interface of the graphics card management library to release the memory segment held by non-sensitive task processes, resulting in compute cards with returned memory. The core utilization rate and memory usage of each compute card with returned memory are queried. Cards with zero core utilization rate and zero memory usage are included in the set of available compute cards.

[0012] Furthermore, the step of statistically analyzing the number of computing cards and available video memory in the available computing card set to determine the actual scheduling capacity of the current GPU cluster, matching computing cards from the available computing card set according to the video memory requirements of the sensitive tasks to be scheduled, and completing the card scheduling of the sensitive tasks according to the expected arrival time includes: For each card-level identifier in the available computing card set, a fault check is performed on the computing card. Card-level identifiers on computing nodes with hardware error logs, heartbeat timeouts, or abnormal driver responses are removed from the available computing card set. The number of computing cards and the cumulative value of available video memory for the remaining card-level identifiers after removing faulty nodes are counted to obtain the actual scheduling capacity and the set of available computing cards after removal. For the video memory requirement value declared in the job description file of the sensitive task to be scheduled, the available video memory size of each card in the set of available computing cards after removal is queried and compared with the video memory requirement value. Card-level identifiers with a video memory size not less than the video memory requirement value are selected, and the card with the smallest difference between the video memory size and the video memory requirement value is selected as the matching computing card. A card-loading scheduling entry is created in the ready queue of the GPU cluster scheduler with the expected arrival time as the trigger time. When the system time reaches the expected arrival time, the loading action of the sensitive task to be scheduled is triggered according to the card-level identifier of the matching computing card, and the card-loading scheduling of the sensitive task is completed.

[0013] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This invention discloses a dynamic scheduling method for computing resources in GPU clusters. It addresses the resource scheduling challenges in GPU clusters where sensitive and non-sensitive tasks coexist. These challenges arise from excessive power allocation favoring sensitive tasks, suppressing the progress of non-sensitive tasks, and the difficulty in accurately timing memory release, leading to a lack of available GPUs after the arrival of sensitive tasks. The invention first collects first resource status data, including power allocation, memory usage, non-sensitive task types, and historical computing density, for each computing card. This data is input into a pre-installed random forest regression model on the management node to assess the estimated remaining time of non-sensitive tasks under the current downclocking conditions. The memory release time is then calculated based on historical execution records. The time interval between this release time and the expected arrival time of sensitive tasks is calculated and compared with a preset threshold using a latency evaluation model within the scheduler. If the threshold is exceeded, it indicates excessive card suppression and occupancy. The power allocation is narrowed to form a second power allocation, power is reallocated, and memory reclamation is triggered according to the release time. After removing faulty nodes, the actual scheduling capacity is verified, and available GPUs are matched according to memory requirements to complete the allocation of sensitive tasks. This invention achieves coordinated predictive control of power quota and memory release timing, balancing the timeliness of sensitive tasks with the efficiency of non-sensitive tasks, and significantly improving the overall resource utilization and scheduling accuracy of GPU clusters. Attached Figure Description

[0014] Figure 1 This is a flowchart of a dynamic scheduling method for computing resources in GPU clusters according to the present invention.

[0015] Figure 2 This is a schematic diagram of a dynamic scheduling method for computing resources in GPU clusters according to the present invention.

[0016] Figure 3 This is another schematic diagram of a dynamic scheduling method for computing resources of GPU clusters according to the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0018] like Figures 1-3 This embodiment of a dynamic scheduling method for computing resources of GPU clusters may specifically include: Step S101: Collect the current power quota and memory usage of each GPU computing card in the GPU cluster, and extract the running characteristics of non-sensitive tasks to obtain the first resource status data.

[0019] By using indicator acquisition probes deployed on the GPU cluster management node, the current power quota value and memory usage of each GPU computing card are read at a fixed sampling period. Card-level identifiers are added to these power quota values ​​and memory usage levels to obtain resource usage records with card-level identifiers. Based on these resource usage records, offline training jobs and data preprocessing jobs with batch processing tags are selected from the job scheduling log. The task type, historical average computation density, and current memory usage of these offline training jobs and data preprocessing jobs are extracted as runtime features. These runtime features are then merged and aligned with the resource usage records according to the card-level identifiers to obtain the first resource status data.

[0020] In one possible implementation, the indicator acquisition probe is deployed as a resident process on the GPU cluster management node. Utilizing the query interface exposed by the GPU management library and job manager, it polls each compute card in the cluster at a fixed sampling period to read the currently effective power quota value and the amount of video memory already occupied by the compute card. The sampling period is typically on the order of seconds, for example, sampling once every 5 seconds, to match the sampling frequency with the frequency of power quota changes.

[0021] Specifically, after each polling cycle, the indicator acquisition probe adds a card-level identifier to the power quota value and memory usage scale. The card-level identifier is formed by concatenating the physical node number and the bus sequence number of the computing card on that node, serving as a unique index in subsequent merging to form a resource usage record with a card-level identifier. Further, offline training jobs and data preprocessing jobs carrying batch processing tags are filtered from the job scheduling log. The batch processing tag is explicitly declared by the submitter in the job description file when the job is submitted, including values ​​such as lenient deadlines and preemptibility; the scheduler uses this to classify the job into a non-sensitive job category. Among the extracted runtime features, the task type is taken from the job type field in the job description file, with values ​​including offline training and data preprocessing; the historical average computation density is obtained by averaging the number of floating-point operations per unit time during each complete execution of the job corresponding to the runtime feature, reflecting the job's consumption level on computing cores under steady-state operation; and the current memory usage is taken from the process-level memory usage statistics returned by the graphics card management library.

[0022] It is understandable that the operational characteristics and the resource usage records are merged and aligned according to the same card-level identifier, so that each record carries five fields: power quota value, video memory usage scale, task type, historical average computing density and current video memory usage, forming structured first resource status data.

[0023] Step S102: Input the first resource status data into the random forest regression model pre-established on the GPU cluster management node, and output the estimated remaining time of non-sensitive tasks.

[0024] The four numerical fields in the first resource status data—power quota, memory usage, historical average computing density, and current memory usage—are scaled according to their respective historical value ranges to ensure that each field value falls within the 0 to 1 range. The task type field is mapped to a 0-1 vector with a length consistent with the number of task categories using one-hot encoding. The ratio of the current power quota to the rated power is then appended to the end of the vector as the frequency reduction magnitude dimension. The scaled four numerical fields, the task type one-hot vector, and the frequency reduction magnitude dimension are sequentially concatenated according to the card level identifier to form a status feature vector that corresponds one-to-one with the card level identifier. The status feature vector serves as the input sample for the subsequent random forest regression model. The system retrieves fully executed non-sensitive job records from the historical job archive of the GPU cluster management node. The situational feature vectors collected by each historical job under different frequency reduction amplitudes are used as training sample features, and the actual time taken by the historical job from the time of suppression to the time of completion of all iterations is used as training sample labels. Multiple sample subsets are generated from the training samples composed of the training sample features and training sample labels using a bootstrap sampling method. A regression decision tree is independently constructed for each sample subset. When splitting nodes, the splitting features and splitting thresholds are selected according to the principle of minimizing the mean square error of features. The multiple regression decision trees are integrated and aggregated to construct a random forest regression model. After obtaining the situation feature vector, each vector is input into the random forest regression model. Each regression decision tree within the random forest regression model recursively routes along the tree structure towards the leaves according to its node splitting rules. In the recursive routing process, at each internal node, the left or right subtree is determined based on the splitting features and splitting threshold stored at that node. The tree then lands on the corresponding leaf node in each regression decision tree and reads the output value of the leaf node. The arithmetic mean of the output values ​​of the leaf nodes given by all regression decision trees is aggregated to obtain the estimated remaining iteration time of the non-sensitive task under the current frequency reduction amplitude under the card level identifier, which is then determined as the estimated remaining time of the non-sensitive task.

[0025] In one possible implementation, the computation process involved in this embodiment occurs on the GPU cluster management node. For the acquired first resource status data, feature normalization, random forest regression model training, and inference of the remaining duration of non-sensitive tasks are performed in sequence, so that the cluster scheduler obtains a time reference for subsequent power quota reallocation.

[0026] It should be noted that each record in the first resource status data corresponds to a GPU computing card in the cluster. The record simultaneously carries five raw fields: the current power quota value of the computing card, the current memory usage, the task type of the non-sensitive job running on the card, the historical average computing density of the non-sensitive job, and the current memory usage of the non-sensitive job. Among them, the task type is a discrete field, which needs to be expanded into a 0-1 vector of length equal to the number of task types through one-hot encoding. The other four are continuous fields with large differences in units and value ranges. For example, the power quota is usually distributed in the range of 50-400 watts, the memory usage range is distributed in the range of 0-80 gigabytes, and the computing density is measured in floating-point operations per second, which can span several orders of magnitude. If the data is directly concatenated and fed into the downstream random forest regression model and subsequent fusion decision module, the continuous fields will be submerged or amplified in distance measurement, gradient update and feature importance evaluation due to the large difference in numerical scale. Therefore, before entering the model, the four continuous fields are subjected to min-max normalization, and the values ​​are compressed to the 0-1 range. Then, they are concatenated with the one-hot encoded segment along the feature dimension to form the final input vector, whose total dimension is equal to 4 plus the number of task categories.

[0027] Specifically, the task type is discrete-value fields, with common values ​​including offline training and data preprocessing. Discrete-value fields are mapped using one-hot encoding, with offline training corresponding to a two-dimensional vector of 1,0 and data preprocessing corresponding to a two-dimensional vector of 0,1. Continuous-value fields such as power quota values ​​and memory usage are mapped using a maximum / minimum scaling method. The scaling formula is y=xa / ba, where x is the current value, a is the historical minimum value of the field, and b is the historical maximum value of the field. a and b are obtained from all historical samples sampled every 5 minutes within a sliding window over the past 30 days. For example, if a computing card has a rated power of 500 watts, and the historical minimum power quota value for the card over the past 30 days is 100 watts and the maximum value is 400 watts, and the current power quota value is 160 watts, then the scaled value is 160-100 divided by 400-100, resulting in 0.2. Memory usage and historical average computing density are processed by calculating their respective historical minimum and maximum values ​​in the same way. Following the scaling result, an additional downclocking dimension is added. This dimension is defined as the ratio of the current power quota value to the rated power of the computing card, and is unrelated to the aforementioned historical maximum value. In the previous example, this dimension was 160 divided by 500, which equals 0.32. Thus, the scaling result of 0.2 reflects the relative position of the current value within the historical range, while the downclocking magnitude of 0.32 reflects the proportion of the current value relative to the hardware's rated capacity. These two dimensions have different origins and meanings. The dimensions are concatenated in the order of task type, power scaling value, memory scaling value, computing density scaling value, and downclocking magnitude to form a situational feature vector.

[0028] For example, suppose there is a computing card in the cluster with a card level identifier of node number 03 and bus sequence number 5, and an offline training job is running. The situation feature vector obtained by the card carries a one-hot encoded segment, a scaled continuous field segment, and a down-frequency magnitude dimension, which is submitted as an input sample to the random forest regression model.

[0029] It is understood that the random forest regression model is a regression model pre-built and trained on the GPU cluster management node. The training samples relied upon for model training come from historical job archives. The GPU cluster management node retains the execution trajectory of all completed non-sensitive jobs, including the situation feature vector collected at each power quota change time during the job's lifecycle, the corresponding frequency reduction magnitude, and the actual time accumulated from the time the job was suppressed to the time when all iterations were completed. The aforementioned actual time serves as the training sample label, and the aforementioned situation feature vector serves as the training sample feature. The two are paired according to the unique job number to obtain the training sample set.

[0030] Preferably, the training sample set is used to generate multiple sample subsets using a bootstrap sampling method. Specifically, the bootstrap sampling involves randomly selecting one sample from the training sample set with replacement each time, repeating this process until the size of the sample subset is the same as the size of the training sample set, thus forming one sample subset. This extraction process is repeated multiple times, and the resulting multiple sample subsets are used to construct a regression decision tree.

[0031] Specifically, a regression decision tree is independently constructed for each sample subset. During the splitting process of each internal node, a candidate dimension subset is first randomly selected from all dimensions of the situation feature vector. Then, splitting features and splitting thresholds are selected from the candidate dimension subset according to the principle of minimizing mean squared error. The mean squared error is the sum of squares of the differences between the training sample labels in the left and right child nodes and the mean labels of each child node after the node splits. The selection objective of splitting features and splitting thresholds is to minimize this sum of squares. When the number of samples in a node is lower than the preset minimum number of split samples, or the node depth reaches the preset maximum depth, splitting stops and the node is set as a leaf node. The output value of the leaf node is the arithmetic mean of all training sample labels within that node.

[0032] In one embodiment, the aforementioned independently constructed multiple regression decision trees are integrated and aggregated to form the random forest regression model. The integration and aggregation takes the form that the random forest regression model internally stores the node splitting rules, splitting features, splitting thresholds, and leaf node output values ​​of all regression decision trees in a parallel manner. Further, during the inference phase, the situational feature vectors produced from the feature normalization stage are input into the random forest regression model line by line. Each regression decision tree within the random forest regression model independently recursively routes along the tree structure towards the leaves according to its node splitting rules. At each internal node, the recursive route determines whether to enter the left or right subtree based on the splitting features and splitting thresholds stored at that node, until it reaches a leaf node and reads the output value of that leaf node.

[0033] Specifically, the arithmetic mean of the output values ​​of the leaf nodes of all regression decision trees is aggregated, and the resulting mean is the estimated remaining iteration time of the non-sensitive job under the current frequency reduction magnitude at that card level. This is determined as the estimated remaining time of the non-sensitive task. The estimated remaining time is output in seconds, reflecting the cumulative running time required for the non-sensitive job to continue completing the remaining iterations from the current moment while maintaining the current power quota.

[0034] Step S103: Add the estimated remaining time to the current system time, and combine it with the historical running records of non-sensitive tasks in the GPU cluster to determine the time when the non-sensitive task releases its video memory.

[0035] The estimated remaining time is summed with the current system time of the GPU cluster management node to obtain the expected completion time of non-sensitive tasks under the current frequency reduction. These expected completion times are registered according to card-level identifiers, with each non-sensitive task running on a compute card corresponding to an independent entry, forming an expected completion time table that carries both the card-level identifier and the expected completion time value. Based on the task type of the non-sensitive task running according to the card-level identifier in the expected completion time table, the time deviation between the expected completion time and the actual completion time of the task type in previous executions is retrieved from the historical execution records of the GPU cluster, along with the release delay between the actual completion time and the memory return trigger time. The arithmetic mean of the time deviation and the release delay is calculated to obtain the historical average deviation and the historical average release delay. The expected completion time, the historical average deviation, and the historical average release delay are superimposed in time axis order. The corrected actual completion time is obtained by adding the historical average deviation to the expected completion time. Then, the memory release time of non-sensitive tasks is obtained by adding the corrected actual completion time to the historical average release delay. The memory release time is registered according to card-level identifier.

[0036] In one possible implementation, this method occurs after obtaining the estimated remaining time of non-sensitive tasks. The goal is to infer when the non-sensitive tasks will return the video memory resources they occupy based on the estimated remaining time and the historical running records saved by the GPU cluster management node, thereby providing an accurate time anchor for subsequent video memory reclamation triggering.

[0037] Specifically, the estimated remaining time for non-sensitive tasks is directly added to the current system time of the GPU cluster management node. System time is referenced in epoch seconds. Let Tr seconds represent the estimated remaining time for the non-sensitive task, and Tn seconds represent the current system time. The expected completion time Te satisfies the relationship: Te = Tn + Tr. All characters in this relationship are in seconds, and the result is also expressed in epoch seconds.

[0038] It should be noted that each compute card only hosts one instance of a non-sensitive task at a time. Therefore, each non-sensitive task running on a compute card corresponds to an independent entry, and all entries are aggregated according to the card-level identifier to form an expected completion time table. The expected completion time table is a two-dimensional data structure with the card-level identifier as the primary key and the expected completion time value as the field, and reserves corresponding fields for writing the actual completion time correction value and the memory release time in subsequent stages. Furthermore, relying solely on the expected completion time is insufficient to reflect the actual point at which a non-sensitive task exits its memory occupation, so two types of historical experience correction items need to be introduced. The GPU cluster management node continuously saves the task type, the expected completion time obtained from each prediction, the iteration end time, and the memory return trigger time for each completed non-sensitive task, constituting the aforementioned historical running record.

[0039] Specifically, entries from past executions are filtered from the historical execution records by task type. For each historical entry, two types of time differences are extracted: the first type is the difference between the expected completion time and the actual completion time, reflecting systematic biases in model prediction; the second type is the difference between the actual completion time and the memory return trigger time, reflecting the release latency experienced by non-sensitive tasks when releasing memory after completing iterations. The release latency is caused by three types of cleanup actions required after the final iteration of a non-sensitive task: checkpoint write to disk, memory allocator return, and job manager cleanup. The cumulative time consumed by these cleanup actions constitutes the release latency. The time distribution of these cleanup actions varies depending on the task type; therefore, a separate set of historical entries is maintained for each task type.

[0040] For example, for several historical entries with the task type of offline training, the arithmetic mean of all first-type time differences is calculated to obtain the historical deviation mean corresponding to this task type; the arithmetic mean of all second-type time differences is calculated to obtain the historical release delay mean corresponding to this task type. Entries with the task type of data preprocessing independently obtain their corresponding historical deviation mean and historical release delay mean in the same way.

[0041] In one embodiment, two superposition operations are performed sequentially along the timeline. The first superposition adds the historical average deviation Tb to the expected completion time Te to obtain the corrected actual completion time Tc, i.e., Tc = Te + Tb, where Tb is in seconds and can be positive or negative; a positive value indicates an early prediction, and a negative value indicates a late prediction. The second superposition adds the historical average release delay Td to the corrected actual completion time Tc to obtain the memory release time Tf for non-sensitive tasks, i.e., Tf = Tc + Td.

[0042] It is understandable that the obtained memory release time Tf is written into the corresponding field of the expected completion time table according to the card level identifier, thereby completing the registration of memory release time for all computing cards carrying non-sensitive tasks. The registered expected completion time table is indexed by the card level identifier and can be directly queried in subsequent stages to avoid temporary duplicate calculations.

[0043] Step S104: Calculate the time interval between the memory release time of the non-sensitive task and the expected arrival time of the sensitive task to be scheduled, and obtain the first delay result.

[0044] The expected arrival times of sensitive tasks to be scheduled are extracted from the queue of sensitive tasks maintained by the GPU cluster scheduler, ordered by the head of the queue. For each memory release time with a card-level identifier, the expected arrival time is paired with the expected arrival time to obtain card-level identifier release arrival time pairs. The difference between the expected arrival time and the memory release time in each release arrival time pair is calculated along the time axis. The expected arrival time is subtracted from the memory release time to obtain the time interval under the corresponding card-level identifier, which is determined as the first latency result.

[0045] In one possible implementation, this embodiment takes into account the registration results of the memory release time, and aims to evaluate the time difference between the time when a non-sensitive task on each computing card exits its occupied memory and the time when a newly arrived, scheduled sensitive task from outside the GPU cluster is expected to be accepted.

[0046] Specifically, the expected arrival time of the sensitive tasks to be scheduled is taken from the sensitive task scheduling queue maintained by the GPU cluster scheduler. The sensitive task scheduling queue is sorted in ascending order of the deadline margin declared by the submitter in the job description file. The first position of the queue stores the sensitive task entry with the smallest margin and the most urgent. The entry also records the time when the task submitter expects to start the calculation, i.e. the expected arrival time, expressed in epoch seconds.

[0047] It should be noted that the memory release time was registered in the previous stage according to the card-level identifier, with each computing card corresponding to an independent value. For example, in a GPU cluster, there are two computing cards with card-level identifiers: node number 03, bus sequence number 5, and node number 03, line sequence number 6. Each card has its own registered memory release time. Each memory release time with a card-level identifier is paired with the expected arrival time taken from the head of the queue. The resulting release arrival time pair consists of two records in the case of two computing cards. Each record carries the card-level identifier, the memory release time value, and the common expected arrival time value. Further, the two times within the release arrival time pair are calculated using a time-axis difference operation. Let the expected arrival time be Ta, the memory release time be Tf, and the time interval Tg satisfy the following relationship: In the relational expression, all characters are in seconds. A positive value for Tg indicates that the time when the sensitive task is expected to start calculation is later than the time when the video memory is released, at which point the video memory has already been freed up; a negative value for Tg indicates that the time when the sensitive task is expected to start calculation is earlier than the time when the video memory is released, at which point the video memory is still occupied by a non-sensitive task when the sensitive task arrives.

[0048] Understandably, after independently performing the difference calculation for each release arrival time pair, all calculation results are aggregated according to the card-level identifier to form the first latency result. The first latency result uses the card-level identifier as an index and the time interval value as a field to reflect the relative gap between the pace of memory release on each computing card in the cluster and the arrival pace of the sensitive tasks to be scheduled.

[0049] Step S105: Analyze the first latency result and the preset threshold using the latency evaluation model preset in the GPU cluster scheduler. If the first latency result exceeds the preset threshold, it is identified that non-sensitive tasks are excessively suppressing and occupying power cards. Based on this, the tilt of the power quota of non-sensitive tasks towards sensitive tasks is narrowed to obtain the second power quota.

[0050] The first latency result is analyzed using a pre-built latency evaluation model in the GPU cluster scheduler. This latency evaluation model is a discriminant function pre-established using a threshold comparison method. The input to the discriminant function is the time interval corresponding to each card-level identifier in the first latency result, and the output is an over-limit indication for that card-level identifier. Each time interval is compared with a preset threshold numerically, card-by-card, marking card-level identifiers whose time intervals exceed the preset threshold, thus forming an over-limit card-level identifier set. The card-level identifier status of non-sensitive tasks is determined based on the presence of card-level identifiers in this over-limit card-level identifier set. If the over-limit card-level identifier set is not empty, it is identified that the non-sensitive tasks corresponding to the card-level identifiers in this set exhibit an excessively suppressed card-level status under the current power quota. This excessively suppressed card-level status refers to the slow iteration progress of non-sensitive tasks at low power levels, causing the memory return time to lag behind the arrival time of sensitive tasks. Based on the identified card-level identifiers that exhibit excessive suppression and card occupation, the power quota tilt that was originally concentrated on sensitive tasks is narrowed. The power quota tilt is represented by a tilt coefficient. The narrowing process involves proportionally reducing the tilt coefficient based on the number of elements in the set of over-limit card-level identifiers, so that the current power quota of non-sensitive tasks is redistributed according to the reduced tilt coefficient, and the redistributed power quota is used as the second power quota.

[0051] In one possible implementation, the latency evaluation model is pre-installed as a resident discrimination component within the GPU cluster scheduler to analyze the first latency result. The goal is to identify whether there are computing cards in the current cluster whose memory release rhythm is significantly behind the arrival rhythm of sensitive tasks.

[0052] Specifically, the latency evaluation model is essentially a discriminant function established by threshold comparison. The input of the discriminant function is the time interval corresponding to each card-level identifier in the first latency result, and the output is the limit indication of the card-level identifier. When the limit indication is 1, it means that the time interval corresponding to the card-level identifier exceeds the preset threshold, and when it is 0, it means that it does not exceed the threshold.

[0053] It should be noted that the preset threshold is a negative value parameter written in advance in the GPU cluster scheduler configuration item according to experience. The value reflects the upper limit of the cluster's tolerance for memory release lagging behind the arrival of sensitive tasks.

[0054] In one embodiment, the preset threshold is set to -30 seconds, meaning that the video memory is allowed to be freed up no later than 30 seconds after a sensitive task arrives. Each time interval carried in the first latency result is compared card-by-card with the preset threshold, and card-level identifiers for time intervals less than the preset threshold are marked as exceeding the limit.

[0055] For example, in a GPU cluster, there are three computing cards with card-level identifiers of node number 03 bus sequence number 5, node number 03 bus sequence number 6, and node number 04 bus sequence number 2, corresponding to time intervals of -12 seconds, -58 seconds, and -45 seconds, respectively. The card-level identifiers of node number 03 bus sequence number 6 and node number 04 bus sequence number 2 are then included in the over-limit card-level identifier set. Further, the number of elements in the over-limit card-level identifier set is checked. If the over-limit card-level identifier set is not empty, it is identified that each card-level identifier in the set corresponds to a non-sensitive task that is excessively suppressing and occupying the card. The physical meaning of the excessive suppression of GPU memory usage is that the power quota originally allocated to non-sensitive tasks has been significantly tilted towards sensitive tasks by the scheduler. The power level of non-sensitive tasks is reduced to below the level required for steady-state operation, and the GPU core operating frequency is reduced accordingly. The time taken for a single iteration is extended several times. However, the video memory resources requested by the non-sensitive task process are continuously held by the non-sensitive task process, resulting in the video memory being returned much later than the time when the sensitive task expects to start computing. As a result, even if a newly arrived sensitive task faces a large number of idle computing cores, it cannot immediately start computing due to insufficient video memory.

[0056] It is understood that the power quota tilt is represented by the tilt coefficient α, which is defined as the ratio of the power allocated to the sensitive task side to the total power budget to the power allocated to the non-sensitive task side. The value ranges from 1 to 10, and the larger the value, the more severe the power tilt towards the sensitive task side.

[0057] Preferably, the tilt coefficient α is adjusted downward according to the number of elements in the over-limit card level identifier set.

[0058] In one embodiment, the down-adjustment action is based on The relationship is expressed as follows: α represents the current tilt coefficient, αnew represents the tilt coefficient after reduction, n represents the number of elements in the set of over-limit card level identifiers, and β represents the reduction step size corresponding to each over-limit card level identifier, which is empirically set to 0.05. In the aforementioned example, the number of elements in the set of over-limit card level identifiers is 2, and the current tilt coefficient is 6. Therefore, the tilt coefficient αnew after reduction is: .

[0059] Specifically, the scheduler redistributes the total power budget by adjusting the tilt coefficient αnew. Sensitive task side and non-sensitive task side each obtain a new power share, which is further distributed to each computing card according to the number of jobs they carry, resulting in a new power quota value for each computing card. The new power quota value is the second power quota.

[0060] Step S106: The power of the computing cards in the GPU cluster is reallocated according to the second power quota, and the memory reclamation of the computing card is triggered based on the memory release time, so as to obtain a set of available computing cards with complete card-level resources.

[0061] The second power quota is distributed to each compute card in the GPU cluster. The corresponding second power quota value is written to the power limit register of the power control unit connected to each compute card. The power control unit adjusts the operating voltage and GPU core frequency of the compute card according to the written register value, so that the operating power of the compute card supporting non-sensitive tasks recovers from the suppressed level to the level corresponding to the second power quota, resulting in a compute card that has completed power redistribution. A timed recycling trigger is set on the compute card that has completed power redistribution according to the card-level identifier at the memory release time. The timed recycling trigger issues a recycling command when the system time reaches the memory release time. The recycling command is transmitted to the memory allocation interface provided by the graphics card management library. The memory allocation interface performs a release action on the memory segment held by the non-sensitive task process. The released memory is returned to the allocatable pool of the compute card, resulting in a compute card with returned memory. For each computing card after the video memory is returned, the core utilization rate and video memory usage are queried. If the core utilization rate and video memory usage of the computing card after the video memory is returned are both zero, it is determined that the core and video memory of the computing card are both idle. The card-level identifier of the computing card is included in the set of available computing cards, thus obtaining a set of available computing cards with complete card-level resources.

[0062] In another implementation of the computing card after the memory is returned, the timed recycling trigger sends a SIGUSR1 pre-notification signal to the non-sensitive task process 30 seconds before the system time reaches the memory release time. After receiving the signal, the process enters the finalization stage, stops receiving new batches of data, writes the intermediate tensors of the current iteration and the training state to persistent storage to complete the checkpoint saving, and then actively calls the memory allocation interface to release the memory segment it holds and exits. If the process has not terminated itself when the memory release time is reached, the trigger first queries the running status and memory read / write activity of the process through the graphics card management library. Only when the activity is zero and there are no unfinished kernel function calls can the forced release be performed. Otherwise, the release action is delayed for 5 seconds and the query is performed again. The maximum number of retries is 3. If it is still not idle, a SIGTERM signal is sent to trigger the graceful termination process. The memory is then reclaimed after the process exits. The recycling command is passed to the video memory allocation interface provided by the graphics card management library. The video memory allocation interface performs a release action on the video memory segment held by the non-sensitive task process. The released video memory is returned to the allocatable pool of the computing card, resulting in a computing card with the video memory returned.

[0063] In one possible implementation, the second power quota is actually applied to the power limit of each computing card, and the occupied video memory is returned to the allocable pool in a timely manner after the non-sensitive tasks are completed, thus aggregating a group of computing cards in which both computing cores and video memory are in an idle state.

[0064] Specifically, the second power quota is distributed to each compute card in the GPU cluster on a card-by-card basis according to the card-level identifier. Each compute card has a power control unit on the onboard level. The power control unit has a reserved power limit register. The power limit register receives the power limit value written by the scheduler and uses it as the hard limit of the instantaneous power consumption of the compute card.

[0065] It should be noted that after receiving the written power limit value, the power control unit adjusts the operating voltage of the computing card and the GPU core frequency according to the preset voltage-frequency level table. For example, if a computing card on the non-sensitive task side was originally suppressed, its operating level was stuck at a low level of 800 MHz and 0.75 volts, corresponding to an operating power of 120 watts. After the second power quota value is issued and written as 220 watts, the power control unit switches to the 1350 MHz and 0.92 volt level accordingly, and the operating power rises back to around 220 watts, thus obtaining a computing card that has completed power redistribution. Furthermore, a timed recycling trigger is attached to each of the computing cards that has completed power redistribution, according to the card-level identifier. The timed recycling trigger is a timed event entry maintained by the GPU cluster scheduler in its own event loop. Each entry is indexed by the card-level identifier and uses the memory release time obtained in the previous stage as the trigger time field. The event loop polls the current system time with millisecond precision. When the current system time reaches or exceeds the memory release time registered by a certain timed event entry, the timed recycling trigger immediately retrieves the corresponding card-level identifier and the node number from the entry, and issues a recycling instruction to the memory management process of the node where the computing card is located based on the memory allocation interface provided by the graphics card management library. The recycling instruction carries the process identifier of the non-sensitive task process and the starting address and length of the memory segment occupied by the process.

[0066] Understandably, after receiving the memory reclamation command, the memory allocation interface performs a release action on the memory segment held by the non-sensitive task process. This release action covers three actions: clearing the memory segment marker, unmapping the page table entry, and reattaching the memory free list. After these three actions are completed, the memory segment originally occupied by the non-sensitive task process returns to the memory allocable pool of the computing card, thus obtaining the computing card with the returned memory.

[0067] In one embodiment, a resource check is initiated for each compute card after the video memory has been returned. Specifically, the resource check involves using the query interface of the graphics card management library to read two real-time indicators for the compute card: the compute core utilization rate and the video memory usage. The compute core utilization rate reflects the percentage of time the GPU cores were occupied by tasks within a past sampling window, while the video memory usage reflects the number of bytes of video memory currently allocated to the compute card.

[0068] Preferably, if the core utilization rate of the computing card after the video memory is returned is zero and the video memory utilization is zero, then it is determined that both the core utilization and video memory of the computing card are in an idle state. The GPU cluster scheduler writes the card-level identifier of the computing card into the set of available computing cards. The set of available computing cards is stored in a set data structure with card-level identifiers as elements. Each computing card corresponding to a card-level identifier in the set has both idle core utilization and idle video memory.

[0069] Step S107: By statistically analyzing the number of computing cards and the available memory in the set of available computing cards, and removing computing nodes with faults, the actual scheduling capacity of the current GPU cluster is determined. Computing cards are matched from the set of available computing cards according to the memory requirements of the sensitive tasks to be scheduled, and the scheduling of sensitive tasks on the cards is completed according to the expected arrival time.

[0070] For each card-level identifier in the available compute card set, a fault check is performed on the compute card. Card-level identifiers on compute nodes with hardware error logs, heartbeat timeouts, or abnormal driver responses are removed from the available compute card set. The number of compute cards remaining after removing faulty nodes is calculated, along with the cumulative value of the available memory on each compute card, to obtain the actual scheduling capacity of the current GPU cluster and the set of available compute cards after removal. Based on the set of available compute cards after removal, for the memory requirement value declared in the job description file for the sensitive task to be scheduled, the available memory size of each card in the set is queried and compared with the memory requirement value. Card-level identifiers with a memory availability value not less than the memory requirement value are selected, and the card with the smallest difference between the available memory size and the memory requirement value among the selected card-level identifiers is chosen as the matching compute card. Based on the card-level identifier corresponding to the matching computing card and the expected arrival time of the sensitive task to be scheduled, a card-up scheduling entry is established in the ready queue of the GPU cluster scheduler. The card-up scheduling entry is triggered by the expected arrival time. When the system time reaches the expected arrival time, the scheduler triggers the loading action of the model weights and batch data of the sensitive task to be scheduled according to the card-level identifier of the matching computing card, so that the model weights and batch data enter the video memory space of the computing card, and the card-up scheduling of the sensitive task is completed.

[0071] In one possible implementation, after excluding computing nodes in an abnormal state, the specific computing card to carry the sensitive tasks to be scheduled is selected based on the actual available hardware scale on the cluster side and the resource requirements on the job side, and the card is loaded according to the arrival time of the job.

[0072] Specifically, a fault check is initiated for each computing card corresponding to each card-level identifier in the set of available computing cards. The fault check covers three types of judgment conditions. The first type of condition is the hardware error log condition. The GPU cluster scheduler reads the hardware error code counter of the computing card from the node's out-of-band management channel. If a new uncorrectable error entry appears in the counter within the most recent sampling window, the card is determined to be hardware abnormal. The second type of condition is the heartbeat timeout condition. The node agent periodically reports live heartbeat messages to the scheduler. If the node where the card is located does not report a heartbeat within three consecutive heartbeat cycles, the node is determined to be unreachable. The third type of condition is the driver response abnormal condition. The scheduler initiates an empty query for the card using the graphics card management library probe interface. If the call returns an error code or times out, the card's driver layer is determined to be abnormal.

[0073] It should be noted that the card-level identifier on a computing node that meets any of the above three judgment conditions is removed from the set of available computing cards, and the remaining card-level identifiers constitute the set of available computing cards after removal.

[0074] Understandably, by performing a scale analysis on the card-level identifiers within the set of available compute cards after the removal of exceptions, the cumulative value of the number of compute cards and the available memory size on each compute card is obtained. These two values ​​together constitute the actual scheduling capacity of the current GPU cluster, reflecting the total amount of hardware that can truly be scheduled after anomalies are eliminated. Furthermore, when a sensitive task to be scheduled is submitted, it declares its memory requirement value in bytes through the memory request field in the job description file. The GPU cluster scheduler reads the memory requirement value from the job description file, queries the available memory size of each card-level identifier in the set of available compute cards after the removal of exceptions, and compares the values ​​with the memory requirement value.

[0075] Preferably, the comparison uses the best fit screening rule.

[0076] Specifically, card-level identifiers with available video memory smaller than the required video memory value are first removed to avoid situations where the card cannot handle the load. Then, the remaining card-level identifiers are sorted in ascending order according to the difference between the available video memory value and the required video memory value. The computing card corresponding to the card-level identifier with the smallest difference is selected as the matching computing card to prevent large video memory computing cards from being occupied by small video memory-sensitive tasks, thus avoiding video memory fragmentation.

[0077] In one embodiment, the scheduler inserts a card-up scheduling entry into its ready queue using the card-level identifier of the matching computing card and the expected arrival time of the sensitive task to be scheduled as elements. The card-up scheduling entry carries the unique number of the sensitive task, the card-level identifier, the node number, and the expected arrival time. The scheduler's event loop polls the current system time with millisecond precision.

[0078] Specifically, when the current system time reaches the expected arrival time registered in the card scheduling entry, the scheduler triggers a loading action for the model weights and batch data of the sensitive task to be scheduled according to the card-level identifier of the matching computing card. The loading action includes retrieving the binary file of the model weights from object storage or a shared file system, transferring it through host memory, and then writing it to the video memory space of the matching computing card via the device bus direct memory access channel. An asynchronous pipeline is established between host memory and video memory for the batch data, ensuring that the model weights and batch data enter the video memory space of the computing card sequentially. At this point, the sensitive task to be scheduled has completed its card scheduling from the scheduling queue to the matching computing card, and the sensitive task directly enters the computation stage on the matching computing card, no longer waiting in line due to insufficient video memory.

[0079] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for dynamic scheduling of computing resources for GPU clusters, characterized in that, The method includes: Collect the current power quota and memory usage of each GPU computing card in the GPU cluster, and extract the running characteristics of non-sensitive tasks to obtain the first resource status data; The first resource status data is input into a random forest regression model pre-built on the GPU cluster management node, and the estimated remaining time of non-sensitive tasks is output. Add the estimated remaining time to the current system time, and combine this with the historical running records of non-sensitive tasks in the GPU cluster to determine the time when the non-sensitive tasks release their GPU memory. Calculate the time interval between the memory release time of the non-sensitive task and the expected arrival time of the sensitive task to be scheduled, and obtain the first delay result; By analyzing the first latency result and the preset threshold using the latency evaluation model pre-set in the GPU cluster scheduler, if the first latency result exceeds the preset threshold, it is identified that non-sensitive tasks are excessively suppressing and occupying cards. Based on this, the tilt of the power quota of non-sensitive tasks towards sensitive tasks is narrowed to obtain the second power quota. The power of the computing cards in the GPU cluster is reallocated according to the second power quota, and the memory reclamation of the computing card is triggered based on the memory release time to obtain a set of available computing cards. The number of computing cards and the available memory in the set of available computing cards are counted to determine the actual scheduling capacity of the current GPU cluster. Computing cards are matched from the set of available computing cards according to the memory requirements of the sensitive tasks to be scheduled, and the scheduling of sensitive tasks on the cards is completed according to the expected arrival time.

2. The method for dynamic scheduling of computing resources for GPU clusters according to claim 1, characterized in that, The process involves collecting the current power quota and memory usage of each GPU in the GPU cluster, extracting the operational characteristics of non-sensitive tasks, and obtaining first resource status data, including: By using indicator acquisition probes deployed on the GPU cluster management node to read the power quota value and memory usage of each GPU computing card at a fixed sampling period, and adding card-level identifiers to the power quota value and memory usage, a resource usage record with card-level identifiers is obtained. Offline training jobs and data preprocessing jobs with batch processing tags are selected from the job scheduling log, and the task type, historical average computing density, and current memory usage are extracted as running features. The running features and the resource usage records are merged and aligned according to the card-level identifiers to obtain the first resource status data.

3. The method for dynamic scheduling of computing resources for GPU clusters according to claim 1, characterized in that, The step of inputting the first resource status data into a random forest regression model pre-established on the GPU cluster management node and outputting the estimated remaining time of non-sensitive tasks includes: After preprocessing the first resource situation data, a situation feature vector corresponding to the card level identifier is established. Completely executed non-sensitive job records are retrieved from the historical job archive of the GPU cluster management node. The situation feature vectors of historical jobs under different frequency reduction amplitudes are used as training sample features, and the actual time consumed from the time the historical job was suppressed to the time all iterations were completed are used as training sample labels. Multiple sample subsets are generated using a self-sampling method. A regression decision tree is independently constructed for each sample subset, and splitting features and splitting thresholds are selected according to the principle of minimizing the feature mean square error. Multiple regression decision trees are integrated and aggregated to obtain the random forest regression model. The situation feature vectors are input into the random forest regression model one by one, and the arithmetic mean of the output values ​​of all leaf nodes to which the regression decision trees fall is aggregated to obtain the estimated remaining time.

4. The method for dynamic scheduling of computing resources for GPU clusters according to claim 3, characterized in that, The process of establishing a situation feature vector corresponding to the card level identifier after preprocessing the first resource situation data includes: normalizing the power quota value, memory usage scale, task type, historical average computing density and current memory usage field by field; mapping the task type to a numerical vector using one-hot encoding; and appending the ratio of the current power quota value to the rated power as the frequency reduction magnitude dimension to the end of the numerical vector to form a situation feature vector corresponding to the card level identifier.

5. The method for dynamic scheduling of computing resources for GPU clusters according to claim 3, characterized in that, The process involves inputting the situation feature vectors one by one into the random forest regression model, and aggregating the arithmetic mean of the output values ​​of the leaf nodes to which all regression decision trees fall. This includes: obtaining the situation feature vectors and inputting them one by one into the random forest regression model; each regression decision tree within the random forest regression model recursively routing along the tree structure towards the leaves according to its node splitting rules; during the recursive routing process, each internal node determines whether to enter the left or right subtree based on the splitting features and splitting thresholds stored in the node; landing on the corresponding leaf node in each regression decision tree and reading the output value of the leaf node; and aggregating the arithmetic mean of the output values ​​of the leaf nodes given by all regression decision trees.

6. The method for dynamic scheduling of computing resources for GPU clusters according to claim 1, characterized in that, The step of adding the estimated remaining time to the current system time and combining it with the historical execution records of non-sensitive tasks in the GPU cluster to determine the memory release time of non-sensitive tasks includes: The estimated remaining time is summed with the current system time of the GPU cluster management node to obtain the expected completion time of the non-sensitive task under the current frequency reduction. This is then recorded according to the card-level identifier to form an expected completion time table. Based on the task type of the non-sensitive task running according to the card-level identifier in the expected completion time table, the time deviation between the expected completion time and the actual completion time, and the release delay between the actual completion time and the memory return trigger time, are retrieved from the historical execution records for each task type in previous executions. The arithmetic mean of the time deviation between the expected completion time and the actual completion time, and the release delay between the actual completion time and the memory return trigger time, are respectively used to obtain the historical deviation mean and the historical release delay mean. The corrected actual completion time is obtained by adding the expected completion time to the historical deviation mean, and then adding the historical release delay mean to obtain the memory release time.

7. The method for dynamic scheduling of computing resources for GPU clusters according to claim 1, characterized in that, The step of calculating the time interval between the memory release time of the non-sensitive task and the expected arrival time of the sensitive task to be scheduled, to obtain the first delay result, includes: Extract the expected arrival time of the sensitive task to be scheduled from the sensitive task queue maintained by the GPU cluster scheduler in order of the first position of the queue, and subtract the memory release time from the expected arrival time to obtain the time interval under the corresponding card level identifier, which is determined as the first latency result.

8. The method for dynamic scheduling of computing resources for GPU clusters according to claim 1, characterized in that, The process involves analyzing the first latency result against a preset threshold using a latency evaluation model pre-configured in the GPU cluster scheduler. If the first latency result exceeds the preset threshold, it identifies an excessive suppression and occupancy of power by non-sensitive tasks. Based on this, the power allocation for non-sensitive tasks is reduced towards sensitive tasks, resulting in a second power allocation, which includes: The latency evaluation model is a discriminant function established by threshold comparison. The input of the discriminant function is the time interval corresponding to each card-level identifier in the first latency result, and the output is the over-limit indication of the card-level identifier. Each time interval is compared with the preset threshold by card according to the numerical value, and card-level identifiers whose time intervals exceed the preset threshold are marked to form an over-limit card-level identifier set. If the over-limit card-level identifier set is not empty, it is identified that the non-sensitive task corresponding to the card-level identifier in the over-limit card-level identifier set has the over-suppression card occupation situation under the current power quota. The over-suppression card occupation situation refers to the slow iteration progress of the non-sensitive task at the low power level, causing the memory return time to lag behind the arrival time of the sensitive task. The tilt coefficient is used as a numerical representation of the tilt amplitude. The tilt coefficient is adjusted proportionally according to the number of elements in the over-limit card-level identifier set, so that the current power quota of the non-sensitive task is redistributed according to the adjusted tilt coefficient to obtain the second power quota.

9. The method for dynamic scheduling of computing resources for GPU clusters according to claim 1, characterized in that, The step of reallocating power to the computing cards in the GPU cluster according to the second power quota, and triggering memory reclamation of the computing card based on the memory release time to obtain a set of available computing cards includes: The second power quota is distributed to each compute card in the GPU cluster. The corresponding second power quota value is written to the power limit register of the connected power control unit. The operating voltage and GPU core frequency of the compute card are adjusted according to the written register value to obtain compute cards that have completed power redistribution. A timed recycling trigger is set on the compute cards that have completed power redistribution according to the card level identifier. When the system time reaches the memory release time, the timed recycling trigger sends a recycling command to the memory allocation interface of the graphics card management library to release the memory segment held by non-sensitive task processes, resulting in compute cards with returned memory. The core utilization rate and memory usage of each compute card with returned memory are queried. Cards with zero core utilization rate and zero memory usage are included in the set of available compute cards.

10. The method for dynamic scheduling of computing resources for GPU clusters according to claim 1, characterized in that, The process of statistically analyzing the number of computing cards and available video memory in the available computing card set to determine the actual scheduling capacity of the current GPU cluster, matching computing cards from the available computing card set according to the video memory requirements of the sensitive tasks to be scheduled, and completing the card scheduling of sensitive tasks according to the expected arrival time includes: For each card-level identifier in the available computing card set, a fault check is performed on the computing card. Card-level identifiers on computing nodes with hardware error logs, heartbeat timeouts, or abnormal driver responses are removed from the available computing card set. The number of computing cards and the cumulative value of available video memory for the remaining card-level identifiers after removing faulty nodes are counted to obtain the actual scheduling capacity and the set of available computing cards after removal. For the video memory requirement value declared in the job description file of the sensitive task to be scheduled, the available video memory size of each card in the set of available computing cards after removal is queried and compared with the video memory requirement value. Card-level identifiers with a video memory size not less than the video memory requirement value are selected, and the card with the smallest difference between the video memory size and the video memory requirement value is selected as the matching computing card. A card-loading scheduling entry is created in the ready queue of the GPU cluster scheduler with the expected arrival time as the trigger time. When the system time reaches the expected arrival time, the loading action of the sensitive task to be scheduled is triggered according to the card-level identifier of the matching computing card, and the card-loading scheduling of the sensitive task is completed.

Images