RoCEv2 network resource scheduling method and system oriented to multiple GPU tasks

Abstract

The invention relates to the technical field of cloud network fusion, in particular to a RoCEv2 network resource scheduling method and system for multi-GPU tasks, and the method comprises the following steps: analyzing a communication portrait file submitted by a job, obtaining a set communication type, the number of participating nodes and the data volume, and obtaining a set communication file; and extracting a flow peak-to-average ratio, gathering communication operation times, a many-to-one communication concurrent flow number and a single GPU concurrent flow number, and calculating and generating a quantized network disturbance index. According to the method, the communication portrait analysis action is introduced before resource scheduling, the set communication type, the number of participating nodes, the data volume, the traffic peak-to-average ratio, the number of times of set communication operation and the number of concurrent streams of various types are obtained in advance, and the network disturbance characteristics of the communication task can be quantified in advance, so that access screening of task disturbance perception is realized; and through joint evaluation of the disturbance index and the background disturbance level of the RoCEv2 network isolation domain, the optimal resource putting position is screened, and the communication interference conflict is reduced.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of cloud network integration, and in particular to a RoCEv2 network resource scheduling method and system for multi-GPU tasks. BACKGROUND

[0002] Cloud network integration refers to a collaborative architecture system of computing resources and network resources, which breaks through the boundary between "cloud computing + network", realizes resource scheduling integration, service deployment automation and business experience optimization, and is a scheduling strategy for communication-intensive tasks in a distributed GPU cluster. Under the RoCEv2 network architecture, the communication demand and network bandwidth resources are matched, and the scheduling strategy is dynamically adjusted to ensure high throughput and low delay of data transmission requirements.

[0003] The prior art mainly relies on resource request parameters in the task submission stage and static link evaluation strategies for allocation in communication scheduling, which makes it difficult to match the real bandwidth consumption characteristics. In the task allocation link, only the node capacity and link load are used for simple selection, which easily causes continuous congestion in the local area. At the same time, there is a lack of feedback mechanism for the actual execution progress of the task, which makes it difficult to dynamically adjust the resource occupation position after the communication task execution deviates, and finally causes the coexistence of link idle and task blocking in the non-equilibrium state. For example, in multi-GPU concurrent training, the early completion of the communication stage of a certain node will cause the original resource allocation to be unable to be recovered, and the subsequent task still needs to wait for the original reserved time period, which reduces the scheduling flexibility and network resource utilization efficiency. Therefore, improvement is needed. SUMMARY

[0004] The purpose of the present application is to solve the shortcomings in the prior art, and to provide a RoCEv2 network resource scheduling method and system for multi-GPU tasks.

[0005] In order to achieve the above-mentioned purpose, the application adopts the following technical scheme, a RoCEv2 network resource scheduling method for multi-GPU tasks, comprising the following steps: Parse the communication image file submitted by the job, obtain the set communication type, the number of participating nodes and the data volume, extract the traffic peak-to-average ratio, the set communication operation frequency, the number of one-to-many communication concurrent streams and the number of single-GPU concurrent streams, and calculate the quantitative network disturbance index; According to the quantitative network disturbance index, obtain the background disturbance level of each RoCEv2 network isolation domain, screen all domains that meet the conditions, obtain a candidate network isolation domain list, select a distribution scheme for the admission of each domain in the candidate network isolation domain list, and establish a task admission and optimal isolation domain distribution decision; According to the task admission and optimal isolation domain allocation decision, the communication phase time points and data volume in the communication profile file are parsed, converted into bandwidth demand at specified time points on the target RoCEv2 network link, a sequenced RoCEv2 bandwidth demand sequence is obtained, and according to the sequenced RoCEv2 bandwidth demand sequence, the bandwidth surplus in the corresponding time slot of the calendar queue is checked, reserved, and a network resource calendar reservation map is established; According to the network resource calendar reservation map, the communication phase occurrence time points of the GPU task are monitored, the time points are subtracted from the reserved time points in the network resource calendar reservation map, a set of execution progress time deviation values is obtained, the set of execution progress time deviation values is applied to the reserved time points of all subsequent communication phases of the task, and a dynamically adjusted network resource scheduling strategy is generated.

[0006] Preferably, the step of obtaining the quantified network disturbance index is: The communication profile file is parsed, the set communication type field, the number of participating nodes field, and the data volume field are located, the records are rearranged in time sequence of communication phases and missing entries are removed, the set communication type, the number of participating nodes, and the data volume are integrated, and a communication profile analysis result is obtained; According to the communication profile analysis result, the start and end time and the transmission byte sequence are read segment by segment to calculate the instantaneous rate and the average rate, and abnormal segments are removed through a threshold value, the number of communication operations is counted and multiple pairs of concurrent and single GPU concurrent are counted in overlapping time windows, and a disturbance factor combination is obtained; According to the disturbance factor combination, the quantified network disturbance index is calculated.

[0007] Preferably, the step of obtaining the candidate network isolation domain list is: The quantified network disturbance index is parsed, the background disturbance level of each RoCEv2 network isolation domain is read and added to the quantified network disturbance index, and the domains that meet the conditions are reserved by comparing with a stable running threshold value, and a candidate network isolation domain list is obtained.

[0008] Preferably, the step of obtaining the task admission and optimal isolation domain allocation decision is: According to the candidate network isolation domain list, the expected disturbance water square deviation of each candidate domain is calculated; According to the expected disturbance water square deviation of each candidate domain, the expected disturbance water square deviations of all candidate domains are compared and the domain corresponding to the minimum value is selected, the admission result of the domain and the allocation relationship of the corresponding task are generated, and a task admission and optimal isolation domain allocation decision is formed.

[0009] Preferably, the step of obtaining the sequenced RoCEv2 bandwidth demand sequence is: According to the task admission and optimal isolated domain allocation decision, the communication phase time point and data volume of the communication profile file are read, overlapping sections are removed and merged according to the start and end time of the communication phase, the required bandwidth of each section is calculated and mapped to the specified time point of the target RoCEv2 network link, and a time-sequenced RoCEv2 bandwidth demand sequence is obtained.

[0010] Preferably, the network resource calendar reservation map acquisition step is: According to the time-sequenced RoCEv2 bandwidth demand sequence, the bandwidth margin of the corresponding time slot in the calendar queue is queried piece by piece, and the reservation is registered if the margin is sufficient, or the time slot is postponed within the job delay range or an alternative network path for the target RoCEv2 network link is planned if the margin is insufficient, to obtain a set of calendar queue reservation states; According to the set of calendar queue reservation states, the time slots of the registered reservations are merged and solidified into the calendar queue, and a mapping is generated for the postponed time slots and the alternative network path and marked as a conflict resolution state, forming a network resource calendar reservation map.

[0011] Preferably, the execution progress time deviation value set acquisition step is: According to the network resource calendar reservation map, the communication phase occurrence time point of the GPU task is continuously monitored, the communication phase occurrence time point and the reserved time point in the network resource calendar reservation map are paired piece by piece according to the communication phase serial number and GPU identifier, the nearest neighbor time matching is performed within the same communication phase start and end range and repeated matching is eliminated, the time difference between the communication phase occurrence time point and the reserved time point is calculated and the signs of the early and late are retained, and the missing entries are recorded as missing and a missing list is established, to obtain an execution progress time deviation value set.

[0012] Preferably, the dynamically adjusted network resource scheduling strategy acquisition step is: According to the execution progress time deviation value set, the time slot of the current communication phase in the calendar queue is located, the execution progress time deviation value of the corresponding communication phase is selected as a unified adjustment amount, the unified adjustment amount is applied to the reserved time points of all subsequent communication phases of the task, and the associated time slot is shifted as a whole in the calendar queue, the bandwidth margin is checked slot by slot and is distinguished as sufficient or insufficient, the reservation operation is performed and a temporary occupation marker is written if the bandwidth margin is sufficient, the time slot is moved forward or backward within the job delay range and the bandwidth margin check is repeated if the bandwidth margin is insufficient, and the alternative network path of the target RoCEv2 network link is searched in the network resource calendar reservation map and the reservation operation is initiated if it is still insufficient, to obtain an adjusted time slot reservation state set. According to the adjusted time slot booking state set, records marked as booked are solidified, a network resource calendar reservation map is updated, original time slots of records that have time slot movement are recovered and temporary occupation marks are cleared, records that use alternative network paths generate link mapping and fallback indications, and are bound to corresponding communication stages, are uniformly written into a task-level change log and a timeline table, and a dynamically adjusted network resource scheduling strategy is formed.

[0013] The application also provides a network resource scheduling system, comprising: The analysis module is configured to analyze a communication image file submitted by a task, obtain a set communication type, a number of participating nodes, and a data volume, extract a traffic peak-to-average ratio, a set communication operation number, a number of one-to-many communication concurrent streams, and a number of single-GPU concurrent streams, and calculate a quantitative network disturbance index; The admission and allocation module is configured to obtain a background disturbance level of each RoCEv2 network isolation domain according to the quantitative network disturbance index, filter all domains that meet the conditions, obtain a candidate network isolation domain list, select an allocation scheme for an admission condition of each domain in the candidate network isolation domain list, and establish a task admission and optimal isolation domain allocation decision; The resource reservation module is configured to analyze a communication stage time point and a data volume in the communication image file according to the task admission and optimal isolation domain allocation decision, convert the communication stage time point and the data volume into a bandwidth requirement at a specified time point on a target RoCEv2 network link, obtain a time-sequenced RoCEv2 bandwidth requirement sequence, check a bandwidth surplus in a corresponding time slot of a calendar queue according to the time-sequenced RoCEv2 bandwidth requirement sequence, perform booking, and establish a network resource calendar reservation map; The dynamic scheduling module is configured to monitor a communication stage occurrence time point of a GPU task according to the network resource calendar reservation map, perform a difference operation on the time point and a reserved time point in the network resource calendar reservation map, obtain a set of execution progress time deviation values, apply the set of execution progress time deviation values to reserved time points of all subsequent communication stages of the task, and generate a dynamically adjusted network resource scheduling strategy.

[0014] Compared with the prior art, the application has the advantages and positive effects that: The application can quantize the network disturbance characteristics of the communication task in advance, so as to realize the access screening of task disturbance perception, by introducing a communication image analysis action before resource scheduling, obtaining the set communication type, the number of participating nodes, the data volume, the peak-to-average ratio, the number of set communication operation times and the number of concurrent flows of multiple types in advance, and by jointly evaluating the disturbance index and the background disturbance level of the RoCEv2 network isolation domain, screening the optimal resource deployment position, reducing communication interference conflicts, by extracting the communication phase time point and data volume, converting it into bandwidth demand, and mapping it to the specific time slot of the RoCEv2 network link, realizing timing bandwidth reservation, matching the task bandwidth demand to the time dimension in the calendar queue and the link path, effectively reducing bandwidth conflicts and resource waste, and by comparing the difference between the actual execution time point and the reserved time point of the task, obtaining the execution progress time deviation, and feeding back the deviation to all reserved time points in the subsequent communication phase, dynamically shifting and adjusting the path in the calendar queue, continuously ensuring that resource use and task actual progress remain synchronized, suppressing resource idling and link conflicts, improving the resource utilization and task stability of the RoCEv2 network, and ensuring that the multi-GPU cluster continuously maintains a communication efficient, path controllable and scheduling orderly running state in a high concurrency environment. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The figure is a schematic diagram of the steps of the application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the application more clear and explicit, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0017] Please refer to Figure 1 The application provides a technical scheme, a RoCEv2 network resource scheduling method for multi-GPU tasks, including the following steps: Parse the communication image file submitted by the job, obtain the set communication type, the number of participating nodes and the data volume, extract the peak-to-average ratio, the number of set communication operation times, the number of one-to-many communication concurrent flows and the number of single-GPU concurrent flows, and calculate the generated quantitative network disturbance index; According to the quantitative network disturbance index, obtain the background disturbance level of each RoCEv2 network isolation domain, screen all domains that meet the conditions, obtain a candidate network isolation domain list, select a distribution scheme for the access of each domain in the candidate network isolation domain list, and establish a task access and optimal isolation domain distribution decision; According to the task admission and optimal isolation domain allocation decision, the communication phase time point and data volume in the communication profile file are analyzed, converted into the bandwidth demand of the specified time point on the target RoCEv2 network link, and a sequenced RoCEv2 bandwidth demand sequence is obtained. According to the sequenced RoCEv2 bandwidth demand sequence, the bandwidth surplus in the corresponding time slot of the calendar queue is checked, reserved, and a network resource calendar reservation map is established; According to the network resource calendar reservation map, the communication phase occurrence time point of the GPU task is monitored, the time point is subtracted from the reserved time point in the network resource calendar reservation map, an execution progress time deviation value set is obtained, the execution progress time deviation value set is applied to the reserved time point of all communication phases of the task, and a dynamically adjusted network resource scheduling strategy is generated.

[0018] The acquisition step of the quantified network disturbance index is: The communication profile file is analyzed, the set communication type field, the number of participating nodes field and the data volume field are located, the records are rearranged in the time order of the communication phase and the missing entries are removed, the set communication type, the number of participating nodes and the data volume are integrated, and a communication profile analysis result is obtained; According to the communication profile analysis result, the start and end time and the transmission byte sequence are read segment by segment to calculate the instantaneous rate and the average rate, and the abnormal segment is removed through the threshold value. The number of set communication operations is counted and the one-to-many concurrent and single GPU concurrent are counted in the overlapping time window to obtain a disturbance factor combination. According to the disturbance factor combination, the quantified network disturbance index is calculated, and the calculation formula is: ; Wherein, Z is the quantified network disturbance index, P is the peak-to-average ratio of traffic, R is the number of set communication operations, M is the number of participating nodes, U is the number of one-to-many communication concurrent streams, and G is the number of single GPU concurrent streams, is a risk weighting coefficient, is a synergistic disturbance coefficient.

[0019] Specifically, the communication profile file, specifically a log file stored in comma-separated values (CSV) format, where each row records a communication event, contains columns including 'timestamp' (nanosecond-level precision), 'communication primitive' (for example, AllReduce, Broadcast), 'participant node list' (identified by ID), 'GPU identification' (globally unique), and 'transmission byte number', first parse this file row by row, extract the 'communication primitive', 'participant node list' and 'transmission byte number' fields according to the 'communication primitive', 'participant node list' and 'transmission byte number' fields, parse the string form of 'participant node list' into the number of nodes, denoted as the number of participant nodes, then, sort all records in ascending order using the 'timestamp' field, establish a communication event sequence arranged in chronological order, after sorting, perform integrity checking on the sequence, define missing entries as any record with empty 'communication primitive', 'participant node number' or 'transmission byte number' fields, or 'transmission byte number' less than or equal to zero, traverse the entire sequence, remove all entries identified as missing, finally, structure and integrate the cleaned and sorted data, integrate the communication type, number of participant nodes and data volume of each communication event into a unified data structure, for example, a list containing multiple dictionary objects, where each dictionary represents a communication event, with keys 'type', 'node number', 'data volume' and 'timestamp', forming the communication profile analysis result required for subsequent analysis.

[0020] According to the communication profile analysis result, that is, a time-ordered communication event list, a fixed-size time window, such as 10 milliseconds, is set to process the data in segments, and in each time window, the total data volume is obtained by accumulating the 'transmission byte number' of all events, and then divided by the window duration (10 milliseconds) to calculate the average rate of the window. At the same time, the data volume of a single event in the window is divided by its duration (if no duration is recorded, a small time unit is used as an approximation) to obtain the instantaneous rate, and the maximum value of the instantaneous rate in all windows is recorded as the peak rate. An abnormal rate threshold is set, which is set with reference to the physical link bandwidth of the RoCEv2 network, for example, for a 200Gbps link, the theoretical upper limit is 25GB / s, considering the protocol overhead and measurement error, the threshold is set to 1.2 times of it, that is, 30GB / s, any calculated instantaneous rate exceeding this threshold is considered abnormal and is removed. In the complete event sequence after removing the abnormal segments, the total number of records is directly counted to obtain the number of set communication operations. Then, in order to count the concurrent streams, a sliding time window with a width of 50 milliseconds and a step of 10 milliseconds is used to traverse the entire event sequence. In each sliding window, check all events with 'communication primitive' of one-to-many type (such as Reduce, Gather), count the number of overlaps in their time intervals, and record the maximum overlap number in all windows as the number of one-to-many communication concurrent streams. At the same time, in the same sliding window, group the events by 'GPU identifier', calculate the number of concurrent communication events participated by each GPU, and record the maximum number of concurrent streams of all GPUs in all windows as the number of single-GPU concurrent streams. Finally, the traffic peak-to-average ratio (the ratio of the peak rate to the average rate of all windows), the number of set communication operations, the number of one-to-many communication concurrent streams, and the number of single-GPU concurrent streams calculated through the above process are combined to form the disturbance factor combination required for subsequent calculation.

[0021] Formula: Logarithmic function Used to smooth the traffic peak-to-average ratio to avoid its disproportionate dominance over the exponential result, while the use of square root aims to balance the disturbance factors of different physical dimensions and integrate them into a unified scale. The formula adds the basic communication frequency disturbance to the complex concurrent congestion disturbance , which reflects the combined effect of the two disturbance sources. The quadratic form inside the congestion term captures the non-linear aggravation effect of concurrent stream quantity growth on network disturbance, and especially through the coefficients and the risks of different concurrent modes are weighted. This structured design makes the final disturbance index not only comprehensive but also highlights the key risk points.

[0022] is the peak-to-average ratio of traffic, which is a key indicator of network traffic burstiness, and its value is equal to the ratio of the peak rate to the average rate of network traffic during task execution. The parameter is obtained by analyzing the communication profile analysis results of the previous step. First, a time window, such as 100 microseconds, is set, and the entire task communication profile data is traversed. The total number of transmitted bytes in each time window is calculated, and the window duration is divided to obtain a series of instantaneous rate values. The maximum value in the series is the peak rate At the same time, the total number of transmitted bytes during the entire task is calculated, and the total duration of the task is divided to obtain the average rate The formula for calculating the peak-to-average ratio of traffic is: For example, in the communication profile analysis of an AI training task, it is monitored that the peak instantaneous traffic reaches 80 GB / s, and the entire task runs for 3600 seconds, with a total communication data volume of 72000 GB, and the average rate is Then the peak-to-average ratio of traffic is .

[0023] is the number of collective communication operations, representing the total number of collective communication operations performed in the entire life cycle of the task, reflecting the frequency of task communication. This parameter is obtained by directly analyzing the communication profile file. After preprocessing the original log file, including sorting by time and removing missing or invalid entries, the remaining valid records are counted. Each valid record corresponds to an independent collective communication operation (such as an AllReduce or a Broadcast). The total number is the value of For example, processing a job log containing millions of communication events, after removing 3% of invalid entries through data cleaning, the remaining valid communication event records are 970000. Then the number of collective communication operations .

[0024] is the number of participating nodes, indicating the total number of computing nodes participating in the multi-GPU task. This parameter is directly read from the communication profile file configured when the job is submitted, and is usually represented as a static metadata field. It represents the distributed scale of the task. For example, a large-scale language model training job is scheduled to be executed on a computing cluster composed of 32 servers, each server acting as a node participating in collective communication. Then the number of participating nodes .

[0025] The number of concurrent many-to-one communication flows represents the maximum number of many-to-one communication operations (such as Reduce and Gather) occurring simultaneously at any given time point during task execution. It reflects the congestion risk at the network's aggregation points. To obtain this parameter, a sliding time window (e.g., window width 50 milliseconds, sliding step size 10 milliseconds) is used to scan the entire communication event sequence. Within each window, all many-to-one communication events are identified, and the maximum number of overlapping time intervals is counted. This process is repeated across all time windows, and the maximum value is taken as the [value / percentage]. For example, when analyzing the communication timing of a job, it was found that within a certain 50-millisecond window, a maximum of 12 different Reduce operations overlapped in time. This leads to the number of concurrent one-to-one communication streams. .

[0026] The single-GPU concurrent stream count refers to the maximum number of communication streams that a single GPU can handle simultaneously during task execution. This parameter measures the load on the network interface controller (NIC) of a single endpoint, and its acquisition method is similar to... Similarly, a sliding time window is also used, but within each window, data is grouped by GPU identifier. Then, the maximum temporal overlap of communication events involving each GPU is calculated. Finally, the global maximum value of this overlap is found across all GPUs and all time windows. For example, analysis revealed that GPU-5 in the job simultaneously participated in 1 AllReduce, 2 Send, and 2 Recv operations within a certain time window, resulting in a total concurrent stream of 5. Meanwhile, the concurrent streams of other GPUs did not exceed 5. Therefore, the number of concurrent streams per GPU... .

[0027] This is a risk-weighted coefficient used to adjust the contribution weight of many-to-one communication concurrency to network disturbances. Its setting is based on the sensitivity of the network topology to "Incast" congestion. Topologies with high bipartite bandwidth, such as Fat-Tree or Clos, have a higher tolerance for this type of congestion. The values ​​are relatively low, while torus or mesh topologies are more sensitive. The value should be set higher, and the specific value of the coefficient should be determined through regression analysis of historical performance data. The calculation formula is as follows: ,in It is the number of many-to-one concurrent flows in historical data. The Pearson correlation coefficient between the latency of the jobs and the average communication latency was calculated by analyzing 5,000 GPU jobs that ran in the data center over the past six months. The risk weighting coefficient is 0.22, and the correlation coefficient with communication delay is 0.22 .

[0028] The synergistic disturbance coefficient quantifies the additional additive disturbance effect when multiple-to-one communication concurrency ( ) occurs simultaneously with single-GPU concurrency ( ). The coefficient is set based on benchmarking by separately increasing and , and both, in a controlled environment to measure the degree of non-linear degradation of network performance, and is set by the formula: where is the delay increment when only is increased, is the delay increment when only is increased, is the delay increment when both and are increased, is the baseline delay. For example, the benchmarking shows that the baseline delay is 10 microseconds, increasing to 12 alone results in a 4-microsecond increase in delay, increasing to 5 alone results in a 2-microsecond increase in delay, and increasing both results in a total of 9-microsecond increase in delay, so the synergistic disturbance coefficient .

[0029] According to the parameters, the calculation is as follows: According to the acquisition process of the aforementioned parameters, a set of specific parameter values is obtained: Traffic peak-to-average ratio ; Set communication operation times ; Number of participating nodes ; Multiple-to-one communication concurrency stream quantity ; Single-GPU concurrency stream quantity ; Risk weighting coefficient ; Synergistic disturbance coefficient ; Substitute these values into the calculation formula for the network disturbance index: ; ; ; ; ; ; ; ; The result shows that the quantified network disturbance index of this multi-GPU task is 225.46, which is a very high value. According to the preset disturbance level grading standard (for example, 0-50 is low disturbance, 50-150 is medium disturbance, and above 150 is high disturbance), this task is determined as a high disturbance task, and a network isolation domain with a very low current background disturbance level must be found for this task, or in the case that the background disturbance of all domains is high, the access request of this task is rejected to avoid causing serious impact on the existing network environment, thereby ensuring the stability of the communication performance of the entire cluster.

[0030] The obtaining step of the candidate network isolation domain list is: The quantified network disturbance index is analyzed, the background disturbance level of each RoCEv2 network isolation domain is read, and the quantified network disturbance index is added to each domain. The domains that meet the conditions are reserved by comparing with the stable running threshold value to obtain the candidate network isolation domain list.

[0031] Specifically, according to the quantified network disturbance index obtained by the foregoing steps, the real-time background disturbance level of all RoCEv2 network isolation domains needs to be obtained first. This background disturbance level is a comprehensive index, which is calculated by periodically (for example, every 10 seconds) collecting the key performance telemetry data of the switches in each domain. The specific calculation is to weight and sum the normalized average queue depth, normalized ECN (Explicit Congestion Notification) marked packet rate, and normalized packet drop rate of all switches in the domain. The weight coefficients are determined according to the correlation between each index and the application performance decline in historical data analysis. For example, the queue depth weight is 0.5, the ECN marking rate is 0.3, and the drop rate is 0.2. Then, the quantified network disturbance index of the task to be scheduled is added to the current background disturbance level of each domain to obtain an expected future disturbance level. Then, the expected disturbance level is compared with a preset stable running threshold value. The threshold value is set based on the statistical analysis of the performance data and network state logs of all completed GPU jobs in the data center in the past year to determine at what disturbance level more than 5% of the jobs have more than 10% performance decline. The 90% of this critical disturbance level is set as the stable running threshold value. For example, if the critical value is 400, the stable running threshold value is set to 360. For each network isolation domain, only when the expected future disturbance level is lower than the stable running threshold value, the domain is considered to be able to carry the new task without causing the deterioration of the overall performance. The identifiers of all domains that pass this check are collected to finally form a list, i.e., the candidate network isolation domain list.

[0032] The task admission and optimal isolated domain allocation decision obtaining step is: According to the candidate network isolated domain list, the expected disturbance variance of each candidate domain is calculated, and the calculation formula is: ; Wherein, is the expected disturbance variance corresponding to the candidate domain j, D is the total number of RoCEv2 network isolated domains, d is the domain index from 1 to D, is the future background disturbance level of the dth RoCEv2 network isolated domain after the candidate domain j is allocated with the task, when , there is , when , there is , is the average future background disturbance level after the candidate domain j is allocated with the task, is the background disturbance level of the kth RoCEv2 network isolated domain, and Z is the quantified network disturbance index; According to the expected disturbance variance of each candidate domain, the expected disturbance variances of all candidate domains are compared, and the domain corresponding to the minimum value is selected to generate the admission result of the domain and the allocation relationship of the corresponding task, and the task admission and optimal isolated domain allocation decision is formed.

[0033] Specifically, the formula is: The above formula provides a globally optimal decision mechanism for selecting one with the least impact on the balance of the entire cluster network ecosystem from multiple network isolated domains that meet the basic admission conditions. By calculating the variance, the dispersion degree of the disturbance level distribution of all domains in the entire network after placing a task in a certain domain is estimated. A lower variance value means that after the task is allocated to the domain , the load (disturbance level) of all domains will be more balanced, avoiding the situation that some domains are excessively idle while other domains become extremely congested due to the addition of new tasks. This strategy helps to improve the resource utilization of the entire cluster and the performance stability of all running tasks. By minimizing the global disturbance variance, this decision process can actively maintain the health of the network environment and prevent the occurrence of global performance bottlenecks caused by local optimal decisions.

[0034] D is the total number of RoCEv2 network isolated domains, representing the number of independent congestion control or management units that the data center network is divided into. This division is usually determined by the network architecture. For example, a cluster with 8 cabinets, each cabinet constitutes an isolated domain, and In this example, it is assumed that the data center has 4 RoCEv2 network isolated domains, i.e. .

[0035] Future background disturbance level of the dth RoCEv2 network isolated domain after assigning task to candidate domain j, which is a conditional value, when calculating the variance of domain , if domain index is itself, then its future disturbance level is the sum of the current background disturbance and the new task disturbance index ; if is not , then its disturbance level remains unchanged, still .

[0036] Future background disturbance level average after assigning task to candidate domain j, the global average disturbance level is recalculated after adding the disturbance index of the new task to the total disturbance of the entire network.

[0037] Background disturbance level of the kth RoCEv2 network isolated domain, this value is obtained by real-time monitoring system, is a quantitative score of multiple indicators such as network delay, packet loss rate, congestion signal, etc. In this example, the monitoring system reports that the background disturbance levels of the current four domains are: , , , .

[0038] Quantitative network disturbance index, the value of this parameter comes from the analysis and calculation results of the task communication portrait in the previous step, according to the aforementioned example, the index value of the high disturbance task calculated is .

[0039] According to the parameters, the calculation is as follows: For example, all four domains pass the stable running threshold check and become candidate domains, and four cases of assigning tasks to domains 1, 2, 3, and 4 need to be calculated respectively.

[0040] Case 1: Task assigned to domain 1 (calculate ): First, calculate the future background disturbance level average after task assignment : ; Then calculate the future background disturbance level of each domain : ; ; ; ; Finally, the variance is calculated : ; ; ; ; Case 2: 15617.84 is calculated; Case 3: 8887.65 is calculated; Case 4: 11181.7 is calculated; The results show that the expected global disturbance variance generated by assigning tasks to four different candidate domains is 6126.63, 15617.84, 8887.65 and 11181.7, respectively.

[0041] According to the expected disturbance variance set of each candidate domain calculated in the previous step, i.e. , the optimal allocation decision process is started. First, check if the candidate network isolation domain list is empty. If it is empty, it is determined that the task cannot be admitted, and a decision record containing the task ID and the "REJECTED" state is generated. If the list is not empty, continue to execute the selection logic, initialize two variables, one for storing the minimum variance value found so far, initially set to a maximum value, and the other for recording the corresponding domain ID, initially empty. Then, iterate through the expected disturbance variance set, compare the variance value of each domain with the current minimum variance value. In the first comparison, assign the value 6126.63 of to the minimum variance variable, and assign the domain ID 1 to the optimal domain variable. Then, compare the value 15617.84 of , as it is greater than the current minimum value, the variable is not changed. Similarly, and also do not trigger updates because they are greater than 6126.63. After the iteration is complete, the value of the minimum variance variable is 6126.63, and the value of the optimal domain variable is 1. According to this, the final decision result is generated, which is a structured data object that clearly indicates that the task is "ADMITTED" and establishes the allocation relationship between the task and the optimal domain, i.e. the task is allocated to the RoCEv2 network isolation domain 1. This decision also includes the basis for selection, i.e. the minimum expected disturbance variance value 6126.63. Finally, the complete record containing the admission status, allocation relationship and decision metric value is solidified to form the task admission and optimal isolation domain allocation decision.

[0042] The obtaining step of the time-sequenced RoCEv2 bandwidth requirement sequence is: According to the task admission and optimal isolated domain allocation decision, the communication phase time points and data volume of the communication portrait file are read, overlapping sections are removed and merged according to the start and end times of the communication phase, the required bandwidth of each section is calculated and mapped to the specified time points of the target RoCEv2 network link, and a time-sequenced RoCEv2 bandwidth requirement sequence is obtained.

[0043] Specifically, according to the task admission and optimal isolated domain allocation decision, first, the task ID and the allocated target RoCEv2 network isolated domain ID are extracted from the decision result, then the corresponding original communication portrait file is re-accessed and loaded using the task ID, and records containing the communication phase start timestamp, end timestamp and transmission data volume are read from it, these records are used to form an initial time period list, then, the list is sorted in ascending order according to the start timestamp, and then the sorted list is traversed to merge overlapping or continuous time periods, the specific method is to maintain a merged result list, start with the first time period of the initial list, compare it with the next time period, if the end time of the current time period is greater than or equal to the start time of the next time period, it is considered that they overlap or are continuous, at this time, the two time periods are merged into a new time period, the start time is the start time of the current period, the end time is the larger value of the two end times, and the data volume is the sum of the data volumes of the two, the new merged time period is used to replace the current period, and the comparison with the subsequent time period is continued, until all time periods are processed, thus a series of independent communication bursts represented by phase time periods and their total data volume are obtained, for each merged time period, the required average bandwidth of the phase is calculated by dividing the total data volume by the duration of the time period (end time minus start time), finally, these bandwidth requirements, their time points, and the specific RoCEv2 network link set calculated according to the node location and the network topology in the target isolated domain are associated to generate a sequence containing multiple entries, each entry is in the format (link ID, start time, end time, required bandwidth), which together form the final time-sequenced RoCEv2 bandwidth requirement sequence.

[0044] The obtaining step of the network resource calendar reservation map is: According to the time-sequenced RoCEv2 bandwidth requirement sequence, the bandwidth surplus is queried in the corresponding time slot of the calendar queue, the reservation is registered if the surplus is sufficient, and the time slot is postponed within the job delay range or an alternative network path is planned for the target RoCEv2 network link if the surplus is insufficient, and a set of calendar queue reservation states is obtained. According to the calendar queue reservation state set, the time slot of the registered reservation is merged and solidified to the calendar queue, the mapping of the postponed time slot and the alternative network path is generated and the conflict resolution state is marked, and the network resource calendar reservation graph is formed.

[0045] Specifically, according to the time-sequenced RoCEv2 bandwidth demand sequence, first, the data structure of the calendar queue needs to be defined, which is a two-dimensional data structure with network link ID as the first index and time slot (for example, fixed width of 10 milliseconds) as the second index, and each cell stores the total amount of bandwidth reserved in the time slot. Then, each bandwidth reservation request (link ID, start time, end time, required bandwidth) in the time-sequenced RoCEv2 bandwidth demand sequence is processed one by one. For a request, first, the start and end times are converted to the corresponding calendar queue time slot index range, and then all time slots in this range are traversed. In each time slot, check whether the remaining amount of the total physical bandwidth of the link after subtracting the reserved bandwidth is greater than or equal to the required bandwidth of the current request. If the bandwidth of all related time slots is sufficient, mark the request as "reservation success" and record it in a temporary reservation state list. If the bandwidth of any time slot is insufficient during the checking process, start the conflict resolution mechanism. The first strategy is time postponement. According to the maximum allowed delay set when the job is submitted, for example, 60 seconds for high-priority jobs and 300 seconds for ordinary jobs, try to shift the reservation time point of all communication stages of the entire task backward by one time slot (10 milliseconds), and then perform full bandwidth capacity check on all communication stages again. Repeat this process until a conflict-free time window is found within the allowed delay range. If found, mark all requests as "postponed reservation success" and record the delay amount. If it is still not found after exceeding the maximum delay, start the second strategy, path re-planning. For the target RoCEv2 network link that causes conflict, query the network topology database to find one or more alternative physical paths from the source to the destination, and perform bandwidth capacity check on the corresponding time slots of all links on the alternative path. If a complete alternative path that meets the bandwidth demand is found, mark the request as "re-routed reservation success" and record the alternative path information. If all strategies fail, mark it as "reservation failure". After the processing results of all requests are summarized, the calendar queue reservation state set is obtained.

[0046] According to the calendar queue reservation state set, all records marked as "reservation success", "delayed reservation success" or "reroute reservation success" are processed to finally determine the network resource allocation. First, the bandwidth values requested by all "reservation success" records are added to the reserved bandwidth values of the corresponding time slots in the calendar queue of the corresponding link. This operation is permanent, i.e. solidified reservation. Then, the "delayed reservation success" records are processed. According to the delay amount recorded, the new reservation time slot range is calculated, and the bandwidth reservation is solidified in these new time slots of the calendar queue. At the same time, a time translation mapping entry in the format (task ID, original communication phase time, adjusted communication phase time) is generated and stored in a global mapping table. For "reroute reservation success" records, the corresponding bandwidth reservation is solidified in the specified time slots on each link included in the alternative network path, and a path mapping entry in the format (task ID, original communication phase link, alternative path link set) is generated and also stored in the mapping table. After completing the solidification of all successful reservations and the generation of mapping, the states of these records are updated to "conflict removed". For records marked as "reservation failure", no operation is performed, and only the failure state is recorded. Finally, the updated calendar queue itself (including all solidified reservations), together with all generated time translation mappings and path mappings, are logically integrated to form a complete and dynamic network resource reservation view, i.e. the network resource calendar reservation graph.

[0047] The acquisition step of the execution progress time deviation value set is as follows: According to the network resource calendar reservation graph, the communication phase occurrence time points of the GPU task are continuously monitored. The communication phase occurrence time points are paired with the reservation time points in the network resource calendar reservation graph according to the communication phase sequence number and the GPU identifier. The nearest neighbor time matching is performed within the same communication phase range, and repeated matching is eliminated. The time difference between the communication phase occurrence time point and the reservation time point is calculated, and the symbols of the early and late are retained. The missing entries are recorded as missing and a missing list is established. The execution progress time deviation value set is obtained.

[0048] Specifically, according to the network resource calendar reservation diagram, first, a background monitoring process is started, which captures the start and end events of each collective communication operation in each GPU task in real time by subscribing to the event hooks provided by the GPU driver or related runtime library (such as NCCL), and these events contain timestamps, communication phase sequence numbers, and the GPU identifier that initiates the operation. When a communication event is detected, its occurrence time point is immediately matched with the time point reserved for the task in the network resource calendar reservation diagram. The matching process is grouped by communication phase sequence number and GPU identifier. For each group, the actual occurrence time point of the communication event is paired with the reserved time point one by one. The pairing rule is to find the reserved point with the smallest absolute time difference from the actual occurrence time point within the reserved communication phase start and end time range. To avoid a reserved point being repeatedly matched by multiple actual events, once a match is successful, the reserved point is marked as used and no longer participates in subsequent matching. After matching is completed, the time difference between each successfully paired actual occurrence time point and reserved time point is calculated. This difference has a positive or negative sign, with a positive value indicating that the actual execution is later than the reserved time (late), and a negative value indicating that it is earlier than the reserved time (early). All these differences constitute the initial time deviation value set. For reserved points that do not find any actual communication events within the reserved time window, they are marked as missing, and their related information (communication phase sequence number, GPU identifier, reserved time) is recorded in a separate missing list. Finally, all calculated time difference values and missing lists are integrated to form the execution progress time deviation value set.

[0049] The obtaining step of the dynamically adjusted network resource scheduling strategy is: According to the execution progress time deviation value set, the time slot of the current communication phase in the calendar queue is located, the execution progress time deviation value of the corresponding communication phase is selected as the unified adjustment amount, the unified adjustment amount is applied to the reserved time points of all subsequent communication phases of the task, and the associated time slot is shifted as a whole in the calendar queue. Check the bandwidth margin and distinguish between sufficient and insufficient. If the bandwidth margin is sufficient, perform the reservation operation and write the temporary occupancy marker. If the bandwidth margin is insufficient, move the time slot forward or backward within the job delay range and repeat the bandwidth margin check. If it is still insufficient, retrieve the alternative network path of the target RoCEv2 network link in the network resource calendar reservation diagram and initiate the reservation operation to obtain the adjusted time slot reservation state set. According to the adjusted time slot reservation state set, the records marked as reserved are solidified and the network resource calendar reservation diagram is updated, the records of time slot movement are recycled and the temporary occupancy marker is cleared, the records using alternative network paths are generated link mapping and rollback indication, and are bound to the corresponding communication phase. Write to the task-level change log and time axis table uniformly to form the dynamically adjusted network resource scheduling strategy.

[0050] Specifically, according to the execution progress time deviation value set, first locate the current communication phase being executed or about to be executed, determine by comparing the current system time with the time axis in the network resource calendar reservation map, once located, filter out all time deviation values belonging to this communication phase from the execution progress time deviation value set, and calculate their average value, this average value is selected as the uniform adjustment amount of this dynamic adjustment, then apply this uniform adjustment amount to the reserved time points of all subsequent communication phases of the task that have not been executed, that is, add this adjustment amount to the start and end time stamps of each reserved time point, generate a new set of adjusted reserved time sequence, then in the calendar queue, all time slots associated with these communication phases are translated as a whole, and each time slot after translation is rechecked for bandwidth margin, the checking result is divided into two categories: sufficient and insufficient, for the time slot with sufficient bandwidth margin, immediately perform the booking operation, write the required bandwidth value, and add a "temporary occupation" mark, for the time slot with insufficient bandwidth margin, within the delay range allowed by the job (for example, the aforementioned 60 seconds or 300 seconds), try to move forward or backward in steps of a time slot, prefer to move forward to try to catch up with the progress, after each move, repeat the bandwidth margin check, until a available position is found or the delay range is exceeded, if the moving strategy fails, retrieve the alternative network path planned in advance for the conflict link in the network resource calendar reservation map, and initiate a new booking operation process for all links on the alternative path, the results (success, failure, moving amount, alternative path) of all these trial booking operations are collected to obtain the adjusted time slot booking state set.

[0051] According to the adjusted time slot booking state set, the final solidification and logging operation is performed on each record in the set. First, all records marked as "booked" are traversed. For these records, the corresponding time slot with a "temporary occupation" mark in the calendar queue is found, and the temporary mark is removed to make it a formal and solidified booking. This operation is equivalent to updating the network resource calendar reservation graph. Then, records that are successfully booked due to time slot movement are processed. In addition to solidifying the new time slot occupation, the original time slot occupied by them also needs to be recovered, i.e., the booking bandwidth value in the original time slot is reduced by the part occupied by the task, and all related temporary occupation marks are cleared. Then, for records that are successfully booked by replacing the network path, a new link mapping relationship is generated, which clearly indicates which set of new links is temporarily replaced for the original network link in a specific communication phase. At the same time, a rollback instruction is attached, indicating that the original path should be tried to be restored at the next dynamic adjustment or task end. This mapping relationship is bound to the corresponding communication phase and stored in the global mapping table. Finally, all the above change operations, including time slot translation, occupation state change, and link mapping switching, are written into a task-level change log indexed by task ID, along with the time stamp and reason of the change (i.e., the execution progress time deviation value relied on), and a time axis reference table is updated synchronously. This reference table clearly shows the correspondence between the original reservation time and the actual reservation time after each adjustment. After all these operations are completed, the network resource scheduling strategy with traceable dynamic adjustment is formed.

[0052] The above is only a preferred embodiment of the present application, and does not limit the present application in other forms. Any skilled person in the art can modify or change the above disclosed technical content to equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments without departing from the technical solution content of the present application still falls within the protection scope of the present application.

Claims

1. A RoCEv2 network resource scheduling method for multi-GPU tasks, characterized in that, The method comprises the following steps: The communication image file submitted by the task is parsed to obtain the collection communication type, the number of participating nodes and the data volume, the peak-to-average traffic ratio, the number of collection communication operations, the number of one-to-many communication concurrent streams and the number of single-GPU concurrent streams are extracted, and the quantitative network disturbance index is calculated and generated; According to the quantitative network disturbance index, the background disturbance level of each RoCEv2 network isolation domain is obtained, all domains meeting the conditions are screened, a candidate network isolation domain list is obtained, the access situation of each domain in the candidate network isolation domain list is selected and a distribution scheme is allocated, and a task access and optimal isolation domain distribution decision is established; According to the task access and optimal isolation domain distribution decision, the communication phase time point and data volume in the communication image file are parsed and converted into the bandwidth demand of the specified time point on the target RoCEv2 network link to obtain a time-sequenced RoCEv2 bandwidth demand sequence, and the time-sequenced RoCEv2 bandwidth demand sequence is used to check the bandwidth surplus in the corresponding time slot of the calendar queue and make a reservation to establish a network resource calendar reservation map; According to the network resource calendar reservation map, the communication phase occurrence time point of the GPU task is monitored, the time point is subtracted from the reserved time point in the network resource calendar reservation map to obtain a set of execution progress time deviation values, the set of execution progress time deviation values is applied to the reserved time points of all subsequent communication phases of the task, and a dynamically adjusted network resource scheduling strategy is generated.

2. The RoCEv2 network resource scheduling method for multi-GPU task according to claim 1, characterized in that, The obtaining step of the quantitative network disturbance index is: The communication image file is parsed to locate the collection communication type field, the number of participating nodes field and the data volume field, the records are rearranged in the time sequence of the communication phase and the missing entries are removed, the collection communication type, the number of participating nodes and the data volume are integrated to obtain a communication image analysis result; According to the communication image analysis result, the start and end time and the transmission byte sequence are read segment by segment to calculate the instantaneous rate and the average rate, and the abnormal segments are removed by threshold, the number of collection communication operations is counted, and the one-to-many concurrent and single-GPU concurrent are counted in the overlapping time window to obtain a disturbance factor combination; According to the disturbance factor combination, the quantitative network disturbance index is calculated.

3. The method of claim 1, wherein, The obtaining step of the candidate network isolation domain list is: The quantitative network disturbance index is parsed to read the background disturbance level of each RoCEv2 network isolation domain, and the background disturbance level of each RoCEv2 network isolation domain is added to the quantitative network disturbance index, the domains meeting the conditions are reserved by comparing the stable running threshold, and the candidate network isolation domain list is obtained.

4. The method of claim 1, wherein, The obtaining step of the task access and optimal isolation domain distribution decision is: According to the candidate network isolation domain list, the expected disturbance square deviation of each candidate domain is calculated; According to the expected disturbance square deviation of each candidate domain, the expected disturbance square deviations of all candidate domains are compared, and the domain corresponding to the minimum value is selected to generate the access result of the domain and the allocation relationship of the corresponding task, and a task access and optimal isolation domain distribution decision is formed.

5. The method of claim 1, wherein, The obtaining step of the time-sequenced RoCEv2 bandwidth demand sequence is: According to the task admission and optimal isolation domain allocation decision, the communication phase time point and data volume of the communication profile file are read, overlapping segments are removed and merged according to the start and end time of the communication phase, the required bandwidth of each segment is calculated and mapped to the specified time point of the target RoCEv2 network link, and the time-sequenced RoCEv2 bandwidth requirement sequence is obtained.

6. The method of claim 1, wherein, The network resource calendar reservation map acquisition step is: According to the time-sequenced RoCEv2 bandwidth requirement sequence, the bandwidth margin of the corresponding time slot in the calendar queue is queried, the reservation is registered if the margin is sufficient, and the time slot is postponed within the job delay range or the alternative network path for the target RoCEv2 network link is planned if the margin is insufficient, to obtain the calendar queue reservation state set; According to the calendar queue reservation state set, the time slots of the registered reservations are merged and solidified into the calendar queue, the mapping of the postponed time slots and the alternative network path is generated and the conflict resolution state is marked, and the network resource calendar reservation map is formed.

7. The method of claim 1, wherein, The acquisition step of the execution progress time deviation value set is: According to the network resource calendar reservation map, the communication phase occurrence time point of the GPU task is continuously monitored, the communication phase occurrence time point is paired with the reserved time point in the network resource calendar reservation map according to the communication phase sequence number and GPU identifier, the nearest neighbor time matching is performed within the same communication phase start and end range, and the repeated matching is eliminated, the time difference between the communication phase occurrence time point and the reserved time point is calculated, and the symbols of the early and late are retained, the missing entries are recorded as missing and the missing list is established, and the execution progress time deviation value set is obtained.

8. The method of claim 1, wherein, The acquisition step of the dynamically adjusted network resource scheduling strategy is: According to the execution progress time deviation value set, the time slot of the current communication phase in the calendar queue is located, the execution progress time deviation value of the corresponding communication phase is selected as the unified adjustment amount, the unified adjustment amount is applied to the reserved time point of all subsequent communication phases of the task, and the associated time slot is shifted as a whole in the calendar queue, the bandwidth margin is checked slot by slot and is distinguished as sufficient or insufficient, the reservation operation is performed and the temporary occupation mark is written if the bandwidth margin is sufficient, the time slot is moved forward or backward within the job delay range and the bandwidth margin check is repeated if the bandwidth margin is insufficient, the alternative network path of the target RoCEv2 network link is retrieved in the network resource calendar reservation map and the reservation operation is initiated if it is still insufficient, and the adjusted time slot reservation state set is obtained. According to the adjusted time slot reservation state set, the records marked as reserved are solidified and the network resource calendar reservation map is updated, the original time slot is recycled and the temporary occupation mark is cleared for the records with time slot movement, the link mapping and rollback indication are generated for the records using the alternative network path, and are bound to the corresponding communication phase, the task-level change log and time axis correspondence table are uniformly written, and the dynamically adjusted network resource scheduling strategy is formed.

9. The network resource scheduling system for multi-GPU task oriented RoCEv2 network resource scheduling method according to any one of claims 1-8, characterized in that, It includes: The analysis module is used for analyzing the communication profile file submitted by the job, obtaining the set communication type, the number of participating nodes and the data volume, extracting the traffic peak-to-average ratio, the set communication operation number, the number of one-to-many communication concurrent streams and the number of single GPU concurrent streams, and calculating the generated quantitative network disturbance index; An admission and distribution module is configured to obtain a background disturbance level of each RoCEv2 network isolated domain according to the quantized network disturbance index, screen all domains meeting the condition, obtain a candidate network isolated domain list, select an allocation scheme for the admission of each domain in the candidate network isolated domain list, and establish a task admission and optimal isolated domain allocation decision; A resource reservation module is configured to parse a communication phase time point and a data volume in a communication profile file according to the task admission and optimal isolated domain allocation decision, convert the communication phase time point and the data volume into a bandwidth demand at a specified time point on a target RoCEv2 network link, obtain a time-sequenced RoCEv2 bandwidth demand sequence, check a bandwidth surplus in a corresponding time slot of a calendar queue according to the time-sequenced RoCEv2 bandwidth demand sequence, make a reservation, and establish a network resource calendar reservation map; A dynamic scheduling module is configured to monitor a communication phase occurrence time point of a GPU task according to the network resource calendar reservation map, perform a difference operation on the time point and a reserved time point in the network resource calendar reservation map, obtain a set of execution progress time deviation values, apply the set of execution progress time deviation values to a reserved time point of a subsequent communication phase of the task, and generate a dynamically adjusted network resource scheduling strategy.

Images