A method and system for cross-cluster management software to schedule heterogeneous GPUs

Abstract

The application provides a method and system for scheduling heterogeneous GPUs by cross-cluster management software, and relates to the technical field of GPU resource scheduling, wherein the method comprises obtaining physical topology data, hardware specification data and historical task records of GPU nodes through cross-cluster management software, establishing an association matrix between GPU units and generating a topology grouping set; based on the topology detection analysis, calculating the communication performance index, constructing a topology graph model and determining the optimal communication path of the multi-card task; combining the multi-objective optimization algorithm to generate a perception scheduling strategy and selecting the optimal GPU combination from the grouping set; in the task execution process, dynamically adjusting the scheduling strategy according to the real-time cluster state and task performance data, maintaining the continuous adaptation of the physical topology structure and the task demand, and finally realizing the efficient collaborative scheduling of cross-cluster heterogeneous GPU resources. The application improves the scheduling efficiency of heterogeneous GPU resources and the overall performance of the system in the distributed computing scenario.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Graphics Processing Unit (GPU) resource scheduling, in particular to a method and system for scheduling heterogeneous GPUs by cross-cluster management software. BACKGROUND

[0002] With the continuous growth of artificial intelligence and large-scale computing demand, efficient scheduling technology of cross-cluster heterogeneous GPU resources is increasingly becoming the key to improving the overall performance of distributed computing. Such technology can be widely applied in cloud gaming, deep learning training, and large-scale scientific computing, etc. and has broad application prospects.

[0003] Some existing technologies use multi-objective optimization algorithms for heterogeneous GPU scheduling, for example, by using genetic algorithms or particle swarm optimization methods to balance multiple objectives such as computing efficiency and energy consumption, thereby achieving preliminary optimized allocation of resources. Some solutions further combine static topology-aware methods, i.e., based on fixed hardware connection topology information to assist in scheduling decision-making.

[0004] However, the existing scheduling methods have limited adaptability in dynamically changing cluster environments. The predetermined scheduling strategy is prone to mismatch with the real-time fluctuating cluster running state and task demand, which leads to a decline in resource utilization efficiency and affects the overall performance of collaborative computing between multiple GPU cards. Therefore, the existing technology generally has a technical problem that efficient resource scheduling cannot be achieved in a dynamic environment. SUMMARY

[0005] The present application provides a method and system for scheduling heterogeneous GPUs by cross-cluster management software to solve the problems of low scheduling efficiency of heterogeneous GPU resources and poor overall system performance in the existing distributed computing scenario.

[0006] To solve the above technical problems, in a first aspect, the present application provides a method for scheduling heterogeneous GPUs by cross-cluster management software, comprising:

[0007] Obtaining physical topology data of multiple GPU nodes in a distributed computing cluster, hardware specification data of GPU units, and historical task execution records by cross-cluster management software;

[0008] Modeling the association relationship between GPU units based on the physical topology data, the hardware specification data, and the historical task execution records, and generating a GPU grouping set;

[0009] topology detection analysis is performed on the GPU group set to construct a topology graph model of cross-cluster management software, and an optimal communication path of a multi-card task is determined in the topology graph model;

[0010] Based on the optimal communication path, a target GPU combination is selected from the GPU group set in combination with a multi-objective optimization algorithm;

[0011] During execution of a computing task by the target GPU combination, a reinforcement learning model is used to dynamically adjust a perception scheduling strategy in combination with cluster running state data and execution performance data, so as to maintain an adaptive relationship between physical topology data and task requirements of a multi-card task, and realize collaborative scheduling of cross-cluster heterogeneous GPU resources.

[0012] Optionally, the selecting, based on the optimal communication path, a target GPU combination from the GPU group set in combination with a multi-objective optimization algorithm comprises:

[0013] Based on a benchmark performance corresponding to the optimal communication path, a multi-objective optimization problem including a communication efficiency target, a resource utilization target and an energy consumption control target is established;

[0014] The multi-objective optimization problem is solved by a multi-objective optimization algorithm to generate a perception scheduling strategy, and the perception scheduling strategy includes an evaluation function and a selection rule of a GPU combination;

[0015] According to the evaluation function and the selection rule, each candidate GPU combination in the GPU group set is calculated and screened to obtain a comprehensive evaluation score of each screened candidate GPU combination;

[0016] Based on the comprehensive evaluation score, all screened candidate GPU combinations are sorted, and a candidate GPU combination with a comprehensive evaluation score greater than a preset evaluation threshold is selected as a target GPU combination.

[0017] Optionally, the solving, by a multi-objective optimization algorithm, the multi-objective optimization problem to generate a perception scheduling strategy comprises:

[0018] The multi-objective optimization problem is decomposed into a first optimization sub-problem, a second optimization sub-problem and a third optimization sub-problem;

[0019] The first optimization sub-problem, the second optimization sub-problem and the third optimization sub-problem are collaboratively solved by a multi-objective optimization algorithm, and a population evolution mechanism is adopted in the solving process to generate a candidate solution set;

[0020] The candidate solution set is sorted to identify a Pareto optimal solution set;

[0021] Based on the pareto optimal solution set, a preset propagation mechanism is introduced to determine a selection rule;

[0022] Key parameters are extracted from the pareto optimal solution set, and an evaluation function of GPU combination is constructed based on the key parameters through an adaptive weighted fusion method;

[0023] The evaluation function and the selection rule are combined to form a perception scheduling strategy.

[0024] Optionally, the combination of cluster running state data and execution performance data uses a reinforcement learning model to dynamically adjust the perception scheduling strategy, including:

[0025] Monitoring and collecting cluster running state data and execution performance data;

[0026] The cluster running state data and the execution performance data are input into the reinforcement learning model, and the policy network of the reinforcement learning model is used to analyze the current state of the distributed computing cluster, and output a scheduling strategy adjustment parameter;

[0027] Based on the scheduling strategy adjustment parameter, the weights of the evaluation function and the threshold values of the selection rule in the perception scheduling strategy are dynamically updated to obtain an updated perception scheduling strategy.

[0028] Optionally, the input of the cluster running state data and the execution performance data into the reinforcement learning model, and the analysis of the current state of the distributed computing cluster by the policy network of the reinforcement learning model, and the output of the scheduling strategy adjustment parameter, include:

[0029] The core usage rate, the video memory occupancy rate and the inter-node network traffic are extracted from the cluster running state data, and the task completion progress and the communication delay time are extracted from the execution performance data;

[0030] The core usage rate, the video memory occupancy rate, the inter-node network traffic, the task completion progress and the communication delay time are multi-dimensionally fused to construct a state vector reflecting the current state of the cluster;

[0031] The state vector is input into the policy network of the reinforcement learning model, and the feature extraction layer of the policy network is used to reduce the dimension of the state vector to extract running key features;

[0032] The running key features are nonlinearly transformed by the hidden layer of the policy network to obtain scheduling features;

[0033] The scheduling features are transmitted to the output layer of the policy network through multi-layer neural network connection, an adjustment output value is generated based on an attention weight distribution mechanism, and a scheduling strategy adjustment parameter is generated based on the adjustment output value.

[0034] Optionally, the association relationship between GPU units is modeled based on the physical topology data, the hardware specification data, and the historical task execution record, and a GPU grouping set is generated, including:

[0035] connection relationship information is extracted from the physical topology data, performance parameter information is extracted from the hardware specification data, and cooperation mode information is extracted from the historical task execution record;

[0036] The connection relationship information, the performance parameter information, and the cooperation mode information are fused to form an initial node feature of each GPU unit;

[0037] Based on the initial node feature, a feature correlation degree between GPU units is calculated;

[0038] GPU units with a feature correlation degree greater than a preset correlation degree threshold are divided into the same group to form a GPU grouping set.

[0039] Optionally, the optimal communication path of the multi-card task in the topology graph model includes:

[0040] In the topology graph model, the path starting point and the path ending point are determined for the communication demand of the multi-card task, and all possible communication paths connecting the path starting point and the path ending point are generated through path discovery processing;

[0041] All possible communication paths are traversed using a path search algorithm, a communication efficiency index of each communication path is calculated, and the communication path with the highest communication efficiency index is selected as the optimal communication path.

[0042] In a second aspect, the present application provides a system for scheduling heterogeneous GPUs by cross-cluster management software, including:

[0043] An acquisition module is configured to acquire, by the cross-cluster management software, physical topology data of a plurality of GPU nodes in a distributed computing cluster, hardware specification data of GPU units, and historical task execution records;

[0044] A modeling module is configured to model the association relationship between GPU units based on the physical topology data, the hardware specification data, and the historical task execution record, and generate a GPU grouping set;

[0045] A calculation module is configured to perform topology detection analysis on the GPU grouping set, construct a topology graph model of the cross-cluster management software, and determine an optimal communication path of a multi-card task in the topology graph model;

[0046] The generating module is configured to select a target GPU combination from the set of GPU groups based on the optimal communication path and in combination with a multi-objective optimization algorithm.

[0047] The adjusting module is configured to dynamically adjust the perception scheduling strategy by using a reinforcement learning model in combination with cluster running state data and execution performance data during execution of the target GPU combination on the computing task, so as to maintain an adaptive relationship between the physical topology data and the task requirements of the multi-card task, and realize collaborative scheduling of the heterogeneous GPU resources across the clusters.

[0048] In a third aspect, the present application provides an electronic device, comprising:

[0049] A memory configured to store a computer program.

[0050] A processor configured to execute the computer program to implement the steps of the method for scheduling heterogeneous GPUs by the cross-cluster management software according to the first aspect.

[0051] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable by a processor to implement the steps of the method for scheduling heterogeneous GPUs by the cross-cluster management software according to the first aspect.

[0052] The technical scheme provided by the present application has the following beneficial effects:

[0053] The present application provides solid data support for subsequent intelligent scheduling decisions by establishing complete cluster resource information; on this basis, the internal relationship between GPU computing units is identified to form logically reasonable resource grouping; further, the communication performance between different GPU units is quantitatively evaluated to construct a topology model that accurately reflects the actual network status; based on the model, the data transmission path with the highest communication efficiency is selected for the task requiring multi-card cooperation; at the same time, a dynamic balance is achieved between multiple optimization objectives such as computing efficiency, energy consumption, and resource utilization, forming a scientific resource selection standard, and finally making the entire scheduling strategy adaptive to changes in the running environment, thereby continuously maintaining the rationality of the resource allocation scheme in a dynamic environment.

[0054] Further, on the basis of priority guarantee of communication performance, the present application further considers factors such as resource utilization and energy consumption control, and can realize balanced optimization of heterogeneous GPU resource allocation by combination of a multi-objective optimization method and a comprehensive scoring mechanism, thereby improving the overall running performance of the distributed computing system.

[0055] These and other aspects of the present application will be more apparent in the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0057] Figure 1 A flow chart of a method for cross-cluster management software to schedule heterogeneous GPUs provided by an embodiment of the present application;

[0058] Figure 2 A specific implementation schematic diagram of a method for cross-cluster management software to schedule heterogeneous GPUs provided by an embodiment of the present application;

[0059] Figure 3 A structural schematic diagram of a system for cross-cluster management software to schedule heterogeneous GPUs provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] The existing cross-cluster GPU scheduling method is difficult to keep synchronization with the real-time changing cluster resource state when coping with dynamically changing computing load and diversified task demand, and is prone to matching lag. This scheduling lag directly leads to the fact that the cluster computing resources cannot be fully utilized, and further affects the execution efficiency of the computing task that needs to be completed by multiple GPUs in cooperation. This phenomenon is particularly obvious in the environment with high degree of heterogeneity of hardware configuration.

[0061] The present application proposes a method for cross-cluster management software to schedule heterogeneous GPUs. The method generates a scheduling strategy that can adapt to the current state of the cluster by constructing a topological graph model reflecting the real-time network status and establishing a multi-objective optimization mechanism. In the actual execution process of the task, the internal parameters of the strategy are further optimized continuously by using a reinforcement learning algorithm, so that the resource allocation scheme can always dynamically match the underlying physical topology structure and the constantly changing computing task demand at the upper level. This method effectively solves the problem that the scheduling strategy in the prior art is disconnected with the dynamic running environment, realizes efficient collaborative utilization of cross-cluster heterogeneous GPU resources, and ultimately improves the overall performance of the distributed computing system.

[0062] In order to enable personnel in the technical field to better understand the present application scheme, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0063] The core of the present application is to provide a method for scheduling heterogeneous GPUs by cross-cluster management software, and a flowchart of one specific embodiment is shown in Figure 1 The method comprises the following steps:

[0064] Step 101: Obtain the physical topology data of multiple GPU nodes in a distributed computing cluster, the hardware specification data of GPU units, and the historical task execution records by cross-cluster management software.

[0065] In the above scheme, the GPU node refers to a physical server or a computing node containing one or more GPU units, and the GPU unit refers to a specific GPU processor in the node. One GPU node can contain multiple GPU units, and these GPU units are connected through interconnection technologies such as NVLink or PCIe;

[0066] The physical topology data refers to the hardware connection relationship information between the GPU nodes, and the physical topology data includes the fast interconnection link connection mode and the bus connection path information.

[0067] The hardware specification data of the GPU unit reflects the heterogeneity, and the hardware specification data contains the difference information of different GPU units in terms of computing power, storage capacity, and communication bandwidth, which reflects the heterogeneous characteristics of the GPU units in terms of model, architecture, and performance.

[0068] The historical task execution record refers to the execution time record, resource usage record, and communication efficiency record of the past computing tasks.

[0069] In actual application, a distributed deep learning training platform needs to schedule a total of 32 GPU nodes for model training. The platform collects the connection state information between the nodes in real time through the agent program deployed on each node, reads the detailed specification parameters of each GPU from the hardware management interface of the device, and obtains the recent historical training task records from the central task management database. After the above data is uniformly preprocessed, a complete resource dataset containing 320 GPU unit information is finally formed.

[0070] Step 102: Model the association relationship between the GPU units based on the physical topology data, the hardware specification data, and the historical task execution record, and generate a GPU grouping set.

[0071] In this embodiment, the step 102 comprises the following flow:

[0072] Step 1021: Extract the connection relationship information from the physical topology data, the performance parameter information from the hardware specification data, and the cooperation mode information from the historical task execution record.

[0073] In the above scheme, the connection relationship information refers to the details of the physical connection path formed between the graphic processor units through the fast interconnection or bus, the performance parameter information includes the hardware characteristic parameters such as the calculation speed, the storage capacity and the communication bandwidth of the graphic processor unit, and the cooperation mode information refers to the record of the data transmission frequency and the cooperative working efficiency between the graphic processor units in the historical task.

[0074] For example, the application identifies the specific fast interconnection connection relationship between eight graphic processor units, obtains the specific calculation capability and the memory size of each unit, and counts the data transmission efficiency between each pair of units in the historical task.

[0075] Step 1022: The connection relationship information, the performance parameter information and the cooperation mode information are fused to form the initial node feature of each GPU unit.

[0076] In the above scheme, the initial node feature is a feature vector representing the comprehensive characteristics of the graphic processor unit formed by fusing multiple information.

[0077] In the embodiment of the application, the connection relationship information is quantified as a connection strength value, the performance parameter information is normalized as a unified standard score, and the cooperation mode information is converted into a cooperation efficiency value in step 1022, and then the three types of quantized information are fused to generate the initial node feature vector of each graphic processor unit; for example, the fast interconnection connection is assigned a higher connection strength value, the bus connection is assigned a lower connection strength value, the calculation capability and the memory size are converted into standard scores within [0, 1], and at the same time the cooperation efficiency record is quantified into a specific cooperation efficiency value, and finally an 8-dimensional feature vector of each graphic processor unit is formed.

[0078] Step 1023: Calculate the feature correlation degree between the GPU units based on the initial node feature.

[0079] In the above scheme, the feature correlation degree is a quantitative index for measuring the similarity degree of two graphic processor units in the topological structure and functional characteristics.

[0080] In the embodiment of the application, the similarity between the feature vectors of each two graphic processor units is calculated based on the above initial node feature vector in step 1023, and the similarity value is taken as the feature correlation degree representing the closeness degree of the unit relationship; for example, the cosine similarity algorithm is used to calculate the cosine value of the angle between each pair of graphic processor unit feature vectors, and the feature correlation degree in the range of 0 to 1 is obtained.

[0081] Step 1024: The GPU units with a feature correlation degree greater than a preset correlation degree threshold are divided into the same group to form a GPU grouping set.

[0082] In the above scheme, the preset correlation degree threshold is a grouping determination standard value set according to actual application requirements.

[0083] In the embodiment of the present application, the feature correlation degrees between all the graphic processor units are compared with the preset correlation degree threshold through step 1024, and the graphic processor units with correlation degrees higher than the threshold are divided into the same group, thereby forming a plurality of graphic processor grouping sets; for example, all the feature correlation degrees form an 8-row 8-column correlation degree matrix, the correlation degree threshold is set to 0.8, the graphic processor units with correlation degrees higher than 0.8 are grouped into the same group, and finally the grouping set is formed.

[0084] The present application realizes intelligent grouping of heterogeneous graphic processor resources by establishing a graphic processor unit correlation relationship model based on multi-source information fusion, and provides a reasonable resource organization basis for subsequent topology-aware scheduling, thereby effectively improving the scientificity of resource management and scheduling efficiency.

[0085] Step 103: performing topology detection analysis on the GPU grouping set, constructing a topology map model of cross-cluster management software, and determining an optimal communication path of a multi-card task in the topology map model.

[0086] In this embodiment, step 103 includes the following processes:

[0087] Step 1031: performing link quality detection analysis on each GPU unit in the GPU grouping set to generate connection state data.

[0088] In the above scheme, the connection state data contains transmission bandwidth measurement values and communication delay measurement values between graphic processor units.

[0089] In the embodiment of the present application, bidirectional link testing is performed on each graphic processor unit in the graphic processor grouping set through step 1031, the maximum transmission bandwidth and the minimum communication delay between each pair of graphic processor units are measured, and connection state data containing specific bandwidth and delay values are generated; for example, in a test group containing 20 graphic processor units, the bandwidth values and delay values between each pair of units are measured by sending test data packets of different sizes.

[0090] Step 1032: calculating a communication performance index between each two GPU units based on the connection state data by using a weighted scoring method.

[0091] In the above scheme, the communication performance index is a quantitative scoring value that comprehensively reflects the communication capability between two graphic processor units.

[0092] In the embodiment of the present application, the communication efficiency index is calculated by using a weighted scoring method based on the bandwidth value and the delay value in the connection state data in step 1032, wherein the bandwidth value can be counted into the score in direct proportion, and the delay value can be counted into the score in inverse proportion, and the two are combined into a single score value by a preset weight coefficient; for example, the bandwidth value is multiplied by the bandwidth weight, and the reciprocal of the delay value is multiplied by the delay weight, and the two are added to obtain the final communication efficiency index.

[0093] Step 1033: The communication efficiency index is standardized and a topology graph model with GPU units as vertices and the standardized communication efficiency index as edge weight is constructed.

[0094] In the above scheme, the topology graph model is a weighted graph structure with GPU units as nodes and the standardized communication efficiency index as edge weight.

[0095] In the embodiment of the present application, all communication efficiency indexes are normalized to the range of 0 to 1 in step 1033, and a topology graph model with GPU units as nodes and the standardized communication efficiency index as edge weight is constructed; for example, the maximum value in all communication efficiency indexes is found, and all indexes are divided by the maximum value to complete standardization, and then a weighted graph model is constructed.

[0096] Step 1034: In the topology graph model, the path starting point and the path ending point are determined for the communication demand of the multi-card task, and all possible communication paths connecting the path starting point and the path ending point are generated by path discovery processing.

[0097] In the above scheme, the path starting point and the path ending point are the starting GPU unit and the target GPU unit specified by the multi-card task, and all possible communication paths are all connected path combinations between the specified path starting point and the path ending point in the topology graph model, which include various connection modes of different numbers and intermediate nodes, including single-hop paths of direct connection and multi-hop paths of forwarding through one or more intermediate GPU units, each path combination represents a complete data transmission path selection, and all feasible connection schemes need to be traversed in the path search process to evaluate the respective communication efficiency indexes.

[0098] In the embodiment of the present application, the starting node and the ending node of the path are determined in the constructed topology graph model according to the specific communication demand of the multi-card cooperative task in step 1034, and all possible communication paths connecting the path starting point and the path ending point are found by using a depth-first search algorithm; for example, the starting GPU unit and the ending GPU unit are determined according to the task requirements, and all connected paths are found by traversing the graph structure.

[0099] Step 1035: traverse all possible communication paths by using a path search algorithm, calculate the communication performance index of each communication path, and select the communication path with the highest communication performance index as the optimal communication path.

[0100] In the above scheme, the communication performance index is the product value of all edge weights on the communication path, and the optimal communication path is the communication path with the highest communication performance index.

[0101] In the embodiment of the present application, step 1035 is used to calculate the product of all edge weights on each communication path as the communication performance index of the path by using a path search algorithm, and the path with the highest cumulative index is selected as the optimal communication path of the task; for example, the continuous product of all standardized communication performance indexes on each path is calculated, and the path with the largest product is selected as the optimal communication path.

[0102] The present application realizes intelligent selection of multi-card task communication paths by establishing an accurate communication performance evaluation system and a topological graph model, thereby effectively improving the data transmission efficiency in a distributed computing scenario.

[0103] Step 104: based on the optimal communication path, combining a multi-objective optimization algorithm, selecting a target GPU combination from the set of GPU groups.

[0104] In this embodiment, step 104 includes the following process, as shown in Figure 2

[0105] Step 1041: based on the benchmark performance corresponding to the optimal communication path, establishing a multi-objective optimization problem containing a communication efficiency target, a resource utilization target, and an energy consumption control target.

[0106] In the above scheme, the benchmark performance is derived from the process of calculating the communication performance index of each communication path and selecting the path with the highest value, and its meaning is that the optimal communication performance level that the GPU unit sequence represented by the optimal communication path can achieve, which is used as a performance benchmark reference for subsequent multi-objective optimization.

[0107] The communication efficiency target is derived from the benchmark performance of the optimal communication path and the real-time network status, and its meaning is to maximize data transmission efficiency while ensuring communication quality; the resource utilization target is derived from the real-time load state and historical usage data of each GPU unit in the set of GPU groups, and its meaning is to improve the overall utilization level of GPU computing resources and video memory resources;

[0108] The energy consumption control target is derived from the power monitoring data of the GPU unit and the cluster energy consumption statistics, and its meaning is to reduce the overall system energy consumption while meeting performance requirements;

[0109] ​In the embodiment of the present application, the multi-objective optimization problem considering the maximization of communication efficiency, the optimization of resource utilization and the minimization of energy consumption is established based on the reference performance value determined by the step 1041; for example, the reference performance is taken as the reference value of the communication efficiency target, and a mathematical programming model containing the above three optimization targets is constructed by combining the actual resource state and energy consumption requirement of the current cluster.

[0110] Step 1042: solving the multi-objective optimization problem by a multi-objective optimization algorithm to generate a perception scheduling strategy, wherein the perception scheduling strategy contains an evaluation function of GPU combination and a selection rule.

[0111] In the above scheme, the evaluation function is the final application function obtained after the multi-objective optimization algorithm is solved, which integrates multiple optimization targets into a single scoring function through fixed weight coefficients, and is used to directly calculate the comprehensive score of the candidate combination; and the purpose of the multi-objective optimization algorithm is to find a Pareto optimal solution set, without pre-setting the weight relationship between the targets.

[0112] The selection rule is a condition set for screening qualified candidate graphics processor combinations.

[0113] The step 1042 can specifically include the following steps:

[0114] A1: decomposing the multi-objective optimization problem into a first optimization sub-problem, a second optimization sub-problem and a third optimization sub-problem.

[0115] A2: solving the first optimization sub-problem, the second optimization sub-problem and the third optimization sub-problem by a multi-objective optimization algorithm, and adopting a population evolution mechanism in the solving process to generate a candidate solution set.

[0116] A3: sorting the candidate solution set to identify a Pareto optimal solution set.

[0117] A4: based on the Pareto optimal solution set, introducing a preset propagation mechanism to determine the selection rule.

[0118] A5: extracting key parameters from the Pareto optimal solution set, and constructing an evaluation function of GPU combination by an adaptive weighted fusion method based on the key parameters.

[0119] In the above scheme, the key parameters refer to the specific values of the balanced solutions in the Pareto optimal solution set that can simultaneously optimize the communication efficiency, resource utilization and energy consumption control three targets, and the key parameters specifically include the weight coefficient of the communication performance index, the threshold parameter of the resource utilization, the specific value of the energy consumption limit and the trade-off proportion relationship between the optimization targets.

[0120] Also, the embodiments of the present application do not make specific limitations on the specific content of the propagation mechanism, and can be specifically set according to actual conditions.

[0121] A6: Combining the evaluation function and the selection rule to form a perception scheduling strategy.

[0122] For example, a plurality of candidate solutions are generated using a multi-objective optimization algorithm, a Pareto optimal solution set is identified through non-dominated sorting, an upper limit of resource usage and a communication delay requirement are determined according to dynamic constraint propagation, and a weighted evaluation function is constructed based on the parameter distribution of the optimal solution set, and then a perception scheduling strategy is constructed in combination with the selection rule.

[0123] Step 1043: According to the evaluation function and the selection rule, each candidate GPU combination in the GPU grouping set is calculated and screened to obtain a comprehensive evaluation score of each screened candidate GPU combination.

[0124] In the above scheme, the comprehensive evaluation score is the calculation score of the candidate graphics processor combination under the evaluation function.

[0125] For example, the score of each candidate combination is calculated using the evaluation function, while checking whether it meets each constraint condition in the selection rule, and only the candidate combination and its score that meet all conditions are retained.

[0126] Step 1044: Based on the comprehensive evaluation score, all screened candidate GPU combinations are sorted, and a candidate GPU combination with a comprehensive evaluation score greater than a preset evaluation threshold is selected as a target GPU combination.

[0127] In the above scheme, the preset evaluation threshold is a score limit for determining whether the candidate graphics processor combination is qualified, and the size of the threshold is not specifically limited by the embodiments of the present application, and can be specifically set according to actual conditions.

[0128] For example, the candidate combinations are sorted in descending order of score, and the best combination with a score higher than the set threshold is selected as the final resource combination for task execution.

[0129] The present application realizes balanced optimization of communication efficiency, resource utilization and energy consumption control through the establishment of a multi-objective optimization scheduling strategy generation mechanism, thereby effectively improving the scheduling quality and system energy efficiency of heterogeneous graphics processor resources.

[0130] Step 105: During the execution of the target GPU combination in the computing task, the perception scheduling strategy is dynamically adjusted using a reinforcement learning model in combination with cluster running state data and execution performance data, to maintain the adaptation relationship between the physical topology data and the task requirements of the multi-card task, and to realize the collaborative scheduling of cross-cluster heterogeneous GPU resources.

[0131] In the above scheme, the task requirement is derived from resource requirements of the to-be-executed computing task, and the resource requirements include the required number of GPUs and the communication bandwidth requirement.

[0132] In this embodiment, the step 105 includes the following process:

[0133] Step 1051: monitoring and collecting cluster running state data and execution performance data.

[0134] In the above scheme, the cluster running state data refers to a set of real-time working parameters of the graphics processor unit during execution of the computing task, and the execution performance data refers to a record of execution progress and performance of the computing task on the graphics processor unit.

[0135] In the embodiments of the present application, during execution of the computing task by the target graphics processor combination, the graphics processor computing core usage rate, the video memory occupancy rate, and the network traffic between nodes are monitored in real time, and the task completion progress and the inter-node communication delay time are recorded, so as to form the cluster running state data and the task execution performance data; for example, the working load data of each graphics processor unit is continuously collected by the deployed monitoring module, and the real-time execution progress information of the task is synchronously obtained by the task manager.

[0136] Step 1052: inputting the cluster running state data and the execution performance data into a reinforcement learning model, analyzing the current state of the distributed computing cluster through a policy network of the reinforcement learning model, and outputting scheduling strategy adjustment parameters.

[0137] In the above scheme, the current state refers to a system state vector composed of the cluster running state data and the execution performance data, which is obtained by monitoring the computing core usage rate, the video memory occupancy rate, the network traffic between nodes, and the task completion progress and the communication delay time of the target GPU combination in real time, and is used to reflect the comprehensive state of the resource usage and the task execution performance of the entire distributed computing cluster at a specific moment.

[0138] The step 1052 can specifically include the following steps:

[0139] B1: extracting the core usage rate, the video memory occupancy rate, and the network traffic between nodes from the cluster running state data, and extracting the task completion progress and the communication delay time from the execution performance data.

[0140] B2: performing multi-dimensional data fusion on the core usage rate, the video memory occupancy rate, the network traffic between nodes, the task completion progress, and the communication delay time, and constructing a state vector for reflecting the current state of the cluster.

[0141] In the above scheme, the state vector is a unified feature representation formed by fusing multiple operating state data and performance data.

[0142] B3: inputting the state vector into a policy network of the reinforcement learning model, performing dimension reduction processing on the state vector through a feature extraction layer of the policy network, to extract operating key features.

[0143] In the above scheme, the operating key features refer to compressed feature vectors extracted from original multi-dimensional state data, which can represent the core operating conditions of the cluster. The physical meanings include a graphics processor computing load concentration feature reflecting the distribution balance of each computing unit workload, a data communication bottleneck feature embodying the congestion degree of inter-node data transmission, and a task execution efficiency feature representing the optimization space of task advancement speed under current resource configuration. These features remove redundant information in the original state data through dimension reduction processing, and retain key operating state indicators affecting scheduling decisions, providing refined decision basis for subsequent policy adjustment.

[0144] Furthermore, the reinforcement learning model, the policy network, the layers used in the internal structure of the policy network, and the corresponding structure design are not specifically limited, and can be set according to actual conditions.

[0145] B4: performing nonlinear transformation on the operating key features through a hidden layer of the policy network, to obtain scheduling features.

[0146] In the above scheme, the scheduling features are deep feature representations obtained through neural network nonlinear transformation.

[0147] B5: transmitting the scheduling features to an output layer of the policy network through multi-layer neural network connection, generating adjustment output values based on an attention weight distribution mechanism, and generating scheduling policy adjustment parameters based on the adjustment output values.

[0148] In the above scheme, the adjustment output values are neural network output policy adjustment instruction values, and the scheduling policy adjustment parameters are a group of values for modifying specific parameters of the scheduling policy.

[0149] In the embodiment of the present application, the key operation indicators are extracted from the collected cluster operation state data through step 1052, and the performance indicators are extracted from the execution performance data. These indicators are fused to construct a state vector and input into the policy network of the reinforcement learning model. The network processes the state vector through its feature extraction layer to obtain condensed operation key features, and then obtains scheduling features through nonlinear transformation of the hidden layer. Finally, the output layer generates adjustment output values based on attention weight distribution, and converts them into specific scheduling strategy adjustment parameter sets. For example, the state vector is composed of the usage rate of a graphics processor computing core, the memory occupancy rate, network traffic, task completion progress, and communication delay time. After processing by the above neural network, a set of scheduling strategy adjustment parameters is output.

[0150] Step 1053: Based on the scheduling strategy adjustment parameters, the weights of the evaluation function and the threshold values of the selection rules in the perception scheduling strategy are dynamically updated to obtain an updated perception scheduling strategy.

[0151] In the embodiment of the present application, the evaluation function weight coefficients and selection rule threshold parameters in the original perception scheduling strategy are adjusted according to the specific values in the scheduling strategy adjustment parameter set through step 1053, thereby forming an updated perception scheduling strategy. For example, the weight proportions of communication efficiency, resource utilization, and energy consumption in the evaluation function are modified according to the adjustment parameters, and the resource usage upper limit and performance requirement threshold in the selection rule are updated.

[0152] The present application realizes real-time adaptation of the scheduling strategy and the operation environment by establishing a dynamic strategy adjustment mechanism based on reinforcement learning, thereby effectively improving the adaptive ability and system performance stability of heterogeneous graphics processor resource scheduling.

[0153] Figure 3 The structure of a system for scheduling heterogeneous GPUs by cross-cluster management software provided in the embodiment of the present application is shown in FIG. Figure 3 The specific implementation part describes the following:

[0154] The acquisition module 31 is configured to acquire, by the cross-cluster management software, physical topology data of a plurality of GPU nodes in a distributed computing cluster, hardware specification data of GPU units, and historical task execution records.

[0155] The modeling module 32 is configured to model the association relationship between the GPU units based on the physical topology data, the hardware specification data, and the historical task execution records, and generate a GPU grouping set.

[0156] The computing module 33 is configured to perform topology detection analysis on the GPU group set, construct a topology graph model of the cross-cluster management software, and determine an optimal communication path of the multi-card task in the topology graph model.

[0157] The generating module 34 is configured to select a target GPU combination from the GPU group set based on the optimal communication path and in combination with a multi-objective optimization algorithm.

[0158] The adjusting module 35 is configured to dynamically adjust a perception scheduling strategy by using a reinforcement learning model in combination with cluster running state data and execution performance data during execution of the computing task by the target GPU combination, so as to maintain an adaptive relationship between physical topology data and task requirements of the multi-card task, and realize collaborative scheduling of the heterogeneous GPU resources across the clusters.

[0159] The system for scheduling heterogeneous GPUs by the cross-cluster management software according to the embodiments of the present application is used to implement the method for scheduling heterogeneous GPUs by the cross-cluster management software as described above, and thus the specific implementation of the system for scheduling heterogeneous GPUs by the cross-cluster management software can be seen from the foregoing method for scheduling heterogeneous GPUs by the cross-cluster management software, and the specific implementation can be referred to the description of the corresponding embodiment part, which will not be repeated here.

[0160] The present application also provides an electronic device, which comprises a memory for storing a computer program and a processor for executing the computer program to implement the steps of the method for scheduling heterogeneous GPUs by the cross-cluster management software according to any of the foregoing embodiments.

[0161] The present application also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the method for scheduling heterogeneous GPUs by the cross-cluster management software according to any of the foregoing embodiments.

[0162] In an exemplary embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0163] The embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in the method for scheduling heterogeneous GPUs by the cross-cluster management software according to any of the foregoing embodiments.

[0164] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of embodiments of the present application and are not intended to limit the scope of the present application. Accordingly, embodiments as described herein contemplate all modifications that come within the scope of the present application as recited by the claims set forth below and any equivalents thereto, with the scope of the present application being measured by the broadest interpretation of those claims set forth below.

[0165] The above provides a method and system for cross-cluster management software scheduling heterogeneous GPUs. The principles and implementation modes of the present application are described herein by applying specific examples, and the above description of the embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method for managing software dispatching of heterogeneous GPUs across a cluster, the method comprising: The method comprises the following steps: Obtaining physical topology data of multiple GPU nodes in a distributed computing cluster, hardware specification data of GPU units and historical task execution records through cross-cluster management software; Modeling the association relationship between GPU units based on the physical topology data, the hardware specification data and the historical task execution records, and generating a GPU grouping set; Performing topology detection analysis on the GPU grouping set, constructing a topology graph model of the cross-cluster management software, and determining an optimal communication path of a multi-card task in the topology graph model; Based on the optimal communication path, combining a multi-objective optimization algorithm, and selecting a target GPU combination from the GPU grouping set; During the execution of the target GPU combination, combining cluster running state data and execution performance data, and dynamically adjusting a perception scheduling strategy by using a reinforcement learning model to maintain the adaptation relationship between physical topology data and task requirements of a multi-card task, and realizing the collaborative scheduling of heterogeneous GPU resources across clusters; The method comprises the following steps: Based on the optimal communication path, a multi-objective optimization problem including a communication efficiency target, a resource utilization target and an energy consumption control target is established; Solving the multi-objective optimization problem by a multi-objective optimization algorithm to generate a perception scheduling strategy, wherein the perception scheduling strategy includes an evaluation function and a selection rule of the GPU combination; According to the evaluation function and the selection rule, each candidate GPU combination in the GPU grouping set is calculated and screened to obtain a comprehensive evaluation score of each screened candidate GPU combination; Based on the comprehensive evaluation score, all screened candidate GPU combinations are sorted, and a candidate GPU combination with a comprehensive evaluation score greater than a preset evaluation threshold is selected as the target GPU combination; The method comprises the following steps: The multi-objective optimization problem is decomposed into a first optimization sub-problem corresponding to the communication efficiency target, a second optimization sub-problem corresponding to the resource utilization target, and a third optimization sub-problem corresponding to the energy consumption control target; The first optimization sub-problem, the second optimization sub-problem and the third optimization sub-problem are collaboratively solved by a multi-objective optimization algorithm, and a population evolution mechanism is adopted in the solving process to generate a candidate solution set; The candidate solution set is sorted to identify a Pareto optimal solution set; Based on the Pareto optimal solution set, a preset propagation mechanism is introduced to determine the selection rule; Key parameters are extracted from the Pareto optimal solution set, and based on the key parameters, an evaluation function of the GPU combination is constructed by an adaptive weighted fusion method; The evaluation function and the selection rule are combined to form a perception scheduling strategy.

2. The method of claim 1, wherein, The method comprises the following steps: Monitoring and collecting cluster running state data and execution performance data; inputting the cluster running state data and the execution performance data into a reinforcement learning model, analyzing a current state of the distributed computing cluster through a policy network of the reinforcement learning model, and outputting a scheduling strategy adjustment parameter; based on the scheduling strategy adjustment parameter, dynamically updating weights of an evaluation function and threshold values of selection rules in the perception scheduling strategy to obtain an updated perception scheduling strategy.

3. The method of claim 2, wherein, The inputting the cluster running state data and the execution performance data into a reinforcement learning model, analyzing a current state of the distributed computing cluster through a policy network of the reinforcement learning model, and outputting a scheduling strategy adjustment parameter comprises: extracting core usage rate, video memory occupancy rate and inter-node network traffic from the cluster running state data, and extracting task completion progress and communication delay time from the execution performance data; performing multi-dimensional data fusion on the core usage rate, the video memory occupancy rate, the inter-node network traffic, the task completion progress and the communication delay time to construct a state vector for reflecting the current state of the cluster; inputting the state vector into the policy network of the reinforcement learning model, performing dimension reduction processing on the state vector through a feature extraction layer of the policy network to extract running key features; performing non-linear transformation on the running key features through a hidden layer of the policy network to obtain scheduling features; passing the scheduling features through a multi-layer neural network connection to the output layer of the policy network, generating an adjustment output value based on an attention weight distribution mechanism, and generating a scheduling strategy adjustment parameter based on the adjustment output value.

4. The method of claim 1, wherein, The modeling of the association relationship between GPU units based on the physical topology data, the hardware specification data and the historical task execution record, and generating a GPU grouping set comprises: extracting connection relationship information from the physical topology data, extracting performance parameter information from the hardware specification data, and extracting cooperation mode information from the historical task execution record; performing feature fusion on the connection relationship information, the performance parameter information and the cooperation mode information to form initial node features of each GPU unit; based on the initial node features, calculating feature association degrees between GPU units; dividing GPU units with feature association degrees greater than a preset association degree threshold to the same group to form a GPU grouping set.

5. The method of claim 1, wherein, The determining of the optimal communication path of the multi-card task in the topology graph model comprises: in the topology graph model, determining a path starting point and a path ending point for the communication demand of the multi-card task, and generating all possible communication paths connecting the path starting point and the path ending point through path discovery processing; adopting a path search algorithm to traverse all possible communication paths, calculating communication performance indicators of each communication path, and selecting a communication path with the highest communication performance indicator as the optimal communication path.

6. A system for managing software dispatching of heterogeneous GPUs across a cluster, the system comprising: comprises: an acquisition module configured to acquire, through cross-cluster management software, physical topology data of a plurality of GPU nodes in a distributed computing cluster, hardware specification data of GPU units, and historical task execution records; The modeling module is used to model the relationships between GPU units based on the physical topology data, the hardware specification data, and the historical task execution records, and generate a GPU group set. The computing module is used to perform topology probing analysis on the GPU group set, construct a topology graph model across the cluster management software, and determine the optimal communication path for multi-GPU tasks in the topology graph model. The generation module is used to select a target GPU combination from the GPU group set based on the optimal communication path and in conjunction with a multi-objective optimization algorithm; The adjustment module is used to dynamically adjust the perceptual scheduling strategy by combining cluster running status data and execution performance data during the execution of computing tasks by the target GPU combination, so as to maintain the adaptation relationship between physical topology data and the task requirements of multi-card tasks and realize the collaborative scheduling of heterogeneous GPU resources across clusters. The step of selecting a target GPU combination from the GPU group set based on the optimal communication path and in conjunction with a multi-objective optimization algorithm includes: Based on the baseline performance corresponding to the optimal communication path, a multi-objective optimization problem is established, which includes communication efficiency objective, resource utilization objective, and energy consumption control objective. The multi-objective optimization problem is solved by a multi-objective optimization algorithm to generate a perceptual scheduling strategy, which includes an evaluation function and selection rules for GPU combination. Based on the evaluation function and the selection rules, each candidate GPU combination in the GPU group set is calculated and filtered to obtain a comprehensive evaluation score for each filtered candidate GPU combination. Based on the comprehensive evaluation score, all selected candidate GPU combinations are sorted, and candidate GPU combinations with a comprehensive evaluation score greater than a preset evaluation threshold are selected as target GPU combinations. The step of solving the multi-objective optimization problem using a multi-objective optimization algorithm to generate a perceptual scheduling strategy includes: The multi-objective optimization problem is decomposed into a first optimization sub-problem corresponding to the communication efficiency objective, a second optimization sub-problem corresponding to the resource utilization objective, and a third optimization sub-problem corresponding to the energy consumption control objective. The first, second, and third optimization subproblems are solved collaboratively using a multi-objective optimization algorithm, and a population evolution mechanism is adopted during the solution process to generate a candidate solution set. The candidate solution set is sorted to identify the Pareto optimal solution set; Based on the Pareto optimal solution set, a preset propagation mechanism is introduced to determine the selection rules; Key parameters are extracted from the Pareto optimal solution set, and an evaluation function for GPU combination is constructed based on the key parameters using an adaptive weighted fusion method. The evaluation function and the selection rule are combined to form a perceptual scheduling strategy.

7. An electronic device, comprising: include: Memory, used to store computer programs; A processor, used to execute the computer program, implements the steps of the method for scheduling heterogeneous GPUs across cluster management software as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method for scheduling the heterogeneous GPUs by the cross-cluster management software according to any one of claims 1 to 5.

Images