Computing power distribution method and device based on heterogeneous graphics processor

Abstract

The invention relates to the technical field of computers, in particular to a computing power distribution method and device based on a heterogeneous graphics processor. The method comprises the steps of obtaining a system performance state quantity of a target computing cluster and computing power demand parameters of multiple tasks; the target computing cluster comprises a plurality of graphics processors, the graphics processors are input into the trained computing power distribution model to generate a corresponding computing power distribution strategy, and then the computing power distribution strategy is verified according to the actually monitored actual resource utilization rate of the target computing cluster and a preset distribution constraint condition; therefore, the reliability of the scheduling strategy can be effectively ensured while the dynamic resource scheduling of the graphics processor is realized.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for allocating computing power based on heterogeneous graphics processors. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of deep learning, large language model (LLM) and other technologies, graphics processing units (GPUs) have become the core computing power resource supporting various high-performance computing tasks.

[0003] However, with the continuous expansion of GPU clusters and the increasing complexity of computing scenarios, current mainstream GPU resource management solutions have several fundamental flaws, resulting in low resource utilization, rigid scheduling, and an inability to adapt to dynamic and complex modern AI workloads. Existing fixed quota allocation models (such as full cards or fixed vGPUs) cannot adapt to the dynamic fluctuations of AI workloads, resulting in both resource idleness and task queuing. Summary of the Invention

[0004] To address the problems of existing technologies, embodiments of this application provide a method, apparatus, electronic device, and storage medium for computing power allocation based on heterogeneous graphics processors. The technical solution is as follows: On the one hand, a computing power allocation method based on heterogeneous graphics processors is provided, which includes: Obtain the system performance status parameters of the target computing cluster and the computing power requirements of multiple tasks; the target computing cluster includes multiple graphics processors; The system performance status parameters of the target computing cluster and the computing power requirements of multiple tasks are input into the computing power allocation model to generate the predicted computing power allocation results for the multiple tasks; the computing power allocation model is a deep reinforcement learning model. Based on the predicted computing power allocation results of the multiple tasks, the predicted system resource utilization rate of the target computing cluster is determined; and the actual system resource utilization rate of the target computing cluster is obtained. When the graphics processors corresponding to the multiple tasks communicate based on the first target communication method, the allocation detection result is determined based on the predicted system resource utilization and actual resource system utilization of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processors. If the allocation detection results indicate that the predicted computing power allocation results of the multiple tasks are feasible, the predicted computing power allocation results of the multiple tasks are determined as the target computing power allocation results of the multiple tasks, and the computing power allocation model is determined as the target computing power allocation model.

[0005] In one exemplary implementation, the predicted computing power allocation result includes the identifier of the target graphics processor allocated to each task; the allocation detection result is determined based on the predicted system resource utilization and actual resource system utilization of the target computing cluster, and / or the predicted computing power allocation results of the multiple tasks and the preset allocation constraints of the graphics processor, including: The difference between the predicted system resource utilization rate and the actual system resource utilization rate of the target computing cluster is determined as the resource deviation value; If the resource deviation value does not exceed the first threshold, determine whether the resource utilization rate and video memory reservation ratio of the target graphics processor meet the preset allocation constraints of the graphics processor; if yes, generate a first detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible; if no, generate a second detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is not feasible. If the resource deviation value exceeds the first threshold, the parameters of the computing power allocation model are adjusted, and computing power allocation prediction is performed on the multiple tasks based on the updated computing power allocation model.

[0006] In an exemplary implementation, the determination of the allocation detection result based on the predicted system resource utilization and actual resource system utilization of the target computing cluster, and / or the predicted computing power allocation results of the multiple tasks and the preset allocation constraints of the graphics processor, includes: When the resource utilization rate and video memory reservation ratio of the target graphics processor meet the preset allocation constraints of the graphics processor, the difference between the predicted system resource utilization rate and the actual resource system utilization rate of the target computing cluster is determined as the resource deviation value. Determine whether the resource deviation value does not exceed a first threshold; if yes, generate a first detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible; if no, generate a second detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is not feasible. If the resource utilization and memory reservation ratio of the target graphics processor do not meet the preset allocation constraints of the graphics processor, the parameters of the computing power allocation model are adjusted, and computing power allocation prediction is performed on the multiple tasks based on the updated computing power allocation model.

[0007] In one exemplary embodiment, the predicted computing power allocation result further includes the resource utilization rate and video memory reservation ratio of the target graphics processor; the preset allocation constraints of the graphics processor include: The load on each graphics processor does not exceed the number of streaming multiprocessors of the graphics processor; The amount of video memory allocated to each graphics processor shall not exceed the capacity of the physical video memory.

[0008] In an exemplary implementation, the predicted computing power allocation result includes the identifier of the target graphics processor allocated to each task; before determining the allocation detection result based on the predicted system resource utilization and actual resource system utilization of the target computing cluster, and / or the predicted computing power allocation results of the multiple tasks and the preset allocation constraints of the graphics processor, the method further includes: The communication method of the target graphics processor is determined from the topology of the graphics processors in the target computing cluster based on the identifier of the target graphics processor.

[0009] In one exemplary implementation, the system performance state variables of the target computing cluster are the state variables of the available graphics processors and the bandwidth utilization rate in the target computing cluster. The computing power requirements of the multiple tasks include the required video memory capacity for each task and the number of multi-stream processors required for each task.

[0010] In one exemplary embodiment, the method further includes: The system performance state of the target computing cluster and the computing power requirement parameters of the multiple tasks are determined as the first state quantity; Determine a second state quantity after executing the multiple tasks based on the predicted computing power allocation results; the second state quantity includes the updated system performance state quantity of the target computing cluster and the updated computing power requirement parameter of the multiple tasks; The first state variable, the prediction computing power allocation result, the reward value corresponding to the prediction computing power allocation result, and the second state variable are stored as experience tuples in the experience replay pool.

[0011] On the other hand, a computing power allocation device is provided, comprising: The acquisition module is used to acquire the system performance status of the target computing cluster and the computing power requirements of multiple tasks; the target computing cluster includes multiple graphics processors; The prediction module is used to input the system performance status of the target computing cluster and the computing power demand parameters of multiple tasks into the computing power allocation model to generate the predicted computing power allocation results of the multiple tasks; the computing power allocation model is a deep reinforcement learning model. The first determining module is used to determine the predicted system resource utilization rate of the target computing cluster based on the predicted computing power allocation results of the multiple tasks; and to obtain the actual system resource utilization rate of the target computing cluster. The first judgment module is used to determine the allocation detection result based on the predicted system resource utilization rate and actual resource system utilization rate of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processor, when the multiple tasks communicate with each other based on the first target communication method. The second judgment module is used to determine the predicted computing power allocation result of the multiple tasks as the target computing power allocation result of the multiple tasks, and to determine the computing power allocation model as the target computing power allocation model, when the allocation detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible.

[0012] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the computing power allocation method based on heterogeneous graphics processors in any of the above aspects.

[0013] On the other hand, a storage medium is provided that stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the computing power allocation method based on heterogeneous graphics processors as described above.

[0014] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a storage medium. A processor of an electronic device reads the computer instructions from the storage medium and executes the computer instructions, causing the electronic device to perform any of the above-described methods for allocating computing power based on heterogeneous graphics processors.

[0015] This application embodiment obtains the system performance status of the target computing cluster and the computing power requirements of multiple tasks. The target computing cluster includes multiple graphics processors, which are input into a trained computing power allocation model to generate a corresponding computing power allocation strategy. Subsequently, the actual resource utilization of the target computing cluster and the preset allocation constraints are used to verify the computing power allocation strategy. This allows for dynamic resource scheduling of graphics processors while effectively ensuring the reliability of the scheduling strategy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a computing power allocation method based on heterogeneous graphics processors provided in an embodiment of this application; Figure 3 This is a structural block diagram of a computing power distribution device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] Please see Figure 1 The diagram shown is an implementation environment provided by an embodiment of this application. The implementation environment includes multiple terminals and / or multiple servers. The multiple terminals, multiple servers, or terminals and servers can communicate with each other via wired or wireless means as needed.

[0021] Specifically, each terminal or server may include one or more graphics processing units (GPUs) or one or more neural processing units (NPUs). The physical layer may include multiple GPU nodes and multiple NPU nodes. Each type of processor node may correspond to an independent terminal or server. Multiple GPU nodes and multiple NPU nodes can communicate with each other through Remote Direct Memory Access (RDMA).

[0022] The physical layer can construct a topology map of hardware resources by being aware of different communication types (such as PCIe / NVLink), thereby quantifying metrics such as streaming multiprocessor units (SM) and HBM bandwidth of each GPU.

[0023] For example, resource pooling is implemented based on vGPU slicing, which can dynamically adjust virtual video memory (1GB~32GB) and computing power quota (10%~100%).

[0024] For example, the task layer includes multiple containers, each running a client runtime environment to run applications. By building a task profile model, characteristics such as IO intensity and memory requirements of AI training / inference tasks can be automatically identified.

[0025] Optionally, terminals include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft.

[0026] It should be noted that the server involved in the embodiments of this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0027] In one exemplary implementation, both the terminal and the server can be node devices in the blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, thus enabling information sharing among multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which consists of multiple blocks, and adjacent blocks are related, ensuring that any data tampering in any block can be detected by the next block. This prevents data tampering in the blockchain and guarantees the security and reliability of the data in the blockchain.

[0028] The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0029] Please see Figure 2 The diagram shows a flowchart of a computing power allocation method based on heterogeneous graphics processors provided in an embodiment of this application. The method may include: S201: Obtain the system performance status of the target computing cluster and the computing power requirements of multiple tasks; the target computing cluster includes multiple graphics processors.

[0030] For example, the target computing cluster may include multiple computing devices (such as terminals and servers), each computing device as a node, and a node may include one or more graphics processors (GPUs). Each GPU can form multiple vGPUs based on vGPU slicing technology. Subsequently, a weighted topology graph of the GPUs can be constructed to dynamically adjust computing power. The aforementioned GPUs can be NVIDIA or domestically produced GPUs.

[0031] For example, this application can abstract NVIDIA / domestic GPUs into a unified vGPU resource pool based on an intelligent scheduling platform; deploy a topology-aware scheduler to automatically identify physical connection topologies such as NVLink, HCCS, and PCIe, and realize the construction of a virtualized resource pool.

[0032] For example, embodiments of this application can determine the communication relationships and types between different GPUs based on hardware detection tools and application interfaces. For instance, `nvidia-smi` can be used to obtain the NVLink topology; the `lspci -tv` command can be used to obtain the PCIe tree structure; the `ibnetdiscover` command can be used to obtain the cross-node RDMA connection status and record the IB switch port mappings of each node, etc. Specifically, communication types can include NVLink (for intra-node communication), HCCS (for intra-node communication), PCIe (for intra-node or inter-node communication), and IB EDR (for inter-node communication).

[0033] For example, a topology graph can be constructed based on the perceived communication between the GPUs. Specifically, hardware resources can be abstracted into a weighted graph based on the following pseudocode: Python class TopologyGraph: def __init__(self): self.nodes = {}# {gpu_id: {numa_node, memory_capacity (memory size)}} self.edges = []# [(src (start), dst (destination), bandwidth (line width), latency (latency), link_type (connection type))] def add_connection(self, src, dst, bw, lat, conn_type): self.edges.append( (src, dst, {'bandwidth': bw, 'latency': lat, 'type': conn_type}) ) (Add edge) In one exemplary embodiment, the system performance status parameters of the target computing cluster are the status parameters of the available graphics processors and bandwidth utilization in the target computing cluster; the computing power requirement parameters of the multiple tasks include the required video memory capacity of each task and the required number of multi-stream processors for each task. Optionally, the available graphics processors are graphics processors that have a preset amount of remaining video memory and a preset number of multi-stream processors, and that are not faulty.

[0034] The state variables of a graphics processing unit can specifically include the GPU's identifier, as well as the remaining video memory and the number of remaining stream processors for each GPU.

[0035] Specifically, you can enable the `with torch.profiler.profile()` context manager to capture data from all stages of forward propagation, back propagation, and optimizer steps.

[0036] If available, the following pseudocode can be used to collect data such as CPU / GPU time consumption, CUDA kernel call count, memory allocation / release events, and operator time distribution.

[0037] profiler = torch.profiler.profile( activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA], schedule=torch.profiler.schedule(wait=1, warmup=1, active=3), on_trace_ready=torch.profiler.tensorboard_trace_handler('. / logs'), record_shapes=True, profile_memory=True ) Extract key features from the Profiler output and construct a multi-dimensional vector: Computationally intensive metrics: FLOPs density: Total number of floating-point operations / Task duration.

[0038] Calculation time percentage: (CUDA kernel time - memory operation time) / total time.

[0039] Video memory characteristics: Peak video memory: Maximum video memory usage (MB) during the task's lifecycle.

[0040] Video memory volatility: the number of video memory allocation / release operations.

[0041] IO characteristics: Data loading latency: The percentage of time the DataLoader thread is blocked.

[0042] PCIe bandwidth utilization: Data transfer rate from Host to Device (GB / s).

[0043] Task metadata: Task type: One-hot encoding for training (0) / inference (1) (converting parameters into a vector of 0s and 1s, such as the red [0,1,0]) Model structural complexity: Normalized value based on ResNet50 (1.0) Example of a feature vector: [0.85, 0.72, 4096, 12.5, 0.15, 8.3, 1, 0, 1.2] The above features can then be processed using Min-Max normalization, quantile discretization, and missing value handling. Min-Max normalization is used to scale metrics such as FLOPs density and memory usage to the [0,1] interval; quantile discretization is used to bin high-variance metrics (such as PCIe bandwidth); and missing value handling is used to fill in missing features (such as multi-GPU tasks) with the mean.

[0044] The aforementioned feature vectors can be used to select all or some features as input to the computing power allocation model, as needed.

[0045] S203: Input the system performance status of the target computing cluster and the computing power demand parameters of multiple tasks into the computing power allocation model to generate the predicted computing power allocation results of the multiple tasks; the computing power allocation model is a deep reinforcement learning model.

[0046] The system performance state variables of the target computing cluster and the computing power requirement parameters of multiple tasks can constitute a state space, which is formed by concatenating the computing power requirement vectors of N tasks, for example: [Task1_Vector, Task2_Vecto+r, ..., TaskN_Vector], and system performance status variables (number of available GPUs, remaining video memory, NVLink bandwidth utilization, etc.).

[0047] For example, suppose there are 3 tasks, namely Task1, Task2 and Task3, and the computing power requirements of each task are as follows: Task1, Task2 and Task3 require 30, 45 and 20 SMs respectively; Task1, Task2 and Task3 require 8GB, 16GB and 4GB of video memory respectively.

[0048] In this way, a reinforcement learning-based elastic allocation engine can monitor in real time the SM utilization rate, the active proportion of warps in each SM, the integer / floating-point throughput of the computing core, HBM bandwidth utilization, HBM or memory read / write bandwidth, L1 / L2 cache hit rate, memory utilization rate, task queue wait time, task priority, streaming multiprocessor (SM) task throughput, overall GPU utilization, power usage and temperature, PCIe bus utilization, and other indicators, and dynamically adjust the computing power allocation ratio; a preemptive scheduling strategy is introduced, allowing high-priority tasks to preempt idle SM units of low-priority tasks.

[0049] In an exemplary embodiment, the method further includes: determining the system performance state of the target computing cluster and the computing power requirement parameters of the multiple tasks as first state variables; determining a second state variable after executing the multiple tasks based on the predicted computing power allocation result; the second state variable includes the updated system performance state of the target computing cluster and the updated computing power requirement parameters of the multiple tasks; storing the first state variable, the predicted computing power allocation result, the reward value corresponding to the predicted computing power allocation result, and the second state variable as an experience tuple in an experience replay pool. The first state variable can be denoted as s_t (t is the current time), the predicted computing power allocation result is the action quantity a, the reward value corresponding to the predicted computing power allocation result is the reward quantity r, the second state variable can be denoted as s_{t+1} (t+1 is the next time), and the formed experience tuple can be denoted as (s_t, a, r, s_{t+1}). Subsequently, when training and updating the computing power allocation model, sample data can be directly obtained from the experience replay pool.

[0050] In this embodiment, the computing power allocation model is specifically a Deep Q-Network (DQN), a deep learning-based reinforcement learning algorithm primarily used to solve reinforcement learning problems based on value functions. Specifically, it employs a neural network to learn the Q-value function.

[0051] Specifically, a DQN network typically includes an input layer, hidden layers, and an output layer. The number of neurons in the input layer is equal to the dimension of the first state variable s_t, such as the route network prediction table, the vehicle's current service state, the vehicle's service priority, the vehicle's navigation route information, and the current ground network state. The hidden layer may include 2-3 fully connected layers, each containing 128-256 neurons, using ReLU as the activation function. The number of neurons in the output layer is equal to the size of the action space, and each neuron outputs the Q-value (expected cumulative reward) for the corresponding action. The final decision, such as the predicted computing power allocation result, can be output subsequently.

[0052] The computing power allocation model can output the predicted computing power allocation results based on the following pseudocode.

[0053] raw_output (torch.Tensor): The raw tensor output by the DQN network, with shape [num_tasks, action_dim].

[0054] Assume the action vector for each task is [gpu_id_raw, compute_quota_raw, mem_raw, ...] return: tuple: A tuple containing the following elements: - gpu_ids (torch.Tensor): GPU IDs assigned to each task, integer tensors.

[0055] - compute_quotas (torch.Tensor): The computing power quota for each task, ranging from [0, 1].

[0056] - mem_reserves (torch.Tensor): The percentage of GPU memory reserved for each task, ranging from [0.1, 1.0].

[0057] The predicted computing power allocation result can be represented as an action space, defined as a multi-dimensional vector, where each element corresponds to a resource allocation strategy for a task. GPU allocation: Integer type, such as [0,1,2] indicates that the task is bound to the specified GPU.

[0058] Computing power quota: Continuous type, 0.1~1.0 represents the proportion of SM units occupied by the GPU.

[0059] Video memory reservation: Continuous type, 0.1~1.0 indicates the pre-allocated percentage of video memory.

[0060] In this embodiment of the application, the training process of the above-mentioned computing power allocation model is as follows: Obtain a training sample set; the training sample set includes the first state quantity, the second state quantity, the reward quantity, and the action quantity mentioned above. Specifically, the training sample set is obtained from the experience replay pool; each sample is a mini-batch (256) for training.

[0061] In this embodiment, the computing power allocation model is a dual-network structure, including an online network and a target network. The target network is the actual target computing power allocation model used. It is not updated through gradient descent, but rather through synchronous updates. That is, the online network is continuously trained and updated, and the current target network is updated to the current online network using preset update parameters (such as preset time steps). The network structures of the online network and the target network are completely identical. Specifically, the parameters of the Online Network can be softly updated to the Target Network every 1000 steps.

[0062] In the DQN model, the training sample data includes state s_t, action a, reward r_t, and new state s_{t+1}. At time step t, under the state s_t of the environment, performing action a will result in reward r_t and a new state s_{t+1}.

[0063] In this embodiment, the state refers to the system performance status of the target computing cluster and the computing power requirements of multiple tasks, reflecting the current system performance and the computing power requirements of the tasks. The action is a specific computing power scheduling operation, and the reward is the reward value for performing the action under the aforementioned environmental state.

[0064] Next, the online network is trained using the training sample set, and a preset training step size is set. The model parameters of the online network are copied to the target network to update the model parameters of the target network. The target network is used to output the action quantities for network switching based on the first state variable.

[0065] Specifically, the training process of the online network based on the training sample set is as follows: The first state variable is input into the online network, and the predicted Q value corresponding to each action variable is determined, denoted as Q(s, a); an action variable is determined using the ε-greedy policy, the selected action variable is executed, and the second state variable and corresponding reward variable are obtained after the action variable is executed. The second state variable is input into the online network to determine the corresponding predicted Q' value; based on the predicted Q' value and the reward amount... Determine the target Q value; the target Q value represents the sum of expected future rewards after taking action a from the current state s. Determine the loss value based on the target Q value and the predicted Q value, and train the online network with the goal of minimizing the loss value until the preset training iteration conditions are met.

[0066] For example, the reward can be implemented based on the following reward function: Reward function def calculate_reward(): # Positive Rewards reward = completed_tasks * 10 reward += (1 / avg_task_delay) * 5 # Delay countdown weighted # Penalty Items reward -= gpu_idle_rate * 20# Idle rate penalty reward -= sla_violations * 50# SLA Default Penalty return reward For example, this computing power allocation model can be Huber Loss to reduce the impact of outliers: loss = F.smooth_l1_loss(q_values, target_q_values) To measure the degree of difference between the predicted Q value and the target Q value.

[0067] For example, exploration and exploitation can be based on an ε-greedy strategy, with an initial ε=0.9 that decays linearly to 0.05 with the number of training steps.

[0068] Preset training iteration conditions may include one or both of the following: preset number of iterations and preset loss value.

[0069] The specific scheduling parameters can be obtained by decoding the values ​​directly output from the computing power allocation model. This can be implemented using the following pseudocode: # 1. GPU-bound decoding: Mapping consecutive values ​​to specific GPU indices #Assume raw_output[:, 0] is a scaled value, for example, in the range [0, num_gpus-1]. gpu_ids = torch.clamp(torch.round(raw_output[:, 0]), 0, self.num_gpus- 1).int() # 2. Computing power quota decoding: Sigmoid is used to ensure that the output is in the range of (0,1), representing the percentage of SM units occupied. compute_quotas = torch.sigmoid(raw_output[:, 1]) # 3. Memory Reservation Decoding: Use Clamp to limit memory reservation to a reasonable range to avoid allocating 0 memory. mem_reserves = torch.clamp(raw_output[:, 2], 0.1, 1.0) return gpu_ids, compute_quotas, mem_reserves def should_trigger_rerun(self, predicted_utilization, actual_utilization, cp_sat_feasible): S205: Determine the predicted system resource utilization rate of the target computing cluster based on the predicted computing power allocation results of the multiple tasks; and obtain the actual system resource utilization rate of the target computing cluster.

[0070] In an exemplary implementation, the predicted computing power allocation result includes the identifier of the target graphics processor allocated to each task; before step S205, the method further includes: determining the communication mode of the target graphics processor from the topology map of the graphics processors of the target computing cluster based on the identifier of the target graphics processor.

[0071] S207: When the graphics processors corresponding to the multiple tasks communicate based on the first target communication method, the allocation detection result is determined based on the predicted system resource utilization rate and actual resource system utilization rate of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processor.

[0072] For example, the first target communication method can be NVLink, HCCS, or IB EDR. When the target graphics processors communicate through the first target communication method, the results output by the computing power allocation model can be further verified based on the actual resource system utilization and preset allocation termination conditions. If either of these two conditions is not met, the following second detection result is generated; only when both conditions are met simultaneously is the following first detection result generated. For details, please see the following explanation: In an exemplary implementation, the predicted computing power allocation result includes the identifier of the target graphics processor allocated to each task; step S207, which involves determining the allocation detection result based on the predicted system resource utilization and actual system resource utilization of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processor, may specifically include: determining the difference between the predicted system resource utilization and the actual system resource utilization of the target computing cluster as a resource deviation value; if the resource deviation value does not exceed a first threshold, determining whether the resource utilization and memory reservation ratio of the target graphics processor meet the preset allocation constraints of the graphics processor; if yes, generating a first detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible; if no, generating a second detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is infeasible; if the resource deviation value exceeds the first threshold, adjusting the parameters of the computing power allocation model, and predicting computing power allocation for the multiple tasks based on the updated computing power allocation model.

[0073] For example, the first threshold can be 20%.

[0074] In another exemplary embodiment, step S207, which involves determining the allocation detection result based on the predicted system resource utilization and actual system resource utilization of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processor, may specifically include: when the resource utilization and memory reservation ratio of the target graphics processor meet the preset allocation constraints of the graphics processor, determining the difference between the predicted system resource utilization and actual system resource utilization of the target computing cluster as a resource deviation value; determining whether the resource deviation value does not exceed a first threshold; if yes, generating a first detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible; if no, generating a second detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is not feasible; when the resource utilization and memory reservation ratio of the target graphics processor do not meet the preset allocation constraints of the graphics processor, adjusting the parameters of the computing power allocation model, and performing computing power allocation prediction for the multiple tasks based on the updated computing power allocation model.

[0075] For example, when the resource utilization and memory reservation ratio of the target graphics processor do not meet the preset allocation constraints of the graphics processor, the computing power allocation model can be updated as above during re-inference, or a conservative backup solution can be temporarily adopted.

[0076] In one exemplary embodiment, the predicted computing power allocation result further includes the resource utilization rate and video memory reservation ratio of the target graphics processor; the preset allocation constraints of the graphics processor include: the load of each graphics processor does not exceed the number of streaming multiprocessors of the graphics processor; and the video memory allocation of each graphics processor does not exceed the capacity of physical video memory.

[0077] In another exemplary embodiment, after step S205, the method further includes: when the graphics processors corresponding to the plurality of tasks communicate based on a second target communication method, and the total bandwidth occupancy under the same second target communication does not exceed a second threshold, determining an allocation detection result based on the predicted system resource utilization and actual resource system utilization of the target computing cluster, and / or the predicted computing power allocation result of the plurality of tasks and the preset allocation constraints of the graphics processors. A detailed explanation of the allocation detection result determination in this case is provided above.

[0078] Of course, when the graphics processors corresponding to the multiple tasks communicate based on the second target communication method, and the total bandwidth usage under the same second target communication exceeds the second threshold, a second detection result is generated.

[0079] For example, to avoid frequent re-inference, a simple cooldown period check can be added, as shown in the pseudocode below: # if self.reasoning_buffer and (time.time() - self.reasoning_buffer[-1]) < 5.0: # For example, to prevent repeated triggering within 5 seconds # trigger = False # reason.append("Currently in cooling-off period") If trigger and reason: print(f"DQN re-inference triggered. Reason: {'; '.join(reason)}") #Record this triggered decision self.reasoning_buffer.append((predicted_utilization, actual_utilization, cp_sat_feasible, reason)) return trigger def get_allocation_plan(self, current_state, monitored_utilization=None, cp_sat_validator=None): S209: If the allocation detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible, the predicted computing power allocation result of the multiple tasks is determined as the target computing power allocation result of the multiple tasks, and the computing power allocation model is determined as the target computing power allocation model.

[0080] To ensure dynamic balance, task migration can be triggered when the SM utilization of a certain GPU is detected to be >95% for more than 5 seconds; high-priority tasks can preempt the NVLink channels of low-priority tasks; and faulty GPUs can be automatically isolated and task flows can be rerouted via PCIe.

[0081] Corresponding to the computing power allocation methods based on heterogeneous graphics processors provided in the above embodiments, this application also provides a computing power allocation device based on heterogeneous graphics processors. Since the computing power allocation device based on heterogeneous graphics processors provided in this application corresponds to the computing power allocation methods based on heterogeneous graphics processors provided in the above embodiments, the implementation methods of the aforementioned computing power allocation methods based on heterogeneous graphics processors are also applicable to the computing power allocation device based on heterogeneous graphics processors provided in this embodiment, and will not be described in detail in this embodiment.

[0082] Please see Figure 3 The diagram shows a structural schematic of a computing power allocation device provided in an embodiment of this application. This device has the function of implementing the computing power allocation method based on heterogeneous graphics processors in the above-described method embodiments. This function can be implemented in hardware or by hardware executing corresponding software. Figure 3 As shown, the computing power allocation device 300 may include: The acquisition module 301 is used to acquire the system performance status of the target computing cluster and the computing power requirements of multiple tasks; the target computing cluster includes multiple graphics processors. The prediction module 303 is used to input the system performance status of the target computing cluster and the computing power demand parameters of multiple tasks into the computing power allocation model to generate the predicted computing power allocation results of the multiple tasks; the computing power allocation model is a deep reinforcement learning model. The first determining module 305 is used to determine the predicted system resource utilization rate of the target computing cluster based on the predicted computing power allocation results of the multiple tasks; and to obtain the actual system resource utilization rate of the target computing cluster. The first judgment module 307 is used to determine the allocation detection result based on the predicted system resource utilization rate and actual resource system utilization rate of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processors, when the multiple tasks communicate with each other based on the first target communication method. The second judgment module 309 is used to determine the predicted computing power allocation result of the multiple tasks as the target computing power allocation result of the multiple tasks, and to determine the computing power allocation model as the target computing power allocation model, when the allocation detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible.

[0083] In an exemplary implementation, the predicted computing power allocation result includes the identifier of the target graphics processor allocated to each task; the first judgment module is used to determine the difference between the predicted system resource utilization rate and the actual resource system utilization rate of the target computing cluster as a resource deviation value; If the resource deviation value does not exceed the first threshold, determine whether the resource utilization rate and video memory reservation ratio of the target graphics processor meet the preset allocation constraints of the graphics processor; if yes, generate a first detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible; if no, generate a second detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is not feasible. If the resource deviation value exceeds the first threshold, the parameters of the computing power allocation model are adjusted, and computing power allocation prediction is performed on the multiple tasks based on the updated computing power allocation model.

[0084] In an exemplary embodiment, the first judgment module is used to determine the difference between the predicted system resource utilization rate and the actual resource system utilization rate of the target computing cluster as a resource deviation value when the resource utilization rate and video memory reservation ratio of the target graphics processor meet the preset allocation constraints of the graphics processor. Determine whether the resource deviation value does not exceed a first threshold; if yes, generate a first detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible; if no, generate a second detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is not feasible. If the resource utilization and memory reservation ratio of the target graphics processor do not meet the preset allocation constraints of the graphics processor, the parameters of the computing power allocation model are adjusted, and computing power allocation prediction is performed on the multiple tasks based on the updated computing power allocation model.

[0085] In one exemplary embodiment, the predicted computing power allocation result further includes the resource utilization rate and video memory reservation ratio of the target graphics processor; the preset allocation constraints of the graphics processor include: The load on each graphics processor does not exceed the number of streaming multiprocessors of the graphics processor; The amount of video memory allocated to each graphics processor shall not exceed the capacity of the physical video memory.

[0086] In one exemplary embodiment, the predicted computing power allocation result includes an identifier of the target graphics processor allocated to each task; the apparatus further includes: The second determining module is used to determine the communication method of the target graphics processor from the topology map of the graphics processors of the target computing cluster based on the identifier of the target graphics processor.

[0087] In one exemplary implementation, the system performance state variables of the target computing cluster are the state variables of the available graphics processors and the bandwidth utilization rate in the target computing cluster. The computing power requirements of the multiple tasks include the required video memory capacity for each task and the number of multi-stream processors required for each task.

[0088] In one exemplary embodiment, the apparatus further includes: The third determining module is used to determine the system performance state quantity of the target computing cluster and the computing power requirement parameters of the multiple tasks as the first state quantity; The fourth determining module is used to determine the second state quantity after executing the multiple tasks based on the predicted computing power allocation results; the second state quantity includes the updated system performance state quantity of the target computing cluster and the updated computing power requirement parameter of the multiple tasks; The storage module is used to store the first state quantity, the prediction computing power allocation result, the reward value corresponding to the prediction computing power allocation result, and the second state quantity as an experience tuple into the experience replay pool.

[0089] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0090] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement any of the computing power allocation methods based on heterogeneous graphics processors provided in the above method embodiments.

[0091] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0092] The embodiments of this application also provide a storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a computing power allocation method based on a heterogeneous graphics processor. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the computing power allocation methods based on heterogeneous graphics processors provided in the above method embodiments.

[0093] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a storage medium. The processor of an electronic device reads the computer instructions from the storage medium and executes the computer instructions, causing the electronic device to perform any of the heterogeneous graphics processor-based computing power allocation methods provided in the above-described method embodiments.

[0094] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0095] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0096] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0097] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a storage medium, such as a read-only memory, a disk, or an optical disk.

[0098] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for allocating computing power based on heterogeneous graphics processors, characterized in that, The method includes: Obtain the system performance status parameters of the target computing cluster and the computing power requirements of multiple tasks; the target computing cluster includes multiple graphics processors; The system performance status parameters of the target computing cluster and the computing power requirements of multiple tasks are input into the computing power allocation model to generate the predicted computing power allocation results for the multiple tasks; the computing power allocation model is a deep reinforcement learning model. Based on the predicted computing power allocation results of the multiple tasks, the predicted system resource utilization rate of the target computing cluster is determined; and the actual system resource utilization rate of the target computing cluster is obtained. When the graphics processors corresponding to the multiple tasks communicate based on the first target communication method, the allocation detection result is determined based on the predicted system resource utilization and actual resource system utilization of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processors. If the allocation detection results indicate that the predicted computing power allocation results of the multiple tasks are feasible, the predicted computing power allocation results of the multiple tasks are determined as the target computing power allocation results of the multiple tasks, and the computing power allocation model is determined as the target computing power allocation model.

2. The computing power allocation method according to claim 1, characterized in that, The predicted computing power allocation result includes the identifier of the target graphics processor allocated to each task; the predicted system resource utilization rate and actual resource system utilization rate based on the target computing cluster, and / or, the allocation detection result determined by the predicted computing power allocation results of the multiple tasks and the preset allocation constraints of the graphics processor, includes: The difference between the predicted system resource utilization rate and the actual system resource utilization rate of the target computing cluster is determined as the resource deviation value; If the resource deviation value does not exceed the first threshold, determine whether the resource utilization rate and video memory reservation ratio of the target graphics processor meet the preset allocation constraints of the graphics processor; if yes, generate a first detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible; if no, generate a second detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is not feasible. If the resource deviation value exceeds the first threshold, the parameters of the computing power allocation model are adjusted, and computing power allocation prediction is performed on the multiple tasks based on the updated computing power allocation model.

3. The computing power allocation method according to claim 1, characterized in that, The predicted system resource utilization and actual system resource utilization based on the target computing cluster, and / or the predicted computing power allocation results of the multiple tasks and the preset allocation constraints of the graphics processor, determine the allocation detection results, including: When the resource utilization rate and video memory reservation ratio of the target graphics processor meet the preset allocation constraints of the graphics processor, the difference between the predicted system resource utilization rate and the actual resource system utilization rate of the target computing cluster is determined as the resource deviation value. Determine whether the resource deviation value does not exceed a first threshold; if yes, generate a first detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible; if no, generate a second detection result; the first detection result indicates that the predicted computing power allocation result of the multiple tasks is not feasible. If the resource utilization and memory reservation ratio of the target graphics processor do not meet the preset allocation constraints of the graphics processor, the parameters of the computing power allocation model are adjusted, and computing power allocation prediction is performed on the multiple tasks based on the updated computing power allocation model.

4. The computing power allocation method according to claim 2 or 3, characterized in that, The predicted computing power allocation results also include the resource utilization rate and video memory reservation ratio of the target graphics processor; The preset allocation constraints of the graphics processor include: The load on each graphics processor does not exceed the number of streaming multiprocessors of the graphics processor; The amount of video memory allocated to each graphics processor shall not exceed the capacity of the physical video memory.

5. The computing power allocation method according to claim 1, characterized in that, The predicted computing power allocation result includes the identifier of the target graphics processor allocated to each task; before determining the allocation detection result based on the predicted system resource utilization and actual system resource utilization of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processor, the method further includes: The communication method of the target graphics processor is determined from the topology of the graphics processors in the target computing cluster based on the identifier of the target graphics processor.

6. The computing power allocation method according to any one of claims 1-5, characterized in that, The system performance state variables of the target computing cluster are the state variables of the available graphics processors and the bandwidth utilization rate in the target computing cluster. The computing power requirements of the multiple tasks include the required video memory capacity for each task and the number of multi-stream processors required for each task.

7. The computing power allocation method according to any one of claims 1-5, characterized in that, The method further includes: The system performance state of the target computing cluster and the computing power requirement parameters of the multiple tasks are determined as the first state quantity; Determine a second state quantity after executing the multiple tasks based on the predicted computing power allocation results; the second state quantity includes the updated system performance state quantity of the target computing cluster and the updated computing power requirement parameter of the multiple tasks; The first state variable, the prediction computing power allocation result, the reward value corresponding to the prediction computing power allocation result, and the second state variable are stored as experience tuples in the experience replay pool.

8. A computing power distribution device, characterized in that, include: The acquisition module is used to acquire the system performance status of the target computing cluster and the computing power requirements of multiple tasks. The target computing cluster includes multiple graphics processors; The prediction module is used to input the system performance status of the target computing cluster and the computing power demand parameters of multiple tasks into the computing power allocation model to generate the predicted computing power allocation results of the multiple tasks; the computing power allocation model is a deep reinforcement learning model. The first determining module is used to determine the predictive system resource utilization rate of the target computing cluster based on the predictive computing power allocation results of the multiple tasks; And obtain the actual system resource utilization rate of the target computing cluster; The first judgment module is used to determine the allocation detection result based on the predicted system resource utilization rate and actual resource system utilization rate of the target computing cluster, and / or the predicted computing power allocation result of the multiple tasks and the preset allocation constraints of the graphics processor, when the multiple tasks communicate with each other based on the first target communication method. The second judgment module is used to determine the predicted computing power allocation result of the multiple tasks as the target computing power allocation result of the multiple tasks, and to determine the computing power allocation model as the target computing power allocation model, when the allocation detection result indicates that the predicted computing power allocation result of the multiple tasks is feasible.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the computing power allocation method based on heterogeneous graphics processors as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the computing power allocation method based on heterogeneous graphics processors as described in any one of claims 1 to 7.

Images