Efficient inference method for large neural network model based on multi-gpgpu

Abstract

The application belongs to the technical field of artificial intelligence and high-performance computing, and specifically relates to a neural network large model efficient inference method based on multiple GPGPUs. The method aims to solve the problems of large communication overhead between multiple processors, uneven load, low resource utilization, and high data transmission delay. Through static analysis and mixed-granularity partitioning of the model computation graph, the computation task is divided into multiple subgraphs; combined with heterogeneous resource perception and dynamic mapping strategy, the subgraphs are allocated to the optimal GPGPU based on a weighted cost function; a global pipeline scheduling plan is constructed using communication topology perception to maximize computation and communication overlap; data is loaded in advance through the host-side hierarchical cache and asynchronous prefetch mechanism to hide transmission delay; and multi-stream concurrent execution and event-based lightweight synchronization are adopted on each GPGPU to reduce the waiting overhead. The application can significantly reduce inference delay, improve throughput and hardware utilization, and has good adaptability and scalability.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and high-performance computing technology, specifically involving an efficient inference method for large neural network models based on multiple GPGPUs. Background Technology

[0002] With the in-depth development of artificial intelligence technology, neural network models centered on deep learning have become a key force driving technological change across various industries. Large-scale pre-trained models, due to their powerful feature extraction and generalization capabilities, have demonstrated outstanding performance in fields such as natural language processing and computer vision, leading to a dramatic increase in reliance on computing infrastructure. As the core link in transforming theoretical models into practical application value, the efficiency of model inference and deployment directly determines the service's response speed and processing capacity.

[0003] Efficient inference for large neural network models has become a key challenge restricting the large-scale application of the technology. To meet their massive computing and storage requirements, the industry generally uses high-performance computing units such as graphics processing units (GPUs) or dedicated neural network processing units (NPUs) for acceleration. The core objective of this technology is to minimize inference latency and maximize data throughput while ensuring model accuracy, thereby supporting real-time or high-concurrency application scenarios.

[0004] In existing technologies, the memory capacity and computing power of a single processor are insufficient to handle neural network models with a large number of parameters, forcing developers to adopt multi-processor parallel computing solutions. However, simple model partitioning and task allocation lead to significant cross-processor communication overhead, creating new performance bottlenecks. Simultaneously, existing parallel strategies struggle to achieve fine-grained scheduling and load balancing of computational tasks when dealing with complex network layer dependencies, resulting in idle computing resources and reduced overall hardware utilization. Furthermore, the data transmission latency between host memory and multi-processor GPU memory further limits end-to-end inference performance. Therefore, there is an urgent need for an efficient inference method for large neural network models based on multiple GPGPUs. Summary of the Invention

[0005] The purpose of this invention is to provide an efficient inference method for large neural network models based on multiple GPGPUs, which can effectively solve the problems in the background technology mentioned above, especially the core technical bottlenecks that are common in the prior art, such as huge communication overhead between multiple processors, unbalanced computing task load, low hardware resource utilization, and significant data transmission latency.

[0006] To achieve the above objectives, the technical solution adopted by this invention is as follows: an efficient inference method for large neural network models based on multiple GPGPUs, comprising the following specific steps:

[0007] Step (1) Static analysis and hybrid granularity partitioning of the model computation graph: Obtain the large neural network model to be inferred and parse it into a directed acyclic computation graph, which contains nodes representing computational operations and edges representing data dependencies; perform static analysis on the computation graph and extract attribute information such as computational load, memory usage and data dependencies of each node; based on the attribute information, use a hybrid granularity partitioning algorithm to divide the computation graph into multiple computational subgraphs, where each computational subgraph is assigned to an independent target graphics processor for execution;

[0008] Step (2) Heterogeneous resource awareness and dynamic mapping of computing tasks: Real-time monitoring of the current load status, available video memory capacity and interconnect bandwidth of each computing unit in the multi-graphics processor cluster; Constructing a dynamic resource view, and combining the computing subgraphs generated in step (1) and their dependencies, applying a task mapping strategy based on a weighted cost function to dynamically allocate each computing subgraph to the optimal target graphics processor. The cost function comprehensively considers computing time, data transmission time and resource idle penalty.

[0009] Step (3) Communication topology awareness and pipeline parallel scheduling: Identify the physical communication topology of the multi-GPU cluster and establish a communication cost matrix; Based on the data dependencies between the communication cost matrix and the computation subgraphs, generate a global execution plan, which constructs a deep pipeline by maximizing the overlap between computation and communication; The execution plan defines the start time, execution order, and specific paths and timings for cross-processor data transfer for each computation subgraph to minimize global execution latency;

[0010] Step (4) Hierarchical data caching and asynchronous prefetching management: A hierarchical data cache is established on the host CPU side to manage the model's weight parameters and inference input data; According to the global execution plan generated in step (3), a data prefetching schedule is created, which transmits the data required for subsequent computation subgraphs asynchronously from the host memory to the target graphics processor's video memory in advance, thereby hiding the data transmission delay;

[0011] Step (5) Multi-stream concurrent execution and low-overhead synchronization control: Create multiple independent computation streams and data transfer streams on each graphics processor; according to the global execution plan, schedule the operations within the computation subgraph to be executed on the computation stream, and schedule the cross-processor data transfer tasks to be executed on the data transfer stream; adopt an event-based lightweight synchronization mechanism to accurately coordinate the execution dependencies between different streams and between different processors, ensuring data consistency while minimizing synchronization waiting overhead.

[0012] Preferably, in step (1), the hybrid granularity partitioning algorithm first performs coarse-grained partitioning of the model by layer to form preliminary hierarchical partitions; then, for specific layers that are computationally intensive or memory-access intensive, fine-grained partitioning is performed at the operator level, splitting a single large operator into multiple sub-operators that can be executed in parallel on different processors. The goal of the partitioning decision is to minimize the total amount of data transmission across partition boundaries, while ensuring that the computational load balance within each partition is greater than 90%.

[0013] Preferably, in step (1), when performing static analysis on the computation graph, the computation characteristics of different operators in the model are quantitatively evaluated and classified into three types: computationally intensive, memory-bandwidth intensive, or communication-intensive. This type information is used as one of the input weights of the hybrid granularity partitioning algorithm to achieve more refined resource matching.

[0014] Preferably, in step (2), the task mapping strategy based on the weighted cost function is solved using a simulated annealing algorithm. The cost function is defined as a weighted sum of computation time, communication time, and synchronization overhead. The weight coefficients can be adaptively adjusted according to the different requirements of the inference task for latency or throughput. The algorithm converges to a quasi-optimal mapping solution within 200 iterations.

[0015] Preferably, in step (3), pipelined parallel scheduling is applied not only to macroscopic parallelism between model layers, but also to microscopic parallelism at the operator level within the layer; for operators with parallelism, such as matrix multiplication, the system automatically decomposes them into multiple blocks and distributes these blocks to multiple stream processor clusters of the same graphics processor in a micropipeline manner, further improving the computational efficiency of a single processor.

[0016] Preferably, the communication topology awareness mechanism uses the underlying hardware driver interface to query and construct the point-to-point communication bandwidth matrix and latency matrix between processors. The selection of the communication path prioritizes the direct link with the highest bandwidth and lowest latency, such as the NVIDIA UC link, and secondly selects the path connected through a high-speed switching chip.

[0017] Preferably, in step (4), the hierarchical data cache is allocated on the host side using page-locked memory to support high-performance direct memory access between the cache and the graphics processor memory; the cache adopts a least recently used replacement strategy and sets a higher cache priority for specific parameter blocks according to the reuse mode of the weight parameters in the model, with a cache hit rate greater than 95%.

[0018] Preferably, in step (4), the time advance of the data prefetching schedule is dynamically calculated. Its value is the expected communication delay before the corresponding data calculation task starts to be executed plus a safety margin time. The safety margin time is set to 15% of the average communication delay to cope with network jitter.

[0019] Preferably, in step (5), the event-based lightweight synchronization mechanism is implemented by recording a completion event at the end of the data transmission stream and inserting an instruction to wait for the event at the beginning of the computation stream that depends on the data; compared with global fence synchronization, this mechanism reduces the synchronization overhead by at least an order of magnitude and reduces unnecessary processor idle waiting.

[0020] Preferably, the method further includes an online performance profiling and feedback optimization module, which continuously collects the actual computing time, communication time and memory usage of each graphics processor during the inference process, and feeds back these real-time data to the task mapping module in step (2) and the parallel scheduling module in step (3) to form a closed-loop optimization system, so that the execution plan can dynamically adapt to the changing system load and input data characteristics.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. Significantly improves inference performance and throughput: By using hybrid granular partitioning and communication topology-aware pipelined parallel scheduling, this invention increases the overlap between computation and communication to over 85%, effectively hiding cross-processor communication overhead. Compared with traditional model parallelism or data parallelism strategies, for large models with hundreds of billions of parameters, end-to-end inference latency is reduced by 40% to 60%, and the total system throughput is increased by 1.5 to 2 times.

[0023] 2. Significantly improve hardware resource utilization: The heterogeneous resource awareness and dynamic task mapping strategy ensures that computing tasks can be continuously and evenly distributed to each graphics processor, avoiding long-term idleness of some processors due to waiting for data or dependencies; the overall average hardware utilization of the system is increased from about 60% in the existing technology to more than 95%, maximizing the return on investment of expensive computing resources.

[0024] 3. Effectively reduce the impact of data transmission latency: By establishing a hierarchical data cache on the host side and combining it with asynchronous data prefetching in the execution plan, this invention decouples data preparation time from actual computation time, and most of the data transmission latency is successfully hidden; this enables even ultra-large-scale models deployed in environments where the host memory is much larger than the total video memory capacity to achieve performance close to that of full in-memory execution.

[0025] 4. Enhanced system versatility and adaptability: The method of this invention does not rely on a specific model structure or fixed hardware configuration. Through a combination of static analysis and online profiling, it can automatically adapt to different neural network models and multi-GPU cluster topologies. The closed-loop feedback optimization mechanism enables it to dynamically adjust the scheduling strategy, maintain efficient operation under changing loads, and has strong robustness and scalability. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall technical architecture of the efficient inference method for large neural network models based on multiple GPGPUs proposed in this invention. Detailed Implementation

[0027] Example 1

[0028] Please refer to Figure 1 To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments.

[0029] Currently, there is no [specific technology available]. To address the aforementioned technical problems, this invention proposes a neural network processor (NPU) and a GPU, and applies them to an efficient inference method for large neural network models based on multiple GPGPUs.

[0030] This embodiment provides an efficient inference method for large neural network models based on multiple GPGPUs. This method aims to address the performance bottlenecks faced by large-scale models during inference on multi-processor clusters through deep analysis of the neural network model, precise awareness of hardware resources, and coordinated scheduling of computation and communication tasks. The execution flow of this method is managed by a central control node or a distributed coordination service to ensure the orderly execution of each step.

[0031] In the above method, step (1) is static analysis of the model computation graph and hybrid granularity partitioning. This step is the foundation of the entire optimization process, and its goal is to transform a macroscopic, holistic neural network model into a series of load-balanced computational task units suitable for parallel processing. This step is specifically decomposed into multiple micro-operations.

[0032] The initial micro-operations involve model acquisition and parsing. The system first loads the large neural network model to be inferred from a specified storage location (such as a local file system or a network repository). The model is in a standard computation graph representation format, such as the Open Neural Network Exchange Format (ONNX), TensorFlow's SavedModel format, or PyTorch's TorchScript format. After loading, a dedicated graph parser module deserializes the model file, constructing an in-memory directed acyclic computation graph (DAG) data structure. In this data structure, each node represents a basic computational operation or operator, such as matrix multiplication (MatMul), convolution, activation function (ReLU), or layer normalization (LayerNorm). Each node internally stores all the attributes of the operation, such as kernel size, stride, and padding. The directed edges in the graph represent data dependencies between nodes, i.e., the output tensor of one node serves as the input tensor of another node. The properties of an edge record the dimension, data type (such as FP32, FP16, INT8), and memory layout of the corresponding tensor.

[0033] The subsequent micro-operations are static analysis and attribute extraction of the computation graph. In this stage, the system traverses every node in the constructed computation graph. For each node, a performance analyzer module performs a static evaluation to quantify its resource requirements. This evaluation process does not involve actual computation execution. Specifically, computational load extraction estimates the required number of floating-point operations (FLOPs) by analyzing the type of operator and the dimension of the input tensor. For example, a matrix multiplication with an input matrix of (M, K) and a weight matrix of (K, N) is estimated to require 2MK*N FLOPs. Memory usage extraction includes two parts: first, the memory usage of the weight parameters, which is directly read from the model file to determine the size of the weight tensor; second, the memory usage of the runtime activation values, which is calculated by deriving the dimension of the output tensor based on the operator's forward propagation formula, thus determining the required memory space. Data dependencies are extracted directly from the graph's topology, identifying the direct predecessor and successor nodes of each node.

[0034] Furthermore, during the static analysis, the system also performs a quantitative evaluation of the computational characteristics of different operators in the model. This evaluation module precisely classifies each operator based on the ratio of its computational load to the total amount of input and output data (i.e., computational intensity). Specifically, operators with high computational intensity (such as large-scale matrix multiplication) are marked as computationally intensive; operators with low computational intensity but requiring the processing of large amounts of data (such as element-wise addition or activation functions) are marked as memory-bandwidth intensive; and operators that require frequent exchange of intermediate results during distributed execution (such as the AllReduce operation) are marked as communication-intensive. These three types of information are encoded as an integer or enumeration type and stored as additional attributes in each computation graph node, providing crucial decision-making basis for subsequent partitioning algorithms to achieve fine-grained matching between operator characteristics and hardware resources.

[0035] The final micro-operation is a hybrid granularity partitioning. Based on all the extracted attribute information, the system initiates a hybrid granularity partitioning algorithm. Its core objective is to minimize the total amount of data transfer across partition boundaries while ensuring a highly balanced computational load within each partition. The algorithm's execution process is layered. First, coarse-grained partitioning is performed, initially partitioning the model according to its inherent network hierarchy (such as the encoder or decoder layer in a Transformer model). This partitioning method conforms to the model's logical structure and typically aggregates closely related operations together, forming a preliminary set of computational subgraphs. Subsequently, the algorithm evaluates the load and memory usage of each preliminary computational subgraph. For specific layers with excessive computational load or memory consumption, especially those marked as computationally intensive or memory-intensive, the algorithm initiates fine-grained partitioning. At the operator level, a single large operator, such as a matrix multiplication of dimension (8192, 8192), is split into multiple sub-operators that can be executed in parallel on different processors. The splitting method can be along the rows or columns of the matrix, generating multiple small-scale matrix multiplication operations. The objective function of the partitioning decision is a combinatorial optimization problem, aiming to minimize the sum of the weights (weights representing data transfer volume) of all edges spanning different computational subgraphs. The constraint is that the ratio of the standard deviation of the computational load to the average load of each computational subgraph is less than a preset threshold. This algorithm ensures that the computational load balance among all the final partitioned computational subgraphs is greater than 90%. Each generated computational subgraph, along with its internal nodes, edges, and data dependencies with external subgraphs, is encapsulated into a new, independent task unit, awaiting subsequent mapping and scheduling.

[0036] In the above method, step (2) is heterogeneous resource awareness and dynamic mapping of computing tasks. The goal of this step is to optimally allocate the computational subgraph generated in step (1) to the physical computing units in the multi-GPU cluster. Here, "optimal" is defined based on a comprehensive cost function that aims to balance computation, communication, and resource utilization.

[0037] The initial step in this process is to monitor and build a dynamic resource view in real time. A resource monitoring daemon within the system polls the status of each graphics processor in the cluster at fixed time intervals (e.g., every 50 milliseconds) by calling underlying hardware driver interfaces (such as the NVIDIA Management Library, NVML) or dedicated monitoring agents. The monitored metrics are multi-dimensional, including: the current load status of the compute unit (stream processor utilization as a percentage), available video memory capacity (in megabytes per second, MB), and point-to-point interconnect bandwidth with all other processors (in gigabytes per second, GB / s). All this real-time data is aggregated into a central data structure to form a dynamic resource view. This view is logically represented as a matrix or dictionary, where the keys are unique identifiers for the graphics processors, and the values ​​are the processor's current complete state vector.

[0038] The next step is to apply a task mapping strategy based on a weighted cost function. The system combines the dynamic resource view with the computational subgraph and its dependencies generated in step (1) to initiate the task mapping module. The goal of this module is to find the most suitable target graphics processor for each computational subgraph. This process employs a heuristic optimization algorithm based on simulated annealing to address the complexity of large-scale mapping problems. Its core is the design of the cost function, which evaluates the appropriate target graphics processor for a given computational subgraph. Assign to a specific graphics processor The expected costs incurred. This cost function. It is defined as a weighted sum of computation time, communication time, and synchronization overhead.

[0039] Specifically, the cost function is defined as:

[0040]

[0041] in, This is the expected computation time of subgraph i on graphics processor j, calculated by dividing the total FLOPs of the subgraph by the processor's time. The theoretical peak performance is calculated and the current load is taken into account. It is an execution subgraph The total transmission time of the required input data is calculated based on the data source (host memory or other GPU), the amount of data, and the interconnect bandwidth in the dynamic resource view. It is a resource idle penalty. When the current load of processor j is low, the penalty is also low, and vice versa. Its purpose is to encourage the system to allocate tasks to processors that are less busy. , , These are the weighting coefficients for each item, and these coefficients are configurable. For example, in an inference task with extremely high end-to-end latency requirements, and The weight will be increased; however, in a scenario where maximizing system throughput is required, The weights will be increased to ensure that all hardware resources are fully utilized. These weights can be adaptively adjusted based on the inference task's metadata or service level agreement.

[0042] The execution flow of the simulated annealing algorithm is as follows: First, a random initial mapping scheme is generated. Then, an iterative process begins. In each iteration, a new candidate scheme is generated by making a small perturbation to the current mapping scheme (e.g., randomly swapping the target processors of the two computational subgraphs). The total cost of the old and new schemes is calculated. If the new scheme has a lower cost, it is accepted. If the new scheme has a higher cost, it is accepted with a probability that decays with a "temperature" parameter, which helps the algorithm escape local optima. The algorithm is configured to converge within 200 iterations, outputting a near-optimal mapping scheme from the computational subgraph to the graphics processor.

[0043] In the above method, step (3) is communication topology awareness and pipeline parallel scheduling. After obtaining the optimal mapping from the computation subgraph to the physical processor, this step is responsible for generating a global, time-accurate execution plan. Its core objective is to maximize the overlap between computation and communication by building a deep pipeline, thereby hiding communication latency and minimizing global execution latency.

[0044] This step first performs communication topology identification and cost matrix construction. A topology-aware module actively queries and constructs the physical communication topology of the processor cluster using the underlying hardware driver interface. This module probes the direct connections between each pair of graphics processors. For example, it identifies which GPU pairs have high-speed direct links (such as NVIDIA NVLink) and which require communication via the PCIe bus and switching chip. Based on the probe results, the system constructs a communication cost matrix. This is an N×N matrix (N is the number of GPUs), where the matrix elements... Stored from the processor To the processor This matrix shows the precise time and available bandwidth required to transmit a unit of data (e.g., 1GB). It includes not only bandwidth information but also basic communication latency. In subsequent path selection, the scheduler prioritizes the lowest-cost path—a direct link with the highest bandwidth and lowest latency—before considering indirect paths connected via high-speed switching chips.

[0045] The next step is to generate a global execution plan. The algorithm for generating this plan takes the computational subgraph dependencies in step (1), the mapping scheme in step (2), and the communication cost matrix as input. The core of the algorithm is to construct a deep pipelined parallel model. This model is not only applied to macro-parallelism between model layers, that is, different network layers (or computational subgraphs) are treated as different stages of the pipeline and allocated to different processors; it is also applied to micro-parallelism at the operator level within a layer. For operators with internal parallelism, such as a large-scale matrix multiplication, the system will automatically decompose it into multiple data blocks. The computation of these blocks is scheduled to multiple stream processor clusters (SMs) on the same graphics processor and executed in a micro-pipeline manner, that is, while one SM is processing the current data block, another SM can start processing the next data block, thereby further improving the computational efficiency and throughput of a single processor.

[0046] The global execution plan is a detailed instruction sequence schedule that precisely defines the start time and execution order of each computational subgraph on the assigned target processor. For cross-processor data dependencies, the plan specifies the exact data transfer path (optimal selection based on the communication cost matrix) and start timing. The scheduling algorithm aims to maximize the overlap between computation and communication. Specifically, data transfer operations are pre-scheduled to execute on a separate hardware DMA engine, while the processor's computational cores simultaneously execute another computational task independent of that data. In this way, the time spent on data transfer is "hidden" by the execution time of the computational tasks. The final execution plan aims to minimize the global execution latency from the input data entering the system to the final output.

[0047] In the above method, step (4) is hierarchical data caching and asynchronous prefetching management. This step is mainly performed on the host (CPU) side and aims to solve the problem that the weights of ultra-large-scale models far exceed the total memory of a single or all GPUs. Through efficient data management and transmission strategies, a virtual memory space with huge capacity and fast access is simulated.

[0048] The core of this step is establishing a hierarchical data cache. This cache is physically allocated in the host's Dynamic Random Access Memory (DRAM). To enable high-performance direct memory access (DMA) between host memory and graphics processor memory, this cache is designated as pinned memory during allocation. Pinned memory is not swapped to disk by the operating system; its physical address is fixed. This allows the GPU's DMA engine to directly read and write to it, avoiding extra copying of data between the host operating system kernel buffer and user space buffer, thus significantly improving data transfer bandwidth. Logically, this cache is hierarchical, assigning different cache priorities to specific parameter blocks based on the importance and reuse pattern of the weight parameters in the model. For example, the attention weight matrix, which is frequently reused in the Transformer model, will be given a higher cache priority. The cache uses the Least Recently Used (LRU) replacement strategy as its basic page replacement algorithm. When the cache is full and new data needs to be loaded, the least recently accessed data block is selected for replacement. Combined with the priority mechanism, high-priority weight blocks are less likely to be replaced even if they have not been accessed for a period of time, thus ensuring that critical parameters remain cached. Through this refined management, the cache hit rate of this hierarchical data cache area can be stably maintained at over 95%.

[0049] Based on the global execution plan generated in step (3), the system creates a data prefetch schedule. This schedule is a precise timeline that details which data (model weights or inference inputs) need to be transferred from host memory to which target GPU's memory at what time during future execution flows. The core of the schedule is calculating the start time of each prefetch operation. This lead time is dynamically calculated and is the planned start time of the computation task served by the corresponding data in the global execution plan, minus an estimated transmission delay, plus a safety margin. The estimated transmission delay is the expected communication delay from the host to the target GPU, obtained based on system benchmarks or historical transmission data. The safety margin is set to 15% of the average communication delay to provide a buffer against network jitter or system load fluctuations that may occur during actual transmission, ensuring that data arrives reliably before the computation task needs it. All prefetch operations are submitted to a dedicated asynchronous data transfer queue and executed automatically in the background by the hardware DMA engine, thus operating in complete parallel with the computation tasks on the CPU and GPU, successfully hiding most of the data transfer delay.

[0050] In the above method, step (5) is multi-stream concurrent execution with low-overhead synchronization control. This step is executed on each independent graphics processor and is responsible for accurately translating the global execution plan into a sequence of underlying hardware instructions. It ensures the correctness of execution through a lightweight synchronization mechanism while maximizing the parallel capabilities of the hardware.

[0051] First, on each graphics processing unit (GPU) participating in the computation, the system creates multiple independent execution queues, or streams, via the driver API. These streams include at least one computation stream for scheduling kernel functions (Kernels) executed by the computation cores, and multiple data transfer streams for scheduling data copy tasks executed by the DMA engine. Separating computation and data transfer into different streams is the hardware foundation for enabling their parallel execution.

[0052] Based on the global execution plan generated in step (33), the scheduling execution module distributes the tasks in the plan. Operations within the computation subgraph, such as matrix multiplication and convolution kernels, are scheduled to be executed sequentially on the computation stream of the target GPU. Cross-processor data transfer tasks, whether from host to device, device to host, or point-to-point between devices, are scheduled to be executed on the corresponding data transfer stream.

[0053] To precisely coordinate execution dependencies between different streams and processors, the system employs a lightweight, event-based synchronization mechanism. When a task (such as data transfer) completes, an instruction is inserted at the end of its stream to record a completion event. This event is essentially a GPU timestamp or a flag. In another stream that depends on the result of this task (such as a computation stream), an instruction to wait for this event is inserted before starting the dependent computation kernel. Only when the GPU detects that the event has been triggered (i.e., the preceding task has completed) will the waiting instruction end, and subsequent computation kernels can begin execution. This event-based synchronization is point-to-point and fine-grained, establishing constraints only between tasks with direct dependencies. Compared to global barrier synchronization, which pauses the execution of all streams on the processor until all previously submitted tasks are completed, leading to unnecessary processor idle waiting, the lightweight synchronization mechanism in this method avoids this global pause, reducing synchronization overhead by at least an order of magnitude, thereby minimizing synchronization waiting overhead and improving the actual utilization of the GPU.

[0054] The method described in this embodiment also includes an online performance profiling and feedback optimization module, which serves as a closed-loop control system throughout the process. During the continuous operation of the inference service, this module continuously collects the actual computation time, communication time, memory usage, and concurrent execution status of each graphics processor in a low-overhead manner through the profiling interface built into the GPU driver. This real-time collected performance data is periodically fed back to the task mapping module in step (2) and the parallel scheduling module in step (3). For example, if online profiling reveals that the actual computation time on a certain GPU is significantly higher than the estimate from static analysis, the feedback system updates the weight of this node in the internal model. When processing subsequent inference requests, the task mapping and parallel scheduling modules re-evaluate their decisions based on this updated and more accurate performance data, generating a more optimized mapping and scheduling scheme. This closed-loop optimization system enables the execution plan to dynamically adapt to changing system loads (e.g., other applications preempting some GPU resources) and input data characteristics (e.g., changes in the distribution of input text length), ensuring that the inference system always operates at high efficiency.

[0055] To illustrate the technical solution of this embodiment more specifically, an application example in a specific scenario is given below.

[0056] The application scenario involves deploying a large autoregressive language model with 175 billion parameters for online inference services on a GPGPU server equipped with eight interconnected GPGPUs via a high-speed NVSwitch.

[0057] During step (1), the system loads the language model and parses it into a computational graph containing hundreds of Transformer decoder layers. The static analysis module estimates the FLOPs and memory requirements of weights and activations for each decoder layer. The hybrid granularity partitioning algorithm first performs coarse-grained partitioning, using a pipelined parallel strategy to divide these hundreds of layers into eight consecutive blocks, each forming a computational subgraph, initially planned to be allocated to one GPGPU. Subsequently, the algorithm enters the fine-grained partitioning stage, identifying large-scale matrix multiplication operations in the self-attention module and feedforward network module within each Transformer layer as computational bottlenecks. For these operations, the algorithm applies a tensor parallel strategy, partitioning the weight matrix and input activations along the model parallel dimension, splitting a large matrix multiplication into eight smaller matrix multiplications executed in parallel on different GPUs, and aggregating the results through the AllReduce operation. Ultimately, the partitioning decision ensures that the computational load balance of each GPGPU exceeds 95% while minimizing the amount of communication data between pipeline stages and within tensor parallelism.

[0058] During step (2), the resource monitoring module detects that one of the GPGPUs (e.g., GPU_3) has slightly less available video memory than other GPUs due to background system tasks. When making decisions, the task mapping strategy based on the weighted cost function uses a memory usage penalty term in the cost function to increase the cost of allocating a computational subgraph requiring larger activation values ​​to GPU_3. After dozens of iterations, the simulated annealing algorithm ultimately allocates a computational subgraph with relatively smaller activation values ​​to GPU_3, swapping its originally planned tasks with those of other GPUs, thus achieving global memory load balancing.

[0059] During step (3), the communication topology awareness module confirms that the eight GPGPUs form a fully connected topology via NVSwitch, with a point-to-point bandwidth of 300GB / s. Based on this information, the global execution plan generator develops a detailed execution schedule for pipeline parallelism and tensor parallelism. This plan divides the entire inference process into multiple micro-batch pipelines. For a micro-batch, when GPU_k is computing the k-th layer block, the scheduling plan arranges for GPU_k-1 to asynchronously transmit its computation results (activation values) to GPU_k via NVLink, while GPU_k+1 also begins prefetching the weights required for its subsequent layer blocks. This deeply overlapping scheduling ensures that the computation pipeline has almost no idle bubbles.

[0060] During step (4), since the 175 billion parameter model (approximately 350GB) far exceeds the total video memory of 8 GPGPUs (e.g., 8 * 80GB = 640GB, but space still needs to be reserved for activation values), most of the weights are stored in the 512GB page-locked memory cache allocated on the host side. Based on the global execution plan, the data prefetch scheduler precisely and in advance transmits the weight blocks required for subsequent pipeline stages such as GPU_k+1 and GPU_k+2 to the target GPU's video memory asynchronously via the PCIe bus, ensuring that the weight data is ready when computation is needed.

[0061] During step (5), on each GPGPU, tensor-parallel computation kernels are submitted to the computation stream, while the AllReduce communication operation for aggregating partial results is submitted to a separate stream using a dedicated communication library (such as NCCL). Before starting, the computation stream waits for an event triggered by the GPU of the previous pipeline stage, indicating that its required input activation values ​​have been transmitted via P2P, thus achieving precise cross-processor synchronization.

[0062] Through the synergistic effect of the above series of steps, the large model with 175 billion parameters can achieve high throughput and low latency inference services on a multi-GPGPU cluster, with an overall system hardware utilization rate of over 95% and end-to-end latency reduced by 50% compared to traditional parallel strategies.

[0063] Example 2

[0064] This embodiment provides an alternative implementation of an efficient inference method for large neural network models based on multiple GPGPUs. Unlike the approach in Embodiment 1, which generates a global execution plan based on static analysis and one-time optimization (such as simulated annealing), this embodiment employs an online adaptive scheduling strategy based on deep reinforcement learning in the dynamic mapping and parallel scheduling stages of computational tasks. This solution is particularly suitable for complex inference scenarios where system load and input request characteristics are highly dynamic and difficult to predict in advance.

[0065] In this embodiment, the basic principles and implementation mechanisms of the static analysis and mixed granularity partitioning of the model computation graph in step (1), the hierarchical data caching and asynchronous prefetching management in step (4), and the multi-stream concurrent execution and low-overhead synchronization control in step (5) are consistent with those of embodiment 1. The core difference is that steps (2) and (3) are replaced by a unified dynamic scheduler driven by a deep reinforcement learning agent (DRL Agent).

[0066] Specifically, this dynamic scheduler is constructed as an intelligent agent that makes continuous decisions. Its operating mode is as follows:

[0067] State Space Definition: At any decision time t, the reinforcement learning agent observes the state of the entire inference system. This state is a high-dimensional vector, composed of:

[0068] A. Dynamic Resource View: Same as in Example 1, it includes information such as the real-time load, available video memory, and interconnect bandwidth of each GPGPU.

[0069] B. Task queue status: A set of computational subgraphs to be scheduled, each subgraph having its own attributes, such as computational load, memory requirements, and whether its data dependencies' predecessor tasks have been completed.

[0070] C. Dependency graph: An adjacency matrix that represents the topological dependencies between tasks to be scheduled.

[0071] D. Communication Cost Matrix: Same as in Example 1, representing the communication overhead between GPGPUs.

[0072] Action Space Definition: At each decision point, the agent's action is to select a computation subgraph from the queue of "ready" computation subgraphs whose data dependencies are satisfied, and assign it a target GPGPU for execution. Therefore, the action space is a discrete combinatorial space with a size of (number of ready tasks × number of GPGPUs).

[0073] Reward Function Design: The design of the reward function is crucial for guiding the agent to learn efficient scheduling strategies. At the completion of each inference request, the agent receives a reward value based on the request's performance metrics. The reward function integrates multiple performance objectives:

[0074] A. Delay Penalty: The higher the actual end-to-end latency compared to the preset Service Level Target (SLO) latency, the greater the penalty.

[0075] B. Throughput Reward: The more inference requests completed per unit of time, the higher the reward.

[0076] C. Resource Utilization Bonus: The higher the average utilization of all GPGPUs during request processing, the higher the bonus.

[0077] D. Communication overhead penalty: The greater the total amount of cross-GPGPU data transfer caused by scheduling decisions, the higher the penalty.

[0078] By weighting and summing these objectives to form a scalar reward signal, the agent learns a scheduling strategy that balances these conflicting objectives.

[0079] Reinforcement Learning Algorithm: This embodiment employs advanced reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG) or Proximal Policy Optimization (PPO). The agent internally consists of two deep neural networks: an Actor network (policy network) and a Critic network (value network). The Actor network is responsible for outputting a specific scheduling action based on the current state (i.e., determining which task runs on which GPU). The Critic network is responsible for evaluating the long-term reward (Q-value) of performing a given action in the current state. During training, the agent continuously collects trajectory data of state, action, and reward by interacting with a high-fidelity system environment simulator or the online system itself. Using this data, both the Actor and Critic networks are simultaneously optimized using gradient descent. The Actor network aims to maximize the evaluation value given by the Critic network, while the Critic network aims to more accurately predict the actual long-term reward.

[0080] Compared to Example 1, the technical advantage of this example lies in its powerful adaptive capability. The simulated annealing algorithm in Example 1 performs a deep optimization of the model and hardware before deployment, generating a near-static optimal execution plan suitable for scenarios with stable loads. In contrast, the reinforcement learning agent in this example can dynamically adjust based on millisecond-level system state changes at runtime. For example, when an external application suddenly consumes some resources of a GPGPU, causing a decrease in its actual performance, the dynamic resource view in the state space immediately reflects this change. In the next decision, the agent's Actor network automatically reduces the probability of allocating new tasks to that GPGPU, dynamically migrating the computational load to other idle GPGPUs. The entire process requires no manual intervention or offline replanning. Similarly, when the characteristics of the input request change (e.g., from processing short text to processing long documents, causing a change in the load distribution of the computational subgraph), the agent can also quickly adapt to the new load pattern through online learning, continuously maintaining the system's high-performance operation. The cost of this approach is a pre-training phase and some decision-making overhead at runtime, but for high-standard inference services requiring extreme robustness and adaptability, the performance stability gains are significant.

[0081] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for efficient inference of large neural network models based on multiple graphics processors, characterized in that, Includes the following steps: The large neural network model is obtained and parsed as a directed acyclic computation graph. The directed acyclic computation graph is statically analyzed to extract the computational load, memory usage and data dependency attributes of each node. Based on the data dependency attributes, a hybrid granularity partitioning algorithm is used to divide the directed acyclic computation graph into multiple computational subgraphs. The system monitors the load status, available video memory, and interconnect bandwidth of each computing unit in a multi-GPU cluster in real time to construct a dynamic resource view. It then applies a task mapping strategy based on a weighted cost function, which combines the computation subgraph and its dependencies with the dynamic resource view to dynamically allocate the computation subgraph to the optimal target GPU. The physical communication topology of the cluster is identified to establish a communication cost matrix, and a global execution plan is generated based on the communication cost matrix and data dependencies. The global execution plan defines the start time, execution order, and cross-processor data transfer path and timing of each computation subgraph by constructing a deep pipeline to maximize the overlap between computation and communication. A hierarchical data cache is established on the host side, and a data prefetching schedule is created according to the global execution plan to asynchronously transfer the data required for subsequent calculations from the host memory to the target graphics processor's video memory. Independent compute and data transfer streams are created on each graphics processor. Compute operations and data transfer tasks are scheduled according to the global execution plan, and a lightweight event-based synchronization mechanism is used to coordinate execution dependencies across streams and processors.

2. The method according to claim 1, characterized in that, The hybrid granularity partitioning algorithm includes the following operations: First, the large neural network model is coarsely divided according to its inherent network hierarchical structure to form preliminary hierarchical partitions, so as to aggregate logically related operations into a preliminary set of computational subgraphs; Subsequently, for specific layers in the initially described set of computational subgraphs that are identified as computationally intensive or memory-access intensive, fine-grained partitioning is performed at the operator level. This fine-grained partitioning splits a single large-scale operator into multiple sub-operators that are executed in parallel on different graphics processors. The decision objective of the hybrid granularity partitioning algorithm is to minimize the total amount of data transfer across the boundaries of different computational subgraphs, while ensuring that the computational load balance among all the final partitioned computational subgraphs meets a preset balance threshold.

3. The method according to claim 2, characterized in that, When performing static analysis on the computation graph, it also includes a quantitative evaluation of the computational characteristics of different operators in the model. This quantitative evaluation is based on the ratio of the computational load of each operator to the total amount of its input and output data, i.e., the computational intensity. The operators are precisely classified into one of the following: computationally intensive, memory-bandwidth intensive, or communication-intensive. This type information is stored as an additional attribute in the corresponding computation graph node as one of the input weights when the hybrid granularity partitioning algorithm makes partitioning decisions, thereby achieving a fine-grained match between operator characteristics and hardware resources.

4. The method according to claim 1, characterized in that, The task mapping strategy based on a weighted cost function employs a heuristic optimization algorithm to solve the mapping scheme. The weighted cost function is defined as a weighted sum of expected computation time, expected communication time, and resource idle penalty. The expected computation time is determined by the total number of floating-point operations in the subgraph, the computational performance of the target graphics processor, and the current load. The expected communication time is calculated based on the amount of data to be transmitted and the interconnect bandwidth in the dynamic resource view. The resource idle penalty incentivizes the system to allocate tasks to processors with lower loads. The weight coefficients of each term are adaptively adjusted according to the different latency or throughput requirements of the inference task.

5. The method according to claim 1, characterized in that, Pipeline parallel scheduling is not only applied to macroscopic parallelism between different network layers of the large neural network model, but also to the allocation of different computational subgraphs as different stages of the pipeline to different graphics processors; The pipelined parallel scheduling is also applied to operator-level micro-parallelism within the network layer.

6. The method according to claim 5, characterized in that, The operator-level micro-parallelism within the layer includes: for operators with internal parallelism, automatically decomposing them into multiple data blocks, and distributing the computation of the multiple data blocks to multiple stream processor clusters of the same graphics processor in a micro-pipeline manner, that is, while one stream processor cluster is processing the current data block, another stream processor cluster starts processing the next data block, so as to improve the computational efficiency and throughput of a single graphics processor.

7. The method according to claim 1, characterized in that, Identifying the physical communication topology of a multi-GPU cluster includes: querying and constructing a point-to-point communication bandwidth matrix and a latency matrix between processors through the underlying hardware driver interface to form the communication cost matrix; when determining the specific path for cross-processor data transmission, the global execution plan prioritizes the direct link with the highest bandwidth and lowest latency in the communication cost matrix, and then selects the indirect path connected through a high-speed switching chip.

8. The method according to claim 1, characterized in that, Establishing a hierarchical data cache on the host CPU side includes: using page-locked memory to allocate cache in the host dynamic random access memory to support high-performance direct memory access between the cache and the graphics processor's video memory; the cache adopts a replacement strategy based on the least recently used rule, and sets higher cache priority for specific parameter blocks according to the reuse mode of the weight parameters in the model to ensure that key parameters are always cached.

9. The method according to claim 8, characterized in that, The data prefetching schedule table determines a start time for each prefetching operation. The start time is determined by dynamically calculating a time advance. The time advance is based on the time when the corresponding data calculation task is scheduled to start execution in the global execution plan, minus the expected communication delay from the host to the target graphics processor, and then adding a safety margin time. The safety margin time is used to cope with system jitter during actual transmission.

10. The method according to claim 1, characterized in that, The specific implementation of the event-based lightweight synchronization mechanism is as follows: insert an instruction to record a completion event at the end of the stream where a data production task is located, and insert an instruction to wait for the completion event to be triggered at the beginning of the stream where a consumption task that depends on the data is located; through this point-to-point dependency coordination method, global fence synchronization is avoided, thereby reducing unnecessary processor idle waiting.

Images