Automatic fusion method, device, system, medium and program product of vllm model

Abstract

The application provides an automatic fusion method, device, system, medium and program product of a vLLM model. On the basis of the vLLM model, the application eliminates PNM calculation cores that cannot be fused, and only focuses on the fusion of SIP calculation cores. Moreover, the overall scheduling power consumption can be automatically optimized according to an energy model.

Description

Technical Field

[0001] This application relates to the field of automatic fusion technology for large models, and in particular to automatic fusion methods, apparatus, systems, media, and program products for vLLM models. Background Technology

[0002] vLLM (Vectorized Large Language Model Inference / Serving System) is a technique for optimizing the inference efficiency of large language models. It significantly improves model performance and resource utilization through various innovative methods. Its core technologies include the following:

[0003] First, vLLM, based on Triton dynamic compilation technology, can merge multiple computational operations (such as matrix multiplication MatMul and layer normalization LayerNorm) into a single kernel. This automatic operator fusion reduces memory accesses, thereby lowering memory latency and improving computational efficiency. Triton dynamic compilation is based on the JIT (Just-In-Time) model, where Triton dynamically compiles code into executable instructions at runtime, offering high flexibility and allowing code optimization based on runtime input data and hardware configuration, such as dynamically adjusting kernel configuration based on the size of the input data. Second, vLLM also employs KV Cache optimization technology, significantly reducing synchronization overhead and further improving inference efficiency by storing key-value pairs contiguously and supporting batch read / write operations. Third, vLLM also has computation graph analysis capabilities, automatically identifying fusionable patterns (such as "MatMul→Bias→GELU") and generating customized code to further optimize performance. Finally, vLLM optimizes memory access through techniques such as register reuse and circular block partitioning, significantly improving cache hit rate and further enhancing overall performance.

[0004] Meanwhile, official tests show that compared to Hugging Face Transformers, vLLM improves throughput by 3 to 4 times while reducing memory usage by 20% to 30%. These optimizations demonstrate that vLLM has significant advantages in the field of large model inference.

[0005] However, despite the significant performance optimizations offered by vLLM, several challenges remain. First, adapting complex models is difficult, especially those employing non-standard architectures (e.g., custom activation functions) or multi-branch structures (e.g., residual connections), making efficient operator fusion challenging. Second, hardware heterogeneity is a significant issue. Different GPU architectures (such as NVIDIA and AMD) require customized optimization strategies, leading to poor technology portability and increased development and maintenance complexity. Furthermore, dynamic shapes are less efficient in variable-length sequence inference because they reduce the efficiency of fusion operations. Finally, existing technologies lack effective debugging tools. The fused code is difficult to debug, and performance analysis tools are insufficient, adding further difficulties to development and optimization. Therefore, existing vLLM models still face the technical challenge of not being able to achieve automatic fusion effectively. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this application is to provide an automatic fusion method, apparatus, system, medium and program product for vLLM models, so as to solve the technical problem that existing vLLM models cannot achieve automatic fusion well.

[0007] To achieve the above and other related objectives, a first aspect of this application provides an automatic fusion method for vLLM models, comprising: reading the computation graph of the vLLM model and loading it into the compiler memory; identifying and removing near-memory computation kernels in the computation graph of the vLLM model to prune the computation graph into pure SIP fusionable subgraphs; anchoring the SIP fusionable subgraphs and the data required by the subgraphs during execution in an L3 cache; determining the baseline number of computation blocks, compiling and generating more computation blocks than the baseline number based on a head / tail mechanism, and searching for the optimal head-tail fusion mode; successively reducing the non-reduce dimension of the intermediate tensors generated during the computation of the head-tail fusion mode; if the reduced intermediate tensors can be stored in an L1 cache and satisfy the capacity and continuity constraints, storing the dimensionality reduction candidate schemes and returning to the step of searching for the optimal head-tail fusion mode for iteration until the capacity and continuity constraints are no longer satisfied; using a pre-built energy model to perform energy calculation on each stored candidate scheme and selecting the fusion mode with the lowest energy.

[0008] In some embodiments of the first aspect of this application, identifying and removing near-memory computational kernels in the computation graph of the vLLM model includes: selecting nodes in the computation graph of the vLLM model that do not conform to the characteristics of near-memory computational kernels as candidates based on operator complexity, tensor shape, and dependencies; if a candidate near-memory computational single node is a predecessor of a certain SIP fusionable subgraph, then the near-memory computational single node can be merged into the SIP fusionable subgraph, otherwise it is removed.

[0009] In some embodiments of the first aspect of this application, the SIP fusion subgraph and the data required by the subgraph during execution are anchored in the L3 cache, which includes: determining the starting operator of the fusion operator generated after the SIP fusion subgraph is compiled as the anchor point and anchoring it in the L3 cache; anchoring the Key Cache and Value Cache to the same L3 cache; and anchoring a single operator that is frequently used on the PNM computing core to the same L3 cache.

[0010] In some embodiments of the first aspect of this application, determining the baseline number of computation blocks, compiling and generating more computation blocks than the baseline number based on the head / tail mechanism, and searching for the optimal head-tail fusion mode includes: determining the baseline number of computation blocks as N; the number of computation blocks actually generated by the compiler as (N+K); wherein, K computation blocks are split into head computation blocks and tail computation blocks.

[0011] In some embodiments of the first aspect of this application, the capacity and continuity constraints include: all tensors generated during the current round of non-reduce dimension reduction are still within the remaining available capacity of the L3 cache, wherein the intermediate tensors stored in the L1 cache do not exceed the upper limit of the L1 cache capacity; and the data boundaries of the head computation block and the tail computation block are intact.

[0012] In some embodiments of the first aspect of this application, the energy model is constructed as follows:

[0013] Energy = ∑(Data transfer volume * Data transfer power consumption) + Power consumption of additional computation + System idle power consumption; where the data transfer volume includes the total number of bytes transferred in a single execution of the entire fusion mode; the power consumption of additional computation includes the power consumption of additional computation and additional data transfer caused by alignment mismatch between the PNM computing core and the SIP computing core; and the system idle power consumption refers to the power consumed by the system when there are no effective computation or data transfer tasks.

[0014] To achieve the above and other related objectives, a second aspect of this application provides an automatic fusion system for vLLM models, comprising: a loading module for reading the computation graph of the vLLM model and loading it into compiler memory; a pruning module for identifying and removing near-memory computational kernels from the computation graph of the vLLM model to prune the computation graph into pure SIP-fused subgraphs; an anchoring module for anchoring the SIP-fused subgraphs and the data required by the subgraphs during execution in the L3 cache; and a block-level search module for determining the baseline number of computational blocks and compiling and generating actual multi-blocks based on a head / tail mechanism. The system uses a number of computational blocks to search for the optimal first-to-last fusion mode. A dimension reduction module is used to successively reduce the non-reduce dimension of the intermediate tensors generated during the computation of the first-to-last fusion mode. If the reduced intermediate tensors can be stored in an L1 cache and satisfy the capacity and continuity constraints, the dimensionality-reduced candidate schemes are stored, and the system returns to the step of searching for the optimal first-to-last fusion mode for iteration until the capacity and continuity constraints are no longer satisfied. An energy calculation module is used to perform energy calculations on each stored candidate scheme using a pre-built energy model and select the fusion mode with the lowest energy.

[0015] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic fusion method of the vLLM model.

[0016] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, enables the computer to implement the automatic fusion method of the vLLM model.

[0017] To achieve the above and other related objectives, a fifth aspect of this application provides a computer apparatus including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the automatic fusion method of the vLLM model.

[0018] As described above, the automatic fusion method, apparatus, system, medium, and program product of the vLLM model in this application have the following beneficial effects: Based on the vLLM model, this application eliminates PNM computing cores that cannot be fused, focusing only on the fusion of SIP computing cores. Furthermore, it can automatically optimize the overall scheduling power consumption according to the energy model. Attached Figure Description

[0019] Figure 1 The diagram shown is a schematic representation of the specific structure of the Transformer layer (LLaMA) in one embodiment of this application.

[0020] Figure 2 The diagram shown is a flowchart illustrating an automatic fusion method for vLLM models in one embodiment of this application.

[0021] Figure 3 The diagram shows the structure of the Transformer layer before and after PNM calculation kernel removal in one embodiment of this application.

[0022] Figure 4 The diagram shown is a structural schematic of an automatic fusion system for vLLM models in one embodiment of this application.

[0023] Figure 5 The diagram shown is a structural schematic of a computer device according to an embodiment of this application. Detailed Implementation

[0024] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0025] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0026] <1> PNM (Processing Near Memory) is a computing architecture designed to improve overall computing efficiency by physically bringing storage and computing units closer together, thereby reducing data transfer latency and power consumption.

[0027] <2> System-in-Package (SIP) is a packaging technology designed to integrate multiple chips with different functions (such as processors, memory, FPGAs, etc.) into a single package to form a complete system.

[0028] <3> MHA (Multi-Head Attention) uses multiple heads to compute attention weights in parallel, enabling the model to simultaneously focus on information from different positions and semantic subspaces within a sequence.

[0029] <4> MLP (Multi-Layer Perceptron) is a feedforward network consisting of two layers of linear transformation and nonlinear activation. It is used to perform feature upscaling and downscaling mapping at each location in the Transformer.

[0030] <5> K cache (Key Cache) caches the key vectors that have already been calculated during the inference phase to avoid redundant calculations and supports incremental generation of long sequences.

[0031] <6> V cache (Value Cache), corresponding to the key vector, caches the generated value vector, and is also used to accelerate autoregressive inference.

[0032] <7> Normalization scales activation values ​​to a stable distribution, mitigating gradient vanishing or exploding during deep network training.

[0033] <8> Softmax, which maps real vectors to probability distributions (summing to 1), is the core nonlinear function in attention weight calculation.

[0034] <9> Batched Matrix Matmul refers to performing multiple sets of matrix multiplications in parallel at once, making full use of the tensor core and improving sequence / batch-level throughput.

[0035] <10> Elementwise multiply refers to multiplying corresponding elements of two identical tensors.

[0036] <11> Elementwise Add refers to adding corresponding positions of two identical tensors to construct residual connections and shortcut paths.

[0037] <12> Transformer is an encoder-decoder architecture based on a self-attention mechanism.

[0038] <13> LLaMA (Large Language Model Meta AI) refers to a series of decoder-only Transformer large language models open sourced by Meta.

[0039] <14> L1 cache, the first-level cache, is the smallest SRAM next to the CPU core, with a latency of 1-3 clock cycles. It is used to store instructions and data, providing instant access for the arithmetic unit.

[0040] <15> L3 cache, a level 3 cache, is a larger SRAM shared by multiple cores, with a latency of 10-40 cycles. It absorbs inter-core communication traffic and reduces the average latency of accessing main memory.

[0041] <16> HBM (High Bandwidth Memory) is a type of 3D stacked DRAM that vertically interconnects multiple layers of memory dies through through silicon vias (TSVs), and is specifically designed for use as a high-bandwidth cache in AI / graphics chips.

[0042] <17> DRAM (Dynamic Random Access Memory) is a traditional dynamic random access memory that stores data by storing electrical charge in capacitors.

[0043] <18> The ALU (Arithmetic Logic Unit) is the core component of a processor responsible for performing all arithmetic and logical operations.

[0044] Figure 1 This demonstrates the structure of a Large Language Model (LLM), specifically the architecture of a vLLM. A vLLM is a deep learning model for natural language processing that processes and generates text through a multi-layered Transformer architecture.

[0045] Figure 1 The left side shows the workflow of vLLM based on the Transformer architecture, specifically:

[0046] Step S11: Input Tokens. This is the raw text data received by the model, typically segmented into a series of tokens. It should be understood that tokens can be words, characters, or smaller linguistic units, depending on the model's design.

[0047] Step S12: Embedding. This step transforms the input tokens into vector representations in a high-dimensional space. The embedding layer learns a vector representation for each token, which captures the semantic information of the token.

[0048] Step S13: Input the vector representations sequentially into the first, second, ..., Lth Transformer layers. The first Transformer layer (Transformer Layer 1) processes the embedded vector representations. This layer includes a self-attention mechanism and a feedforward neural network to capture dependencies between different positions in the sequence. The second Transformer layer (Transformer Layer 2) continues to process the output of the previous layer (Transformer Layer 1). Each Transformer layer further extracts and transforms features, enhancing the model's understanding of the input sequence. Until the final Transformer Layer L, after multiple layers of processing, the input data is transformed into higher-level representations containing rich semantic information.

[0049] Step S14: Output the prediction head after processing through multiple Transformer layers. This step transforms the output of the Transformer layers into the final prediction result, performing the final prediction based on the output of the Transformer layers. This typically involves converting the output of the Transformer layers into predicted labels.

[0050] Step S15: Predict the next token. In the text generation task, the model predicts the next token based on the previous tokens, thereby generating continuous text.

[0051] In summary, the input text is first segmented into tokens and converted into embedding vectors. These vectors are then fed into multiple Transformer layers, each processed by a self-attention mechanism and a feedforward neural network to extract and transform features. The output of the final Transformer layer is fed into a prediction head, which predicts the next token based on the preceding tokens. This process can be iterated to generate a continuous sequence of text.

[0052] Figure 1 The right side shows the specific structure of a Transformer layer (LLaMA), which is a core component of vLLM. Each Transformer layer contains the following key modules:

[0053] The normalization module, located at the beginning of the Transformer layer, normalizes the input first. This step is performed before the Multi-Head Attention (MHA) and Multi-Layer Perception modules, and its purpose is to stabilize the training process. By adjusting the distribution of the input data to have a uniform scale, the normalization module helps the model converge faster and improves training stability.

[0054] Multi-Head Attention (MHA) is the core mechanism of the Transformer layer, enabling the model to capture information in parallel across multiple different representation subspaces. The workflow of the MHA mechanism includes the following key steps:

[0055] First, the input hidden states are normalized using a normalization module. Second, the normalized hidden states are then weighted by three different weight matrices (W). q W k W v The input sequence is transformed into query (Q), key (K), and value (V) vectors, respectively. These vectors form the basis of the multi-head attention mechanism. The query vector is used to find information relevant to other parts of the input sequence, while the key and value vectors are used to determine the relevance weights and actual content of the information, respectively. q W k W v This is the weight matrix used to generate the query Q, key K, and value V. Furthermore, to improve computational efficiency, especially when processing long sequences, the model uses a key cache (KCache) and a value cache (VCache) to store and reuse the key (K) and value (V), thus avoiding the recalculation of the same key (K) and value (V) at each attention head. Next, the query (Q) and key (K) are multiplied by batch matrix multiplication to obtain an attention score matrix; this matrix is ​​normalized using the Softmax function and converted into attention weights. Softmax ensures that the sum of all weights is 1, allowing the model to learn the importance of each value (V) for the current query (Q). Then, the attention weights, the value (V) vector, and the result of the Softmax function are multiplied by batch matrix multiplication (O), and the result is multiplied by another weight matrix (W). oThe hidden state is subjected to a linear transformation to adjust the dimension of the output or to perform further feature extraction. This output is then residually connected to the hidden state of the input (elementwise add), and then passed through a normalization layer again to prepare it for the next processing stage (such as a multilayer perceptron).

[0056] The Multi-Layer Perceptron (MLP) module is another important component of the Transformer layer. It is used to further process and transform the output of the Multi-Head Attention (MHA) module. The specific operation steps are as follows:

[0057] First, the output of the Multi-Head Attention (MHA) module is normalized using a normalization layer to stabilize gradients during training, making the model training more stable. Second, the normalized output is controlled by a gate mechanism, which determines which information can pass through, which needs to be suppressed, and which information should be passed to the next layer. The output obtained through the gate mechanism undergoes non-linear processing using a sigmoid linear unit (SiLU) activation function to help the model learn more complex patterns. Simultaneously, the normalized output is further expanded using a feature expansion layer (Up). Up performs a linear transformation on the features using a weight matrix (Wo) to increase the feature dimensionality and capture richer information. Next, the result of the non-linear processing using the SiLU activation function and the result processed by the Up linear transformation layer are multiplied elementwise. The result is then input into a feature compression layer (Down) for feature compression. The feature compression layer (Down) also performs a linear transformation on the features using a weight matrix (Wo), but it reduces the dimensionality of the features to select the most important information. Finally, the output of the feature compression layer (Down) is element-wise added to the output of the original multi-head attention module (MHA). This residual connection helps alleviate the vanishing gradient problem in deep networks and promotes information flow. Ultimately, the result of the element-wise addition operation is the output of the multilayer perceptron (MLP) module, which is either passed to the next Transformer layer or used as the final output.

[0058] Therefore, in this way, the Multilayer Perceptron (MLP) plays a crucial role in the Transformer layers. It not only increases the model's non-linear expressive power but also helps the model learn and represent complex features more effectively through feature expansion and compression. This structure enables the Transformer model to handle various complex sequence data tasks, such as natural language processing and computer vision.

[0059] The automatic fusion method for vLLM models provided in this application, in Figure 1 Based on the provided Transformer layer structure, certain improvements have been made. The chip computing cores include PNM computing cores and SIP computing cores. Since PNM computing cores cannot be merged, the focus is on the fusion of SIP computing cores. The technical solution provided in this application can automatically optimize the overall scheduling power consumption based on the energy model. The specific process of the automatic fusion method of the vLLM model in this application is as follows: Figure 2 As shown, it includes the following steps.

[0060] Step S21: Read the computation graph of the vLLM model and load it into the compiler memory.

[0061] It's important to note that the computational graph of a vLLM model is also called a network, or a "network definition" or "IR graph." The computational graph consists of elements such as nodes, edges, and attributes. Nodes refer to each operator (OP), such as MatMul, Softmax, Add, ReLU, LayerNorm, etc. Edges refer to tensors or intermediate feature maps, indicating which operator the data flows from to which operator. Attributes refer to the properties of each node, such as operator type, input / output shape, data type, layout, and whether the shape is dynamic. Therefore, reading the computational graph of a vLLM model involves loading the entire directed acyclic graph (DAG) consisting of "operators-tensors-attributes" into the compiler's memory, which is then used to subsequently remove PNM computational kernels and retain SIP-fusionable subgraphs.

[0062] Step S22: Identify and remove nearby computational kernels in the computation graph of the vLLM model to prune the computation graph into a SIP-compatible subgraph.

[0063] Since the previous step loaded the entire computation graph, which contains both PNM and SIP computation kernels, this step performs a pruning operation. The aim is to remove all PNM computation kernels from the vLLM model's computation graph, leaving only a pure SIP fusionable subgraph.

[0064] It should be noted here that PNM (Processing Near Memory) computing cores refer to near-memory computing cores, which will be referred to as PNM computing cores below. They reduce latency and power consumption caused by data movement by placing computing logic (such as accelerators, lightweight cores, and reconfigurable logic) close to memory. PNM computing cores are typically implemented in the following ways: embedding processing units in the logic layer of 3D stacked memory, integrating computing logic in the memory controller or memory module, or packaging memory and computing dies together using advanced packaging technologies. Therefore, PNM computing cores have high bandwidth and low latency, but are limited by process technology and area. They typically perform simple, regular, and highly parallel operations and do not support operator fusion because the resulting composite operators are often too complex for PNM hardware resources to complete in one go. SIP (System-In-Package) computing cores refer to system-in-package computing cores, which are computing units that integrate multiple dies such as processors, caches, and accelerators into a single package using advanced packaging technologies. It features a complete ALU, register file, and cache system, supports complex operator fusion and multi-precision computation, can directly execute fusion kernels generated by the compiler, and works in collaboration with other computing cores through shared on-chip caches (such as L3), thereby achieving a balance between power consumption, area, and bandwidth, and can be applied to high-performance AI inference and graphics computing scenarios.

[0065] In the embodiments of this application, the specific implementation of identifying and eliminating near-memory computational kernels in the computation graph of the vLLM model includes: selecting nodes in the computation graph of the vLLM model that do not meet the characteristics of near-memory computational kernels as candidates based on operator complexity, tensor shape, and dependencies; if a candidate near-memory computational single node is a predecessor of a certain SIP fusionable subgraph, then the near-memory computational single node can be merged into the SIP fusionable subgraph, otherwise it is eliminated.

[0066] It's worth noting that if the predecessor of a SIP-fusionable subgraph is a PNM computation core, then the PNM computation core can be fused into that subgraph. In other words, when the upstream node of a SIP-fusionable subgraph happens to be a single operator executed by a PNM computation core, the compiler can directly incorporate that PNM operator into this SIP subgraph, forming a new, larger fusion kernel. Therefore, the PNM is no longer an independent node, but is absorbed into the fusion scope of SIP, thereby further reducing data movement and lowering power consumption.

[0067] Specifically, nodes that require reduction or nonlinear normalization are removed based on operator complexity; nodes with irregular tensor shapes are removed based on tensor shape; and nodes with fusion dependencies and data layout dependencies are removed based on dependencies. PNM computation kernels typically possess the characteristics of simplicity, regularity, and high parallelism. The computation of a PNM computation kernel is usually pure computation (no control flow, no reduction), and most operations are matrix multiplication or element-wise operations. The input and output tensor shapes are fixed, and memory is contiguous, making it suitable for high-bandwidth, low-latency scenarios requiring near-memory computation. Therefore, based on operator complexity, tensor shape, and dependencies, nodes in the computation graph of the vLLM model that do not conform to the characteristics of near-memory computation kernels are removed.

[0068] For example, nodes exhibiting reductions or non-linear normalizations such as Softmax, LayerNorm, and ReduceSum should be removed based on operator complexity. Nodes with irregular tensor shapes, such as those with dynamic batches, sequence lengths, padding, or non-2^n alignment, should be removed based on tensor shape. Nodes with dependencies should be removed if they have been merged into a larger kernel (e.g., Matmul+Bias+GELU merged into a fused op), or if there are transpose / reshape rearrangements upstream or downstream.

[0069] Specifically, in the embodiments of this application, nodes that are independent, static, non-fused, non-rearranged, or perform element-wise operations are removed from the computation graph. The final removed PNM computation kernels include: query (Q), key (K) and value (V) vectors, batched matrix multiplication, dot product (O), gate mechanism, feature expansion (Up), feature contraction (Down), and element-wise multiply. For ease of understanding, Figure 3 The diagram shows the structure of the Transformer layer before and after the removal of the PNM computation core. The left side of the diagram shows the structure before removal, and the right side shows the structure after removal. The comparison between the two clearly shows the PNM computation core that was removed.

[0070] Step S23: Anchor the SIP fusionable subgraph and the data required by the subgraph during execution in the L3 cache.

[0071] Specifically, since SIP fusionable subgraphs are often large in size and have high bandwidth requirements, and the Key Cache and Value Cache must reside in the L3 cache to be shared by all computing cores, it is also necessary to fix the single operators used frequently on the PNM computing core in the L3 cache to avoid cross-chip transfer.

[0072] Specifically, this step can be achieved through the following sub-steps:

[0073] First, the starting operator of the fusion operator generated after compiling the SIP fusion subgraph is determined as the anchor point and anchored to the L3 cache. It should be understood that the fusion operator is a large kernel formed by merging multiple smaller operators. Anchoring the starting operator of the fusion operator to the L3 cache allows the kernel to directly read data from the L3 cache upon startup, eliminating the need to move data from main memory (HBM / DRAM). This step involves determining the anchor point, which is the first operator scheduled in the entire inference chain around which all subsequent operators revolve. The compiler fixes it in the L3 cache because the L3 cache has sufficient capacity and bandwidth to allow the fusion kernel to directly obtain input upon startup, eliminating the latency of moving data from HBM / DRAM.

[0074] Secondly, the Key Cache and Value Cache are anchored to the same L3 cache. During the autoregressive phase, the Key Cache and Value Cache are read and written every time a token is generated, which is extremely frequent. To avoid moving the token across chips every time, the Key Cache and Value Cache are also anchored to the same L3 cache, sharing the same high-speed cache with the fusion operator. This way, updates to the Key Cache and Value Cache are performed locally within the L3 cache, optimizing both bandwidth and latency.

[0075] Finally, frequently used individual operators on the PNM compute core are anchored to the same L3 cache. For individual operators that ultimately reside on the PNM compute core, if their input size is small and their usage frequency is high, they will also be anchored in the L3 cache due to the high cost of cross-chip reads. Therefore, the L3 cache is a shared cache pool for the PNM compute core and the SIP compute core. SIP fusion operators are read from the L3 cache, and simple PNM operators are also read from the L3 cache, minimizing main memory bandwidth pressure throughout the inference chain.

[0076] Step S24: Determine the baseline number of computation blocks, compile and generate more computation blocks than the baseline number based on the head / tail mechanism, and search for the optimal head-tail fusion mode.

[0077] Specifically, the baseline number of computation blocks is determined to be N. The actual number of computation blocks generated by the compiler is (N+K), where the additional K computation blocks are split into head blocks and tail blocks, i.e., head block + tail block = K blocks. After dividing the entire computation chain into N standard computation blocks and K head / tail blocks, the compiler tries various connection methods. The connection method that merges multiple reads and writes into one, eliminates cross-block synchronization, and maximizes computational density is usually determined as the optimal head-tail fusion mode.

[0078] It should be understood that the head / tail mechanism refers to adding a head block and a tail block to a chain of computational blocks, specifically designed to capture and merge cross-boundary computational patterns, preventing the beginning and end from being cut off due to block splitting. For example, assuming the original model has N computational blocks, the compiler generates K more computational blocks, the first few called head blocks and the last few called tail blocks. The head block is responsible for including operators that can still be merged before block 0; the tail block is responsible for including operators that can still be merged after block (N-1). In this way, even if the entire model is divided into N blocks, the head / tail mechanism can merge the original cross-boundary small operators into a larger fusion kernel, ensuring that the computational chain remains unbroken at both the beginning and end. It should also be understood that the capacity of the L3 cache is a hard constraint; K cannot be infinitely increased, otherwise the working set will overflow the L3 cache. The head block and tail block must also fall within the L3 cache to ensure the integrity of the beginning and end logic.

[0079] Step S25: The intermediate tensor generated during the calculation of the first-to-last fusion mode is successively reduced in non-reduce dimension; if the reduced intermediate tensor can be stored in the L1 cache and satisfies the capacity and continuity constraints, the candidate scheme of dimensionality reduction is stored, and the process is returned to the step of searching for the optimal first-to-last fusion mode for iteration until the capacity and continuity constraints are no longer satisfied.

[0080] It's important to clarify here that the non-reduce dimension refers to the dimension that is summed / reduced in a single matrix multiplication, while the dimension that remains unchanged is the non-reduce dimension. For example, in C = A × B, where A has shape [batch, seq, head, d_k] and B has shape [batch, seq, head, d_k], the reduce dimension is d_k; batch, seq, and head are the non-reduce dimensions. Reducing the non-reduce dimension means reducing the size of batch, seq, or head, thus reducing the overall tensor volume. Furthermore, intermediate tensors refer to the working set generated during the computation of the fusion mode that needs to be immediately consumed by subsequent operators.

[0081] In embodiments of this application, the capacity and continuity constraints include:

[0082] (1) All tensors (inputs, intermediate activations, and outputs) generated during the current round of non-reduce dimension reduction are still within the remaining available capacity of the L3 cache, of which the intermediate tensors stored in the L1 cache do not exceed the capacity limit of the L1 cache.

[0083] (2) The data boundaries of the head and tail computation blocks are intact. The starting index of the head computation block and the ending index of the tail computation block are still seamlessly connected with the original main computation chain after the dimension is reduced. The head or tail computation blocks should not be truncated or have data holes due to the size reduction, thereby destroying the pipeline continuity across computation blocks.

[0084] In the embodiments of this application, the storage content for storing the dimensionality reduction candidate schemes includes the non-reduce dimension after dimensionality reduction and the corresponding fusion kernel description (operator chain, tile size, head / tail boundary, etc.). These candidate schemes are used for scoring after subsequent energy calculation. It should be understood that the fusion kernel description is a "blueprint" generated by the compiler for the hardware after merging several operators into a large kernel. It consists of three parts: first, the operator chain, which refers to the original operator sequence that is merged together, such as "Matmul→Bias→GELU"; second, the tile size, which is the specific size of the submatrix or subtensor processed by each computation block at one time, such as 128×64; and finally, the head / tail boundary, which indicates the start and end indices of the head and tail computation blocks that are additionally generated to maintain the continuity of cross-block data, ensuring that the entire fusion kernel does not have data truncation or repeated loading at the beginning and end.

[0085] In the embodiments of this application, each intermediate tensor that can be stored in the L1 cache after each round of scaling and meets the capacity and continuity constraints is stored as a feasible candidate solution, along with its dimension and the fusion mode after scaling. This process is iterated to obtain several such candidate solutions.

[0086] Step S26: Calculate the energy of each stored candidate scheme using the pre-built energy model and select the fusion mode with the lowest energy.

[0087] In the embodiments of this application, the energy model is constructed as follows:

[0088] Energy = ∑(Data transfer volume * Data transfer power consumption) + Additional calculated power consumption + System idle power consumption; Formula (1)

[0089] The data transport volume includes all bytes transported in a single execution of the fused mode; the power consumption of additional computation includes the power consumption of additional computation and data transport caused by alignment mismatch between the PNM and SIP computing cores; the system idle power consumption refers to the power consumed by the system when there are no effective computation or data transport tasks, typically the power consumed to maintain basic states such as clock, voltage, and registers. Generally speaking, the shorter the time, the less the system idle power consumption. That is to say, if the same task can be completed faster, the idle power consumption will decrease, and the overall energy consumption will also decrease.

[0090] The data transport volume includes all bytes transported in a single execution of the entire fusion mode; the power consumption of the additional computation includes the power consumption of the additional computation and transport volume caused by alignment mismatch between the PNM computing core and the SIP computing core. It should be understood that the additional workload caused by alignment mismatch can be divided into two parts:

[0091] First, PNM is redundant. The PNM computation kernel uses 128B wide vector units. If the length of the reduce dimension is not an integer multiple of 128B, zeros must be padded at the end or the redundant elements must be masked. These calculations consume ALU cycles and energy, which is redundant computation.

[0092] Secondly, there is SIP redundancy. The kernel of the SIP computing core is usually loaded with a minimum granularity of 64B. When the N or K dimension size is not a multiple of 64B, padding or masking is also required, resulting in the waste of additional memory bandwidth and computing cycles.

[0093] Therefore, the energy model constructed in this application embodiment will convert the additional computation and additional transportation caused by these two alignment mismatches into energy consumption and include them in the total energy, so as to give priority to the size alignment scheme during scheduling and avoid redundant overhead.

[0094] When calculating energy, the data transport volume refers to the total number of bytes transported in a single execution of the entire head-to-tail fusion mode, including the size of the input tensor read from the L3 cache and the size of the output tensor finally written back to the L3 cache. If some tensors fall into the L1 cache after compression, the actual transport volume between the L1 cache and the registers is added separately.

[0095] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" mean examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0096] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0097] The above section, with reference to specific embodiments, provides a detailed explanation of the automatic fusion method for vLLM models provided in this application. The following section will further describe the automatic fusion system, medium, and apparatus for vLLM models also provided in this application.

[0098] Figure 4 This document illustrates a schematic diagram of an automatic fusion system for vLLM models according to an embodiment of this application. The automatic fusion system for vLLM models in this application specifically includes: a loading module 41, a pruning module 42, an anchoring module 43, a block-level search module 44, a dimension reduction module 45, and an energy calculation module 46.

[0099] The loading module 41 reads the computation graph of the vLLM model and loads it into the compiler memory; the pruning module 42 identifies and removes near-memory computation kernels from the computation graph of the vLLM model to prune the computation graph into a pure SIP fusionable subgraph; the anchoring module 43 anchors the SIP fusionable subgraph and the data required by the subgraph during execution in the L3 cache; the block-level search module 44 determines the baseline number of computation blocks, compiles and generates more computation blocks than the baseline number based on the head / tail mechanism, and searches for the optimal head-tail fusion mode; the dimension reduction module 45 successively reduces the non-reduce dimension of the intermediate tensors generated during the computation of the head-tail fusion mode; if the reduced intermediate tensors can be stored in the L1 cache and meet the capacity and continuity constraints, the dimensionality reduction candidate scheme is stored, and the search for the optimal head-tail fusion mode is returned to iterate until the capacity and continuity constraints are no longer met; the energy calculation module 46 performs energy calculation on each stored candidate scheme using a pre-built energy model and selects the fusion mode with the lowest energy.

[0100] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0101] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0102] Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of this application. Figure 5 As shown, the computer device includes at least one processor 501, a memory 502, at least one network interface 503, and a user interface 505. The various components in the device are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general will label all buses as bus systems.

[0103] The user interface 505 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0104] It is understood that memory 502 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0105] In this embodiment of the invention, the memory 502 is used to store various types of data to support the operation of the electronic terminal 500. Examples of this data include: any executable program for operation on the electronic terminal 500, such as the operating system 5021 and application programs 5022; the operating system 5021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 5022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The implementation of the XX method provided in this embodiment of the invention may be included in the application program 5022.

[0106] The methods disclosed in the above embodiments of the present invention can be applied to processor 501, or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 501 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0107] In an exemplary embodiment, the electronic terminal 500 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0108] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figures 1 to 3 Automatic fusion method for vLLM models in any of the embodiments shown.

[0109] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to perform the above-described method.

[0110] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0111] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0112] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0116] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0117] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0119] In summary, this application provides an automatic fusion method, apparatus, system, medium, and program product for the vLLM model. Based on the vLLM model, this application eliminates PNM computing cores that cannot be fused, focusing only on the fusion of SIP computing cores. Furthermore, it can automatically optimize overall scheduling power consumption based on the energy model. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0120] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An automatic fusion method for vLLM models, characterized in that, include: Read the computation graph of the vLLM model and load it into the compiler's memory; In the computation graph of the vLLM model, nearby computation kernels are identified and removed to prune the computation graph into a pure SIP fusionable subgraph; The SIP fusionable subgraph and the data required by the subgraph during execution are anchored in the L3 cache; Determine the baseline number of computation blocks, compile and generate more computation blocks than the baseline number based on the head / tail mechanism, and search for the optimal head-tail fusion mode; The intermediate tensor generated during the computation of the first-to-last fusion mode is successively reduced in non-reduce dimension. If the reduced intermediate tensor can be stored in the L1 cache and satisfies the capacity and continuity constraints, the candidate scheme for dimensionality reduction is stored and the process is returned to the step of searching for the optimal first-to-last fusion mode for iteration until the capacity and continuity constraints are no longer satisfied. Energy calculations are performed on each stored candidate scheme using a pre-built energy model, and the fusion mode with the lowest energy is selected.

2. The automatic fusion method for vLLM models according to claim 1, characterized in that, Identifying and removing near-memory computational kernels in the computational graph of the vLLM model includes: Based on operator complexity, tensor shape, and dependencies, nodes in the computation graph of the vLLM model that do not meet the characteristics of the nearest-memory computation kernel are selected as candidates. If a candidate nearest-memory computation node is a predecessor of a certain SIP fusionable subgraph, then the nearest-memory computation node can be merged into the SIP fusionable subgraph; otherwise, it is eliminated.

3. The automatic fusion method for vLLM models according to claim 1, characterized in that, The SIP-compatible subgraph and the data required by the subgraph during execution are anchored in the L3 cache, which includes: The starting operator of the fusion operator generated after compiling the SIP fusion subgraph is determined as the anchor point and anchored in the L3 cache; Anchor the Key Cache and Value Cache to the same L3 cache; Anchor the single operator that is frequently used on the PNM computation core to the same L3 cache.

4. The automatic fusion method for vLLM models according to claim 1, characterized in that, The determination of the baseline number of computational blocks involves compiling and generating more computational blocks than the baseline number based on a head / tail mechanism, and searching for the optimal head-tail fusion mode, which includes: The base number of computation blocks is determined to be N; the actual number of computation blocks generated by the compiler is (N+K); among them, K computation blocks are split into header computation blocks and tail computation blocks.

5. The automatic fusion method for vLLM models according to claim 1, characterized in that, The capacity and continuity constraints include: all tensors generated during the current round of non-reduce dimension reduction are still within the remaining available capacity of the L3 cache, wherein the intermediate tensors stored in the L1 cache do not exceed the upper limit of the L1 cache capacity; and the data boundaries of the head and tail computation blocks are intact.

6. The automatic fusion method for vLLM models according to claim 1, characterized in that, The energy model is constructed as follows: Energy = ∑(Data transfer volume * Data transfer power consumption) + Power consumption of additional calculations + System idle power consumption; The data transport volume includes the total number of bytes transported in a single execution of the entire fusion mode; the power consumption of the additional computation includes the power consumption of the additional computation and transport volume caused by the alignment mismatch between the PNM computing core and the SIP computing core; the system idle power consumption refers to the power consumed by the system when there are no effective computation or data transport tasks.

7. An automatic fusion system for vLLM models, characterized in that, include: The loading module is used to read the computation graph of the vLLM model and load it into the compiler's memory; The pruning module is used to identify and remove nearby computational kernels in the computation graph of the vLLM model, so as to prune the computation graph into a pure SIP fusionable subgraph; An anchoring module is used to anchor the SIP fusionable subgraph and the data required by the subgraph during execution in the L3 cache; A block-level search module is used to determine the baseline number of computation blocks, compile and generate more computation blocks than the baseline number based on the head / tail mechanism, and search for the optimal head-tail fusion mode. The dimension reduction module is used to successively reduce the non-reduce dimension of the intermediate tensor generated during the calculation of the first-and-last fusion mode. If the reduced intermediate tensor can be stored in the L1 cache and satisfies the capacity and continuity constraints, the candidate solution for dimension reduction is stored and the process is returned to the step of searching for the optimal first-and-last fusion mode for iteration until the capacity and continuity constraints are no longer satisfied. The energy calculation module is used to perform energy calculations on each stored candidate scheme using a pre-built energy model and select the fusion mode with the lowest energy.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the automatic fusion method of the vLLM model according to any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, enables the computer to implement the automatic fusion method of the vLLM model as described in any one of claims 1 to 6.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the automatic fusion method for the vLLM model according to any one of claims 1 to 6.

Images