Large model reasoning method with efficient storage for many-core supercomputing

Abstract

The invention discloses a storage-efficient large model reasoning method for many-core supercomputing, and relates to the field of machine learning. The overall architecture of the method is based on video memory optimization of Attention and FFN modules in a Transform model, and high storage efficiency in a reasoning stage is realized through matrix block calculation and dynamic video memory management. The method comprises the following steps: firstly, carrying out parameter partitioning and serial calculation: in an Attention module, vertically cutting a Q and K parameter matrix into a plurality of sub-blocks along a column direction, and keeping the input complete; serially calculating the product of each sub-block and the input to obtain a local Q matrix and a local K matrix, and immediately performing QKT block multiplication to obtain a partial attention score; aggregating calculation results of all the sub-blocks to obtain a complete attention score matrix; afterwards, normalizing the attention score by using Softmax, carrying out serial multiplication on the normalized attention score and a V block subjected to delay calculation, and splicing a result to obtain Attention output; optimizing an FFN module, vertically cutting parameters of a full connection layer into sub-blocks, inputting complete data, sequentially carrying out serial calculation on the complete data and the sub-blocks, and splicing nonlinear transformation results of all the sub-blocks in real time.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and particularly to a storage-efficient large model inference method for many-core supercomputing. Background Art

[0002] In the context of the rapid development of artificial intelligence technology and its deep integration with industries, large language models with hundreds of billions of parameters based on the Transformer architecture (such as GPT-4, PaLM, etc.) have become the core engines driving fields such as natural language processing, intelligent assistants, and scientific computing. However, with the exponential growth of model scale (the total number of parameters breaking through from billions to trillions) and the extension of application scenarios to fields such as edge computing and real-time interaction, traditional inference methods face a fundamental contradiction between the demand for video memory resources and the hardware's carrying capacity. Currently, how to reduce video memory occupancy has become a new topic. For example, Dao et al. proposed using the FlashAttention technology to optimize the memory access and calculation process of the attention mechanism. Specifically, however, the existing technologies have the following bottlenecks:

[0003] Square-law contradiction between video memory occupancy and model scale:

[0004] During the Transformer inference process, the attention mechanism needs to generate complete Q (query), K (key), and V (value) matrices (with dimensions of N×d_model, where N is the sequence length and d_model is the model dimension), and jointly occupy the video memory with the intermediate activation values of the FFN layer. Generally speaking, the larger the model scale, the larger the encoding dimension d_model of the word vectors, and the longer the maximum input sequence length N supported, which increases the video memory pressure during the inference process and results in a square-law growth in the overall video memory demand. This characteristic forms a fundamental contradiction with the exponential expansion of the large model's parameter scale.

[0005] Video memory redundancy dilemma in parallel computing

[0006] The parameter matrix can be horizontally sliced for multi-GPU parallel computing. Although the computing efficiency is improved, due to the need to synchronously store complete intermediate results (such as block copies of Q, K, and V) among devices, the video memory utilization rate is less than 30%, and the communication overhead grows non-linearly with the increase in the number of blocks.

[0007] Lightweight deployment requirements in edge scenarios

[0008] In low-video-memory environments such as industrial Internet of Things and mobile terminals, the model needs to perform real-time inference under strict resource constraints. However, due to the excessively high video memory peak of the existing methods, developers are forced to adopt model compression technologies with serious accuracy loss, restricting the large-scale implementation of AI technology.

[0009] Analysis of the disadvantages of existing technologies:

[0010] Explosive growth of video memory in traditional Transformer architectures

[0011] During the standard Transformer inference process, the attention mechanism needs to generate complete Q, K, V matrices (dimension N×d_model) and the attention score matrix (dimension N×N), which together with a large number of intermediate activation values generated by the FFN module occupy the video memory. The video memory occupancy grows according to the square law with the input sequence length N and the word vector encoding dimension d_model.

[0012] Global storage dependence of intermediate results

[0013] Insufficient storage optimization in many-core supercomputing environments

[0014] Existing many-core optimization solutions have significant defects:

[0015] Blocked parameter caching technology: Although it reduces the parameter loading frequency through inter-core asynchronous pipelines, it does not solve the video memory redundancy problem of intermediate calculation results (such as QK T matrices, FFN activation values).

[0016] Mixed-precision quantization strategy: FP16 / INT8 quantization causes attention distribution shift and a decrease in inference accuracy in long sequence scenarios.

[0017] Static topology mapping method: Parameter distribution based on a priori communication heat maps is difficult to adapt to dynamic sparse activation patterns. Summary of the Invention

[0018] The object of the present invention is to address the above deficiencies and propose a storage-efficient large model inference method for many-core supercomputing. When constructing a storage-efficient inference framework for many-core supercomputing, it breaks through the technical bottlenecks of traditional methods in video memory occupancy, intermediate result storage, and dynamic adaptability, and realizes lossless inference of large models with hundreds of billions of parameters in video memory-constrained environments.

[0019] The present invention specifically adopts the following technical solutions:

[0020] A storage-efficient large model inference method for many-core supercomputing, including block optimization of the Attention module and block optimization of the FFN module, specifically including the following steps. First, perform parameter blocking and serial calculation:

[0021] In the Attention module, vertically cut the Q and K parameter matrices into multiple sub-blocks along the column direction, and keep the input intact; serially calculate the product of each sub-block and the input to obtain local Q and K matrices, and immediately perform QK TChunk multiplication yields partial attention scores; aggregating the calculation results of all sub-chunks gives the complete attention score matrix; subsequently, the attention scores are normalized using Softmax and serially multiplied with the V chunks calculated with a delay, and the concatenated results yield the Attention output;

[0022] Optimization of the FFN module: The parameters of the fully connected layer are vertically sliced into sub-chunks, and the complete data is input and serially calculated with the sub-chunks in turn. The non-linear transformation results of each sub-chunk are concatenated in real time to reduce the intermediate video memory occupancy; specifically, through dynamic video memory release, that is, once the calculation of a sub-chunk is completed, the video memory of its parameters and intermediate results is immediately released, ensuring that the peak video memory is only determined by the calculation requirements of a single sub-chunk; it also has multi-core adaptability. Through pipeline scheduling, the chunk tasks are mapped to the multi-core processor, and subsequent parameters are pre-loaded during serial calculation to balance video memory optimization and calculation efficiency.

[0023] Preferably,

[0024] In parameter chunking and serial calculation, it includes the balance of video memory and calculation in the chunking strategy. The specific process is as follows:

[0025] Calculation of peak video memory:

[0026] The peak video memory of a single layer is determined by the maximum sub-chunk calculation requirement;

[0027] Attention module: The serial calculation of Q and K in chunks and the delayed chunk calculation of V further compress the peak video memory from (3×N×d_model + N 2 ) to (2×N×d_model / M + N 2 )

[0028] FFN module: The collaborative chunk calculation of the fully connected layer and the non-linear layer further compresses the peak video memory from (2×N×d_model×4) to (2×N×d_model×4 / M).

[0029] Time delay control:

[0030] Pipeline scheduling: Utilizing the GPU multi-core architecture, pre-load the K1 parameters when calculating Q1 to reduce the data transfer overhead;

[0031] Calculation-release-reuse strategy: Through timely video memory release and parameter prefetching, partially offset the time delay caused by serial execution,

[0032] Chunk granularity trade-off:

[0033] Balance between video memory and calculation efficiency: The smaller the chunking granularity, the lower the video memory occupancy, but the serial calculation time increases; the chunking granularity needs to be dynamically adjusted according to the hardware video memory capacity.

[0034] Preferably,

[0035] The video memory status monitoring layer has also been reported.

[0036] Deploy a video memory occupancy sampler on each GPU node, and report real-time video memory usage data to the central scheduler at a granularity of 10 ms, mark the nodes with high video memory occupancy as "computation-intensive ready" status, and ensure that they can obtain subsequent chunk tasks preferentially.

[0037] Pipeline scheduling engine

[0038] Multi-stage pipeline design: Decompose the computing task into four stages of "parameter prefetching → computing → video memory release → result aggregation", and reduce the end-to-end latency to 1.5 times that of a single stage through a parallel pipeline.

[0039] Dependency-aware scheduling: Use a directed acyclic graph (DAG) to model the data dependency relationship between sub-blocks, and ensure that dependent subtasks are preferentially executed when the pipeline progresses.

[0040] Fault tolerance and elastic expansion

[0041] Video memory overflow fallback: If the video memory of a certain node is insufficient, immediately migrate the current sub-block calculation to a low-load node, and trigger the checkpoint mechanism to restore the calculation state.

[0042] Dynamic resource pool: Support adding new GPU nodes during runtime, and the scheduler automatically incorporates them into the task allocation pool and rebalances the chunking strategy.

[0043] The present invention has the following beneficial effects:

[0044] (1) Vertical parameter segmentation and dynamic video memory management

[0045] Different from the traditional horizontal segmentation, the present invention cuts the parameter matrices of the Q, K, V, and FFN layers along the column dimension (vertical cut) into multiple sub-blocks, calculates the results of a single sub-block serially in order, and immediately releases its video memory after the calculation is completed. For example, for an input with a dimension of N×d_model, the parameter matrix with a dimension of d_model×d_model is cut into M sub-blocks, and the dimension of each sub-block is d_model×(d_model / M). At any time, only the intermediate result with a dimension of N×(d_model / M) is stored in the video memory. In this way, the storage pressure of the video memory on each intermediate result is reduced to 1 / M (M is the number of chunks) of the original.

[0046] (2) Dynamic release mechanism for intermediate results

[0047] The video memory dynamic release strategy through block matrix multiplication avoids storing the complete intermediate matrices (such as Q, K, V values and FFN activation values). Optimization of Q, K, V value calculation: The Q, K, V parameter matrices are sliced by column, and the local Q, K, V values are calculated block by block. The corresponding partial attention scores are calculated using the local Q and K. Immediately after the calculation is completed, the video memory occupancy of the local Q and K values is released, thus avoiding the storage of the complete Q, K, V values. Optimization of FFN activation values: The sub-block results are directly passed after the non-linear transformation (such as GeLU), avoiding the storage of the complete intermediate matrix.

[0048] (3) Optimization for many-core supercomputing adaptation

[0049] Combined with the parallel pipeline characteristics of many-core supercomputing, a video memory-computation load balancing strategy is designed: pipeline scheduling, preloading the K1 parameter synchronously when calculating Q1, reducing the data transfer overhead and offsetting the time delay of serial execution. Dynamic task allocation, according to the video memory occupancy status of the computing nodes, preferentially allocate the block tasks (such as Q_i, W_i) of the high-load nodes to maximize the utilization rate of hardware resources.

[0050] The inference method described in this application can reduce the single-node video memory requirement of the 100-billion parameter model to 1 / M (M is the number of blocks) of the traditional method without sacrificing the computing accuracy. At the same time, it avoids the communication overhead problem in the long sequence and large-scale block scenarios of the traditional method, improving the overall inference efficiency. Description of the Drawings

[0051] Figure 1 It is a schematic diagram of the block optimization of the Attention module;

[0052] Figure 2 It is a schematic diagram of the block optimization of the FFN module. Detailed Embodiments

[0053] The following further describes the detailed embodiments of the present invention in conjunction with the drawings and specific embodiments:

[0054] As the core foundation of the current large language model, the inference process of the Transformer architecture is mainly composed of two major modules: the multi-head attention mechanism (Multi-Head Attention) and the feed-forward neural network (FFN). The following details its calculation logic and the source of video memory requirements:

[0055] Multi-Head Attention

[0056] Input and parameter matrix projection:

[0057] The input sequence (with dimension N×d_model, where N is the sequence length and d_model is the model dimension) generates query (Q), key (K), and value (V) matrices through three independent linear transformation weight matrices W_Q, W_K, and W_V (each with dimension d_model×d_model). [1-2] :

[0058] Q = X × W Q , K = X × W K , V = X × W V

[0059] where the dimensions of Q, K, and V are all N×d_model.

[0060] Attention score calculation:

[0061] The attention scores are calculated through the scaled dot-product attention formula:

[0062]

[0063] First, calculate the product of Q and the transpose of K (QK^T) to obtain a matrix with dimension N×N, representing the similarity between the query and the key. Then divide by √d_k (where d_k is the dimension of the key vector, usually d_k = d_model / h and h is the number of attention heads) to stabilize the gradient, and normalize it to a probability distribution through softmax. Finally, multiply the attention weights by the value matrix V to generate the final output, with dimension N×d_model.

[0064] Multi-head concatenation and output projection:

[0065] Split Q, K, and V into h sub-matrices according to the number of attention heads h (each sub-matrix has dimension N×(d_model / h)), calculate the attention output independently for each head, and after concatenation, restore the dimension through a linear projection layer W_O (with dimension d_model×d_model). In terms of VRAM requirements, the projection matrices of Q, K, and V (3×d_model 2 ) and the intermediate QK^T matrix (N 2 ) are the main sources of VRAM peak.

[0066] Feed-forward neural network (FFN)

[0067] Dimensionality increase transformation:

[0068] The input sequence (with dimension N×d_model) is increased in dimension through the first fully connected layer W_1 (with dimension d_model×4d_model) to obtain intermediate activation values with dimension N×4d_model, where GeLU is used as the activation function.

[0069] H hidden= GeLU(X × W1)

[0070] H hidden The dimension is N × 4d_model, and GeLU is the Gaussian Error Linear Unit activation function.

[0071] Dimensionality reduction and restoration:

[0072] The intermediate activation values are reduced to the original dimension through the second fully connected layer W_2 (with a dimension of 4d_model × d_model), and the final output dimension is N × d_model, which is consistent with the input dimension.

[0073] H out = H hidden × W2

[0074] In terms of VRAM requirements, the weights of the dimensionality increase layer W_1 (4d_model 2 ) and the intermediate activation values (4Nd_model) are the key factors for VRAM occupancy.

[0075] Technical contradictions in VRAM occupancy

[0076] The first aspect is the storage of parameter matrices. The Q / K / V projections (3d_model 2 ) and the FFN weights (4d_model 2 + d_model 2 ) require high-dimensional VRAM. The second aspect is the explosion of intermediate results. The QK^T matrix (N 2 ) and the FFN intermediate activation values (4Nd_model

[0077] ) grow linearly or quadratically with the sequence length N.

[0078] For example, when d_model = 512 and N = 1024, the VRAM occupied by a single-layer QK^T matrix is 1024 × 1024 × 4B ≈ 4MB; if d_model = 4096 and N = 2048, the VRAM requirement for QK^T is 2048 × 2048 × 4B ≈ 16MB, and the FFN intermediate activation values reach 2048 × 16384 × 4B ≈ 128MB.

[0079] It is worth noting that the LoRA technology proposed by Hu et al. significantly reduces the number of parameters in the Transformer model through low-rank matrix factorization, thereby reducing the storage and computational requirements, providing a new idea for solving the VRAM bottleneck of large-scale models.

[0080] Storage optimization for large model inference in a many-core supercomputing environment

[0081] In a many-core supercomputing environment, three mainstream technical routes for storage optimization in large model inference have been formed:

[0082] Chunk Parameter Caching and Dynamic Loading

[0083] The SW-TF framework based on the ShenWei many-core architecture proposes a parameter chunk prefetching technique, which disassembles the Transformer parameter matrix into sub-chunks according to the calculation stage, and realizes the parallelization of calculation and parameter loading through inter-core asynchronous pipelines. For example, in the multi-head attention calculation, the Q / K / V projection matrices are stored in the LDM by slicing according to the attention head and layer dimensions, and the block loading order is predicted through the calculation dependency relationship, reducing the global memory access frequency. However, this method does not solve the problem of video memory explosion of intermediate results (such as the QKT matrix), and the inter-core load balancing depends on manual tuning.

[0084] Mixed Precision and Quantization Compression

[0085] The hierarchical mixed precision scheme for ShenWei TaihuLight adopts the FP16 / INT8 mixed precision strategy and implements residual quantization compensation (RQC) for activation values and weight gradients in the forward propagation. Experiments show that this method can reduce the storage occupancy of a model with hundreds of billions of parameters by 48%, but introduces two key problems: The first problem is that the accumulation of quantization errors leads to the deviation of the attention distribution, and the inference accuracy drops by 1.2% in the scenario of N = 4096 long sequences. This phenomenon has been widely discussed in mixed precision training. As mentioned by Micikevicius et al., although mixed precision training significantly reduces the storage requirements, the accumulation of quantization errors may affect the accuracy of the model. The second problem is that the broadcast of low-precision parameters requires the reconstruction of the many-core communication protocol, increasing the instruction overhead by 15%. When conducting large-scale neural network training on ShenWei TaihuLight, the broadcast of low-precision parameters requires the reconstruction of the many-core communication protocol to adapt to the mixed precision strategy, but this process will bring additional instruction overhead.

[0086] In addition, although quantization techniques can significantly improve the inference efficiency of the model, a trade-off needs to be made between quantization accuracy and model performance. The tensor decomposition acceleration technology (swTensor) under the ShenWei architecture needs to consider the impact of quantization errors on model accuracy while achieving efficient computing. It is worth noting that Fedus et al. proposed a Switch Transformers architecture based on sparse activation, which expands the model scale without significantly increasing computing resources through a dynamic sparsity mechanism, providing a new idea for the efficient inference of large-scale models.

[0087] Topology-Aware Parameter Distribution

[0088] Topology-aware parameter distribution technology significantly improves communication efficiency by optimizing the distribution of parameters in the supercomputer network. For example, Kim et al. proposed a highly scalable Dragonfly topology in [reference], which can effectively reduce communication latency and bandwidth consumption by optimizing network connections and data transmission paths. This topology design provides an efficient communication foundation for large-scale parallel computing, but its static partitioning mode is difficult to adapt to dynamic sparse activation scenarios.

[0089] In addition, Huang et al. proposed the GPipe framework in [reference], which optimizes the training process of large-scale neural networks through pipeline parallel technology. However, GPipe mainly focuses on the pipeline partitioning of computing tasks and does not fully consider the impact of the memory hierarchy characteristics on storage efficiency, which may become a performance bottleneck in dynamic sparse activation scenarios.

[0090] The mathematical principle relied on by this application is based on the mathematical properties of block matrix multiplication. For matrix multiplication C = A × B, if matrix B is vertically sliced into N sub-blocks [B1, B2,..., B_N], then C can be decomposed into

[0091]

[0092] That is, after each sub-block is independently calculated, the final result is obtained by accumulation. This property allows the present invention to decompose large-scale matrix multiplication into multiple small-scale computing tasks, and the video memory only needs to store the current sub-block and some intermediate results. For example, in the QKV generation stage, the parameter matrix W_QKV (d_model × 3d_model) is sliced into N vertical sub-blocks (d_model × 3d_model / N), and the input sequence X (N × d_model) is multiplied by each sub-block in turn to generate local Q, K, and V matrices. After each sub-block calculation is completed, its video memory is immediately released, and only the concatenated complete Q, K, and V are retained.

[0093] The storage-efficient large model inference method for many-core supercomputers proposed in this application includes block optimization of the Attention module and block optimization of the FFN module, and specifically includes the following steps. First, parameter block division and serial calculation are performed:

[0094] In the Attention module, the Q and K parameter matrices are vertically sliced into multiple sub-blocks along the column direction, and the input remains complete; the product of each sub-block and the input is serially calculated to obtain local Q and K matrices, and immediately perform QK T Block multiplication to obtain partial attention scores; aggregate the calculation results of all sub-blocks to obtain a complete attention score matrix; then use Softmax to normalize the attention scores and serially multiply them with the V blocks calculated later, and splice the results to obtain the Attention output;

[0095] The FFN module is optimized by vertically slicing the fully connected layer parameters into sub-blocks. The complete input data is then serially calculated with the sub-blocks. The nonlinear transformation results of each sub-block are spliced in real time, reducing intermediate video memory usage. Specifically, this is achieved through dynamic video memory release. After each sub-block calculation is completed, the video memory for its parameters and intermediate results is immediately released, ensuring that the peak video memory is determined only by the computational requirements of a single sub-block. The module also has multi-core adaptability, mapping block tasks to multi-core processors through pipeline scheduling, preloading subsequent parameters in serial calculations, and balancing video memory optimization with computational efficiency.

[0096] Parameter block and serial calculation include the memory and computing balance of the block strategy. The specific process is as follows:

[0097] Calculation of peak memory usage:

[0098] The peak value of a single-layer video memory is determined by the computing requirements of the largest sub-block;

[0099] Attention module: The serial block calculation of Q and K and the delayed block calculation of V further reduce the peak value of video memory from (3×N×d_model+N 2 ) is compressed to (2×N×d_model / M+N 2 )

[0100] FFN module: The collaborative block calculation of the fully connected layer and the nonlinear layer further compresses the peak memory from (2×N×d_model×4) to (2×N×d_model×4 / M).

[0101] Time delay control:

[0102] Pipeline scheduling: Utilizing the GPU multi-core architecture, K1 parameters are preloaded when calculating Q1, reducing data handling overhead.

[0103] Compute-release-reuse strategy: timely release of video memory and parameter prefetching can partially offset the time delay caused by serial execution.

[0104] Block granularity tradeoffs:

[0105] Balance between video memory and computing efficiency: The smaller the segmentation granularity, the lower the video memory usage, but the serial computing time increases; the segmentation granularity needs to be dynamically adjusted according to the hardware video memory capacity.

[0106] Also reported to the memory status monitoring layer

[0107] A memory occupancy sampler is deployed on each GPU node to report real-time memory usage data to the central scheduler at a 10ms granularity. Nodes with high memory usage are marked as "computationally intensive ready" to ensure they receive priority in subsequent block tasks.

[0108] Pipeline scheduling engine

[0109] Multi-stage pipeline design: Decompose the computing task into four stages: "parameter prefetching → computing → video memory release → result aggregation", and reduce the end-to-end latency to 1.5 times that of a single stage through a parallel pipeline;

[0110] Dependency-aware scheduling: Use a directed acyclic graph (DAG) to model the data dependencies between sub-blocks, ensuring that dependent subtasks are executed first when the pipeline advances;

[0111] Fault tolerance and elastic expansion

[0112] Video memory overflow fallback: If a node runs out of video memory, immediately migrate the current sub-block computation to a low-load node and trigger a checkpoint mechanism to restore the computation state;

[0113] Dynamic resource pool: Support adding new GPU nodes at runtime. The scheduler automatically incorporates them into the task allocation pool and rebalances the chunking strategy.

[0114] The following takes the number of split sub-blocks = 2 as an example to illustrate the calculation process of this method

[0115] Chunk optimization of the Attention module, such as Figure 1 shown

[0116] (1) Vertical splitting of the Q / K / V parameter matrices

[0117] 1. Parameter splitting: Evenly divide the parameter matrices of Q and K (with dimensions of 512×512) into two sub-matrices (Q1, Q2 and K1, K2) along the column direction (vertical splitting) (n is greater than or equal to 2), and the size of each sub-matrix is 512×256.

[0118] 2. Input processing: Keep the input matrix X (with dimensions of N×512) intact to avoid additional splitting overhead.

[0119] Serial calculation: First calculate X×Q1 to obtain the sub-result Q_part1 (N×256), and the video memory occupancy is N×256. After releasing the video memory occupied by Q_part1, calculate X×Q2 to obtain Q_part2 (N×256). Calculation strategy for Q_part1K: The parameter splitting of K is the same as that of Q, but its transposed form (K1 T , K2 T ) needs to be retained for subsequent chunked multiplication.

[0120] 3. Delayed calculation of V: After the calculations of Q and K are completed, do not calculate V immediately. Instead, perform the chunked calculation of V after the Softmax stage to further reduce the peak video memory.

[0121] (2) Attention score calculation

[0122] 1. Multiply the input by the Q and K parameter sub - matrices to obtain the Q and K sub - blocks

[0123] Q1 and K1, and immediately multiply Q1 and K1 T to obtain the sub - attention matrix block

[0124] 2. Release the video memory space occupied by Q1 and K1

[0125] 3. Calculate the Q and K sub - blocks Q2 and K2, multiply Q2 by K2T to obtain another attention matrix block, and accumulate the two sub - blocks to obtain the complete attention score

[0126] 4. Release the video memory space occupied by Q2 and K2

[0127] 5. Normalize the attention score using Softmax

[0128] (3) Block - based calculation and result concatenation of V

[0129] 1. Delayed split of V: The parameter matrix of V adopts the same vertical split strategy as Q / K (V1, V2, size 512×256), but the calculation is delayed until after Softmax is completed.

[0130] 2. Block - based calculation and concatenation: Serially multiply the attention score matrix after Softmax with the sub - blocks of V (V1, V2): Calculate Softmax_block×V1 to obtain the sub - output Attn_part1 (N×256). After releasing the video memory of V1, calculate Softmax_block×V2 to obtain Attn_part2 (N×256). Combine Attn_part1 and Attn_part2 to generate the complete Attention output (N×512).

[0131] Block optimization of the FFN module, as Figure 2 shown

[0132] (1) Vertical split strategy of the parameter matrix

[0133] 1. Parameter split: Split the fully - connected weight matrix of the FFN layer (such as size 512×512) along the column direction into two sub - blocks (W1, W2, size 512×256).

[0134] 2. Input processing: Keep the input matrix intact (N×512) to avoid the additional overhead introduced by splitting.

[0135] 3. Serial Computation: First, calculate the input × W1 to obtain the intermediate result FFN_part1 (N × 256). After releasing the video memory of FFN_part1, calculate the input × W2 to obtain FFN_part2 (N × 256). If the two parts of the result are merged, the complete linear output of the FFN layer can be restored without loss.

[0136] (2) Serial Execution of Nonlinear Transformation

[0137] First, perform block-by-block activation. The calculation result of each sub-block (such as FFN_part1) is immediately applied with an activation function (such as GeLU) to avoid storing the complete intermediate result. The activated sub-blocks are directly passed to the next layer or the concatenation step, and the video memory occupancy is always maintained at the single sub-block level (N × 256).

[0138] 3. Video Memory and Computation Balance of Blocking Strategy

[0139] Peak Video Memory Calculation:

[0140] The peak video memory of a single layer is determined by the maximum sub-block calculation requirement. For example, after splitting a 512×512 matrix into 2 blocks (512×256) and multiplying it with the input (N × 512), the intermediate result becomes two matrices with dimensions N × 256. Only one intermediate result is retained in the video memory at the same time, and the peak video memory is reduced by 50%.

[0141] Attention Module: The block-by-block serial calculation of Q and K and the delayed block-by-block calculation of V further compress the peak video memory from (3 × N × d_model + N 2 ) to (2 × N × d_model / M + N 2 ).

[0142] FFN Module: The collaborative block-by-block calculation of the fully connected layer and the nonlinear layer further compresses the peak video memory from (2 × N × d_model × 4) to (2 × N × d_model × 4 / M).

[0143] Time Delay Control:

[0144] Pipeline Scheduling: Utilize the multi-core architecture of the GPU to preload the K1 parameter when calculating Q1 to reduce the data transfer overhead.

[0145] Compute-Release-Reuse Strategy: Through timely release of video memory and prefetching of parameters, partially offset the time delay caused by serial execution.

[0146] Trade-off of Blocking Granularity:

[0147] Video Memory and Computation Efficiency Balance: The smaller the splitting granularity (such as 4 blocks), the lower the video memory occupancy, but the serial calculation time increases; the splitting granularity needs to be dynamically adjusted according to the hardware video memory capacity.

[0148] Dynamic Load Balancing and Pipeline Control

[0149] 1. Video Memory Status Monitoring Layer

[0150] Deploy a video memory occupancy sampler on each GPU node to report real-time video memory usage data to the central scheduler at a granularity of 10 ms. Mark the nodes with high video memory occupancy as "compute-intensive ready" to ensure that they can obtain subsequent chunk tasks (such as K2 T 、V2) preferentially.

[0151] 2. Pipeline Scheduling Engine

[0152] Multi-stage pipeline design: Decompose the computing task into four stages: "parameter prefetching → computing → video memory release → result aggregation", and reduce the end-to-end latency to 1.5 times that of a single stage (4 times for traditional serial behavior) through parallel pipelines.

[0153] Dependency-aware scheduling: Use a directed acyclic graph (DAG) to model the data dependency relationships between sub-blocks to ensure that dependent subtasks are preferentially executed when the pipeline advances.

[0154] 3. Fault Tolerance and Elastic Expansion

[0155] Video memory overflow fallback: If a node has insufficient video memory, immediately migrate the current sub-block computation to a low-load node and trigger the checkpoint mechanism to restore the computation state.

[0156] Dynamic resource pool: Support adding new GPU nodes at runtime. The scheduler automatically incorporates them into the task allocation pool and rebalances the chunking strategy.

[0157] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions, or substitutions made by those skilled in the art within the essence of the present invention should also fall within the protection scope of the present invention.

Claims

1. A storage-efficient large model inference method for many-core supercomputing, characterized in that, Including the block optimization of the Attention module and the block optimization of the FFN module, specifically including the following steps. First, perform parameter block division and serial calculation: In the Attention module, vertically cut the Q and K parameter matrices into multiple sub-blocks along the column direction, and keep the input intact; serially calculate the product of each sub-block and the input to obtain local Q and K matrices, and immediately perform block multiplication of QKT to obtain partial attention scores; Aggregate the calculation results of all sub-blocks to obtain a complete attention score matrix; then use Softmax to normalize the attention scores, and serially multiply them with the V blocks calculated with delay, and splice the results to obtain the Attention output; Optimization of the FFN module. Vertically cut the fully connected layer parameters into sub-blocks, and the complete input data is serially calculated with the sub-blocks in turn. The non-linear transformation results of each sub-block are spliced in real time to reduce the intermediate video memory occupancy; specifically, through dynamic video memory release, that is, after each sub-block calculation is completed, immediately release the video memory of its parameters and intermediate results to ensure that the peak video memory is only determined by the calculation requirements of a single sub-block; it also has multi-core adaptability. Through pipeline scheduling, map the block tasks to the multi-core processor, and preload the subsequent parameters during serial calculation to balance video memory optimization and calculation efficiency.

2. The method for efficient large model inference with storage for multi-core supercomputing according to claim 1, characterized in that In the parameter block division and serial calculation, it includes the balance of video memory and calculation of the block strategy. The specific process is as follows: Peak video memory calculation: The peak video memory of a single layer is determined by the maximum sub-block calculation requirement; Attention Module: The block serial calculation of Q and K and the delayed block calculation of V further compress the peak video memory from (3×N×d_model+N 2 ) to (2×N×d_model / M+N 2 ); FFN module: The collaborative block calculation of the fully connected layer and the non-linear layer further compresses the peak video memory from (2×N×d_model×4) to (2×N×d_model×4 / M); Time delay control: Pipeline scheduling: Utilize the GPU multi-core architecture to preload the K1 parameter when calculating Q1 to reduce the data transfer overhead; Calculation-release-reuse strategy: Through timely video memory release and parameter prefetching, partially offset the time delay caused by serial execution, Block granularity trade-off: Balance of video memory and calculation efficiency: The smaller the segmentation granularity, the lower the video memory occupancy, but the serial calculation time increases; the segmentation granularity needs to be dynamically adjusted according to the hardware video memory capacity.

3. The storage-efficient large model inference method for many-core supercomputing according to claim 1, characterized in that It also reports a video memory status monitoring layer Deploy a video memory occupancy sampler on each GPU node to report real-time video memory usage data to the central scheduler at a granularity of 10ms, and mark the nodes with high video memory occupancy as "computation-intensive ready" status to ensure that they can obtain subsequent block tasks preferentially; Pipeline scheduling engine Multi-stage pipeline design: Decompose the calculation task into four stages of "parameter prefetching → calculation → video memory release → result aggregation", and reduce the end-to-end delay to 1.5 times that of a single stage through parallel pipelines; Dependency-aware scheduling: Use a directed acyclic graph DAG to model the data dependency relationship between sub-blocks to ensure that dependent sub-tasks are preferentially executed when the pipeline advances; Fault tolerance and elastic expansion Video memory overflow fallback: If the video memory of a certain node is insufficient, immediately migrate the current sub-block calculation to a low-load node and trigger a checkpoint mechanism to restore the calculation state; Dynamic resource pool: It supports adding new GPU nodes at runtime. The scheduler automatically incorporates them into the task allocation pool and rebalances the chunking strategy.

Images