Model reasoning acceleration method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a model reasoning acceleration method and device, electronic equipment and a nonvolatile storage medium. The method comprises the steps that in the process that a natural language processing model carries out general matrix calculation, a first matrix and a second matrix are divided into a plurality of blocks respectively, general matrix calculation is executed through an image processing unit, and the first matrix and the second matrix are matrixes needing general matrix calculation; loading the blocks from a global memory of the image processing unit to a shared memory of the image processing unit, and performing matrix operation by adopting threads corresponding to the blocks in the shared memory; and writing a result obtained by calculation of each thread back to a corresponding position in the global memory to obtain a matrix calculation result corresponding to matrix calculation of the first matrix and the second matrix. The technical problem of low model reasoning calculation efficiency caused by high memory access delay and low data reuse rate of a global memory of an image processing unit during reasoning of a large model in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of large model computational inference technology, and more specifically, to a model inference acceleration method, apparatus, electronic device, and non-volatile storage medium. Background Technology

[0002] Currently, large language models (LLMs) in the field of natural language processing have shown great potential in tasks such as text generation, machine translation, and question answering systems due to their superior performance. However, as the scale of the models continues to increase and the number of model parameters surges, the inference process of LLMs faces severe challenges, especially in production environments with limited computing resources. The inference process of LLMs is constrained by factors such as large memory and GPU memory usage and low parallelism capabilities, resulting in technical problems such as low computational efficiency of model inference.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a model inference acceleration method, apparatus, electronic device, and non-volatile storage medium to at least solve the technical problem of low model inference computation efficiency caused by high memory access latency and low data reuse rate of global memory of image processing unit during large model inference in related technologies.

[0005] According to one aspect of the embodiments of this application, a method for accelerating model inference is provided, comprising: dividing a first matrix and a second matrix into multiple blocks during the general matrix calculation of a natural language processing model, wherein the general matrix calculation is performed by an image processing unit, and both the first matrix and the second matrix are matrices that need to be calculated using the general matrix calculation; loading the blocks from the global memory of the image processing unit to the shared memory of the image processing unit, and performing matrix operations using threads corresponding to the blocks in the shared memory, wherein one thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory; and writing the results calculated by each thread back to the corresponding location in the global memory to obtain the matrix calculation results corresponding to the matrix calculation of the first matrix and the second matrix.

[0006] Optionally, the size of the block is determined based on the size of the shared memory and the size of the thread block, wherein the thread block is the smallest unit for the image processing unit to execute instructions, and each thread block contains a fixed number of threads.

[0007] Optionally, the matrix calculation result includes: a result matrix; loading blocks from global memory into shared memory and performing matrix operations on the blocks in shared memory includes: allocating a corresponding thread block for each block, wherein each thread in the thread block is used to load a portion of the data of the block into shared memory; and performing matrix operations on the blocks in shared memory after all threads have completed the operation of loading data into shared memory, wherein each thread is responsible for calculating a sub-region of the result matrix.

[0008] Optionally, the size of the first matrix is ​​M×K, the size of the second matrix is ​​K×N, the size of the block corresponding to the first matrix is ​​BM×BK, and the size of the block corresponding to the second matrix is ​​BK×BN; for the block corresponding to the first matrix, each thread moves... For each floating-point number, corresponding to the block of the second matrix, each thread moves... There are 3 floating-point numbers, where TM×TN is the size of the sub-region that each thread is responsible for when calculating the result matrix. M, K, N, BM, BK, BN, TM, and TN are all positive integers.

[0009] Optionally, during the data processing process of the natural language processing model in the feedforward network layer, the method further includes: determining a sparsity pattern, wherein the sparsity pattern is used to indicate the proportion of non-zero elements that should be retained in the feedforward network layer; determining a binary mask matrix based on the sparsity pattern, wherein the binary mask matrix includes all mask combinations that conform to the sparsity pattern; using a target controller function, based on the binary mask matrix, performing mask feature selection on the data parameters involved in data processing in the feedforward network layer to obtain sparsified data parameters; and using the sparsified data parameters to perform calculations in the feedforward network layer.

[0010] Optionally, the mask feature selection for the data parameters involved in data processing in the feedforward network layer based on the binary mask matrix includes: determining the mask index corresponding to the data parameter based on the binary mask matrix, wherein the mask index is used to selectively retain some data in the data parameter; and applying the mask index to the data parameter to obtain the sparsified data parameter.

[0011] Optionally, the data parameters include: model parameters and input data, wherein the model parameters include the weight matrix and bias vector in the feedforward network layer, and the input data is the feature vector input to the feedforward network layer; the target controller function includes: a hard sampling controller function corresponding to the model parameters and a soft sampling controller function corresponding to the input data, wherein the hard sampling controller function is used to directly calculate the mask index corresponding to the model parameters, and the soft sampling controller function is used to calculate the probability value corresponding to the input data and determine the mask index corresponding to the input data based on the probability value.

[0012] According to another aspect of the embodiments of this application, a model inference acceleration device is also provided, comprising: a block segmentation module, used to divide a first matrix and a second matrix into multiple blocks during the general matrix calculation process of a natural language processing model, wherein the general matrix calculation is performed by an image processing unit, and both the first matrix and the second matrix are matrices that need to be calculated using the general matrix calculation; a computation module, used to load the blocks from the global memory of the image processing unit into the shared memory of the image processing unit, and to perform matrix operations using threads corresponding to the blocks in the shared memory, wherein one thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory; and a write-back module, used to write the results calculated by each thread back to the corresponding location in the global memory, thereby obtaining the matrix calculation results corresponding to the matrix calculation of the first matrix and the second matrix.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes a model inference acceleration method during runtime.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes a model inference acceleration method by running the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of a model inference acceleration method.

[0016] In this embodiment, during the general matrix calculation in the natural language processing model, the first matrix and the second matrix are divided into multiple blocks. The general matrix calculation is performed by the image processing unit, and both the first and second matrices are matrices that require general matrix calculation. The blocks are loaded from the global memory of the image processing unit to the shared memory of the image processing unit, and the corresponding threads in the shared memory are used to perform matrix operations. One thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory. The results calculated by each thread are written back to the corresponding location in the global memory. The method of performing matrix calculations on the first and second matrices reduces the number of global memory accesses by dividing matrix operations into blocks and utilizing GPU shared memory, effectively reducing GPU memory usage and improving the computation-to-memory ratio, making the computation more efficient. Furthermore, the use of an N:M sparse mechanism in the feedforward network layer intelligently selects non-zero values ​​for calculation, avoiding a large number of invalid zero-value multiplications, further reducing computational load and improving model inference speed. This solves the technical problem of low model inference efficiency caused by high memory access latency and low data reuse rate of the global memory of the image processing unit during large models in related technologies. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for accelerating model inference, according to an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of a method for accelerating model inference according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a method flow for an LLM automated inference process provided according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of a computational process for accelerating large-scale model inference according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an N:M dilution calculation process provided according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of a GEMM calculation process provided according to an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of a matrix calculation process using shared GEMM memory according to an embodiment of this application;

[0025] Figure 8 This is a schematic diagram showing the efficiency test comparison results when the generation length is 2048 according to an embodiment of this application;

[0026] Figure 9 This is a schematic diagram showing the efficiency test comparison results when the generation length is 1024, according to an embodiment of this application.

[0027] Figure 10 This is a schematic diagram of a model inference acceleration device provided according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] To facilitate a better understanding of the embodiments of this application by those skilled in the art, some technical terms or nouns involved in the embodiments of this application are explained as follows:

[0031] TensorRT-LLM is an open-source library for defining, optimizing, and performing inference on large language models (LLMs) in production environments. It retains the core functionality of FasterTransformer in open-source Python and works with TensorRT's deep learning compiler to accelerate LLM inference.

[0032] auto-regressive: Currently, LLM inference mainly relies on auto-regressive decoding. Each decoding step can only produce one output token, and the historical output content needs to be concatenated and used as the input of LLM before the next decoding step can be performed.

[0033] A warp is the basic execution unit of a GPU. A warp contains 32 parallel threads, which execute in SMIT mode. This means that all threads execute the same instruction, and each thread uses its own data to execute that instruction. Threads within a warp must reside in the same block. If the number of threads in a block is not an integer multiple of the warp size, some of the extra threads will remain active within the warp. In other words, even if the number of threads is not a multiple of the warp size, the hardware will use special methods to make the warp larger, although these threads will be active. It's important to note that even these active threads will consume GPU resources.

[0034] Computing Hardware Performance: GPU computing power calculation formulas are primarily used to evaluate and compare the differences in computing capabilities among different graphics processing units (GPUs). GPU computing power is typically expressed in floating-point operations per second (FLOPS), representing the number of floating-point operations a GPU can perform per second. Generally, GPU computing power depends on factors such as its architecture, core frequency, number of stream processors, and arithmetic logic units per stream processor.

[0035] Flash attention is an algorithm that reorders attention calculations, speeding up attention computations and reducing memory usage without requiring any approximations.

[0036] Gumbel sampling is a technique for drawing samples from a discrete probability distribution. It is often used to avoid the complexity of directly sampling a discrete distribution, especially when dealing with discrete random variables in deep learning and reinforcement learning. The Gumbel sampling method is based on the Gumbel distribution, which is an extreme value distribution often used to simulate the distribution of maximum or minimum values.

[0037] As the scale of large language models continues to expand, the training and usage costs become extremely high. In terms of time and memory, the extremely high inference cost is a major bottleneck for the large-scale application of LLM to solve real-world tasks. The main limitations of LLM inference are its large memory and GPU memory consumption and low parallelism. The large memory and GPU memory consumption is primarily due to the introduction of KV cache technology in LLM to accelerate the Q, K, and V calculations in the Transformer self-attention mechanism. During the decoding phase, the K-V cache is used to store intermediate calculations to prevent redundant computations. For an inference computation with a batch size of 512 and a single input length of 2048, the memory and GPU memory consumed by the KV cache are three times that required by the original model, and the inference cost of the attention mechanism increases exponentially with the length of the input sequence. The low parallelism is mainly because LLM uses an autoregressive computation form similar to n-grams in the generation process. Due to the design limitations of the Transformer, subsequent generation process calculations, self-attention mechanism calculations, and positional encoding calculations all rely on the results of previous calculations, ultimately preventing parallel computation across the entire sequence. This leads to a series of problems, including the high cost of deploying LLM.

[0038] The goals of LLM inference optimization can be summarized as follows: reducing model memory footprint by using fewer GPU devices and less GPU memory; reducing computational complexity by decreasing the number of computational FLOPs required for inference; and reducing inference latency to speed up execution. This can be achieved primarily through various methods in the following processes to reduce memory and GPU memory usage for inference, ultimately resulting in faster inference computation time.

[0039] 1) Utilizing various parallelisms: Vertically scaling models across a large number of GPUs. Intelligent parallelism of model components and data makes it possible to run models containing trillions of parameters.

[0040] 2) Memory Offloading: Unused data is temporarily offloaded to the CPU and read back when needed later. This helps reduce memory usage but can lead to higher latency.

[0041] 3) Intelligent ingredient strategy: For example, Effective Transformer packs consecutive sequences together to remove filler within a batch.

[0042] 4) Network compression techniques: such as pruning, quantization, and distillation. In terms of parameter count or bit width, smaller models should require less memory and run faster.

[0043] 5) Target model-specific architecture improvements: Many architectural changes, especially changes to the attention layer, have helped improve the converter's decoding speed.

[0044] Currently, related technologies primarily accelerate large model inference through methods such as knowledge distillation, model quantization, Mixture of Experts (MoE), and model pruning and dilution. However, these methods mainly focus on optimizing the model structure itself, neglecting the core bottlenecks in the inference process—the efficiency of General Matrix Multiplication (GEMM) and Feed-Forward Network (FFN) layer computations. These computations are the core reason for the performance degradation of LMM inference, and none of the LMM solutions mentioned above have offered satisfactory solutions to these issues.

[0045] To address the aforementioned issues, this application provides two solutions: firstly, introducing dynamic shard memory for general matrix multiplication in GPU computing to alleviate GPU resource issues caused by global memory; and secondly, introducing dynamic shard memory into the FNN layer of the model network. Sparse mode pairs The layer is diluted, skipping the zero-value vector and directly applying it. Non-zero weights in the layers are calculated to accelerate LLM inference. This will be explained in detail below.

[0046] According to an embodiment of this application, a method embodiment for accelerating model inference is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or electronic device) for implementing a model inference acceleration method is shown. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0048] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or electronic device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0049] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model inference acceleration method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned model inference acceleration method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0051] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or electronic device).

[0052] Under the above operating environment, this application provides a method for accelerating model inference. Figure 2This is a schematic diagram of a method for accelerating model inference according to an embodiment of this application, as shown below. Figure 2 As shown, the method includes the following steps:

[0053] In step S202, during the general matrix calculation process of the natural language processing model, the first matrix and the second matrix are divided into multiple blocks respectively. The general matrix calculation is performed by the image processing unit. Both the first matrix and the second matrix are matrices that need to be calculated using the general matrix.

[0054] Step S204: Load the blocks from the global memory of the image processing unit to the shared memory of the image processing unit, and use the thread corresponding to the block in the shared memory to perform matrix operations. Here, one thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory.

[0055] Step S206: Write the result calculated by each thread back to the corresponding location in global memory to obtain the matrix calculation result corresponding to the matrix calculation of the first matrix and the second matrix.

[0056] By using the steps described above, matrix operations are divided into blocks and shared memory is utilized by the GPU, which greatly reduces the number of global memory accesses, effectively reduces video memory usage, and improves the computation-to-memory access ratio, making computation more efficient. This solves the technical problem of low computational efficiency in large models during inference due to high memory access latency and low data reuse rate of the global memory of the image processing unit.

[0057] The method for accelerating model inference in steps S202 to S206 of the embodiments of this application will be further described below.

[0058] Figure 3 This is a schematic diagram of a method flow for an LLM automated inference process provided in an embodiment of this application, as shown below. Figure 3 As shown, the task of LLM inference is to generate tokens from an input sequence to complete sentences or answer questions. Its inference data flow can be divided into two typical stages with similar operations: a pre-filling stage and a decoding stage. In the pre-filling stage, the system "understands" the input sequence (e.g., "What is quantum cryptography?"). Each token (in...) Figure 3The input sequence (represented as a word) is encoded as an embedding vector, while the input sequence forms a matrix. The main output of the pre-padding stage is a new token that is predicted as the next token in the input sequence (e.g., "encrypted call" in the figure). The decoding stage is responsible for generating the output sequence, where the output tokens from the pre-padding stage are used as inputs to the decoding stage. The decoding stage is performed stepwise, with each output token serving as an input token for the next decoding step (e.g., "encrypted" is further used as input).

[0059] The reasoning steps in LLM are as follows: Figure 3 Steps ① to ⑥ are shown in the diagram, where ① and ⑤ represent linear projection layers; ②, ③, and ④ mainly perform Transformer attention calculations; and ⑥ mainly performs feedforward network calculations. Here, the prompt processing in the LLM inference process can be simply referred to as the preprocessing stage, and the token prediction stage can be referred to as the decoding stage. The operators in these two stages are basically the same, mainly differing in the shape of the input matrix.

[0060] The linear projection layer mainly performs fully connected computations, that is, it performs dot product operations between the input vector and each weight matrix. , , , They are in , , and The projection above. Generated through linear projection during the preprocessing stage. , , The vector matrix is ​​generated again through projection operations during the decoding stage. and This is a subsequent calculation step as part of the preprocessing.

[0061] Attention calculation mainly involves three operational steps (②-④), which correspond to... and Perform dot product operation Computation and attention and The dot product operation, the final attention value is obtained through The calculation is as follows:

[0062]

[0063] The feedforward neural network layer mainly performs two calculations, the first of which is ⑥ ( The second ⑥ () expands the feature dimension to enhance the representation. This is used to recover the dimensions of the features and serve as the final output.

[0064] Based on the above reasoning process, this application proposes a method to accelerate large model inference by combining block-based GEMM shared memory and N:M sparse FFN layers. The specific calculation process is as follows: Figure 4 As shown, it is specifically divided into N:M fine-grained structured sparse FFN layer computation and block-shared GEMM computation, which will be introduced in detail below.

[0065] First of all, Fine-grained structured sparsity The process of layer computing is described below.

[0066] In the baseline Transformer model, decoding speed largely depends on... The execution efficiency of the layer, such as Figure 3 Step 6 of the calculation is shown in the diagram. Each A layer typically contains two multilayer perceptrons (MLPs) and a layer between them. Activation function. Dimension of the activation vector between two fully connected layers. Typically, it's in other locations (such as model dimensions). 4 to 8 times that of ). Activation functions often introduce a large number of zero values. These zero values ​​exhibit a fixed structure during the activation process, and sometimes even force the calculation of only one non-zero value within a block, resulting in a large number of invalid calculations.

[0067] Sparsity techniques can reduce computation by removing a large number of zero values ​​from a sparse matrix, transforming it into a more compact form. This is illustrated in the embodiments of this application. The basic idea behind fine-grained structured sparsity is to select N positions that might be zero from M consecutive weight values, treat them as zero, and remove them. This method allows the sparse weight matrix to be compressed and stored, keeping only the non-zero elements, and using an index matrix to record the positions of these non-zero elements in the original matrix. This process is similar to selectively deleting weights from consecutive weight regions, enabling the maximum deletion of consecutive regions for each token while maintaining the dynamic changes in the sparsity pattern, thus achieving the coarsest granularity.

[0068] Based on this, the embodiments of this application propose a... Fine-grained structured sparsity methods are used to optimize Layers and The calculations are performed to improve the inference efficiency of the model, as detailed below.

[0069] In some embodiments of this application, during the data processing of the natural language processing model in the feedforward network layer, the method further includes: determining a sparsity pattern, wherein the sparsity pattern is used to indicate the proportion of non-zero elements that should be retained in the feedforward network layer; determining a binary mask matrix based on the sparsity pattern, wherein the binary mask matrix includes all mask combinations that conform to the sparsity pattern; using a target controller function, based on the binary mask matrix, performing mask feature selection on the data parameters involved in data processing in the feedforward network layer to obtain sparsified data parameters; and using the sparsified data parameters to perform calculations in the feedforward network layer.

[0070] Specifically, the sparsity mode, i.e. N:M, needs to be determined first. For example, a sparsity mode of 2:4, or 1:4 and 4:8 can be set. This application does not limit this.

[0071] For a parameter block containing M consecutive parameters The objective of this application's embodiments is to determine an optimal binary mask. Ensure that given input data At this time, the original performance of the model can be maintained even after sparse weights are applied. The mask M must contain exactly the... These zero values ​​form a discrete candidate set. (i.e., the aforementioned binary mask matrix), its size is = There are 10 different candidate schemes, that is, all mask combinations that conform to the sparse pattern.

[0072] Based on the binary mask matrix, mask features can be selected for the data parameters involved in data processing in the feedforward network layer, as follows.

[0073] In some embodiments of this application, the mask feature selection of data parameters involved in data processing in the feedforward network layer based on the binary mask matrix includes: determining the mask index corresponding to the data parameter based on the binary mask matrix, wherein the mask index is used to selectively retain part of the data in the data parameter; and applying the mask index to the data parameter to obtain the sparsified data parameter.

[0074] The data parameters include model parameters and input data. The model parameters include the weight matrix and bias vector in the feedforward network layer, and the input data is the feature vector input to the feedforward network layer. The target controller function includes a hard sampling controller function corresponding to the model parameters and a soft sampling controller function corresponding to the input data. The hard sampling controller function is used to directly calculate the mask index corresponding to the model parameters, and the soft sampling controller function is used to calculate the probability value corresponding to the input data and determine the mask index corresponding to the input data based on the probability value.

[0075] Specifically, for The calculations in this application embodiment are for model parameter features (i.e., the aforementioned model parameters). and And input features (i.e., the input data mentioned above). Two sampling functions were designed: (i.e., the aforementioned hard sampling controller function) and (i.e., the soft sampling controller function mentioned above), due to the parameters and It is a fixed value, only needs to be calculated once, therefore it can be used. Formal implementation Sampling is performed to obtain the index of this parameter on the binary mask M, which can be called hard sampling, as shown in the following formula:

[0076]

[0077] in, It is uniformly distributed random noise. Known as Gumbel noise, Indicates the importance or probability of model parameters.

[0078] For input features Since each input value is different, a gentler sampling method needs to be designed to prevent brute-force sampling from affecting the semantics of the input features. In this embodiment, a more flexible sampling method can be used. Specifically, during the sampling process, first through The sampled points are converted into probability values, and then the specific mask matrix is ​​indexed by the probability values. The position of is shown in the following formula.

[0079]

[0080] The temperature term This is a hyperparameter used to control the difficulty of the sampling index. When hour, Approaching .

[0081] The final computational flow for N:M fine-grained structured sparseness is as follows: Figure 5 As shown, a special design is needed for... and as well as sparse matrix ,one Layer sparse matrix and a controller The specific formula is as follows:

[0082]

[0083]

[0084]

[0085] Each activation in the formula Corresponding to , and ,in, These represent the sparsed weight matrix, input eigenvector, and bias vector, respectively; controller Used to implement the formula , , The choice of non-zero values, among which... and These are the parameters of the controller, where M and N are the dimensions of the matrix. This formula is derived through... During inference, it is determined which columns among M consecutive weight values ​​should be non-zero. Therefore, a controller load parameter is pre-calculated during inference. When calculating the layer weight matrix, some columns with zero values ​​are removed, and the non-zero columns are selected and loaded into memory for subsequent inference.

[0086] On the other hand, this application also provides a method for block-sharing GEMM computation, which will be described in detail below.

[0087] exist Figure 3 In steps ① through ⑥ of the inference process shown, whether in the preprocessing or decoding stage, the data is loaded onto the GPU for computation. During this process, the GPU primarily performs matrix operations; each GPU operation is a GEMM operation. Additionally, the decoding stage involves a large amount of... The computation of fully connected layers thus optimizes the general matrix and The computation of fully connected layers can improve computational efficiency and save inference memory.

[0088] GEMM is a core component of various computing processes and a standard technique for evaluating computing hardware performance (FLOPS). GEMM is defined as follows:

[0089]

[0090] Soon Matrix (First matrix) and Perform matrix multiplication on the second matrix and scale the result. times, then with scaling A matrix of times Add them together and put the final result into (In the result matrix). This calculation process assumes... The shape is , The shape is ,but The shape is The main part is Matrix multiplication, according to the definition of matrix multiplication, yields the following formula:

[0091]

[0092] The computational load, memory access ratio, and memory access ratio in this calculation process are shown in the following formula:

[0093]

[0094]

[0095]

[0096] like Figure 6 As shown, the calculation of each element in this process requires Multiplication and The addition is performed once, and the total calculation is... Requires execution Second floating-point operations, additional and Both scaling and shrinking require If there are n floating-point operations, then the total number of floating-point operations is 1. ,because Therefore, the number of floating-point operations is usually approximately equal to .

[0097] In actual computation, since the smallest unit of instruction execution on a GPU is a warp (32 threads), read and write operations within the same warp can be partially merged. For 32 threads in a warp, in each loop, the matrix needs to be read... The same element in the matrix (1 transmission), and the matrix Calculating 32 consecutive elements requires two global memory loads to complete a single multiplication-accumulation operation, resulting in an extremely low memory access ratio and hindering effective data reuse. Therefore, this embodiment introduces shared memory to reduce redundant memory reads, as detailed below.

[0098] First, the first matrix A and the second matrix B are divided into multiple blocks. Then, the blocks are loaded from the GPU's global memory into shared memory, and the corresponding threads in the shared memory are used to perform matrix operations. One thread is used to perform matrix operations on one block, and the access speed of shared memory is faster than that of global memory. Finally, the results calculated by each thread are written back to the corresponding location in global memory to obtain the matrix calculation results corresponding to the matrix calculations of the first and second matrices.

[0099] The size of the blocks is determined based on the size of the shared memory and the size of the thread blocks. The thread block is the smallest unit in which the image processing unit executes instructions, and each thread block contains a fixed number of threads.

[0100] In some embodiments of this application, the matrix calculation result includes: a result matrix; loading blocks from global memory into shared memory and performing matrix operations on the blocks in shared memory includes: allocating a corresponding thread block for each block, wherein each thread in the thread block is used to load a portion of the data of the block into shared memory; and performing matrix operations on the blocks in shared memory after all threads have completed the operation of loading data into shared memory, wherein each thread is responsible for calculating a sub-region of the result matrix.

[0101] Specifically, the first matrix A and the second matrix B are divided into matrices. and Stored in shared memory, for matrix partitioning Each thread moves data each time. Each block is a floating-point number, and then each block is computed by a thread, where each thread is responsible for calculating the matrix. Each element. All subsequent data required for calculations is read from shared memory, eliminating some duplicates in this step. , Matrix memory read.

[0102] The first matrix has a size of M×K, the second matrix has a size of K×N, the block size corresponding to the first matrix is ​​BM×BK, and the block size corresponding to the second matrix is ​​BK×BN. For the block corresponding to the first matrix, each thread moves data in one pass. For each floating-point number, corresponding to the block of the second matrix, each thread moves... There are 3 floating-point numbers, where TM×TN is the size of the sub-region that each thread is responsible for when calculating the result matrix. M, K, N, BM, BK, BN, TM, and TN are all positive integers.

[0103] In addition, considering the limitations of shared memory capacity, it is possible to Each read on the dimension Dividing the matrix into blocks of varying sizes, this loop requires K / BK iterations to complete the entire matrix multiplication operation, yielding the result of the block computation. The memory access ratio computation for each block after optimization using shared memory is shown in the following formula:

[0104]

[0105]

[0106]

[0107] The optimized calculation process is as follows: Figure 7 As shown, according to the above formula, we can know and The larger the value, the higher the computation-to-memory access ratio and the better the performance; the actual number of reads in the optimized computation process is reduced from the original... become By introducing block shared memory, it is possible to support very large [scale / size] data. and The calculation.

[0108] To enable those skilled in the art to better understand the embodiments of this application, the above process steps are further illustrated below with examples.

[0109] Taking 2:4 sparse as an example, first initialize a binary mask matrix. Here in every It must contain exactly two zeros to produce a value with discrete candidate set By randomly selecting two values ​​from the four values ​​and setting them to 0, the final sorted result is as follows: = = 6, there are a total of 6 candidate sets, as shown below:

[0110]

[0111] For a model parameter matrix of 4x4, during the model loading phase, the FNN layer in the decoding parameters is of the form shown below. First take out and For each row of these parameters, the controllers function performs mask feature selection to select values ​​that conform to the 2:4 sparse form, as shown below.

[0112] For 4x4 model parameters First calculate the parameters in the first row, then use... Perform parameter sampling. The first row is sampled using a noise variable, then a one-hot index is generated, and the location is then determined using the index. The specific rows in the matrix are used to obtain the mask matrix, and finally, feature selection is performed using the mask matrix to obtain the diluted result. The parameters are as follows:

[0113]

[0114]

[0115]

[0116] Similarly, for The second row of parameters is used to obtain the index of the mask matrix corresponding to the second row, as shown in the following formula:

[0117]

[0118] Then, using the mask matrix, feature selection is performed on the row corresponding to the index to obtain the diluted result. The parameters are shown in the following formula:

[0119]

[0120] Finally, after processing each row, the diluted parameter feature set is obtained. and As shown in the following formula:

[0121]

[0122]

[0123] On the other hand, for the FNN layer In calculation The dilution steps and the above as well as The dilution steps are the same, but for direct use of input... The conventional method of calculating the mask matrix index can introduce noise that is sensitive to the input features. Therefore, a more moderate method is needed to obtain the mask matrix, avoiding brute-force approaches to extreme values. Calculation thus changes the input The original semantic information is preserved. Based on this, a mild index calculation function is introduced. The specific calculation steps are shown in the following formula:

[0124] For the input feature matrix :

[0125]

[0126] First, process the first line:

[0127]

[0128]

[0129] Similarly for The second row uses the following formula to obtain the index of the mask matrix corresponding to the second row:

[0130]

[0131] Then, feature selection is performed using the row corresponding to the index in the mask matrix to obtain the diluted result. The input features are shown in the following formula:

[0132]

[0133] Finally, after processing each line, the diluted input is obtained. The feature set is shown in the following formula:

[0134]

[0135] After obtaining the diluted , and After that, you can use the regular method. The algorithm performs the calculation as shown in the following formula, and the diluted calculation result can be obtained.

[0136]

[0137] Furthermore, regarding block-based shared GEMM computation, taking M=512, K=512, N=512, BM = BN = 128, BK = 8, TM = TN = 8 as an example, without using shared memory, the computational load, memory access load, and computation-to-memory ratio are 524288, 134217728, and 256, respectively, as shown in the following formula:

[0138]

[0139]

[0140]

[0141] After using shared memory, the computational load, memory access load, and computation-to-memory ratio are 16640, 524288, and 32, respectively. All computational metrics have decreased significantly, as shown in the following formula:

[0142]

[0143]

[0144]

[0145] Furthermore, to evaluate the inference accuracy of the optimized large language model, this application selected Qwen2.5-7B, LLaMA3-8B, and ChatGLM3-6B as benchmark models and conducted experiments on the NVIDIA A10 GPU platform. Since the maximum sequence length supported by the TensorRT-LLM (TRT-LLM) framework is 2048, the input token length was uniformly set to 2048 in all experiments. Considering that some models do not support the Flash Attention mechanism, to ensure experimental fairness, Flash Attention was not enabled for inference in any model. In addition, the experimental parameters were set as follows: beam search width (beam_search) was 1, batch size (batch_size) was 1, and maximum number of generated tokens (max_new_tokens) was 100.

[0146] Regarding the evaluation metrics, the ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Sum scores of each model's output were calculated and compared with the performance of the original model. The experimental results are shown in the table below.

[0147]

[0148] The data in the table shows that there are certain differences in inference accuracy under different models and optimization methods. Overall, the model optimized by the method in this embodiment improves the accuracy of inference by 0.04 and 0.02 in ROUGE-2 and ROUGE-Sum metrics compared to the TRT-LLM and the original BF16 method, respectively; while it slightly decreases in ROUGE-1 and ROUGE-L metrics, decreasing by 0.03 and 0.02, respectively.

[0149] Specifically, in the Qwen2.5-7B model, the optimized model in this embodiment improves the ROUGE-1 and ROUGE-Sum inference by 0.02 and 0.01 respectively compared to TRT-LLM and the original BF16 inference, but decreases the ROUGE-2 and ROUGE-L inference by 0.03 and 0.01 respectively.

[0150] In the ChatGLM3-6B model, the optimized model in this embodiment improves the ROUGE-2 and ROUGE-Sum inference metrics by 0.06 and 0.01 respectively compared to TRT-LLM and the original BF16, while decreasing the ROUGE-1 and ROUGE-L metrics by 0.04 and 0.01 respectively.

[0151] In summary, compared with the TRT-LLM inference method and the original BF16 inference method, the proposed method shows a smaller gap in inference accuracy and even outperforms in some metrics. This indicates that the method in this application can achieve efficient model inference while maintaining or even improving inference accuracy, demonstrating good practicality and promotional value.

[0152] Meanwhile, this application embodiment also tested the acceleration performance of the proposed solution, mainly testing two metrics: generation speed (token / s) and throughput speed (request / s). Generation speed includes not only the generation process but also the time for context understanding, thus representing the actual number of tokens processed (understanding input and generating output) per second. Throughput speed represents the average number of requests the system can process per second under extreme conditions with no request gaps. Flash attention was not enabled for inference in the experiments. Two main experiments were designed: one with a maximum input length of max_length=2048, max_new_token=2048, num-prompts=100, beam_search=1, batch=1, seed=0, and the other with max_length=1024, max_new_token=1024, num-prompts=100, beam_search=1, batch=1, seed=0. Specific test results are as follows: Figure 8 and Figure 9 As shown.

[0153] Experimental results show that with `max_length=2048` and `max_new_token=2048`, the highest throughput speedup reaches 2.73, and the generation speedup reaches 2.91. With `max_length=1024` and `max_new_token=1024`, the highest throughput speedup reaches 3.29, and the generation speedup reaches 3.65. Furthermore, in both experimental environments, some models achieved speedups exceeding those of the TensorRT_LLM framework without sacrificing inference accuracy, thus demonstrating the effectiveness of this approach in accelerating LLM inference.

[0154] In summary, this application proposes a general matrix algorithm based on shared memory. By setting shared memory instead of global memory, the memory usage for LLM inference can be effectively reduced, and the computation time for inference can also be reduced. Furthermore, this application proposes an N:M fine-grained sparsity approach to dilute the FNN computation in LLM. By selecting non-zero values ​​in the matrix and removing zero values, a large number of zero-multiplication transports in the LLM inference process can be reduced, thereby shortening the model inference time.

[0155] According to an embodiment of this application, an embodiment of a model inference acceleration device is also provided. Figure 10 This is a schematic diagram of a model inference acceleration device provided according to an embodiment of this application. Figure 10 As shown, the device includes:

[0156] The block module 1000 is used to divide the first matrix and the second matrix into multiple blocks during the general matrix calculation process of the natural language processing model. The general matrix calculation is performed by the image processing unit. Both the first matrix and the second matrix are matrices that need to be calculated using the general matrix.

[0157] The operation module 1002 is used to load blocks from the global memory of the image processing unit to the shared memory of the image processing unit, and to perform matrix operations using threads corresponding to the blocks in the shared memory. Here, one thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory.

[0158] The write-back module 1003 is used to write the results calculated by each thread back to the corresponding location in global memory, so as to obtain the matrix calculation results corresponding to the matrix calculation of the first matrix and the second matrix.

[0159] Optionally, the size of the block is determined based on the size of the shared memory and the size of the thread block, wherein the thread block is the smallest unit for the image processing unit to execute instructions, and each thread block contains a fixed number of threads.

[0160] Optionally, the matrix calculation result includes: a result matrix; loading blocks from global memory into shared memory and performing matrix operations on the blocks in shared memory includes: allocating a corresponding thread block for each block, wherein each thread in the thread block is used to load a portion of the data of the block into shared memory; and performing matrix operations on the blocks in shared memory after all threads have completed the operation of loading data into shared memory, wherein each thread is responsible for calculating a sub-region of the result matrix.

[0161] Optionally, the size of the first matrix is ​​M×K, the size of the second matrix is ​​K×N, the size of the block corresponding to the first matrix is ​​BM×BK, and the size of the block corresponding to the second matrix is ​​BK×BN; for the block corresponding to the first matrix, each thread moves... For each floating-point number, corresponding to the block of the second matrix, each thread moves... There are 3 floating-point numbers, where TM×TN is the size of the sub-region that each thread is responsible for when calculating the result matrix. M, K, N, BM, BK, BN, TM, and TN are all positive integers.

[0162] Optionally, the natural language processing model further includes the following steps during data processing in the feedforward network layer: determining a sparsity pattern, wherein the sparsity pattern indicates the proportion of non-zero elements to be retained in the feedforward network layer; determining a binary mask matrix based on the sparsity pattern, wherein the binary mask matrix includes all mask combinations that conform to the sparsity pattern; using a target controller function to perform mask feature selection on the data parameters involved in data processing in the feedforward network layer based on the binary mask matrix to obtain sparsified data parameters; and using the sparsified data parameters to perform computation on the feedforward network layer.

[0163] Optionally, the mask feature selection for the data parameters involved in data processing in the feedforward network layer based on the binary mask matrix includes: determining the mask index corresponding to the data parameter based on the binary mask matrix, wherein the mask index is used to selectively retain some data in the data parameter; and applying the mask index to the data parameter to obtain the sparsified data parameter.

[0164] Optionally, the data parameters include: model parameters and input data, wherein the model parameters include the weight matrix and bias vector in the feedforward network layer, and the input data is the feature vector input to the feedforward network layer; the target controller function includes: a hard sampling controller function corresponding to the model parameters and a soft sampling controller function corresponding to the input data, wherein the hard sampling controller function is used to directly calculate the mask index corresponding to the model parameters, and the soft sampling controller function is used to calculate the probability value corresponding to the input data and determine the mask index corresponding to the input data based on the probability value.

[0165] It should be noted that each module in the above-mentioned model inference acceleration device can be a program module (e.g., a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0166] It should be noted that the model inference acceleration device provided in this embodiment can be used to execute... Figure 2The model inference acceleration method shown above is also applicable to the embodiments of this application, and will not be repeated here.

[0167] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following model inference acceleration method by running the computer program: During the general matrix calculation process of the natural language processing model, the first matrix and the second matrix are divided into multiple blocks, wherein the general matrix calculation is performed by an image processing unit, and both the first and second matrices are matrices requiring general matrix calculation; the blocks are loaded from the global memory of the image processing unit to the shared memory of the image processing unit, and matrix operations are performed using threads corresponding to the blocks in the shared memory, wherein one thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory; the results calculated by each thread are written back to the corresponding location in the global memory to obtain the matrix calculation results corresponding to the matrix calculation of the first and second matrices.

[0168] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the model inference acceleration method described in various embodiments of this application: During the general matrix calculation process of the natural language processing model, the first matrix and the second matrix are respectively divided into multiple blocks, wherein the general matrix calculation is performed by an image processing unit, and both the first matrix and the second matrix are matrices that require general matrix calculation; the blocks are loaded from the global memory of the image processing unit to the shared memory of the image processing unit, and matrix operations are performed using threads corresponding to the blocks in the shared memory, wherein one thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory; the results calculated by each thread are written back to the corresponding location in the global memory to obtain the matrix calculation results corresponding to the matrix calculation of the first matrix and the second matrix.

[0169] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0170] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0175] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for accelerating model inference, characterized in that, include: In the process of performing general matrix calculation in the natural language processing model, the first matrix and the second matrix are divided into multiple blocks respectively. The general matrix calculation is performed by the image processing unit. The first matrix and the second matrix are both matrices that need to be calculated. The blocks are loaded from the global memory of the image processing unit into the shared memory of the image processing unit, and matrix operations are performed using threads corresponding to the blocks in the shared memory. One thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory. The result calculated by each thread is written back to the corresponding location in the global memory to obtain the matrix calculation result corresponding to the matrix calculation of the first matrix and the second matrix.

2. The model inference acceleration method according to claim 1, characterized in that, The size of the block is determined based on the size of the shared memory and the size of the thread block, wherein the thread block is the smallest unit for the image processing unit to execute instructions, and each thread block contains a fixed number of threads.

3. The model inference acceleration method according to claim 2, characterized in that, The matrix calculation result includes: a result matrix; loading the block from global memory to shared memory and performing matrix operations on the block in the shared memory includes: Each block is assigned a corresponding thread block, wherein each thread in the thread block is used to load a portion of the data of the block into the shared memory; Once all threads have completed loading data into the shared memory, matrix operations are performed on the blocks in the shared memory, where each thread is responsible for calculating a sub-region of the resulting matrix.

4. The model inference acceleration method according to claim 3, characterized in that, The first matrix has a size of M×K, the second matrix has a size of K×N, the size of the block corresponding to the first matrix is ​​BM×BK, and the size of the block corresponding to the second matrix is ​​BK×BN; for the block corresponding to the first matrix, each thread moves... For each floating-point number, corresponding to the block of the second matrix, each thread moves... There are 10 floating-point numbers, where TM×TN is the size of the sub-region that each thread is responsible for when calculating the result matrix, and M, K, N, BM, BK, BN, TM, and TN are all positive integers.

5. The model inference acceleration method according to claim 1, characterized in that, During the data processing process in the feedforward network layer of the natural language processing model, the method further includes: Determine a sparse pattern, wherein the sparse pattern is used to indicate the proportion of non-zero elements that should be retained in the feedforward network layer; Based on the sparse pattern, a binary mask matrix is ​​determined, wherein the binary mask matrix includes all mask combinations that conform to the sparse pattern; Using the target controller function, based on the binary mask matrix, mask feature selection is performed on the data parameters involved in data processing in the feedforward network layer to obtain the sparsified data parameters; The feedforward network layer is computed using the sparsified data parameters.

6. The model inference acceleration method according to claim 5, characterized in that, Based on the binary mask matrix, the mask feature selection for the data parameters involved in data processing in the feedforward network layer includes: Based on the binary mask matrix, the mask index corresponding to the data parameter is determined, wherein the mask index is used to selectively retain a portion of the data in the data parameter; The mask index is applied to the data parameters to obtain the sparsified data parameters.

7. The model inference acceleration method according to claim 6, characterized in that, The data parameters include: model parameters and input data, wherein the model parameters include the weight matrix and bias vector in the feedforward network layer, and the input data is the feature vector input to the feedforward network layer; the target controller function includes: a hard sampling controller function corresponding to the model parameters and a soft sampling controller function corresponding to the input data, wherein the hard sampling controller function is used to directly calculate the mask index corresponding to the model parameters, and the soft sampling controller function is used to calculate the probability value corresponding to the input data and determine the mask index corresponding to the input data based on the probability value.

8. A model inference acceleration device, characterized in that, include: The block module is used to divide the first matrix and the second matrix into multiple blocks during the general matrix calculation process of the natural language processing model. The general matrix calculation is performed by the image processing unit, and the first matrix and the second matrix are both matrices that need to be calculated. The computation module is used to load the block from the global memory of the image processing unit to the shared memory of the image processing unit, and to perform matrix operations using a thread corresponding to the block in the shared memory, wherein one thread is used to perform matrix operations on one block, and the access speed of the shared memory is faster than the access speed of the global memory. The write-back module is used to write the results calculated by each thread back to the corresponding location in the global memory, so as to obtain the matrix calculation results corresponding to the matrix calculation of the first matrix and the second matrix.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the model inference acceleration method according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the model inference acceleration method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the model inference acceleration method according to any one of claims 1 to 7.

Images