Inference acceleration method and device for neural network model, medium and product

Abstract

The invention provides a reasoning acceleration method and device for a neural network model, a medium and a product. A reasoning acceleration method for a neural network model is characterized by comprising the steps that before the neural network model is deployed, an original weight matrix of a to-be-accelerated target network layer in the neural network model is recognized, and the size of the original weight matrix is n * m; the original weight matrix is decomposed into a first low-rank matrix and a second low-rank matrix through a low-rank decomposition algorithm, the size of the first low-rank matrix is n * k, the size of the second low-rank matrix is k * m, and k is smaller than the minimum value of n and m; and in a reasoning stage of the neural network model, for the input data, executing the following operations: executing matrix multiplication of the input data and the first low-rank matrix to obtain an intermediate result; and performing matrix multiplication of the intermediate result and the second low-rank matrix to obtain a final output result.

Description

Technical Field

[0001] This disclosure relates to methods, apparatuses, non-transitory computer-readable media, and computer program products for accelerating inference for neural network models. Background Technology

[0002] With the rapid development of artificial intelligence technology, the scale of neural network models is constantly expanding. The core computational operation of neural network models—large-scale matrix multiplication (e.g., matrix multiplication in fully connected layers and projection layers in attention mechanisms)—has become a computational bottleneck restricting model inference speed and deployment costs, consuming enormous computing power and memory bandwidth. Especially during the model inference stage, how to reduce computational latency and improve throughput is a technical challenge that continues to attract industry attention. Summary of the Invention

[0003] In one aspect, this application discloses a method for accelerating inference in a neural network model, comprising: before deploying the neural network model, identifying the original weight matrix of the target network layer to be accelerated in the neural network model, wherein the size of the original weight matrix is ​​n×m, where n and m are positive integers, representing the number of rows and columns of the original weight matrix, respectively; decomposing the original weight matrix into a first low-rank matrix and a second low-rank matrix using a low-rank decomposition algorithm, wherein the size of the first low-rank matrix is ​​n×k, and the size of the second low-rank matrix is ​​k×m, where k is a positive integer, n and k represent the number of rows and columns of the first low-rank matrix, respectively, k and m represent the number of rows and columns of the second low-rank matrix, and k is less than the minimum value of n and m; and during the inference phase of the neural network model, for input data, performing the following operations: performing matrix multiplication of the input data with the first low-rank matrix to obtain an intermediate result; and performing matrix multiplication of the intermediate result with the second low-rank matrix to obtain a final output result.

[0004] In one aspect, this application discloses an inference acceleration device for a neural network model, comprising: a decomposition preprocessing module, configured to: identify the original weight matrix of the target network layer to be accelerated in the neural network model before deploying the neural network model, wherein the size of the original weight matrix is ​​n×m, where n and m are positive integers, representing the number of rows and columns of the original weight matrix, respectively; and decompose the original weight matrix into a first low-rank matrix and a second low-rank matrix using a low-rank decomposition algorithm, wherein the size of the first low-rank matrix is ​​n×k, the size of the second low-rank matrix is ​​k×m, where k is a positive integer, n and k represent the number of rows and columns of the first low-rank matrix, k and m represent the number of rows and columns of the second low-rank matrix, and k is less than the minimum of n and m; and a matrix calculation engine, configured to perform the following operations for input data during the inference phase of the neural network model: perform matrix multiplication of the input data with the first low-rank matrix to obtain an intermediate result; and perform matrix multiplication of the intermediate result with the second low-rank matrix to obtain a final output result.

[0005] In other respects, this application discloses a non-transitory computer-readable medium storing instructions and a computer program product including instructions. These instructions, when executed by one or more processors, cause the processors to perform the methods described in this application. Attached Figure Description

[0006] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be noted that, in accordance with standard practice in the art, the features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the features may be arbitrarily increased or decreased.

[0007] Figure 1 A schematic diagram of an inference acceleration method for a neural network model according to an embodiment of this application is shown.

[0008] Figure 2 A schematic diagram of an inference acceleration device for a neural network model according to an embodiment of this application is shown.

[0009] Figure 3 A schematic diagram of a computing device that can implement embodiments of this application is shown. Detailed Implementation

[0010] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.

[0011] As mentioned earlier, the core computational operation of neural network models, namely large-scale matrix multiplication, has become a computational bottleneck restricting the model's inference speed and deployment cost, consuming enormous computing resources and memory bandwidth. To address this issue, existing technologies have explored several approaches, primarily employing the following two categories.

[0012] The first type of method involves weight sparsification and sparse computation. This involves setting a large number of elements in the weight matrix to zero using techniques such as pruning, creating a sparse matrix that skips zero-value operations during computation, thus saving computational resources. However, this type of method has significant drawbacks: First, unstructured sparse patterns are difficult to utilize efficiently on general-purpose hardware (such as GPUs and general-purpose AI accelerators), requiring dedicated sparse computation units and complex index decoding circuits, resulting in complex hardware design and limited energy efficiency improvements; second, highly irregular memory access patterns severely reduce memory access efficiency, offsetting some of the computational gains.

[0013] The second category of methods involves low-rank approximation and adapter techniques. In the field of model fine-tuning, techniques such as LoRA (Low-Rank Adaptation) achieve efficient parameter fine-tuning by decomposing the weight matrix W or the weight matrix update ΔW into the product of two low-rank matrices (e.g., A and B, where A∈R^{n×r}, B∈R^{r×m}, r << n, m). However, existing techniques mainly focus on parameter optimization and storage savings during the training phase, without deeply utilizing this decomposition structure to optimize computational complexity and data handling during the inference phase.

[0014] Therefore, there is an urgent need for a neural network inference acceleration solution that can achieve a reduction in computational complexity comparable to sparse computing, while avoiding its hardware incompatibility and irregular memory access defects.

[0015] To address the aforementioned problems, this paper discloses various implementation methods that provide a method and apparatus for accelerating neural network matrix multiplication based on low-rank decomposition. Specifically, the dense weight matrix M (n×m) to be accelerated in the neural network model is pre-decomposed into the product of two consecutive low-rank matrices A (n×k) and B (k×m) before inference, i.e., M ≈ A * B, where the dimensionality reduction factor k satisfies k << min(n, m). This decomposition is mathematically equivalent to applying a specific pattern of ultra-high sparsity structured sparsity to the original weight matrix M, thereby fundamentally reducing the number of computational operands (FLOPs) and the amount of weight data transported during inference while maintaining the algorithm's accuracy essentially intact.

[0016] Figure 1A schematic diagram of an inference acceleration method 100 for a neural network model according to an embodiment of this application is shown. Method 100 can be applied to various neural network layers that rely on large-scale matrix multiplication. Figure 1 As shown, the method 100 may include steps S102, S104 and S106.

[0017] In step S102, before deploying the neural network model, the original weight matrix M of the target network layer to be accelerated in the neural network model is identified. The original weight matrix M has a size of n×m, where n and m are positive integers, representing the number of rows and columns of the original weight matrix M, respectively.

[0018] In some embodiments, the target network layer can be a computationally intensive layer in a neural network model, such as, but not limited to, a fully connected feedforward layer or an attention output projection layer in a Transformer model. The original weight matrix M can be a dense weight matrix. As used in this paper, a dense weight matrix refers to a matrix in a neural network model where weight parameters are stored as a two-dimensional array and most elements are non-zero. In complex neural network models such as large language models, matrix multiplication operations associated with dense weight matrices are a major source of computational overhead and a key bottleneck restricting model inference speed.

[0019] In step S104, the original weight matrix M is decomposed into a first low-rank matrix A and a second low-rank matrix B using a low-rank decomposition algorithm, such that M ≈ A * B, where the size of the first low-rank matrix A is n×k and the size of the second low-rank matrix B is k×m, where k is a positive integer, n and k represent the number of rows and columns of the first low-rank matrix A, respectively, and k and m represent the number of rows and columns of the second low-rank matrix B, respectively, and k is less than the minimum value of n and m.

[0020] In some embodiments, low-rank decomposition algorithms may include, but are not limited to, singular value decomposition truncation algorithms, structured pruning and reconstruction algorithms, or training-based adapter fine-tuning algorithms. It should be noted that, for simplicity and ease of understanding, this paper uses the example of using a low-rank decomposition algorithm to decompose the original weight matrix M into two low-rank matrices (i.e., a first low-rank matrix A and a second low-rank matrix B). However, it is understood that low-rank decomposition algorithms can be used to decompose a large, high-rank original weight matrix M into two or more smaller low-rank matrices.

[0021] In some embodiments, k is much smaller than the minimum of n and m, i.e., k << min(n, m). For example, when n=m=1024, k can take the value 64 or 128. The specific value of k can be determined according to actual needs, as described below.

[0022] In some embodiments, before decomposing the original weight matrix M into a first low-rank matrix A and a second low-rank matrix B using a low-rank decomposition algorithm in step S104, method 100 may further include the following step: determining the value of k based on the computational and storage characteristics of the target hardware platform where the neural network model is deployed. Specifically, the optimal rank k value can be determined based on the computational and storage characteristics of the target hardware platform where the neural network model is deployed (e.g., memory bandwidth, number of computing units, cache size, etc.). Furthermore, determining the optimal rank k value can be further based on a user-preset tolerance for accuracy loss. The choice of k value directly affects the trade-off between computational load and model accuracy: the smaller the k value, the lower the computational load but the greater the potential accuracy loss; the larger the k value, the higher the accuracy but the less significant the acceleration effect. For example, for GPU platforms with powerful parallel computing capabilities, a larger k value can be chosen to maintain higher accuracy; for resource-constrained edge devices, a smaller k value can be chosen to obtain a higher speedup ratio. In this way, a balance can be achieved between model accuracy loss and computational acceleration gains.

[0023] In step S106, during the inference phase of the neural network model, for the input data x, the following operations are performed: matrix multiplication of the input data x with the first low-rank matrix A is performed to obtain an intermediate result h, i.e., h = A * x; and matrix multiplication of the intermediate result h with the second low-rank matrix B is performed to obtain the final output result y, i.e., y = B * h.

[0024] In some embodiments, during the inference phase of the neural network model, the decomposed first low-rank matrix A and second low-rank matrix B are loaded and stored to replace the original weight matrix M in the calculation. The input data x can be an input vector or a matrix. The final output result y obtained in step S106 is mathematically approximately equivalent to the calculation using the original weight matrix M, but physically achieves a significant reduction in computational load and data transfer volume.

[0025] From a computational complexity perspective, the decomposition operation at step S104 transforms a single dense matrix multiplication y = Mx with a complexity of O(nm) into two consecutive low-rank matrix multiplications with complexities of O(nk) and O(km): h = Ax, y = Bh. The total computational complexity is reduced from O(nm) to O(k*(n+m)). When k is sufficiently small, this reduction in computational complexity is equivalent to the effect achieved by sparsifying the original weight matrix M to the theoretical sparsity ratio of (1 - k / (min(n,m))).

[0026] Based on Method 100, hardware-friendly computational remapping can be achieved. Because the low-rank computational flow (x → A → h → B → y) is mapped to the target computing hardware, this computational pattern has a completely regular data layout and access pattern: matrices A and B are both dense small matrices, eliminating the irregular indexing, zero-value skipping, and data compression / decompression overhead required by traditional sparse computation. The computation process can be efficiently pipelined, and the size (k-dimensional) of the intermediate result h is much smaller than the original output dimension (m-dimensional), significantly reducing the occupation of on-chip cache or registers and the intermediate data write-back bandwidth.

[0027] Figure 2 A schematic diagram of an inference acceleration device 200 for a neural network model according to an embodiment of this application is shown. Figure 2 As shown, the device 200 may include a decomposition preprocessing module 210 and a matrix calculation engine 220.

[0028] The decomposition preprocessing module 210 can be used to: identify the original weight matrix M of the target network layer to be accelerated in the neural network model before deploying the neural network model, wherein the size of the original weight matrix M is n×m, where n and m are positive integers, representing the number of rows and columns of the original weight matrix M, respectively; and decompose the original weight matrix M into a first low-rank matrix A and a second low-rank matrix B using a low-rank decomposition algorithm, such that M ≈ A * B, wherein the size of the first low-rank matrix A is n×k, the size of the second low-rank matrix B is k×m, where k is a positive integer, n and k represent the number of rows and columns of the first low-rank matrix A, k and m represent the number of rows and columns of the second low-rank matrix B, respectively, and k is less than the minimum value of n and m.

[0029] In some embodiments, the target network layer can be a computationally intensive layer in a neural network model, such as, but not limited to, a fully connected feedforward layer in a Transformer model, or an attention output projection layer. The original weight matrix M can be a dense weight matrix.

[0030] In some embodiments, the low-rank decomposition algorithm may include, but is not limited to: singular value decomposition truncation algorithm, structured pruning and reconstruction algorithm, or training-based adapter fine-tuning algorithm. Similar to the combination above... Figure 1 As described in Method 100, a large high-rank original weight matrix M can be decomposed into two or more small low-rank matrices using a low-rank decomposition algorithm.

[0031] In some embodiments, k is much smaller than the minimum of n and m, i.e., k << min(n, m). The specific value of k can be determined according to actual needs. In some embodiments, the decomposition preprocessing module 210 can also be used to determine the value of k based on the computational and storage characteristics of the target hardware platform for deploying the neural network model. Specifically, similar to the above combination Figure 1 As described in method 100, the optimal rank k value can be determined based on the computational and storage characteristics (e.g., memory bandwidth, number of computing units, cache size, etc.) of the target hardware platform where the neural network model is deployed. Furthermore, the optimal rank k value can be further determined based on a user-preset tolerance for accuracy loss.

[0032] The matrix computation engine 220 can be used in the inference phase of a neural network model to perform the following operations on input data x: perform matrix multiplication of input data x with a first low-rank matrix A to obtain an intermediate result h, i.e., h = A * x; and perform matrix multiplication of intermediate result h with a second low-rank matrix B to obtain the final output result y, i.e., y = B * h.

[0033] In some embodiments, the matrix computing engine 220 is designed for a target hardware platform (such as an ASIC, FPGA, or custom AI accelerator), for example, it may be a general-purpose processor (such as a CPU or GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), or any other form of computing unit. The matrix computing engine 220 can be configured to efficiently perform consecutive small-scale dense matrix multiplications without requiring dedicated circuitry for sparse computation. The memory subsystem of the matrix computing engine 220 can be optimized for sequential access to small-scale dense matrices A and B. In other words, the matrix computing engine 220 can be optimized for dense small matrix multiplications, enabling the loading of a first low-rank matrix A and a second low-rank matrix B in sequential access mode without requiring index decoding or zero-value skipping logic necessary for sparse matrix computation.

[0034] In some embodiments, the apparatus 200 may further include a runtime scheduler. During the inference phase, the runtime scheduler is able to sequentially invoke the matrix calculation engine 220 to perform multiplication of the first low-rank matrix A and the second low-rank matrix B according to the model structure, and manage the temporary storage and transfer of intermediate results h.

[0035] Understandably, in combination Figure 1 Other technical details of the described inference acceleration method 100 for neural network models can be similarly applied. Figure 2 The inference acceleration device 200 for neural network models is described in detail herein.

[0036] Embodiments of this application may include a non-transitory computer-readable medium. The medium stores instructions that, when executed by one or more processors, cause the processors to perform the methods described herein.

[0037] Embodiments of this application may also include a computer program product. This computer program product includes instructions that, when executed by one or more processors, cause the processors to perform the methods described herein.

[0038] The inference acceleration method, apparatus, non-transitory computer-readable medium, and computer program product for neural network models according to embodiments of this application have at least the following beneficial effects: 1. Significantly Improved Computational Efficiency: By transforming a dense matrix multiplication with a complexity of O(n×m) into two matrix multiplications with complexities of O(n×k) and O(k×m) respectively, the total complexity is reduced to O(k×(n+m)). When k is much smaller than n and m, the computational load is significantly reduced, which is mathematically equivalent to applying high-sparseness structured sparsity to the original weight matrix. Therefore, by constructing a low-rank approximation mathematically, this application achieves a computational complexity reduction equivalent to that of extremely high sparsity pruning (O(n×m) → O(k×(n+m))), without needing to handle irregular computational graphs.

[0039] 2. Hardware Friendliness: Both the first and second low-rank matrices after decomposition are regular dense matrices. The computation process involves continuous regular matrix multiplication, making it suitable for efficient execution on existing highly optimized dense matrix computation units (such as GPU TensorCores and TPU matrix operation units). The target hardware can load data in sequential access mode, eliminating the need for complex index decoding or zero-value skipping logic required for sparse matrix computation. This avoids the complex control logic and memory access overhead required by traditional sparse computation, resulting in low hardware implementation cost and high energy efficiency.

[0040] 3. Significantly reduced data handling: The total data volume (n×k + k×m) for loading and storing the two low-rank matrices is much smaller than the data volume (n×m) for loading and storing the original weight matrix, reducing memory bandwidth pressure. Simultaneously, the dimension k of the intermediate result is smaller than the original output dimension m, reducing on-chip cache usage and on-chip / off-chip data write-back bandwidth.

[0041] 4. Accuracy and acceleration can be balanced: By selecting an appropriate k value based on the computing and storage characteristics of the target hardware platform, a significant speedup can be achieved within an acceptable range of accuracy loss, thus achieving a balance between model accuracy loss and computational acceleration gains, meeting the needs of different application scenarios.

[0042] Figure 3A schematic diagram is shown of a computing device 300 that can implement embodiments according to this application. This computing device 300 can be used to perform the various methods described above in conjunction with embodiments of this application, for example, in conjunction with... Figure 1 The method described. For example... Figure 3 As shown, computing device 300 may include bus 302 or other communication mechanism for transmitting information, and one or more processors 304 coupled to bus 302 for processing information. The one or more processors 304 may include, for example, one or more general-purpose microprocessors.

[0043] like Figure 3 As shown, in some embodiments, computing device 300 may further include main memory 306 coupled to bus 302, which is used to store information and instructions executed by one or more processors 304. For example, main memory 306 includes, but is not limited to, random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 306 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 304. When these instructions are stored in a storage medium accessible to one or more processors 304, they can cause computing device 300 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 308 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical disks or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0044] like Figure 3 As shown, in some embodiments, computing device 300 may further include one or more communication interfaces or network interfaces 310 coupled to bus 302. Network interface 310 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 310 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0045] The various processes, methods, and algorithms described in the preceding sections can be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by these code modules. The processes and algorithms can be implemented, partially or fully, in dedicated circuit systems.

[0046] When the functions disclosed herein are implemented as software functional units and sold or used as standalone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes instructions to cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0047] Some embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0048] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to as the “service system”) that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device capable of installing platform applications.

[0049] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations should be considered to fall within the scope of this disclosure. Additionally, certain methods or processes may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and the blocks or states associated with them may be executed in other suitable orders. For example, the described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0050] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). The algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly turn the computer into an executable function but may learn from training data to produce a predictive model of the executable function.

[0051] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementations that operate to perform one or more of the operations or functions described herein.

[0052] Similarly, the methods described herein can be implemented at least in part by a processor, where one or more specific processors are instances of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also operate to support the execution of related operations in a “cloud computing” environment or as the execution of related operations in a “Software as a Service” (SaaS) context. For example, at least some operations can be performed by a group of computers (as an example of a machine containing processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0053] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In some embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0054] Throughout this specification, multiple examples may be implemented as components, operations, or structures of a single example. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configurations may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as separate components. These and other variations, modifications, additions, and improvements also fall within the scope of this document.

[0055] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0056] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0057] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0058] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A method for accelerating inference in neural network models, characterized in that, include: Before deploying the neural network model, the original weight matrix of the target network layer to be accelerated in the neural network model is identified, wherein the size of the original weight matrix is ​​n×m, where n and m are positive integers, representing the number of rows and columns of the original weight matrix, respectively; The original weight matrix is ​​decomposed into a first low-rank matrix and a second low-rank matrix using a low-rank decomposition algorithm. The first low-rank matrix has a size of n×k, and the second low-rank matrix has a size of k×m, where k is a positive integer, n and k represent the number of rows and columns of the first low-rank matrix, respectively, and k and m represent the number of rows and columns of the second low-rank matrix, respectively, with k being less than the minimum of n and m. During the inference phase of the neural network model, the following operations are performed on the input data: Perform matrix multiplication of the input data with the first low-rank matrix to obtain an intermediate result; and Perform a matrix multiplication between the intermediate result and the second low-rank matrix to obtain the final output result.

2. The method according to claim 1, characterized in that, The target network layer includes: a fully connected feedforward network layer in the Transformer model, or an attention output projection layer.

3. The method according to claim 1, characterized in that, The low-rank decomposition algorithms include: singular value decomposition truncation algorithm, structured pruning and reconstruction algorithm, or training-based adapter fine-tuning algorithm.

4. The method according to claim 1, characterized in that, Before using the low-rank decomposition algorithm to decompose the original weight matrix into the first low-rank matrix and the second low-rank matrix, the method further includes: The value of k is determined based on the computing and storage characteristics of the target hardware platform where the neural network model is deployed.

5. An inference acceleration device for neural network models, characterized in that, include: The decomposition preprocessing module is used for: Before deploying the neural network model, the original weight matrix of the target network layer to be accelerated in the neural network model is identified, wherein the size of the original weight matrix is ​​n×m, where n and m are positive integers, representing the number of rows and columns of the original weight matrix, respectively; and The original weight matrix is ​​decomposed into a first low-rank matrix and a second low-rank matrix using a low-rank decomposition algorithm. The first low-rank matrix has a size of n×k, and the second low-rank matrix has a size of k×m, where k is a positive integer, n and k represent the number of rows and columns of the first low-rank matrix, respectively, and k and m represent the number of rows and columns of the second low-rank matrix, respectively, with k being less than the minimum of n and m. A matrix calculation engine is used during the inference phase of the neural network model to perform the following operations on the input data: Perform matrix multiplication of the input data with the first low-rank matrix to obtain an intermediate result; and Perform a matrix multiplication between the intermediate result and the second low-rank matrix to obtain the final output result.

6. The apparatus according to claim 5, characterized in that, The target network layer includes: a fully connected feedforward network layer in the Transformer model, or an attention output projection layer.

7. The apparatus according to claim 5, characterized in that, The low-rank decomposition algorithms include: singular value decomposition truncation algorithm, structured pruning and reconstruction algorithm, or training-based adapter fine-tuning algorithm.

8. The apparatus according to claim 5, characterized in that, The decomposition preprocessing module is also used for: The value of k is determined based on the computing and storage characteristics of the target hardware platform where the neural network model is deployed.

9. A non-transitory computer-readable medium storing instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-4.

10. A computer program product, comprising instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-4.

Images