Mama accelerator and application method thereof

Abstract

The invention relates to a Mama accelerator and an application method thereof. The Mama accelerator comprises a systolic array and a random access memory interface, wherein the systolic array receives off-chip instructions and data; the systolic array comprises a weight cache, a feature cache, a control module, a post-processing unit and a hybrid computing array composed of a plurality of computing units, wherein the weight cache is used for storing weight data; the feature cache is used for storing input data; the control module is used for controlling interaction between off-chip data and on-chip data through the random access memory interface; the post-processing unit is used for calculating an activation function and carrying out normalization and bias processing on the accumulated output data; the calculation unit is used for executing a multiply-accumulate operation, an element multiplication operation and an element addition operation according to different calculation modes and weights. Therefore, multiply-accumulate calculation and element multiply-add calculation can be simultaneously realized in the calculation unit, the calculation efficiency of the Mama network is remarkably improved, and the energy consumption and the calculation delay are reduced.

Description

Technical Field

[0001] This application relates to the field of integrated circuit technology, and particularly to a Mamba accelerator and its application method. Background Art

[0002] With the development of computer technology, various large language models have emerged. Among them, large language models based on Transformer have achieved great success in recent years due to their attention mechanism. However, they show the disadvantage of low efficiency on long sequences. Recently, architectures based on state space models have shown great potential in various fields, and the Mamba model is one of the most successful variants, featuring fast inference and linear scaling of sequence length.

[0003] In traditional technologies, accelerating the Mamba model still faces many problems. Among them, the main problem is that the Mamba model contains a large number of non-matrix multiplication calculations (such as element-wise multiplication and element-wise addition), and these calculations are difficult to be supported on existing Central Processing Unit / Processor (CPU) or Graphics Processing Unit (GPU). Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a Mamba accelerator and its application method that can simultaneously implement multiply-accumulate calculations and element-wise multiply-add calculations in a computing unit, significantly improve network computing efficiency, and reduce energy consumption and latency.

[0005] In a first aspect, this application provides a Mamba accelerator, which includes: a systolic array and a random memory interface. The systolic array receives off-chip instructions and data based on the random memory interface. The systolic array includes: a weight cache, a feature cache, a control module, a post-processing unit, and a hybrid computing array composed of multiple computing units, where:

[0006] The weight cache is used to store weight data;

[0007] The feature cache is used to store input data;

[0008] The control module is used to control the interaction between off-chip data and on-chip data through the random memory interface;

[0009] The post-processing unit is used to calculate the activation function and perform normalization and bias processing on the accumulated output data;

[0010] The computing unit is configured to perform multiply-accumulate operations, element-wise multiplication operations, and element-wise addition operations according to different computing modes and weights.

[0011] In one embodiment, the random access memory interface includes: a double data rate synchronous dynamic random access memory (DDR) interface.

[0012] In one embodiment, the computing unit includes: a mode controller, a weight register, an input register, a multiplier, a shifter, an adder, a state space cache, and a plurality of gating units, where:

[0013] The mode controller is configured to control the computing mode of the computing unit;

[0014] The weight register is configured to store the weights for computing;

[0015] The input register is configured to store the input data for computing;

[0016] The multiplier is configured to perform multiplication operations;

[0017] The adder is configured to perform addition operations;

[0018] The shifter is configured to perform shift operations;

[0019] The state space cache is configured to store and / or retrieve data for multiplication operations and addition operations in parallel;

[0020] The gating units are configured to complete the switching between different computing modes.

[0021] In one embodiment, the computing unit further includes: an accumulation register, which is configured to store the accumulation result and pass the accumulation result to the next computing unit.

[0022] In one embodiment, the state space cache includes: at least two multiplication blocks and addition blocks, and each multiplication block and each addition block complete the storage and retrieval of data based on the gating units.

[0023] In one embodiment, the mode controller is specifically configured to: control the computing mode of the computing unit to switch between a convolution mode, a fully connected mode, an element-wise multiplication mode, and an element-wise addition mode;

[0024] Wherein, the computing paths of the input register, the weight register, and the accumulation register form the computing paths of the convolution layer and the fully connected layer in the Mamba network; the multiply-accumulate calculation in the Mamba network is completed by the multiplier and the adder.

[0025] In one embodiment, the input register and weight register in the computing unit configure the input data and weight data through on-chip storage;

[0026] The output cache of the computing unit configures the output data through on-chip storage;

[0027] The input register and weight register in the computing unit interact with off-chip DDR under the control of the control module to complete data transmission and storage.

[0028] In one embodiment, the gating unit includes: a first gating unit, a second gating unit, a third gating unit, a fourth gating unit, a fifth gating unit, and a sixth gating unit;

[0029] The first input terminal of the multiplier is connected to the input register, the second input terminal of the multiplier is connected to the weight register through the first gating unit, and the second input terminal of the multiplier is connected to the first multiplication block of the state space cache through the second gating unit;

[0030] After the multiplier completes the multiplication operation, one path of the multiplication operation result is transmitted to the first input terminal of the adder through the third gating unit, and the other path of the multiplication operation result is stored in the second multiplication block of the state space cache, or any adder block, through the shifter and the fourth gating unit; the first multiplication block and the second multiplication block are different;

[0031] The first input terminal of the adder is connected to the state space cache through the third gating unit and the fourth gating unit; the second input terminal of the adder is connected to the previous computing unit through the fifth gating unit, and the second input terminal of the adder is connected to the adder block of the state space cache through the fifth gating unit and the fourth gating unit;

[0032] After the adder completes the addition operation, one path of the addition operation result is stored in the multiplication block, or adder block, of the state space cache through the shifter and the fourth gating unit; the other path of the addition operation result is transmitted to the accumulation register;

[0033] The state space cache is connected to the DDR interface and the output cache through the sixth gating unit.

[0034] In a second aspect, the present application also provides an application method of a Mamba accelerator, which uses the Mamba accelerator as described in any one of the first aspects to perform multiply-accumulate, and / or element-wise multiply-add calculations according to the calculation process of the Mamba model.

[0035] In a third aspect, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the method steps described in the second aspect are implemented.

[0036] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method steps described in the second aspect are implemented.

[0037] In a fifth aspect, the present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method steps described in the second aspect are implemented.

[0038] For the above Mamba accelerator and its application method, by setting a systolic array and a random access memory interface, the systolic array receives off-chip instructions and data based on the random access memory interface; the systolic array includes: a weight cache, a feature cache, a control module, a post-processing unit, and a hybrid computing array composed of multiple computing units, where: the weight cache is used to store weight data; the feature cache is used to store input data; the control module is used to control the interaction between off-chip data and on-chip data through the random access memory interface; the post-processing unit is used to calculate the activation function and perform normalization and bias processing on the accumulated output data; the computing unit is used to perform multiply-accumulate operations, element-wise multiplication operations, and element-wise addition operations according to different computing modes and weights. Thus, it is possible to simultaneously implement multiply-accumulate calculations and element-wise multiply-add calculations in the computing unit, and significantly improve the computing efficiency of the Mamba network, reduce energy consumption and computing latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other related drawings can be obtained without creative efforts based on these drawings.

[0040] Figure 1 It is a schematic diagram of the overall architecture of an accelerator in an embodiment;

[0041] Figure 2 It is a calculation flowchart of an accelerated Mamba neural network in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] In order to make the objectives, technical solutions, and advantages of the present application more clearly understood, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] Exemplarily, an embodiment of the present application provides a Mamba accelerator, which includes: a systolic array and a random access memory interface. The systolic array receives off-chip instructions and data based on the random access memory interface. The systolic array includes: a weight cache, a feature cache, a control module, a post-processing unit, and a hybrid computing array composed of multiple computing units, where: the weight cache is used to store weight data; the feature cache is used to store input data; the control module is used to control the interaction between off-chip data and on-chip data through the random access memory interface; the post-processing unit is used to calculate the activation function and perform normalization and bias processing on the accumulated output data; the computing unit is used to perform multiply-accumulate operations, element-wise multiplication operations, and element-wise addition operations according to different computing modes and weights.

[0044] Among them, the computing unit can adopt 16x16 matrix operations, but the embodiments of the present application are not limited thereto.

[0045] Among them, the post-processing unit can meet the requirements of the Mamba network activation function, normalization, and bias calculation.

[0046] Exemplarily, the random access memory interface includes: a double data rate synchronous dynamic random access memory (DDR) interface.

[0047] Exemplarily, the computing unit includes: a mode controller, a weight register, an input register, a multiplier, a shifter, an adder, a state space cache, and several gating units, where: the mode controller is used to control the computing mode of the computing unit; the weight register is used to store the weights for calculation; the input register is used to store the input data for calculation; the multiplier is used to perform multiplication operations; the adder is used to perform addition operations; the shifter is used to perform shift operations; the state space cache is used to store and / or retrieve data for multiplication and addition operations in parallel; the gating unit is used to complete the switching between different computing modes.

[0048] In this embodiment, each computing unit includes a mode controller to control the computing mode, a weight register to store the computing weights, an input register to store the computing inputs, and an accumulation register to store the accumulation results and pass them to the next unit. The hybrid mode calculation is implemented through a multiplier, a shifter, and an adder.

[0049] Exemplarily, the computing unit further includes: an accumulation register, which is used to store the accumulation result and transfer the accumulation result to the next computing unit.

[0050] Exemplarily, the state space cache includes: at least two multiplication blocks and addition blocks, and each multiplication block and each addition block complete the storage and retrieval of data based on the gating unit.

[0051] Among them, at least two multiplication blocks and at least two addition blocks can realize the parallel storage and retrieval of element multiplication and addition.

[0052] Exemplarily, the mode controller is specifically used to: control the computing mode of the computing unit to switch between the convolution mode, the fully connected mode, the element multiplication mode, and the element addition mode; wherein, the computing paths of the input register, the weight register, and the accumulation register form the computing paths of the convolution layer and the fully connected layer in the Mamba network; the multiply-accumulate calculation in the Mamba network is completed by the multiplier and the adder.

[0053] In this embodiment, the computing unit can flexibly switch the computing mode through the mode controller, so as to meet the operator requirements according to the computing process of the Mamba network.

[0054] Exemplarily, the input register and the weight register in the computing unit configure the input data and the weight data through on-chip storage; the output cache of the computing unit configures the output data through on-chip storage; the input register and the weight register in the computing unit interact with the off-chip DDR under the control of the control module to complete data transmission and storage. Thereby, the computing efficiency of the Mamba network can be significantly improved.

[0055] Exemplarily, the gating unit includes: a first gating unit, a second gating unit, a third gating unit, a fourth gating unit, a fifth gating unit, and a sixth gating unit; the first input end of the multiplier is connected to the input register, the second input end of the multiplier is connected to the weight register through the first gating unit, and the second input end of the multiplier is connected to the first multiplication block of the state space cache through the second gating unit.

[0056] Optionally, after the multiplier completes the multiplication operation, one path of the multiplication operation result is transmitted to the first input end of the adder through the third gating unit, and the other path of the multiplication operation result is stored in the second multiplication block of the state space cache or any addition block through the shifter and the fourth gating unit; the first multiplication block and the second multiplication block are different.

[0057] In this embodiment, one input of the element multiplication comes from the input register, and the other input comes from any multiplication block in the state space cache. After the multiplication calculation is controlled by the gating unit, a shift is performed and the result is stored in another multiplication block or addition block in the state space cache for the next calculation.

[0058] Exemplarily, the first input terminal of the adder is connected to the state space cache through the third gating unit and the fourth gating unit; the second input terminal of the adder is connected to the previous calculation unit through the fifth gating unit, and the second input terminal of the adder is connected to the addition block in the state space cache through the fifth gating unit and the fourth gating unit.

[0059] Optionally, after the adder completes the addition operation, one result of the addition operation is stored in the multiplication block or addition block in the state space cache through the shifter and the fourth gating unit; the other result of the addition operation is transmitted to the accumulation register.

[0060] In this embodiment, the two inputs of the element addition respectively come from two addition blocks in the state space cache. After passing through the gating unit, the element addition is completed, and then it is stored in the multiplication block or addition block in the state space cache through the shifter for the next calculation.

[0061] Optionally, the state space cache is connected to the DDR interface and the output cache through the sixth gating unit.

[0062] It should be noted that in the above embodiment, the six gating units complete the mixed calculation of multiple modes and the storage and retrieval of data in the state space cache.

[0063] When analyzing the Mamba model in this embodiment, for different calculation modes, the mode controller will enable the corresponding calculation mode in the calculation unit. In addition, data will be transmitted through the DDR interface to the on-chip cache of the calculation array and the state space cache of the calculation unit. When the calculation starts, for the multiply-accumulate calculation, the array output will be transmitted to the output cache for the next calculation; for the element multiply-add calculation, the intermediate result will be stored in the state space cache of the calculation unit as the data source for the next element calculation. Of course, the state space cache can also transmit data to the output cache through the gating unit.

[0064] In this embodiment, in view of the characteristics that element vector multiplication and addition account for a large proportion in the Mamba network and matrix calculation is inefficient, a hybrid computing systolic array is designed to optimize the network computing efficiency, which can effectively reduce latency and improve computing throughput compared with the CPU and GPU architectures. In this embodiment, the on-chip storage distribution is also used to effectively reduce the interaction between off-chip and on-chip data and reduce the computing power consumption of large model networks. In addition, through the analysis of the Mamba model and hardware configuration, end-to-end Mamba network computing can be completed.

[0065] Exemplarily, as Figure 1 shown, a schematic diagram of the overall architecture of an accelerator is provided. The accelerator includes a 16x16 computing unit, a feature cache, a weight cache, an accumulation cache, an output cache, a post-processing unit, a control module, a DDR interface, etc. Among them, each computing unit includes: a mode controller, a weight register, an input register, an accumulation register, a multiplier, a shifter, an adder, six gating units, and a state space cache, and each state space cache contains two multiplication blocks and two addition blocks.

[0066] In this embodiment, for the calculation of the fully connected layer and convolutional layer of the Mamba network, Figure 1 the accelerator stores the input of the network through an input buffer, while the weight values are stored in the weight cache of the computing array. After the computing array obtains the multiply-accumulate value, the calculation result is accumulated through the accumulation module and then the output result is stored in the output buffer. The output buffer can transmit the output to the post-processing unit for activation function, normalization and bias operations. In addition, the result can also be transmitted to the input buffer as the input result of the next operation instead of being transmitted to the off-chip DDR to reduce the data transfer power consumption.

[0067] In this embodiment, for the element multiplication and element addition operations of the Mamba network, the input is first stored through the input buffer and transmitted to the state space cache of the computing unit through the off-chip DDR. At the beginning of the element multiplication operation, the input register of the computing unit is multiplied by the output of any multiplication block. After obtaining the output, no addition processing is performed, but a shift operation is performed through the shifter, and the output returns to any block of the state space cache for the next element multiplication or element addition. For the element addition operation, the adder and shifter of the computing unit are required. Among them, the input of the adder comes from the two addition blocks of the state space cache, and the shift operation is the same as that in the element multiplication. Finally, the output also exists in the state space cache.

[0068] It should be noted that the output of the state space cache can be transmitted to the output cache through the gating unit, so as to realize the flexible switching of these four operations in the computing array.

[0069] Exemplarily, as Figure 2As shown, an accelerated Mamba neural network calculation flowchart is provided, and its main process is as follows: The network input first passes through a fully connected layer, and the output is copied into two parts. One part is used as the residual after passing through an activation function once and performs an element-wise multiplication operation with the output of the other path. The other part first performs a convolution calculation, then an activation function, and then enters the state space model calculation. The state space model mainly consists of element-wise multiplication, element-wise addition, and fully connected layer calculations. The output of the state space model performs an element-wise multiplication operation and then passes through a fully connected layer calculation to obtain the output.

[0070] In this embodiment, by analyzing the Mamba network and configuring the accelerator, the end-to-end Mamba network calculation can be efficiently completed.

[0071] Exemplarily, the present application further provides an application method of a Mamba accelerator, which can use the Mamba accelerators in the above various embodiments and perform multiply-accumulate and / or element-wise multiply-add calculations according to the calculation process of the Mamba model.

[0072] In an exemplary embodiment, a computer device is provided, and the computer device can be a server. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it realizes an application method of a Mamba accelerator.

[0073] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are realized.

[0074] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are realized.

[0075] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps in the above method embodiments.

[0076] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0077] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, Resistive Random Access Memory (ReRAM), Magnetoresistive Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene memory, etc. Volatile memory can include Random Access Memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), etc. The database involved in the embodiments provided in this application can include at least one of a relational database and a non-relational database. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, Artificial Intelligence (AI) processors, etc., and are not limited thereto.

[0078] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0079] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application should be subject to the appended claims.

Claims

1. A Mamba accelerator, characterized in that: The Mamba accelerator includes: a systolic array and a random access memory interface, wherein the systolic array receives off-chip instructions and data based on the random access memory interface; the systolic array includes: a weight cache, a feature cache, a control module, a post-processing unit, and a hybrid computing array composed of multiple computing units, wherein: The weight cache is used to store weight data; The feature cache is used to store input data; The control module is used to control the interaction between off-chip data and on-chip data through the random access memory interface; The post-processing unit is used to calculate the activation function, and perform normalization and bias processing on the accumulated output data; The computing unit is used to perform multiplication-accumulation operations, element-wise multiplication operations, and element-wise addition operations according to different computing modes and weights.

2. The Mamba accelerator according to claim 1, characterized in that: The random access memory interface includes: a double rate synchronous dynamic random access memory DDR interface.

3. The Mamba accelerator according to claim 1, characterized in that: The computing unit includes: a mode controller, a weight register, an input register, a multiplier, a shifter, an adder, a state space cache, and a plurality of gating units, wherein: The mode controller is used to control the calculation mode of the calculation unit; The weight register is used to store the calculated weight; The input register is used to store the input data of the calculation; The multiplier is used to perform a multiplication operation; The adder is used to perform addition operation; The shifter is used to perform a shift operation; The state space cache is used to store and / or retrieve data of multiplication and addition operations in parallel; The gating unit is used to complete the switching between different computing modes.

4. The Mamba accelerator according to claim 3, characterized in that: The calculation unit further includes: an accumulation register, which is used to store an accumulation result and transfer the accumulation result to the next calculation unit.

5. The Mamba accelerator according to claim 3, characterized in that: The state space cache includes: at least two multiplication blocks and an addition block, each multiplication block and each addition block completes the storage and retrieval of data based on a gating unit.

6. The Mamba accelerator according to claim 3, characterized in that: The mode controller is specifically used to: control the computing mode of the computing unit to switch between a convolution mode, a full connection mode, an element multiplication mode, and an element addition mode; Among them, the calculation path of the input register, the weight register and the accumulation register constitutes the calculation path of the convolution layer and the fully connected layer in the Mamba network; the multiplication and accumulation calculation in the Mamba network is completed by the multiplier and the adder.

7. The Mamba accelerator according to claim 3, characterized in that: The input registers and weight registers in the computing unit configure input data and weight data through on-chip storage; The output cache of the computing unit configures output data through on-chip storage; The input register and the weight register in the computing unit interact with the off-chip DDR under the control of the control module to complete data transmission and storage.

8. The Mamba accelerator according to claim 3, characterized in that: The gating unit comprises: a first gating unit, a second gating unit, a third gating unit, a fourth gating unit, a fifth gating unit and a sixth gating unit; The first input terminal of the multiplier is connected to the input register, the second input terminal of the multiplier is connected to the weight register through a first gating unit, and the second input terminal of the multiplier is connected to the first multiplication block of the state space cache through a second gating unit; After the multiplier completes the multiplication operation, one multiplication operation result is transmitted to the first input terminal of the adder through the third gating unit, and another multiplication operation result is stored in the second multiplication block of the state space cache or any addition block through the shifter and the fourth gating unit; the first multiplication block and the second multiplication block are different; The first input end of the adder is connected to the state space cache through the third gating unit and the fourth gating unit; the second input end of the adder is connected to the previous calculation unit through the fifth gating unit, and the second input end of the adder is connected to the addition block of the state space cache through the fifth gating unit and the fourth gating unit; After the adder completes the addition operation, one path of the addition operation result is stored in the multiplication block or the addition block of the state space cache through the shifter and the fourth gating unit; the other path of the addition operation result is transmitted to the accumulation register; The state space buffer is connected to the DDR interface and the output buffer through the sixth gating unit.

9. A method for applying a Mamba accelerator, characterized in that: Using the Mamba accelerator as claimed in any one of claims 1 to 8, multiplication and accumulation and / or element multiplication and accumulation calculations are performed according to the calculation process of the Mamba model.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to claim 9 are implemented.

Images