Universal configurable Transformer neural network hardware accelerator and its implementation method

Abstract

The present invention discloses a hardware accelerator of a universal configurable Transformer neural network and an implementation method thereof, comprising a control unit, a computing unit and a DMA unit, wherein the control unit monitors the status of the computing unit and the DMA unit and decides whether to send instructions; the computing unit reads data from a fixed sRAM position and outputs it to a fixed position for the next calculation, and when it detects that the calculation command queue is empty, the calculation result is output to an sRAM storage position shared with the DMA unit; the DMA unit interacts with the computing unit through the sRAM, puts the calculation data into a loading target address stack brake position, and stores the shared data of the sRAM storage position back to a specified position according to the storage address queue. The present invention achieves a high input data reuse rate and reduces the memory access demand by setting the core operation and the secondary operation; reduces the instruction fetching and decoding time of a large number of instructions by filling the configuration table with the instruction stream, and improves the overall execution efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network models, and in particular relates to a hardware accelerator of a universal configurable Transformer neural network and an implementation method thereof. Background Art

[0002] With the rapid development of large Transformer-based language models, the scale of their parameters continues to expand, and the demand for computing power has therefore increased. In order to meet the computing power requirements of large models, many teams have begun to focus on enhancing hardware support for Transformers, and some researchers have begun to turn their attention to more general chip architectures. However, due to the large dimensions of matrices in large models, the dedicated architectures proposed in the past often require a large number of instructions to complete the calculation of a matrix, and the calculation instructions repeatedly access the same set of data, resulting in a large number of memory access requirements; and specialized ASIC chips are not flexible enough to cope with future changes in model structure.

[0003] Chinese Patent CN114897133A-A universal configurable Transformer hardware accelerator and its implementation method, which can meet all the computing requirements of the entire network layer through the computing engine array, Softmax operation module and LayerNorm operation module; the computing engine array improves the computing parallelism in the accelerator, speeding up the calculation speed while improving the utilization rate of the data loaded into the internal cache of the accelerator; through the various storage areas of the on-chip cache module, the feature data, bias matrix data, weight data and parameter data can be stored separately, improving the efficiency of subsequent data reading, reducing the delay of the memory access process, and improving the computing efficiency; the parallelism of the computing engine, the number of computing engines, and the computing parallelism of the nonlinear layer can all be configured through the control module, and the model deployment and implementation can be completed according to different computing resources, which can be widely used in the field of neural network model technology. However, the patent design is aimed at the calculation of Transformer, and its architecture design includes design components centered around the three modules of matrix multiplication, Softmax, and LayerNorm, resulting in insufficient flexibility.

[0004] Chinese Patent CN117391096A-A Vision Transformer encoder hardware accelerator based on FPGA and its calculation method, including: processor module, DDR memory module, data scheduling module, matrix storage module, residual addition module, first matrix multiplication module, GeLU operation module, second matrix multiplication module, third matrix multiplication module, softmax calculation module; the processor module is used to dynamically schedule the accelerator and perform auxiliary calculations, while the operation array adopts a data parallel calculation strategy with fixed output, strong network scale adaptability, high input and output data reuse, and efficient on-chip and off-chip data transfer. This hardware accelerator is a dedicated accelerator for the Vision Transformer neural network, which can effectively improve the computing speed and efficiency of the Vision Transformer neural network and effectively reduce computing power consumption. Dedicated hardware acceleration modules face the difficulty of insufficient flexibility.

[0005] Chinese Patent CN117010460A-A Swin-Transformer reasoning hardware accelerator, including a three-layer storage hierarchy consisting of external storage, static random access memory SRAM and register Reg, two quantization units, an outer product matrix multiplication array and three nonlinear computing units; the invention uses the unique computing characteristics of Swin Transformer to design a dedicated hardware acceleration architecture for application on the end side, and verifies it on the FPGA platform. The energy efficiency of the accelerator is 15.14 times higher than that of the NVIDIA RTX 2080Ti GPU, and 2.02 to 3.11 times higher than that of the existing Transformer accelerator. The chip is a dedicated accelerator that uses a customized processing unit array and lacks flexibility.

[0006] Chinese Patent CN115545177A - A Transformer hardware accelerator based on FPGA; it includes a data preprocessing module, a computing unit and a controller; the computing unit includes a matrix processing unit and a nonlinear processing unit; the nonlinear processing unit includes an activation function unit, a layer normalization module and a Softmax module; the data preprocessing module is deployed on the CPU, the controller and the computing unit are deployed on the FPGA, the data preprocessing module preprocesses the text to be processed, converts it into an integer sequence and inputs it into the FPGA, the data is scheduled in the computing unit to complete the calculation process, and finally the output result is obtained. This invention only needs to use fewer logical resources during calculation, and can get the result in a shorter time, while not affecting the final calculation accuracy. This invention basically refers to the structural design of Transformer and lacks support for other models.

[0007] Chinese Patent CN114118344A-Hardware accelerator applied to Transformer neural network and its calculation method, the hardware accelerator includes: data storage module, pulsation array module, bias addition module, dot product attention function operation module, activation function operation module, residual addition module, layer normalization function operation module and control module. The pulsation array module includes multiple basic operation units for performing matrix calculations, and the control module is used to control the calculation process of the hardware accelerator. Based on the pulsation array module that can perform all matrix calculations in the Transformer neural network, the above-mentioned hardware accelerator can effectively improve the calculation speed and efficiency of the Transformer neural network. This application is different from its technical solution. Summary of the invention

[0008] In view of the above problems, the present invention provides a universal configurable Transformer neural network hardware accelerator and an implementation method thereof, aiming to support large-scale matrix and vector operations.

[0009] According to a first aspect of an embodiment of the present disclosure, a hardware accelerator of a general configurable Transformer neural network is provided, comprising a control unit, a computing unit and a DMA unit, wherein:

[0010] The control unit includes a controller, an instruction cache and a scheduler, wherein the controller stores instructions to be executed, the instruction cache is used to cache instructions, and the scheduler is used to monitor the status of the calculation unit and the DMA unit and decide whether to send instructions;

[0011] The calculation unit includes a matrix multiplier and a special calculator. When the matrix multiplier and the special calculator perform calculation tasks, they both read data from a fixed SRAM location and output them to a fixed location for the next calculation. When it is detected that the calculation command queue is empty, the calculation result is output to an SRAM storage location shared with the DMA unit.

[0012] The DMA unit interacts with the computing unit through the sRAM. The DMA unit puts the computing data into the loading target address stack braking position. After the computing unit completes the calculation, the DMA unit stores the shared data of the sRAM storage position back to the designated position according to the storage address queue.

[0013] According to a second aspect of an embodiment of the present disclosure, a method for implementing a hardware accelerator of a general configurable Transformer neural network is provided, wherein the hardware accelerator is controlled by filling a configuration table with an instruction stream, the control unit disassembles the content of the configuration table and dispatches it to the computing unit and the DMA unit, and the computing unit and the DMA unit generate a command queue according to the dispatched configuration table and execute it.

[0014] In some embodiments, the computing unit implements computing by combining core computing with secondary computing. When the core computing configuration table is set to empty, the core computing is skipped and secondary computing is performed.

[0015] In some embodiments, when the instruction stream is a composite computing instruction, the content of the computing unit configuration table is split into instructions for different computing.

[0016] In some embodiments, the computing unit performs calculations according to computing instructions. When the instruction queue is empty, it re-reads the configuration information of the next group, forms a new instruction queue and starts executing the instructions of the next group. When all groups in the configuration table have been executed, the computing unit sends a feedback signal back to the control unit, and communicates with the DMA unit to clear the configuration table information, reset the signal and prepare to process the next configuration table.

[0017] In some embodiments, the DMA unit disassembles the content of the configuration table into three address queues and sets a transposition flag register; the three address queues are respectively a source address queue of a load instruction, a target address queue of a load instruction, and a target address queue of a storage instruction; the DMA unit moves the data of the source address to the target address according to a first-in-first-out rule based on whether the target address memory is full; based on the calculation unit placing the calculation result in an sRAM space shared with the DMA unit, the DMA unit moves the data from the shared sRAM space to the sRAM space pointed to by the target address of the storage instruction according to whether the data exists and the order of the storage address queues.

[0018] In some embodiments, the configuration table ID number of the DMA unit and the configuration table ID number of the computing unit start to run after confirming that they are consistent with each other.

[0019] In some embodiments, when loading data, the DMA unit continues to carry out the transfer until it detects that the memory at the target address is full, and then transfers the data to a temporary storage location; when the DMA unit stores a data back, it will try again to transfer the temporarily stored data to the target address, and if successful, it will initiate the next memory access request and continue the transfer work;

[0020] When the address queue of the DMA unit is empty, re-read the configuration information of the next group, form a new instruction queue and start executing the address of the next group;

[0021] When all group configuration tables in the DMA unit configuration table are executed, the DMA unit stops running and sends a feedback signal to the control unit, and communicates with the computing module to eliminate the current configuration information, set the signal and wait for the next configuration table.

[0022] In some embodiments, the content of the configuration table is mainly configuration matrix information, and matrix multiplication and vector multiplication are organized as core calculations. According to the number of core calculations, the same configuration table is divided into multiple groups. One group is responsible for providing the information required for a core operation and a secondary calculation. The data required in the group include the dimension of the matrix, the source address of the load instruction, the target address of the load instruction, the target address of the storage instruction, the core calculation type and the secondary calculation type.

[0023] In some embodiments, instruction configuration data of the self-attention module and the feed-forward network are integrated into the ROM in the hardware accelerator.

[0024] In some embodiments, when computing a Transformer neural network, the implementation process includes:

[0025] The Transformer neural network is regarded as three configuration tables of self-attention, inter-layer normalization and feed-forward neural network. Each residual block is completed by a configuration table that does not configure the core operation but configures the secondary operation;

[0026] The self-attention configuration table sets matrix multiplication as the core operation. The first three times are matrix multiplications without configuring secondary operations. The fourth time configures ReduceMax, Mask, and SoftMax secondary operations in sequence. The fifth time is matrix multiplication without configuring secondary operations.

[0027] After the respective attention configuration tables are completed, an additional matrix multiplication is configured, and no secondary operations are configured;

[0028] The inter-layer normalization configuration table includes multiple groups of operations;

[0029] The feedforward neural network includes two sets of matrix multiplications, the first set of secondary operations is vector Gelu, and the second set of core calculations is matrix multiplication;

[0030] During execution, the form information is sent to the corresponding execution hardware, which checks whether the configuration table is completed and starts execution.

[0031] The disclosed embodiment provides a general configurable Transformer neural network hardware accelerator and an implementation method thereof. The general configurable Transformer neural network hardware accelerator achieves a high input data reuse rate, reduces memory access requirements, and avoids repeated memory access by setting core operations and secondary operations, and reduces the instruction fetch and decoding time of a large number of instructions by filling the configuration table with instructions, thereby improving the overall execution efficiency and providing a new solution for balancing versatility and high performance. In summary, the present invention has good data reuse and good scalability.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present invention and, together with the description, are used to explain the principles of the present invention.

[0034] Figure 1 is a diagram of the hardware accelerator architecture of a universal configurable Transformer neural network in an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of the classification of computing units in an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of a hardware accelerator implementation of a universal configurable Transformer neural network in an embodiment of the present invention;

[0037] Figure 4 This is an example of the calculation process in the universal configurable Transformer in the embodiment of the present invention;

[0038] Figure 5 Schematic diagram of the decoding layer configuration of the Transformer neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0040] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0041] Reference Figure 1 , Figure 2, a universal configurable Transformer neural network hardware accelerator, comprising a control unit, a computing unit and a DMA unit, wherein the control unit comprises a controller, an instruction cache and a scheduler, the controller stores instructions to be executed, the instruction cache is used to cache instructions, and the scheduler is used to monitor the status of the computing unit and the DMA unit and decide whether to send instructions;

[0042] The calculation unit includes a matrix multiplier and a special calculator. When the matrix multiplier and the special calculator perform calculation tasks, they both read data from a fixed SRAM location and output it to a fixed location for the next calculation. When it is detected that the calculation command queue is empty, the calculation result is output to an SRAM storage location shared with the DMA unit.

[0043] The DMA unit interacts with the computing unit through the sRAM. The DMA unit puts the computing data into the loading target address stack braking position. After the computing unit completes the calculation, the DMA unit stores the shared data of the sRAM storage position back to the designated position according to the storage address queue.

[0044] like Figure 1 As shown, the FTC (Fusion Transformer Core, generalized Transformer core) architecture of the embodiment is an architecture of a computing core, which can perform a part of the computing tasks under the control of the upper-level controller and become a part of the computing network; it can also run independently, accept input signals (instruction streams), perform calculations, and output results. The basic components of the FTC architecture include a control unit, a computing unit, and a DMA unit, as well as a control bus and a storage system. The FTC can be connected to a controller through an internal bus to realize parallel computing of multiple FTCs. The functions of the external controller include monitoring whether each FTC is empty and distributing the computing tasks. In hardware implementation, the FTC is composed of a control module, a computing module, a DMA module, a bus, and on-chip storage resources. Multiple FTCs can be controlled by a higher-level configuration system, and the configuration system splits the computing tasks to the FTCs for calculation.

[0045] like Figure 2 As shown in the figure, the calculation unit includes a matrix multiplier and a special calculator. When each module is executed, data is taken from a fixed SRAM location and output to a fixed output location of the next calculation. When the last calculation detects that the command queue is empty, it outputs to the SRAM storage location shared by the DMA unit and the calculation unit.

[0046] In addition to the above-mentioned units and modules, the hardware accelerator of the general configurable Transformer neural network may also include other components. However, since these components are irrelevant to the contents of the embodiments of the present disclosure, their illustration and description are omitted here.

[0047] like Figure 3 As shown, the hardware accelerator implementation method of the above-mentioned general configurable Transformer neural network controls the hardware accelerator by filling the configuration table with an instruction stream, the control unit disassembles the content of the configuration table and dispatches it to the computing unit and the DMA unit, and the computing unit and the DMA unit generate a command queue according to the dispatched configuration table and execute it.

[0048] In a specific embodiment, Figure 3 The FTC (Fusion Transformer Core) accelerator controls the hardware based on a mixture of instruction streams and configuration tables. Instructions are responsible for filling the configuration table. This accelerator uses configuration tables to simplify the large number of instructions brought by matrix operations, and avoids repeated memory access operations by fusing calculations, that is, using core calculations in conjunction with secondary calculations. This is because there is a dependency between the same group of calculations in the same configuration table, that is, the core calculation and the secondary calculation, that is, the result of the previous calculation needs to be given to the next calculation as the input of the next calculation. Figure 4 As shown, taking a typical SoftMax as an example, only the necessary input information (left multiplication / right multiplication / bias matrix) needs to be accessed to complete a complete calculation, without having to write the data back and read the data in for each calculation.

[0049] Filling the configuration table with instructions also provides greater flexibility for the accelerator. On this basis, the instruction configuration data of the self-attention module and the feedforward network are integrated into the ROM in the hardware accelerator to simplify the instructions of the Transformer system, which is currently the most widely used. The configuration tables of the DMA (Direct Memory Access) unit and the computing unit are aligned according to the ID number of the table.

[0050] like Figure 3 As shown in the figure, the instruction stream is divided into ordinary calculation instructions and compound calculation instructions. All calculation operations are implemented by filling in the configuration table. The configuration table content is disassembled by the control unit and dispatched to the calculation unit and DMA unit, which generates a command queue and executes it according to the disassembled configuration table.

[0051] The computing unit implements the calculation in the form of core calculation and secondary calculation. When the core calculation configuration table is set to empty, the core calculation is skipped and the secondary calculation is performed. Specifically, in order to implement ordinary calculation, the core calculation is allowed to be set to empty. That is, the core calculation can be skipped and the secondary calculation can be performed.

[0052] When the instruction stream is a compound calculation instruction, the content of the calculation unit configuration table is split into instructions for different calculations. Specifically, when performing compound calculations, in the calculation unit, the content of the configuration table is split into instructions for different calculations. Including matrix multiplication calculation instructions corresponding to the minimum calculation unit, vector multiplication / addition / subtraction calculation instructions, vector gelu instructions, vector natural exponent instructions, vector addition tree instructions, vector mask operations, vector Reduce Max operations, scalar reciprocal instructions, and scalar reciprocal root instructions. The addition of instruction types does not affect the design of the accelerator architecture, and more computing modules can be added to the current architecture.

[0053] After receiving the configuration table, the calculation unit forms a calculation instruction. The calculation unit performs calculations according to the calculation instructions. When the instruction queue is empty, the configuration information of the next group is re-read, a new instruction queue is formed and the instructions of the next group are started. When all groups in the configuration table are executed, the calculation unit sends a feedback signal back to the control unit, and communicates with the DMA unit to clear the configuration table information, reset the signal and prepare to process the next configuration table. Specifically, the execution method of the calculation instruction is as follows. The calculation controller checks according to the calculation instruction. As long as the conditions are met and the execution unit is idle, the instruction can be executed to realize the parallel operation of the instruction. When the instruction queue is empty, the configuration information of the next group is re-read, a new instruction queue is formed and the instructions of the next group are started. When all groups in the configuration table are executed, the calculation unit sends a feedback signal back to the control unit, shakes hands with the DMA unit, clears all information in the configuration table, resets all signals and prepares to process the next configuration table.

[0054] The DMA unit disassembles the content of the configuration table into three address queues and sets a transposition flag register; the three address queues are respectively a source address queue of a load instruction, a target address queue of a load instruction and a target address queue of a storage instruction; the DMA unit moves the data of the source address to the target address according to the first-in-first-out rule based on whether the target address memory is full; based on the calculation unit placing the calculation result in the sRAM space shared with the DMA unit, the DMA unit moves the data from the shared sRAM space to the sRAM space pointed to by the target address of the storage instruction according to whether the data exists and the order of the storage address queues.

[0055] The configuration table ID number of the DMA unit and the configuration table ID number of the computing unit start running after confirming that they are consistent with each other. Specifically, the configuration table ID number forming the DMA configuration must be consistent with the configuration table ID number of the computing module. The DMA unit and the computing unit shake hands and confirm that the ID numbers are consistent before starting running.

[0056] When loading data, the DMA unit continues to move data until it detects that the memory at the target address is full, and then moves the data to a temporary storage location; when the DMA unit stores a data back, it will try again to move the temporarily stored data to the target address, and if successful, it will initiate the next memory access request and continue the moving work;

[0057] When the address queue of the DMA unit is empty, re-read the configuration information of the next group, form a new instruction queue and start executing the address of the next group;

[0058] When all group configuration tables in the DMA unit configuration table are executed, the DMA unit stops running and sends a feedback signal to the control unit, and communicates with the computing module to eliminate the current configuration information, set the signal and wait for the next configuration table.

[0059] The configuration table content is mainly based on configuration matrix information, and matrix multiplication and vector multiplication are organized as core calculations. According to the number of core calculations, the same configuration table is divided into multiple groups. One group is responsible for providing the information required for one core operation and one secondary calculation. The data required in the group includes the dimension of the matrix, the source address of the load instruction, the target address of the load instruction, the target address of the storage instruction, the core calculation type, and the type of secondary calculation. It should be noted that there can be multiple calculations, which are executed according to the filling order.

[0060] When a new configuration instruction is received, the instruction is executed immediately to fill in the configuration table with relevant information, that is, the contents of the configuration table are filled with instructions.

[0061] When receiving the execution command, check whether the current configuration table is complete. If complete, split the configuration table and send it to the corresponding unit. If not complete, report an error and exit all commands, and stop running.

[0062] The control unit consists of a controller, an instruction cache, and a scheduler. The controller is responsible for storing the descriptors that need to be executed. The descriptors represent the start and end addresses of the instructions that need to be executed in the DDR. The instruction cache is responsible for storing the retrieved instructions. The scheduler is responsible for monitoring the status of the calculation unit and the DMA unit, that is, checking whether the corresponding execution components are idle and deciding whether to send the current configuration table. The instructions that fill the configuration table do not need to be checked and are executed directly.

[0063] The computing unit divides operators into different priorities. Generally, matrix multiplication / vector multiplication calculation is used as the core, among which matrix multiplication occupies the vast majority of the calculation, so the most computing resources are allocated to form MFU, and other calculations are classified as SFU. Multiple secondary calculations can be set and executed in sequence. Generally, vector addition and subtraction calculations, vector gelu calculations, vector natural exponential calculations, vector ReduceMax calculations, scalar reciprocal calculations, and scalar reciprocal root calculations are used as secondary calculations. The FTC architecture design of the present invention revolves around one or several calculations as the core, integrating other calculations to avoid the occurrence of repeated memory access operations. Data is input from the source operand position of the first calculation, and the result is output from the last calculation.

[0064] The DMA unit interacts with the computing unit through SRAM. The DMA unit puts the data required for the core calculation into the location specified by the load target address stack. After the computing unit is completed, the data is placed in the shared area, and the DMA unit stores the data back to the specified location according to the storage address queue.

[0065] like Figure 5 As shown, when calculating the Transformer, if the configuration signal configured in the ROM is not used, the process of implementing the calculation includes:

[0066] Transformer is considered as three configuration tables of self-attention, inter-layer normalization and feed-forward neural network. Each residual block is completed by a configuration table that does not configure the core operation but configures the secondary operation.

[0067] The self-attention configuration table sets five matrix multiplications as core operations. The first three matrix multiplications are not configured with secondary operations. The fourth is configured with ReduceMax, Mask, and SoftMax secondary operations (executed sequentially). The fifth core calculation is matrix multiplication and no secondary operations are configured.

[0068] After all self-attentions are completed, an additional matrix multiplication is configured, and no secondary operations are configured;

[0069] The inter-layer normalization configuration table contains multiple groups of operations, involving vector addition, addition tree, reciprocal multiplication, vector subtraction, scalar reciprocal multiplication, vector multiplication, etc.

[0070] The feedforward neural network contains two sets of matrix multiplications. The first set of secondary operations is vector Gelu, and the second set of core calculations is matrix multiplication.

[0071] During execution, the form information is sent to the corresponding execution hardware, which checks whether the configuration table is completed and starts execution.

[0072] It can be understood that the contents of the above hardware accelerator embodiments are applicable to the present method embodiments, the functions specifically implemented by the present method embodiments are the same as those of the above hardware accelerator embodiments, and the beneficial effects achieved are also the same as those achieved by the above hardware accelerator embodiments.

[0073] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The above method may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner, i.e., according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0074] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer programs described above include a plurality of instructions that may be executed by one or more processors.

[0075] Further, the above method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or a portion thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself.

[0076] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0077] In summary, the present invention provides a universal configurable Transformer neural network hardware accelerator and its implementation method. The universal configurable Transformer neural network hardware accelerator achieves a high input data reuse rate by setting core operations and secondary operations, integrating the interdependent calculations before and after, and reducing the memory access requirements and avoiding repeated memory accesses; by filling the configuration table with instructions, the instruction fetch and decoding time of a large number of instructions is reduced, the overall execution efficiency is improved, and a new solution is provided for balancing versatility and high performance. In summary, the present invention has good data reuse and good scalability.

[0078] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0079] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A general configurable Transformer neural network hardware accelerator, characterized in that: It includes a control unit, a calculation unit and a DMA unit, wherein: The control unit includes a controller, an instruction cache and a scheduler, wherein the controller stores instructions to be executed, the instruction cache is used to cache instructions, and the scheduler is used to monitor the status of the calculation unit and the DMA unit and decide whether to send instructions; The calculation unit includes a matrix multiplier and a special calculator. When the matrix multiplier and the special calculator perform calculation tasks, they both read data from a fixed SRAM location and output them to a fixed location for the next calculation. When it is detected that the calculation command queue is empty, the calculation result is output to an SRAM storage location shared with the DMA unit. The DMA unit interacts with the computing unit through the sRAM. The DMA unit puts the computing data into the loading target address stack braking position. After the computing unit completes the calculation, the DMA unit stores the shared data of the sRAM storage position back to the designated position according to the storage address queue.

2. A method for implementing a hardware accelerator of a universal configurable Transformer neural network as claimed in claim 1, characterized in that: The hardware accelerator is controlled by filling a configuration table with an instruction stream. The control unit disassembles the content of the configuration table and distributes it to the computing unit and the DMA unit. The computing unit and the DMA unit generate a command queue according to the distributed configuration table and execute it.

3. The implementation method according to claim 2, characterized in that: The computing unit implements the calculation in the form of core calculation combined with secondary calculation. When the core calculation configuration table is set to empty, the core calculation is skipped and the secondary calculation is performed.

4. The implementation method according to claim 2, characterized in that: When the instruction stream is a composite computing instruction, the content of the computing unit configuration table is split into instructions for different computing.

5. The implementation method according to claim 2, characterized in that: The computing unit performs calculations according to the calculation instructions. When the instruction queue is empty, the configuration information of the next group is re-read to form a new instruction queue and start executing the instructions of the next group. When all groups in the configuration table have been executed, the computing unit sends a feedback signal back to the control unit and communicates with the DMA unit to clear the configuration table information, reset the signal and prepare to process the next configuration table.

6. The implementation method according to claim 2, characterized in that: The DMA unit disassembles the configuration table content into three address queues and sets a transposition flag register; the three address queues are respectively a source address queue of a load instruction, a target address queue of a load instruction, and a target address queue of a store instruction; the DMA unit moves the data of the source address to the target address according to a first-in-first-out rule based on whether the target address memory is full; Based on the calculation unit placing the calculation result in the sRAM space shared with the DMA unit, the DMA unit moves the data from the shared sRAM space to the sRAM space pointed to by the target address of the storage instruction according to whether the data exists and the order of the storage address queue.

7. The implementation method according to claim 2, characterized in that: The configuration table ID number of the DMA unit and the configuration table ID number of the calculation unit start to run after confirming that they are consistent with each other.

8. The implementation method according to claim 2, characterized in that: When loading data, the DMA unit continues to move data until it detects that the memory at the target address is full, and then moves the data to a temporary storage location; when the DMA unit stores a data back, it will try again to move the temporarily stored data to the target address, and if successful, it will initiate the next memory access request and continue the moving work; When the address queue of the DMA unit is empty, re-read the configuration information of the next group, form a new instruction queue and start executing the address of the next group; When all group configuration tables in the DMA unit configuration table are executed, the DMA unit stops running and sends a feedback signal to the control unit, and communicates with the computing module to eliminate the current configuration information, set the signal and wait for the next configuration table.

9. The implementation method according to claim 2, characterized in that: The content of the configuration table is mainly configuration matrix information, and matrix multiplication and vector multiplication are organized as core calculations. According to the number of core calculations, the same configuration table is divided into multiple groups. One group is responsible for providing the information required for one core operation and one secondary calculation. The data required in the group include the dimension of the matrix, the source address of the load instruction, the target address of the load instruction, the target address of the storage instruction, the core calculation type and the secondary calculation type.

10. The implementation method according to any one of claims 2 to 9, characterized in that: When computing a Transformer neural network, the implementation process includes: The Transformer neural network is regarded as three configuration tables of self-attention, inter-layer normalization and feed-forward neural network. Each residual block is completed by a configuration table that does not configure the core operation but configures the secondary operation; The self-attention configuration table sets matrix multiplication as the core operation. The first three times are matrix multiplications without configuring secondary operations. The fourth time configures ReduceMax, Mask, and SoftMax secondary operations in sequence. The fifth time is matrix multiplication without configuring secondary operations. After the respective attention configuration tables are completed, an additional matrix multiplication is configured, and no secondary operations are configured; The inter-layer normalization configuration table includes multiple groups of operations; The feedforward neural network includes two sets of matrix multiplications, the first set of secondary operations is vector Gelu, and the second set of core calculations is matrix multiplication; During execution, the form information is sent to the corresponding execution hardware, which checks whether the configuration table is completed and starts execution.

Images