Automatically configured deep neural network accelerator system and implementation method thereof

Abstract

The invention discloses an automatically configured deep neural network accelerator system and an implementation method thereof, and the system comprises a deep neural network accelerator and a code generation module. The deep neural network accelerator comprises an off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a calculation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer and an output buffer, and corresponding RTL codes are generated through a code generation module according to user requirements. The RTL code is transmitted to the controller, the calculation module and the DMA module are configured through the controller according to the RTL code, a configured deep neural network accelerator is obtained, and neural network calculation is executed through the deep neural network accelerator. The adaptive capacity of the neural network accelerator to various application scenes is improved, the neural network calculation efficiency is also improved, and the method can be widely applied to the technical field of neural network accelerators.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network accelerators, and in particular to an automatically configured deep neural network accelerator system and an implementation method thereof. Background Art

[0002] In the prior art, the problem of automatically generating flexible hardware accelerators for different deep neural networks lies in four aspects. First, different operations mean different computing modes and circuit structures. Unifying the computing process of various operations under limited hardware resources is a major challenge for hardware flexibility. Second, deformable attention and deformable convolution involve sampling and dynamic irregular access of irregular receptive fields, which increases the overhead of bilinear interpolation and affects data reuse. Third, directly using ChatGPT to generate code is not effective, and its generation capabilities in specific areas need to be improved, especially RTL code. Fourth, ChatGPT has restrictions on the length of text generation, which makes it unable to meet the generation requirements of RTL code in ultra-large-scale integrated circuit design. The above problems need to be solved urgently. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide an automatically configured deep neural network accelerator system and an implementation method thereof, which can automatically generate corresponding computing kernels according to different configuration information, thereby improving the adaptability of the neural network accelerator to various application scenarios and improving the computing efficiency of the neural network.

[0004] The first technical solution adopted by the present invention is:

[0005] An automatically configured deep neural network accelerator system includes a deep neural network accelerator and a code generation module, wherein the deep neural network accelerator includes an off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a calculation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer, wherein:

[0006] The DMA module and the controller are both connected to the off-chip memory signal through the bus interface, the data segmentation module and the data integration module are both connected to the DMA module signal, the input end of the weight buffer, the input end of the offset buffer and the first input end of the input buffer are all connected to the output end of the data segmentation module, the output end of the offset buffer is connected to the second input end of the input buffer through the address generation engine, the output end of the input buffer and the output end of the weight buffer are both connected to the input end of the calculation module, the output end of the calculation module is connected to the input end of the output buffer through the non-linear operation module, the output end of the output buffer is connected to the input end of the data integration module, the calculation module is connected to the controller signal, and the output end of the code generation module is connected to the input end of the controller.

[0007] Further, the calculation module includes a matrix multiplication block, a matrix transposition unit, an intermediate buffer and a SoftMax unit, the output end of the input buffer and the output end of the weight buffer are both connected to the input end of the matrix multiplication block, the first output end of the matrix multiplication block is connected to the input end of the matrix transposition unit, the output end of the matrix transposition unit is connected to the input end of the SoftMax unit through the intermediate buffer, the second output end of the matrix multiplication block, the output end of the intermediate buffer and the output end of the SoftMax unit are all connected to the input end of the output buffer through the nonlinear operation module, and the matrix transposition unit and the SoftMax unit are both connected to the controller signal.

[0008] Furthermore, the matrix multiplication block includes multiple processing units, each of which includes multiple multipliers and a first accumulator, the output end of the input buffer and the output end of the weight buffer are both connected to the input end of the processing unit, and the output result of the matrix multiplication block is obtained by connecting the calculation results of each processing unit.

[0009] Furthermore, the matrix transposition unit includes a first register array and a second register array, the first register array is used to perform a read operation in the first stage and a write operation in the second stage, and the second register array is used to perform a write operation in the first stage and a read operation in the second stage.

[0010] Further, the SoftMax unit includes an exponential module, a second accumulator, a divider and a FIFO memory, the input end of the exponential module is connected to the output end of the intermediate buffer, the first output end of the exponential module is connected to the first input end of the divider through the FIFO memory, the second output end of the exponential module is connected to the second input end of the divider through the second accumulator, and the output end of the divider is connected to the input end of the output buffer through the nonlinear operation module.

[0011] Furthermore, the address generation engine is used to obtain the absolute coordinates of the two-dimensional offset of the offset buffer input, and superimpose the absolute coordinates with the initial input activation value to obtain the offset input address, and then transmit the offset input address to the input buffer.

[0012] Furthermore, the code generation module is used to generate corresponding RTL code according to user requirements and transmit the RTL code to the controller.

[0013] Further, the code generation module generates the RTL code through the following steps:

[0014] Obtain a user demand prompt file / modify demand condition file, input the user demand prompt file / modify demand condition file and a preset template file into ChatGPT, and obtain a target Python file;

[0015] Run the target Python file to obtain the RTL code.

[0016] The second technical solution adopted by the present invention is:

[0017] A method for implementing an automatically configured deep neural network accelerator system, which is implemented by the above-mentioned automatically configured deep neural network accelerator system, comprises the following steps:

[0018] Generate corresponding RTL code according to user requirements through the code generation module, and transmit the RTL code to the controller;

[0019] The controller configures the computing module and the DMA module according to the RTL code to obtain a configured deep neural network accelerator;

[0020] Neural network calculations are performed by the deep neural network accelerator.

[0021] Further, the performing of neural network calculations by the deep neural network accelerator specifically includes:

[0022] Load input data from the off-chip memory to the DMA module, and segment the input data through the data segmentation module to obtain activation data, offset data and weight data, and then store the activation data in the input buffer, store the offset data in the offset buffer, and store the weight data in the weight buffer;

[0023] Obtaining the activation data, the offset data, and the weight data through a calculation module, and performing a matrix multiplication calculation, a convolution operation, a transposed convolution operation, a deformable convolution operation, a hole convolution operation, an attention operation, or a deformable attention operation according to the activation data, the offset data, and the weight data to obtain a calculation result;

[0024] Performing activation operation on the calculation result through a nonlinear operation module to obtain output data, and storing the output data into an output buffer;

[0025] The output data is integrated and processed by the data integration module to obtain integrated output data, and the integrated output data is written back to the off-chip memory by the DMA module.

[0026] The beneficial effects of the present invention are as follows: the present invention provides an automatically configured deep neural network accelerator system and an implementation method thereof, the automatically configured deep neural network accelerator system includes a deep neural network accelerator and a code generation module, the deep neural network accelerator includes an off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a calculation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer, the code generation module generates corresponding RTL codes according to user requirements, and transmits the RTL codes to the controller, the controller configures the calculation module and the DMA module according to the RTL code, obtains a configured deep neural network accelerator, and performs neural network calculations through the deep neural network accelerator. The present invention can automatically generate corresponding calculation kernels according to different configuration information, improves the adaptability of the neural network accelerator to various application scenarios, and also improves the efficiency of neural network calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the structure of an automatically configured deep neural network accelerator system provided by an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of the structure of a matrix multiplication block provided in an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the structure of a matrix transposition unit provided in an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of the structure of a SoftMax unit provided in an embodiment of the present invention;

[0031] Figure 5 A signal flow diagram of an address generation engine provided by an embodiment of the present invention;

[0032] Figure 6 A schematic diagram of the working principle of the code generation module provided in an embodiment of the present invention;

[0033] FIG. 7( a) is a schematic diagram of a conditional file writing format for a convolution operation provided by an embodiment of the present invention;

[0034] FIG. 7( b ) is a schematic diagram of a format for writing a conditional file for an attention operation provided by an embodiment of the present invention;

[0035] FIG. 7( c ) is a schematic diagram of a conditional file writing format for a deformable attention operation provided by an embodiment of the present invention;

[0036] Figure 8 A flowchart of the steps of an implementation method of an automatically configured deep neural network accelerator system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0038] In the description of the present invention, the meaning of "a plurality" is more than two. If there is a description of "a first" or "a second", it is only used to distinguish the technical features, and it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this article have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are only for describing specific embodiments, not for limiting the present invention.

[0039] Compared to CNN, Transformer has a larger receptive field, which gives it more complex representation capabilities. However, using dense attention in models including ViT incurs a lot of memory usage and computational cost. In addition, dense attention may exceed irrelevant parts of the receptive field, thereby affecting the performance of the model. On the other hand, the data in the sparse attention adopted in models such as Swin Transformer is agnostic, which may limit the ability to model long-range relationships. To alleviate these problems, the researchers developed a deformable attention mechanism inspired by deformable convolutional networks, in which the position of key-value pairs in the attention is selected in a data-dependent manner. This flexible approach allows the attention module to focus on relevant areas and capture more informative features.

[0040] Similar to deformable convolution, the hardware implementation of deformable attention will increase the overhead of bilinear interpolation and deformable offset. Bilinear interpolation is the weighted average sum of neighboring pixels of the final sample value, which will introduce a large number of multiplications in the sampling process. In order to reduce the total amount of calculation (mainly multiplication), the offset can be integer-valued during inference to eliminate the redundant multiplications generated during the interpolation process. In addition, there are other algorithmic improvements that can be more conducive to the hardware implementation of deformable attention, such as nonlinear approximation and offset sharing mechanism.

[0041] In CNN and Transformer, the key calculation is matrix multiplication, which consists of multiple multiplication-accumulation units. In order to support various matrix multiplication operations, different computing modes and memory accesses of different operations are considered to find similar computing forms and build reusable data paths. ChatGPT can be used in this process to assist hardware designers by generating RTL code for specific circuits based on different configuration information. In order to achieve unified calculation of different operations, some traditional accelerators for deformable convolution involve a large amount of memory resources, such as register arrays and FIFOs to store multiplexed data, thereby reducing the frequency of memory access.

[0042] In recent years, deep neural networks such as CNN, GAN and Transformer have played a vital role in various fields due to their powerful performance. However, the different memory access and complex computing modes of these networks have brought great difficulties to their hardware deployment on resource-constrained hardware devices. Manually designing dedicated accelerators for specific networks will incur additional manpower costs and time overhead. These accelerators may also lack versatility and flexibility and cannot be used for complex multi-task application scenarios. Therefore, it is crucial to develop an automated framework with a flexible hardware architecture for various networks.

[0043] In the present invention, an efficient and flexible deep neural network accelerator architecture is proposed, which can support different neural network computing operations. In addition, in order to flexibly deploy and implement the designed accelerator, the present invention proposes a framework that uses ChatGPT's excellent text generation capabilities to automatically generate RTL code. The designed framework was tested on FPGA, and the experimental results show that the framework can flexibly generate various computing kernels according to different configuration information, and the proposed hardware accelerator supports more neural network operations than other similar accelerators, with satisfactory hardware efficiency. The embodiments of the present invention are described in detail below.

[0044] Reference Figure 1 The present invention provides an automatically configured deep neural network accelerator system, including a deep neural network accelerator and a code generation module, wherein the deep neural network accelerator includes an off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a calculation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer, wherein:

[0045] The DMA module and the controller are both connected to the off-chip memory signal through the bus interface, the data segmentation module and the data integration module are both connected to the DMA module signal, the input end of the weight buffer, the input end of the offset buffer and the first input end of the input buffer are all connected to the output end of the data segmentation module, the output end of the offset buffer is connected to the second input end of the input buffer through the address generation engine, the output end of the input buffer and the output end of the weight buffer are both connected to the input end of the calculation module, the output end of the calculation module is connected to the input end of the output buffer through the nonlinear operation module, the output end of the output buffer is connected to the input end of the data integration module, the calculation module is connected to the controller signal, and the output end of the code generation module is connected to the input end of the controller.

[0046] Specifically, the present invention can automatically generate corresponding computing kernels according to different configuration information, thereby improving the adaptability of the neural network accelerator to various application scenarios and also improving the computing efficiency of the neural network.

[0047] Reference Figure 1, further as an optional implementation, the calculation module includes a matrix multiplication block, a matrix transposition unit, an intermediate buffer and a SoftMax unit, the output end of the input buffer and the output end of the weight buffer are both connected to the input end of the matrix multiplication block, the first output end of the matrix multiplication block is connected to the input end of the matrix transposition unit, the output end of the matrix transposition unit is connected to the input end of the SoftMax unit through the intermediate buffer, the second output end of the matrix multiplication block, the output end of the intermediate buffer and the output end of the SoftMax unit are all connected to the input end of the output buffer through the nonlinear operation module, and the matrix transposition unit and the SoftMax unit are both connected to the controller signal.

[0048] Reference Figure 2 , further as an optional implementation, the matrix multiplication block includes multiple processing units, each processing unit includes multiple multipliers and a first accumulator, the output end of the input buffer and the output end of the weight buffer are both connected to the input end of the processing unit, and the output result of the matrix multiplication block is obtained by connecting the calculation results of each processing unit.

[0049] Specifically, p and q represent the input and output channel parallelism, respectively. The processing unit contains p multipliers and an accumulator, and the matrix multiplication block is stacked by q processing units. The input activation value is broadcast to the q processing units. At the same time, the weights of the filter are also fed to the q processing units. The results calculated in each processing unit are then connected and the final data is output.

[0050] Reference Figure 3 As an optional implementation, the matrix transposition unit includes a first register array and a second register array, the first register array is used to perform a read operation in the first stage and a write operation in the second stage, and the second register array is used to perform a write operation in the first stage and a read operation in the second stage. ''

[0051] Specifically, in order to implement matrix transposition on-chip, a ping-pong storage method can be used to organize and transfer data using a register array. In phase 1, array 1 performs a read operation, while array 2 performs a write operation. Similarly, in phase 2, array 1 performs a write operation, while array 2 performs a read operation. During a write operation, data is transferred in a horizontal direction. When performing a read operation, data is transferred in a vertical direction. The matrix transposition unit can process n×n matrix transpositions.

[0052] Reference Figure 4, further as an optional implementation, the SoftMax unit includes an exponential module, a second accumulator, a divider and a FIFO memory, the input end of the exponential module is connected to the output end of the intermediate buffer, the first output end of the exponential module is connected to the first input end of the divider through the FIFO memory, the second output end of the exponential module is connected to the second input end of the divider through the second accumulator, and the output end of the divider is connected to the input end of the output buffer through the nonlinear operation module.

[0053] Specifically, the exponential module processes the input data to obtain the exponent. The exponential stream has two routes, one directly flows into the accumulator to calculate the denominator of the divider, and the other flows into the FIFO (first in, first out) memory used as the numerator of the divider. Because after the SoftMax processing, each data is limited to (0,1), the integer part is zero, so the integer part can be discarded.

[0054] Reference Figure 5 , further as an optional implementation, the address generation engine is used to obtain the absolute coordinates of the two-dimensional offset of the offset buffer input, and superimpose the absolute coordinates with the initial input activation value to obtain the offset input address, and then transmit the offset input address to the input buffer.

[0055] Specifically, m and n represent the depth of the offset buffer and the input buffer, respectively, Height represents the height of the input feature map, Inch represents the number of input channels, and p represents the parallelism of the input channels. The data of the offset buffer is spliced ​​by two offsets from different dimensions. Since these two-dimensional coordinates from the offset buffer are the relative coordinates of the original feature values ​​in space, it is also necessary to obtain the absolute coordinates. After superimposing them with the initial input activation value on the two-dimensional plane, the offset input address can be obtained. Among them, there is a multiplexer that can select the data path according to the requirements.

[0056] The inconsistency of common operations mainly comes from different receptive fields and sampling methods. The present invention proposes a unified sampling algorithm that can convert different calculation processes into the same operation based on similar characteristics. Here, they are all converted into matrix multiplication.

[0057] The specific calculation process of different operations using the unified sampling algorithm is explained below through formulas.

[0058]

[0059] Formula group (1) is the algorithm of deformable attention and standard attention. ΩΔ is a set of dynamic input offsets, Ω represents the receptive field of the input feature map, and p nrepresents the element position of each matrix or the relative coordinates in the kernel window, N represents the size of the matrix or kernel window, Δp n represents the position offset in two-dimensional space, Q, K and V represent the matrices of query, key and value respectively. and Represents the vector of weights and activation values. The subscripts of all vectors and scalars represent their spatial position coordinates. Taking deformable attention as an example, the process of calculating the key and value matrices involves the offset of the sampling position. The offset is limited to an integer value in a small range. It can be simply regarded as a change in the rows or columns of the matrix, and reduces the redundant multiplication operations generated during the bilinear interpolation process. The process of calculating the Z matrix should actually be regarded as two rounds of matrix multiplication, but because it is similar to the process of calculating the query matrix and it is shortened to reduce the length, that is, the attention type operation includes five rounds of matrix multiplication. The process of calculating standard attention is similar to that of deformable attention, just remove the offset of the key and value matrices.

[0060]

[0061] Formula (2) is the matrix multiplication algorithm, which is the most basic operator of the unified sampling algorithm.

[0062]

[0063] Formula group (3) is the algorithm of deformable convolution and convolution. Since convolution is a kind of matrix multiplication, it is not difficult to convert it into the above form. The difference between deformable convolution and convolution is also whether there is an offset. The standard convolution is the case where the offset of the deformable convolution is set to 0. Where (i, j) is the center position of the standard convolution window, l x and l y express pn The maximum absolute value between the horizontal and vertical coordinates of , k represents the kernel size.

[0064]

[0065] Formula group (4) is the algorithm of transposed convolution. If it is directly treated as a round of convolution, the result is sparse matrix multiplication, which will greatly reduce the efficiency. Here, according to the size of the step size, the convolution kernel is decomposed into multiple sub-convolution kernels, multiple rounds of operations are performed, and the results are spliced ​​together in order, which perfectly solves the sparse matrix multiplication problem of transposed convolution. Among them, s represents the step size, and the convolution kernel is divided into s×s sub-convolution kernels. Although the weight receptive field is divided into multiple, it can greatly improve the efficiency. The weight offset is in the set (including l×l two-dimensional offsets Δp n ). s Indicates that in the weight receptive field Ωw The position in.

[0066] As a further optional implementation, the code generation module is used to generate corresponding RTL code according to user requirements and transmit the RTL code to the controller.

[0067] Reference Figure 6 As an optional implementation, the code generation module generates RTL code by the following steps:

[0068] Obtain the user demand prompt file / modify demand condition file, input the user demand prompt file / modify demand condition file and the preset template file into ChatGPT, and obtain the target Python file;

[0069] Run the target Python file to get the RTL code.

[0070] Specifically, an embodiment of the present invention utilizes ChatGPT to assist in the design of a neural network hardware accelerator. According to application requirements, RTL code for implementing corresponding operations can be generated by interacting with ChatGPT. Since the scale of RTL code for ultra-large-scale integrated circuits is very large, it usually exceeds the word limit that ChatGPT can generate, so a shorter Python code template generated by ChatGPT is developed, and then this Python file is used to generate RTL code. First, two files are provided to ChatGPT: a prompt file that outlines the required requirements and a template file that provides a template for the generated Python code. If the requirements need to be modified, the updated information can be provided to ChatGPT through a conditional file, and ChatGPT will generate a modified Python file as feedback. Then, you only need to run the generated Python file to obtain the RTL code, and then simulate it to verify the function.

[0071] As shown in Figures 7(a) to (c), it is a schematic diagram of the conditional file writing format for the convolution operation, attention operation and deformable attention operation provided by the embodiment of the present invention. "Operation type" indicates the required operation to be performed. "Activation function" is to select nonlinear activation operations, such as ReLU, Leaky ReLU and GELU. It is worth noting that if the attention type operation is selected, the nonlinear SoftMax operation will be automatically performed. "Feature map size" needs to fill in the length, width, input channel and output channel of the input feature map. In "Filter parameter", the kernel size, stride and padding need to be filled in. It should be noted that the attention type operation should set the kernel size to 1. "Input channel parallelism", "Output channel parallelism" and "Batch sizeparallelism" are input channel parallelism, output channel parallelism and batch parallelism, respectively, which are hardware parameters that determine the parallelism of the entire circuit. "Buffer config" contains the configuration information of each buffer used in the accelerator.

[0072] Reference Figure 8 The embodiment of the present invention provides a method for implementing an automatically configured deep neural network accelerator system, which is implemented by the above-mentioned automatically configured deep neural network accelerator system, and includes the following steps:

[0073] S101, generating corresponding RTL code according to user requirements through a code generation module, and transmitting the RTL code to a controller;

[0074] S102, configuring the computing module and the DMA module according to the RTL code through the controller to obtain a configured deep neural network accelerator;

[0075] S103, performing neural network calculations through a deep neural network accelerator.

[0076] As an optional implementation, neural network calculations are performed by a deep neural network accelerator, which specifically includes:

[0077] S1031, loading input data from the off-chip memory to the DMA module, and segmenting the input data through the data segmentation module to obtain activation data, offset data and weight data, and then storing the activation data in the input buffer, storing the offset data in the offset buffer, and storing the weight data in the weight buffer;

[0078] S1032, obtaining activation data, offset data, and weight data through a calculation module, and performing matrix multiplication calculation, convolution operation, transposed convolution operation, deformable convolution operation, hole convolution operation, attention operation, or deformable attention operation according to the activation data, offset data, and weight data to obtain a calculation result;

[0079] S1033, performing activation operation on the calculation result through the nonlinear operation module to obtain output data, and storing the output data into the output buffer;

[0080] S1034, integrating the output data through the data integration module to obtain integrated output data, and writing the integrated output data back to the off-chip memory through the DMA module.

[0081] Specifically, in the data preparation stage, the input data is loaded from the off-chip memory into the DMA, and then the data is split and assigned to the corresponding buffers. The input buffer and offset buffer of the computing kernel contain N blocks, which store the corresponding activation values ​​and offset values ​​respectively. The weight buffer has only one block and is broadcast to N matrix multiplication blocks. The obtained results are stored in the corresponding blocks of the output buffer, which are also divided into N blocks. The address generation engine generates offset addresses by sampling and then reads data from the input buffer. The nonlinear operation module executes the activation function. After calculating a batch of data, the results can be stored in the output buffer, or other operations can be performed according to specific requirements. Finally, the output data will be written back to the off-chip memory through DMA.

[0082] In some optional embodiments, the conditional file writing format corresponding to the convolution operation can generate a corresponding hardware accelerator by changing the corresponding parameters. In the data preparation stage, the input data is loaded from the off-chip memory into the DMA, and then the data is split and assigned to the corresponding buffer. Because the receptive field of the standard convolution is not offset, the multiplexer in the address generation engine will select the path with an output of 0. The nonlinear operation module of this embodiment performs the ReLU activation function. After calculating the result, the matrix multiplication block directly stores the data in the output buffer, and then writes it back to the off-chip memory through DMA.

[0083] In some optional embodiments, the conditional file writing format corresponding to the attention operation can generate the corresponding hardware accelerator by changing the corresponding parameters. In the data preparation stage, the input data is loaded from the off-chip memory into the DMA, and then the data is segmented and assigned to the corresponding buffer. Because the receptive field of the standard attention is not offset, the multiplexer in the address generation engine will select the path with an output of 0. The attention type automatically performs the nonlinear operation of SoftMax and automatically sets the kernel size to 1. The attention operation will perform five rounds of operations in the matrix multiplication block. The first three rounds will calculate the values ​​of the query, key and value matrices respectively and store them in the intermediate buffer, where the key value will be transposed by the matrix transposition module and then stored in the intermediate buffer. The last two rounds are to read data from the intermediate buffer for SoftMax operation, and then store the results in the output buffer, and then write them back to the off-chip memory through DMA.

[0084] In some optional embodiments, the conditional file writing format corresponding to the deformable attention operation can generate the corresponding hardware accelerator by changing the corresponding parameters. In the data preparation stage, the input data is loaded from the off-chip memory into the DMA, and then the data is split and allocated to the corresponding buffer. Because the receptive field of the deformable attention is offset, the address generation engine will read the data from the offset buffer to calculate the offset address of the input buffer. The attention type will automatically perform the nonlinear operation of SoftMax and automatically set the kernel size to 1. The deformable attention operation will perform five rounds of operations in the matrix multiplication block. The first three rounds will calculate the values ​​of the query, key and value matrices respectively and store them in the intermediate buffer, where the key and value have a receptive field offset, and the key value will be transposed by the matrix transposition module and then stored in the intermediate buffer. The last two rounds are to read data from the intermediate buffer for SoftMax operation, and then store the results in the output buffer, and then write them back to the off-chip memory through DMA.

[0085] The above is an explanation of the automatically configured deep neural network accelerator system and implementation method of an embodiment of the present invention. It can be recognized that the present invention proposes a unified sampling algorithm, which can convert various operations into an adaptive algorithm based on matrix multiplication. Based on the algorithm, a deep neural network accelerator with low hardware complexity, high efficiency and flexibility is designed. The deep neural network accelerator effectively handles multiple types of operations in a unified input and output layout format, alleviating the problems related to irregular memory access caused by deformable attention and deformable convolution; based on ChatGPT, the present invention proposes an automatic code generation strategy, which generates concise Python code through a template file, and then uses it to generate the corresponding RTL code, which solves the problem of text length limitation when generating large-scale RTL code; the present invention also proposes an automated deep neural network accelerator hardware design framework based on ChatGPT, which can automatically generate the RTL code of the corresponding neural network accelerator according to given conditions, and it has advantages in hardware efficiency and hardware implementation agility.

[0086] It is understandable that in terms of hardware flexibility, since irregular sampling of the receptive field can easily lead to memory access conflicts, many accelerators do not support complex operations such as deformable attention, deformable convolution, and transposed convolution. The unified sampling algorithm proposed in the present invention, its dynamic sampling mode eliminates memory access conflicts, making it suitable for various complex scenarios. In addition, compared with accelerators suitable for single or multiple operations, the present invention can support a wider range of operations, which means that it can be applied to the acceleration of more tasks. At the system level, the new automated hardware design framework based on ChatGPT conversations of the present invention can not only automatically generate code files through conversations, but also flexibly select operation types according to specific requirements to meet hardware acceleration of various types of networks.

[0087] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The above methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner, according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0088] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer programs described above include a plurality of instructions that may be executed by one or more processors.

[0089] Further, the above method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or a portion thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself.

[0090] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0091] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. An automatically configured deep neural network accelerator system, characterized in that: It includes a deep neural network accelerator and a code generation module, wherein the deep neural network accelerator includes an off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a calculation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer, wherein: The DMA module and the controller are both connected to the off-chip memory signal through the bus interface, the data segmentation module and the data integration module are both connected to the DMA module signal, the input end of the weight buffer, the input end of the offset buffer and the first input end of the input buffer are all connected to the output end of the data segmentation module, the output end of the offset buffer is connected to the second input end of the input buffer through the address generation engine, the output end of the input buffer and the output end of the weight buffer are both connected to the input end of the calculation module, the output end of the calculation module is connected to the input end of the output buffer through the non-linear operation module, the output end of the output buffer is connected to the input end of the data integration module, the calculation module is connected to the controller signal, and the output end of the code generation module is connected to the input end of the controller.

2. The automatically configured deep neural network accelerator system according to claim 1, characterized in that: The calculation module includes a matrix multiplication block, a matrix transposition unit, an intermediate buffer and a SoftMax unit. The output end of the input buffer and the output end of the weight buffer are both connected to the input end of the matrix multiplication block, the first output end of the matrix multiplication block is connected to the input end of the matrix transposition unit, the output end of the matrix transposition unit is connected to the input end of the SoftMax unit through the intermediate buffer, the second output end of the matrix multiplication block, the output end of the intermediate buffer and the output end of the SoftMax unit are all connected to the input end of the output buffer through the nonlinear operation module, and the matrix transposition unit and the SoftMax unit are both connected to the controller signal.

3. The automatically configured deep neural network accelerator system according to claim 2, characterized in that: The matrix multiplication block includes multiple processing units, each of which includes multiple multipliers and a first accumulator. The output end of the input buffer and the output end of the weight buffer are both connected to the input end of the processing unit. The output result of the matrix multiplication block is obtained by connecting the calculation results of each processing unit.

4. The automatically configured deep neural network accelerator system according to claim 2, characterized in that: The matrix transposition unit comprises a first register array and a second register array, wherein the first register array is used to perform a read operation in a first stage and a write operation in a second stage, and the second register array is used to perform a write operation in the first stage and a read operation in the second stage.

5. The automatically configured deep neural network accelerator system according to claim 2, characterized in that: The SoftMax unit includes an exponential module, a second accumulator, a divider and a FIFO memory, wherein the input end of the exponential module is connected to the output end of the intermediate buffer, the first output end of the exponential module is connected to the first input end of the divider through the FIFO memory, the second output end of the exponential module is connected to the second input end of the divider through the second accumulator, and the output end of the divider is connected to the input end of the output buffer through the nonlinear operation module.

6. The automatically configured deep neural network accelerator system according to claim 1, characterized in that: The address generation engine is used to obtain the absolute coordinates of the two-dimensional offset of the offset buffer input, and superimpose the absolute coordinates with the initial input activation value to obtain the offset input address, and then transmit the offset input address to the input buffer.

7. The automatically configured deep neural network accelerator system according to claim 1, characterized in that: The code generation module is used to generate corresponding RTL code according to user requirements and transmit the RTL code to the controller.

8. The automatically configured deep neural network accelerator system according to claim 7, characterized in that: The code generation module generates the RTL code by the following steps: Obtain a user demand prompt file / modify demand condition file, input the user demand prompt file / modify demand condition file and a preset template file into ChatGPT, and obtain a target Python file; Run the target Python file to obtain the RTL code.

9. A method for implementing an automatically configured deep neural network accelerator system, for implementing the automatically configured deep neural network accelerator system according to any one of claims 1 to 8, characterized in that: The following steps are involved: Generate corresponding RTL code according to user requirements through the code generation module, and transmit the RTL code to the controller; The controller configures the computing module and the DMA module according to the RTL code to obtain a configured deep neural network accelerator; Neural network calculations are performed by the deep neural network accelerator.

10. The method for implementing an automatically configured deep neural network accelerator system according to claim 9, characterized in that: The performing of neural network calculations by the deep neural network accelerator specifically includes: Load input data from the off-chip memory to the DMA module, and segment the input data through the data segmentation module to obtain activation data, offset data and weight data, and then store the activation data in the input buffer, store the offset data in the offset buffer, and store the weight data in the weight buffer; Obtaining the activation data, the offset data, and the weight data through a calculation module, and performing a matrix multiplication calculation, a convolution operation, a transposed convolution operation, a deformable convolution operation, a hole convolution operation, an attention operation, or a deformable attention operation according to the activation data, the offset data, and the weight data to obtain a calculation result; Performing activation operation on the calculation result through a nonlinear operation module to obtain output data, and storing the output data into an output buffer; The output data is integrated and processed by the data integration module to obtain integrated output data, and the integrated output data is written back to the off-chip memory by the DMA module.

Images