Automatically configured deep neural network accelerator system and method of implementing the same

Abstract

This invention discloses an automated configuration deep neural network accelerator system and its implementation method, including a deep neural network accelerator and a code generation module. The deep neural network accelerator includes off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a computation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer. The code generation module generates corresponding RTL code according to user requirements and transmits the RTL code to the controller. The controller configures the computation module and the DMA module according to the RTL code to obtain the configured deep neural network accelerator, which then performs neural network computation. This invention improves the adaptability of neural network accelerators to various application scenarios and also improves the computational efficiency of neural networks, making it widely applicable in the field of neural network accelerator technology.

Description

Technical Field

[0001] This invention relates to the field of neural network accelerator technology, and in particular to an automated configuration deep neural network accelerator system and its implementation method. Background Technology

[0002] In existing technologies, the challenges of automatically generating flexible hardware accelerators for different depth neural networks lie in four aspects. First, different operations imply different computational modes and circuit structures; unifying the computational process for various operations under limited hardware resources is a major challenge to hardware flexibility. Second, deformable attention and deformable convolution involve sampling irregular receptive fields and dynamically irregular access, which increases the overhead of bilinear interpolation and affects data reuse. Third, directly using ChatGPT to generate code yields poor results; its generation capabilities in specific domains, especially RTL code, need improvement. Fourth, ChatGPT has limitations on the length of generated text, making it unsuitable for generating RTL code in VLSI design. These problems urgently need to be addressed. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide an automated configuration deep neural network accelerator system and its implementation method, which can automatically generate corresponding computing kernels based on different configuration information, thereby improving the adaptability of the neural network accelerator to various application scenarios and also improving the computational efficiency of the neural network.

[0004] The first technical solution adopted in this invention is:

[0005] An automated configuration deep neural network accelerator system includes a deep neural network accelerator and a code generation module. The deep neural network accelerator includes off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a computation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer, wherein:

[0006] The DMA module and the controller are both connected to the off-chip memory via the bus interface. The data segmentation module and the data integration module are both connected to the DMA module. The input terminals of the weight buffer, the offset buffer, and the first input terminal of the input buffer are all connected to the output terminal of the data segmentation module. The output terminal of the offset buffer is connected to the second input terminal of the input buffer via the address generation engine. The output terminals of the input buffer and the weight buffer are both connected to the input terminal of the calculation module. The output terminal of the calculation module is connected to the input terminal of the output buffer via the nonlinear operation module. The output terminal of the output buffer is connected to the input terminal of the data integration module. The calculation module is connected to the controller. The output terminal of the code generation module is connected to the input terminal of the controller.

[0007] Furthermore, the calculation module includes a matrix multiplication block, a matrix transpose unit, an intermediate buffer, and a SoftMax unit. The output terminals of the input buffer and the weight buffer are both connected to the input terminal of the matrix multiplication block. The first output terminal of the matrix multiplication block is connected to the input terminal of the matrix transpose unit. The output terminal of the matrix transpose unit is connected to the input terminal of the SoftMax unit through the intermediate buffer. The second output terminal of the matrix multiplication block, the output terminal of the intermediate buffer, and the output terminal of the SoftMax unit are all connected to the input terminal of the output buffer through the nonlinear operation module. The matrix transpose unit and the SoftMax unit are both connected to the controller signal.

[0008] Furthermore, the matrix multiplication block includes multiple processing units, each of which includes multiple multipliers and a first accumulator. The output of the input buffer and the output of the weight buffer are both connected to the input of the processing unit. The output of the matrix multiplication block is obtained by connecting the calculation results of each processing unit.

[0009] Furthermore, the matrix transpose unit includes a first register array and a second register array. The first register array is used to perform a read operation in the first stage and a write operation in the second stage. The second register array is used to perform a write operation in the first stage and a read operation in the second stage.

[0010] Furthermore, the SoftMax unit includes an exponent module, a second accumulator, a divider, and a FIFO memory. The input of the exponent module is connected to the output of the intermediate buffer. The first output of the exponent module is connected to the first input of the divider through the FIFO memory. The second output of the exponent module is connected to the second input of the divider through the second accumulator. The output of the divider is connected to the input of the output buffer through the nonlinear operation module.

[0011] Furthermore, the address generation engine is used to obtain the absolute coordinates of the two-dimensional offset of the offset buffer input, and superimpose the absolute coordinates with the initial input activation value to obtain the offset input address, and then transmit the offset input address to the input buffer.

[0012] Furthermore, the code generation module is used to generate corresponding RTL code according to user requirements and transmit the RTL code to the controller.

[0013] Furthermore, the code generation module generates the RTL code through the following steps:

[0014] Obtain the user requirement prompt file / modify the requirement condition file, and input the user requirement prompt file / modify the requirement condition file and the preset template file into ChatGPT to obtain the target Python file;

[0015] Run the target Python file to obtain the RTL code.

[0016] The second technical solution adopted in this invention is:

[0017] A method for implementing an automatically configured deep neural network accelerator system, comprising the following steps:

[0018] The code generation module generates corresponding RTL code according to user requirements and transmits the RTL code to the controller.

[0019] The controller configures the computation module and DMA module according to the RTL code to obtain a configured deep neural network accelerator.

[0020] Neural network computations are performed using the deep neural network accelerator.

[0021] Furthermore, the execution of neural network computation via the deep neural network accelerator specifically includes:

[0022] Input data is loaded from off-chip memory into the DMA module, and the input data is segmented by the data segmentation module to obtain activation data, offset data and weight data. The activation data is then stored in the input buffer, the offset data is stored in the offset buffer, and the weight data is stored in the weight buffer.

[0023] The activation data, offset data, and weight data are obtained through the calculation module, and matrix multiplication, convolution, transpose convolution, deformable convolution, dilated convolution, attention, or deformable attention operations are performed based on the activation data, offset data, and weight data to obtain the calculation results.

[0024] The calculation result is activated by a nonlinear operation module to obtain output data, and the output data is stored in an output buffer.

[0025] The output data is integrated by the data integration module to obtain integrated output data, and then the integrated output data is written back to the off-chip memory by the DMA module.

[0026] The beneficial effects of this invention are as follows: This invention provides an automatically configured deep neural network accelerator system and its implementation method. The automatically configured deep neural network accelerator system includes a deep neural network accelerator and a code generation module. The deep neural network accelerator includes off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a computation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer. The code generation module generates corresponding RTL code according to user requirements and transmits the RTL code to the controller. The controller configures the computation module and the DMA module according to the RTL code to obtain the configured deep neural network accelerator, which then performs neural network computation. This invention can automatically generate corresponding computation kernels based on different configuration information, improving the adaptability of the neural network accelerator to various application scenarios and also improving the computational efficiency of neural networks. Attached Figure Description

[0027] Figure 1 A schematic diagram of an automated configuration deep neural network accelerator system provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the structure of a matrix multiplication block provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of the matrix transpose unit provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of the SoftMax unit provided in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the signal flow of the address generation engine provided in an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram illustrating the working principle of the code generation module provided in an embodiment of the present invention;

[0033] Figure 7(a) is a schematic diagram of the condition file writing format for convolution operations provided in an embodiment of the present invention;

[0034] Figure 7(b) is a schematic diagram of the condition file writing format for attention operations provided in an embodiment of the present invention;

[0035] Figure 7(c) is a schematic diagram of the condition file writing format for deformable attention operations provided in an embodiment of the present invention;

[0036] Figure 8 A flowchart illustrating the steps of an automated configuration method for a deep neural network accelerator system provided in this embodiment of the invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0038] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order in which the indicated technical features are presented. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and not for limiting the invention.

[0039] Compared to CNNs, Transformers have a larger receptive field, which gives them more complex representation capabilities. However, utilizing dense attention in models including ViT incurs significant memory consumption and computational costs. Furthermore, dense attention may extend beyond irrelevant parts of the receptive field, impacting model performance. On the other hand, the data in sparse attention employed in models like the Swin Transformer is agnostic, which may limit the ability to model long-distance relationships. To mitigate these issues, researchers have developed a deformable attention mechanism inspired by deformable convolutional networks, in which the position of key-value pairs within the attention is selected in a data-dependent manner. This flexible approach allows the attention module to focus on relevant regions and capture more informational features.

[0040] Similar to deformable convolution, hardware implementations of deformable attention incur overhead from bilinear interpolation and deformable offsets. Bilinear interpolation is a weighted average of the neighboring pixels of the final sample value, which introduces a large number of multiplications during sampling. To reduce the overall computational cost (mainly multiplications), offsets can be rounded to integer values ​​during inference to eliminate redundant multiplications introduced during interpolation. Furthermore, other algorithmic improvements are available that are more conducive to hardware implementations of deformable attention, such as nonlinear approximation and offset sharing mechanisms.

[0041] In CNNs and Transformers, the key computation is matrix multiplication, which consists of multiple multiply-accumulate units. To support various matrix multiplication operations, different computational patterns and memory accesses for different operations are considered to find similar computational forms and build reusable data paths. ChatGPT can be used in this process to assist hardware designers by generating RTL code for specific circuits based on different configuration information. To achieve unified computation for different operations, some traditional accelerators for deformable convolutions involve large amounts of memory resources, such as register arrays and FIFOs, to store multiplexed data, thereby reducing the frequency of memory accesses.

[0042] In recent years, deep neural networks such as CNNs, GANs, and Transformers have played a crucial role in various fields due to their powerful performance. However, the different memory access methods and complex computational patterns of these networks pose significant challenges to their hardware deployment on resource-constrained devices. Manually designing dedicated accelerators for specific networks would result in additional human and time costs, and these accelerators may also lack versatility and flexibility, failing to address complex multi-tasking application scenarios. Therefore, developing an automated framework with flexible hardware architecture for various networks is essential.

[0043] This invention proposes a highly efficient and flexible deep neural network accelerator architecture that supports various neural network computational operations. Furthermore, to facilitate flexible deployment and implementation of the designed accelerator, this invention proposes a framework that leverages ChatGPT's excellent text generation capabilities to automatically generate RTL code. The framework was tested on an FPGA, and experimental results show that it can flexibly generate various computational kernels based on different configuration information. The proposed hardware accelerator supports more neural network operations than other similar accelerators and exhibits satisfactory hardware efficiency. The embodiments of this invention are described in detail below.

[0044] Reference Figure 1 This invention provides an automated configuration deep neural network accelerator system, including a deep neural network accelerator and a code generation module. The deep neural network accelerator includes off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a computation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer, wherein:

[0045] Both the DMA module and the controller are connected to the external memory via a bus interface. The data segmentation module and the data integration module are both connected to the DMA module. The inputs of the weight buffer, the offset buffer, and the first input of the input buffer are all connected to the output of the data segmentation module. The output of the offset buffer is connected to the second input of the input buffer via the address generation engine. The outputs of the input buffer and the weight buffer are both connected to the input of the calculation module. The output of the calculation module is connected to the input of the output buffer via the nonlinear operation module. The output of the output buffer is connected to the input of the data integration module. The calculation module is connected to the controller. The output of the code generation module is connected to the controller.

[0046] Specifically, the present invention can automatically generate corresponding computing kernels based on different configuration information, which improves the adaptability of neural network accelerators to various application scenarios and also improves the computing efficiency of neural networks.

[0047] Reference Figure 1As a further optional implementation, the calculation module includes a matrix multiplication block, a matrix transpose unit, an intermediate buffer, and a SoftMax unit. The output terminals of the input buffer and the weight buffer are both connected to the input terminal of the matrix multiplication block. The first output terminal of the matrix multiplication block is connected to the input terminal of the matrix transpose unit. The output terminal of the matrix transpose unit is connected to the input terminal of the SoftMax unit through the intermediate buffer. The second output terminal of the matrix multiplication block, the output terminal of the intermediate buffer, and the output terminal of the SoftMax unit are all connected to the input terminal of the output buffer through a nonlinear operation module. The matrix transpose unit and the SoftMax unit are both connected to the controller signal.

[0048] Reference Figure 2 As an optional implementation, the matrix multiplication block includes multiple processing units, each of which includes multiple multipliers and a first accumulator. The output of the input buffer and the output of the weight buffer are both connected to the input of the processing unit. The output of the matrix multiplication block is obtained by connecting the calculation results of each processing unit.

[0049] Specifically, p and q represent the parallelism of the input and output channels, respectively. Each processing unit contains p multipliers and one accumulator, and a matrix multiplication block is formed by stacking q processing units. Input activation values ​​are broadcast to the q processing units. Simultaneously, filter weights are also fed into the q processing units. The results computed in each processing unit are then concatenated to output the final data.

[0050] Reference Figure 3 As a further optional implementation, the matrix transpose unit includes a first register array and a second register array. The first register array is used to perform a read operation in a first stage and a write operation in a second stage, while the second register array is used to perform a write operation in the first stage and a read operation in the second stage.

[0051] Specifically, to implement matrix transpose on-chip, a ping-pong memory method can be used, employing register arrays to organize and transmit data. In stage 1, array 1 performs read operations, while array 2 performs write operations. Similarly, in stage 2, array 1 performs write operations, while array 2 performs read operations. During write operations, data is transmitted horizontally. During read operations, data is transmitted vertically. The matrix transpose unit can handle the transpose of an n×n matrix.

[0052] Reference Figure 4As an optional implementation, the SoftMax unit includes an exponent module, a second accumulator, a divider, and a FIFO memory. The input of the exponent module is connected to the output of the intermediate buffer. The first output of the exponent module is connected to the first input of the divider through the FIFO memory. The second output of the exponent module is connected to the second input of the divider through the second accumulator. The output of the divider is connected to the input of the output buffer through the nonlinear operation module.

[0053] Specifically, the exponent module processes the input data to obtain the exponent. The exponent stream has two paths: one flows directly into the accumulator to calculate the denominator of the divider, and the other flows into the FIFO (First-In-First-Out) memory to be used as the numerator of the divider. Because each data point is restricted to (0,1) after SoftMax processing, the integer part is always zero, and therefore, the integer part can be discarded.

[0054] Reference Figure 5 As an optional implementation, the address generation engine is used to obtain the absolute coordinates of the two-dimensional offset of the offset buffer input, and superimposes the absolute coordinates with the initial input activation value to obtain the offset input address, and then transmits the offset input address to the input buffer.

[0055] Specifically, m and n represent the depths of the offset buffer and input buffer, respectively; Height represents the height of the input feature map; In_ch represents the number of input channels; and p represents the input channel parallelism. The offset buffer data is concatenated from two offsets from different dimensions. Since these two-dimensional coordinates from the offset buffer are relative coordinates of the original feature values ​​in space, absolute coordinates are also needed. These are then superimposed with the initial input activation values ​​on the two-dimensional plane to obtain the offset input address. A multiplexer can be used to select the data path as required.

[0056] Inconsistencies in common operations mainly stem from different receptive fields and sampling methods. This invention proposes a unified sampling algorithm that can transform different computational processes into the same operations based on similar characteristics. Here, they are all transformed into matrix multiplication.

[0057] The following formulas illustrate the specific calculation process for different operations using the uniform sampling algorithm.

[0058]

[0059] Formula (1) is the algorithm for deformable attention and standard attention. Where Ψ ΩΔ It is a set of dynamic input offsets, where Ω represents the receptive field of the input feature map, and p nThis represents the element position of each matrix or its relative coordinates within the kernel window, where N represents the size of the matrix or kernel window, and Δp... n Representing the positional offset in two-dimensional space, Q, K, and V represent the query, key, and value matrices, respectively. and Vectors representing weights and activation values, with indices for all vectors and scalars indicating their spatial coordinates. Taking deformable attention as an example, the process of calculating the key and value matrices involves offsets of sampling positions. By limiting these offsets to small integer values, they can be simply viewed as changes to matrix rows or columns, reducing redundant multiplication operations generated during bilinear interpolation. Calculating the Z matrix also involves two rounds of matrix multiplication, but since it's similar to calculating the query matrix and for brevity, it's simplified; therefore, attention-type operations involve five rounds of matrix multiplication. The process of calculating standard attention is similar to deformable attention, except the offsets for calculating the key and value matrices are removed.

[0060]

[0061] Formula (2) is the matrix multiplication algorithm, which is the most basic operator of the unified sampling algorithm.

[0062]

[0063] Formula group (3) is the algorithm for deformable convolution and convolution. Since convolution is a type of matrix multiplication, it is not difficult to transform it into the above form. The difference between deformable convolution and convolution is whether there is an offset. Standard convolution is the case where the offsets of deformable convolution are all set to 0. Where (i,j) is the center position of the standard convolution window, l x and l y express pn The maximum absolute value between the horizontal and vertical coordinates, where k represents the kernel size.

[0064]

[0065] Formula (4) is the algorithm for transposed convolution. If it is directly treated as a single round of convolution, the result is sparse matrix multiplication, which will cause a significant drop in efficiency. Here, based on the stride, the convolution kernel is decomposed into multiple sub-convolution kernels, and multiple rounds of operation are performed. The results are then concatenated in an orderly manner, thus perfectly solving the sparse matrix multiplication problem of transposed convolution. Here, s represents the stride, and the convolution kernel is divided into s×s sub-convolution kernels. Although the receptive field of the weights is divided into multiple parts, it can greatly improve efficiency. The offsets of the weights are all in the set (including l×l two-dimensional offsets Δp) n (in) p s Indicates the weighted receptive field Ωw The position in the middle.

[0066] As an optional implementation, the code generation module is used to generate corresponding RTL code according to user requirements and transmit the RTL code to the controller.

[0067] Reference Figure 6 As an optional implementation, the code generation module generates RTL code through the following steps:

[0068] Obtain the user requirement prompt file / modify the requirement condition file, input the user requirement prompt file / modify the requirement condition file and the preset template file into ChatGPT to obtain the target Python file;

[0069] Run the target Python file to obtain the RTL code.

[0070] Specifically, this invention utilizes ChatGPT to assist in the design of neural network hardware accelerators. Based on application requirements, RTL code implementing corresponding operations can be generated through interaction with ChatGPT. Since the RTL code for very large-scale integrated circuits is extremely large, often exceeding the word limit that ChatGPT can generate, a shorter Python code template was developed for ChatGPT to generate. This Python file is then used to generate the RTL code. First, two files are provided to ChatGPT: a prompt file outlining the required specifications and a template file providing the template for the generated Python code. If the requirements need to be modified, updated information can be provided to ChatGPT through a condition file, and ChatGPT will generate a modified Python file as feedback. Then, simply running the generated Python file yields the RTL code, which can then be simulated to verify functionality.

[0071] Figures 7(a) to (c) illustrate the condition file writing format for convolution operations, attention operations, and deformable attention operations provided in embodiments of the present invention. "Operation type" indicates the desired operation to be performed. "Activation function" selects a non-linear activation operation, such as ReLU, Leaky ReLU, and GELU. Notably, if an attention-type operation is selected, a non-linear SoftMax operation will be automatically executed. "Feature map size" requires the length, width, input channels, and output channels of the input feature map. In "Filter parameter," the kernel size, stride, and padding need to be entered. Note that for attention-type operations, the kernel size should be set to 1. "Input channel parallelism," "Output channel parallelism," and "Batch size parallelism" are the input channel parallelism, output channel parallelism, and batch size parallelism, respectively; these are hardware parameters that determine the parallelism of the entire circuit. "Buffer configuration" contains configuration information for the various buffers used in the accelerator.

[0072] Reference Figure 8 This invention provides a method for implementing an automatically configured deep neural network accelerator system, which includes the following steps:

[0073] S101. Generate corresponding RTL code according to user requirements through the code generation module, and transmit the RTL code to the controller;

[0074] S102. The controller configures the computing module and DMA module according to the RTL code to obtain the configured deep neural network accelerator.

[0075] S103. Perform neural network computations using a deep neural network accelerator.

[0076] As a further optional implementation, neural network computation is performed using a deep neural network accelerator, specifically including:

[0077] S1031. Load input data from off-chip memory into the DMA module, and perform data segmentation processing on the input data through the data segmentation module to obtain activation data, offset data and weight data. Then store the activation data in the input buffer, store the offset data in the offset buffer, and store the weight data in the weight buffer.

[0078] S1032. The activation data, offset data, and weight data are obtained through the calculation module, and matrix multiplication, convolution, transpose convolution, deformable convolution, dilated convolution, attention, or deformable attention are performed based on the activation data, offset data, and weight data to obtain the calculation results.

[0079] S1033. The calculation results are activated by the nonlinear operation module to obtain the output data, and the output data is stored in the output buffer.

[0080] S1034. The output data is integrated and processed by the data integration module to obtain integrated output data, and the integrated output data is written back to the off-chip memory through the DMA module.

[0081] Specifically, in the data preparation phase, input data is loaded from off-chip memory into DMA, then segmented and allocated to corresponding buffers. The computation kernel's input and offset buffers contain N blocks, storing the corresponding activation and offset values ​​respectively. The weight buffer has only one block, broadcast to N matrix multiplication blocks. The obtained results are stored in the corresponding block of the output buffer, also divided into N blocks. The address generation engine generates offset addresses by sampling and then reads data from the input buffer. The nonlinear operation module executes the activation function. After computing a batch of data, the results can be stored in the output buffer or used to perform other operations according to specific requirements. Finally, the output data is written back to off-chip memory via DMA.

[0082] In some optional embodiments, the corresponding hardware accelerator can be generated by changing the parameters of the condition file format corresponding to the convolution operation. During the data preparation phase, input data is loaded from off-chip memory into DMA, then the data is segmented and allocated to the corresponding buffers. Because the receptive field of standard convolution does not shift, the multiplexer in the address generation engine selects the path with an output of 0. The nonlinear operation module in this embodiment executes the ReLU activation function. After calculating the result, the matrix multiplication block directly stores the data in the output buffer and then writes it back to off-chip memory via DMA.

[0083] In some optional embodiments, the corresponding hardware accelerator can be generated by changing the parameters of the condition file format corresponding to the attention operation. During the data preparation phase, input data is loaded from off-chip memory into DMA, then the data is segmented and allocated to the corresponding buffers. Because the receptive field of standard attention does not shift, the multiplexer in the address generation engine selects the path with an output of 0. The attention type automatically performs the SoftMax non-linear operation and automatically sets the kernel size to 1. The attention operation performs five rounds of computation in the matrix multiplication block. The first three rounds calculate the values ​​of the query, key, and value matrices and store them in an intermediate buffer, where the key value is transposed by the matrix transpose module before being stored in the intermediate buffer. The last two rounds read data from the intermediate buffer, perform the SoftMax operation, store the result in the output buffer, and then write it back to off-chip memory via DMA.

[0084] In some optional embodiments, the corresponding hardware accelerator can be generated by changing the corresponding parameters according to the condition file writing format corresponding to the deformable attention operation. During the data preparation phase, input data is loaded from off-chip memory into DMA, then the data is segmented and allocated to the corresponding buffers. Because the receptive field of the deformable attention is shifted, the address generation engine reads data from the offset buffer to calculate the offset address of the input buffer. The attention type automatically performs the SoftMax non-linear operation and automatically sets the kernel size to 1. The deformable attention operation performs five rounds of operations in the matrix multiplication block. The first three rounds calculate the values ​​of the query, key, and value matrices and store them in the intermediate buffer, where the receptive field of the key and value is shifted, and the key value is transposed by the matrix transpose module before being stored in the intermediate buffer. The last two rounds read data from the intermediate buffer, perform the SoftMax operation, store the result in the output buffer, and then write it back to off-chip memory via DMA.

[0085] The above describes the automated configuration deep neural network accelerator system and implementation method according to embodiments of the present invention. It can be understood that the present invention proposes a unified sampling algorithm that can convert various operations into an adaptive algorithm based on matrix multiplication. Based on this algorithm, a low-hardware-complexity, efficient, and flexible deep neural network accelerator is designed. This deep neural network accelerator effectively handles multiple types of operations with a unified input / output layout format, alleviating problems related to irregular memory access caused by deformable attention and deformable convolution. Based on ChatGPT, the present invention proposes an automatic code generation strategy that generates concise Python code from template files, which is then used to generate corresponding RTL code. This solves the problem of text length limitations when generating large-scale RTL code. The present invention also proposes an automated deep neural network accelerator hardware design framework based on ChatGPT. This framework can automatically generate RTL code for the corresponding neural network accelerator according to given conditions, and it has advantages in hardware efficiency and hardware implementation agility.

[0086] Understandably, regarding hardware flexibility, irregular sampling of the receptive field easily leads to memory access conflicts, causing many accelerators to not support complex operations such as deformable attention, deformable convolution, and transposed convolution. However, the unified sampling algorithm proposed in this invention eliminates memory access conflicts with its dynamic sampling mode, making it suitable for various complex scenarios. Furthermore, compared to accelerators applicable to a single or multiple operations, this invention can support a wider range of operations, meaning it can be applied to accelerate more tasks. At the system level, this invention's novel automated hardware design framework based on ChatGPT sessions can not only automatically generate code files through dialogue but also flexibly select operation types according to specific requirements to meet the hardware acceleration needs of various network types.

[0087] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The methods described above can be implemented using standard programming techniques—including implementation in a computer program on a non-transitory computer-readable storage medium configured to allow the computer to operate in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0088] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The aforementioned computer programs include a plurality of instructions executable by one or more processors.

[0089] Furthermore, the above methods can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention also includes the computer itself.

[0090] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including a specific visual depiction of physical and tangible objects generated on the display.

[0091] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. An automated configuration deep neural network accelerator system, characterized in that, It includes a deep neural network accelerator and a code generation module. The deep neural network accelerator includes off-chip memory, a bus interface, a DMA module, a controller, a data segmentation module, a data integration module, an address generation engine, a computation module, a nonlinear operation module, an input buffer, an offset buffer, a weight buffer, and an output buffer, wherein: The DMA module and the controller are both connected to the off-chip memory via the bus interface. The data segmentation module and the data integration module are both connected to the DMA module. The input of the weight buffer, the input of the offset buffer, and the first input of the input buffer are all connected to the output of the data segmentation module. The output of the offset buffer is connected to the second input of the input buffer via the address generation engine. The outputs of the input buffer and the weight buffer are both connected to the input of the calculation module. The output of the calculation module is connected to the input of the output buffer via the nonlinear operation module. The output of the output buffer is connected to the input of the data integration module. The calculation module is connected to the controller. The output of the code generation module is connected to the input of the controller. The code generation module generates corresponding RTL code according to user requirements and transmits the RTL code to the controller. The controller configures the computation module and DMA module according to the RTL code to obtain a configured deep neural network accelerator, and performs the following neural network computation steps through the deep neural network accelerator: Input data is loaded from off-chip memory into the DMA module, and the input data is segmented by the data segmentation module to obtain activation data, offset data and weight data. The activation data is then stored in the input buffer, the offset data is stored in the offset buffer, and the weight data is stored in the weight buffer. The activation data, offset data, and weight data are obtained through the calculation module, and matrix multiplication, convolution, transpose convolution, deformable convolution, dilated convolution, attention, or deformable attention operations are performed based on the activation data, offset data, and weight data to obtain the calculation results. The calculation result is activated by a nonlinear operation module to obtain output data, and the output data is stored in an output buffer. The output data is integrated by the data integration module to obtain integrated output data, and then the integrated output data is written back to the off-chip memory by the DMA module.

2. The automated configuration deep neural network accelerator system according to claim 1, characterized in that: The calculation module includes a matrix multiplication block, a matrix transpose unit, an intermediate buffer, and a SoftMax unit. The output terminals of the input buffer and the weight buffer are both connected to the input terminal of the matrix multiplication block. The first output terminal of the matrix multiplication block is connected to the input terminal of the matrix transpose unit. The output terminal of the matrix transpose unit is connected to the input terminal of the SoftMax unit through the intermediate buffer. The second output terminal of the matrix multiplication block, the output terminal of the intermediate buffer, and the output terminal of the SoftMax unit are all connected to the input terminal of the output buffer through the nonlinear operation module. The matrix transpose unit and the SoftMax unit are both connected to the controller signal.

3. The automated configuration deep neural network accelerator system according to claim 2, characterized in that: The matrix multiplication block includes multiple processing units, each of which includes multiple multipliers and a first accumulator. The output of the input buffer and the output of the weight buffer are both connected to the input of the processing unit. The output of the matrix multiplication block is obtained by connecting the calculation results of each processing unit.

4. The automated configuration deep neural network accelerator system according to claim 2, characterized in that: The matrix transpose unit includes a first register array and a second register array. The first register array is used to perform a read operation in a first stage and a write operation in a second stage. The second register array is used to perform a write operation in the first stage and a read operation in the second stage.

5. The automated configuration deep neural network accelerator system according to claim 2, characterized in that: The SoftMax unit includes an exponent module, a second accumulator, a divider, and a FIFO memory. The input of the exponent module is connected to the output of the intermediate buffer. The first output of the exponent module is connected to the first input of the divider through the FIFO memory. The second output of the exponent module is connected to the second input of the divider through the second accumulator. The output of the divider is connected to the input of the output buffer through the nonlinear operation module.

6. The automated configuration deep neural network accelerator system according to claim 1, characterized in that: The address generation engine is used to obtain the absolute coordinates of the two-dimensional offset of the offset buffer input, and superimpose the absolute coordinates with the initial input activation value to obtain the offset input address, and then transmit the offset input address to the input buffer.

7. The automated configuration deep neural network accelerator system according to claim 1, characterized in that, The code generation module generates the RTL code through the following steps: Obtain the user requirement prompt file / modify the requirement condition file, and input the user requirement prompt file / modify the requirement condition file and the preset template file into ChatGPT to obtain the target Python file; Run the target Python file to obtain the RTL code.

8. A method for implementing an automatically configured deep neural network accelerator system, used to implement it through an automatically configured deep neural network accelerator system as described in any one of claims 1 to 7, characterized in that, Includes the following steps: The code generation module generates corresponding RTL code according to user requirements and transmits the RTL code to the controller. The controller configures the computation module and DMA module according to the RTL code to obtain a configured deep neural network accelerator. Neural network computations are performed using the deep neural network accelerator.

Images