A general CNN accelerator based on row-fixed data flow

Abstract

The present invention provides a universal CNN accelerator based on fixed-row data streams, comprising an AHB bus interface module, a control register module, a state controller module, a DMA data handling module, an on-chip BRAM buffer module, a PE array module, an adder tree module, and an external DDR memory module. The present invention has the beneficial effect of configuring optimal parameters matching the neural network structure based on network parameters through the control register module, flexibly adjusting the array to adapt to different neural network model structures, and achieving superior acceleration and computational speed within certain resource constraints.

Description

Technical Field

[0001] The present invention relates to the technical field of accelerators, and in particular to a universal CNN accelerator based on row-fixed data streams. Background Art

[0002] In recent years, with the advancement of artificial intelligence, deep neural networks have achieved remarkable results in areas such as text recognition, object detection, face recognition, and speech recognition. As application scenarios become more complex, deep neural network models are also growing in size, requiring significant amounts of storage and computation. Therefore, accelerating and implementing deep neural network algorithms on hardware has become a critical issue in the field of machine learning.

[0003] In the early days, convolutional neural network accelerators primarily relied on CPUs (Central Processing Units) and GPUs (Graphic Processing Units) to accelerate convolutional neural networks. However, using CPUs to execute algorithms with large amounts of data and computational complexity resulted in slow performance, while using GPUs, while faster, also resulted in high power consumption. Therefore, to improve acceleration efficiency and reduce power consumption, fully customized ASIC (Application Specific Integrated Circuit) accelerators or semi-custom FPGA (Field Programmable Gate Array) accelerators are often used on real-time, high-efficiency, and low-energy edge mobile devices to accelerate the forward inference process of convolutional networks.

[0004] Based on the above analysis, it is necessary to propose a convolutional neural network accelerator based on row-fixed data flow and parameterized configurable storage and computing resources to achieve excellent energy efficiency and versatility indicators. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a universal CNN accelerator based on row fixed data stream with better energy efficiency and universality indicators.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a universal CNN accelerator based on fixed row data flow, including: an AHB bus interface module, a control register module, a state controller module, a DMA data handling module, an on-chip BRAM buffer module, a PE array module, an addition tree module and an external DDR storage module;

[0007] The AHB bus interface module is electrically connected to an external module for transmitting instructions and data transmitted from the microcontroller module and the external storage module;

[0008] The control register module is used to receive instructions and data transmitted from the AHB bus and generate parameter information and save it in the status command register and network parameter register. The parameter information is sent to the PE array module for folding and spatial mapping, and the parameter information is sent to the state controller module so that the state controller module issues different commands.

[0009] The state controller module is used to receive parameter information and generate corresponding state commands to the DMA data handling module and the PE array module, so that the DMA data handling module performs data transfer and the PE array module performs data calculation;

[0010] The DMA data handling module is used to receive the corresponding command and start the read and write data transmission between the AHB bus and the on-chip BRAM buffer module. When the data transmission is completed, it returns a signal to the status command register, and the status command register generates the next command according to the completion signal;

[0011] The PE array module is used to receive the corresponding command, start reading the data of the on-chip BRAM buffer module and perform calculations to generate the part and the calculation results, and return a completion signal to the state controller module after the calculation is completed;

[0012] The on-chip BRAM buffer area module is used to store weight data, input feature map data, and output feature map data;

[0013] The addition tree module is used to receive partial sum results from the PE array module, accumulate the partial sum results, obtain a final calculation result, and pass it to the on-chip BRAM cache module.

[0014] Furthermore, the control register module includes a network parameter register and a status command register;

[0015] The AHB bus interface module is used to configure the parameters of the network parameter registers, including feature map size, convolution kernel size, number of input channels, number of convolution kernels, padding, stride step size, external DDR memory module address, DMA data transfer module read and write direction, and number of parallel processing channels;

[0016] The AHB bus interface module is used to configure the parameters of the status command register and control the working status of the accelerator, including: DMA operation enable, weight loading enable, PE array configuration enable, and feature map data loading convolution calculation start enable.

[0017] Furthermore, the state controller module performs the state cycle task by reading the network parameter register and the state command register.

[0018] Furthermore, the four state loop tasks include: the state loop of the DMA data handling module reading and writing data, the state loop of the PE array module loading weight data, the state loop of the PE array module folding and spatial mapping, and the state loop of the PE array module loading input feature map data and performing convolution operations.

[0019] Furthermore, the DMA data transport module is responsible for data transport between the external DDR storage module and the on-chip BRAM cache module according to the AHB protocol. The data transported by the DMA data transport module includes weight data, input feature map data and output feature map data.

[0020] Furthermore, the on-chip BRAM cache module includes three pseudo-dual-port SRAM caches, namely a weight cache for caching weight data, an input feature map cache for caching input feature map data, and an output feature map cache for caching output feature map data.

[0021] Furthermore, the PE array module is provided with multiple groups of Pe_Array_groups, the number of Pe_Array_groups is set according to the folding and spatial mapping mode of the PE array, and each group of Pe_Array_group is composed of n*n PE array units, where n is the convolution kernel size.

[0022] Furthermore, each column of n PE array units is connected by a ROUTER for accumulating the output parts and results of a column of array units.

[0023] Furthermore, each ROUTER includes a fifo for storing partial sums and results. When a new partial sum is transmitted, the value of the fifo is read and added to the new value before being stored in the fifo.

[0024] Furthermore, the addition tree module includes an accumulator and an output feature map buffer area, and the accumulator is used to accumulate the output partial sum of Pe_Array_group to obtain the output feature map partial sum and write it into the output feature map buffer area.

[0025] The beneficial effects of the present invention are as follows: a universal CNN accelerator based on fixed-row data flow mainly includes: a control register module, a state controller module, a DMA data transfer module, an on-chip BRAM buffer module, a PE array module, an adder tree module, an external DDR storage module, and an AHB bus interface module. Based on network parameters, the control register module is used to configure optimal parameters matching the neural network structure, and the array is flexibly adjusted to adapt to different scenarios of neural network model structures, achieving a relatively good acceleration effect and computing rate within certain resource constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the mechanisms shown in these drawings without paying any creative work.

[0027] Figure 1 Schematic diagram of the overall architecture of a universal CNN accelerator based on fixed row data streams of the present invention;

[0028] Figure 2 Schematic diagram of the workflow of the universal CNN accelerator based on fixed row data stream of the present invention;

[0029] Figure 3 A schematic diagram of an atomic operation performed by a PE array of the present invention;

[0030] Figure 4 A schematic diagram of a stripe operation performed by a PE array of the present invention;

[0031] Figure 5 A schematic diagram of a channel operation performed by the PE array of the present invention;

[0032] Figure 6 A schematic diagram of a group operation performed by the PE array of the present invention;

[0033] Figure 7 This is a state cycle diagram of DMA reading and writing data in the state controller of the present invention.

[0034] Figure 8 State cycle diagram for loading weights for the PE array in the state controller of the present invention.

[0035] Figure 9 This is a state cycle diagram for PE array folding and space mapping in the state controller of the present invention.

[0036] Figure 10 A state cycle diagram for loading the input feature map and performing convolution operations on the PE array in the state controller of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] It should be noted that the descriptions of "first", "second", etc. in the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0039] like Figure 1 As shown, an embodiment of the present invention is: a universal CNN accelerator based on a fixed row data stream, comprising: an AHB bus interface module 100, a control register module 200, a state controller module 300, a DMA data handling module 400, an on-chip BRAM buffer module 500, a PE array module 600, an addition tree module 700 and an external DDR memory module 800;

[0040] AHB bus interface module 100, AHB bus interface module 100 is electrically connected to the external DDR memory module 800, for transmitting instructions and data transmitted from the CPU and the external memory module;

[0041] The control register module 200 is electrically connected to the AHB bus interface module 100, the PE array module 600, and the state controller module 300, and is configured to receive instructions and data transmitted via the AHB bus, generate parameter information, and store it in the network parameter register 210 and the state command register 220. The control register module 200 sends the parameter information to the state controller module 300, causing the state controller module 300 to issue different commands, and sends the parameter information to the PE array module 600 for folding and spatial mapping.

[0042] The state controller module 300 is electrically connected to the control register module 200 and the PE array module 600. The state controller module 300 is electrically connected to the DMA data handling module 400 and is configured to receive parameter information and generate corresponding state commands to the DMA data handling module 400 and the PE array module 600, thereby enabling the DMA data handling module 400 to perform data transfer and the PE array module 600 to perform data operations.

[0043] DMA data handling module 400, DMA data handling module 400 is electrically connected to AHB bus module 100 and on-chip BRAM buffer module 500, respectively, for receiving corresponding commands, controlling read and write data transmission between AHB bus module 100 and on-chip BRAM buffer module 500, and when the data transmission is completed, a signal is returned to the state controller module 300, and the state controller module 300 generates the next command according to the completion signal;

[0044] The PE array module 600 is electrically connected to the adder tree module 700 and is used to receive corresponding commands, start reading data from the on-chip BRAM buffer module 500 and perform calculations to generate partial sums of calculation results. Upon completion of the calculations, the module returns a completion signal to the state controller module 300.

[0045] On-chip BRAM buffer module 500, which is electrically connected to the PE array module 600 and is used to store weight data, input feature map data, and output feature map data. When the PE array module 600 needs to use data for calculation, it is responsible for data transmission with the PE array module 600;

[0046] The adder tree module 700 is used to receive the partial sum results from the PE array module 600 , accumulate the partial sum results, obtain the final calculation result, and pass it to the on-chip BRAM buffer module 500 .

[0047] The control register module 200 includes a network parameter register 210 and a status command register 220;

[0048] The AHB bus interface module 110 is used to configure the parameters of the network parameter register 210, including the feature map size, convolution kernel size, number of input channels, number of convolution kernels, padding, stride step size, external DDR memory module address, DMA data transfer module read and write direction, and number of parallel processing channels;

[0049] The AHB bus interface module 110 is used to configure the parameters of the status command register 220 to control the working state of the accelerator, including: DMA operation enable, weight loading enable, PE array configuration enable, and feature map data loading convolution calculation start enable.

[0050] The state controller module 300 performs the state cycle task by reading the network parameter register 210 and the state command register 220 .

[0051] Among them, the four state loop tasks include: the state loop of the DMA data handling module 400 reading and writing data, the state loop of the PE array module 600 loading weight data, the state loop of the PE array module 600 folding and spatial mapping, and the state loop of the PE array module 600 loading input feature map data and performing convolution operations.

[0052] Among them, the DMA data transfer module 400 is responsible for data transfer between the external DDR storage module 800 and the on-chip BRAM cache module 500 according to the AHB protocol. The data transferred by the DMA data transfer module 400 includes weight data, input feature map data and output feature map data.

[0053] Among them, the on-chip BRAM cache module 500 includes three pseudo-dual-port SRAM caches, namely a weight cache for caching weight data, an input feature map cache for caching input feature map data, and an output feature map cache for caching output feature map data.

[0054] The PE array module 600 is provided with multiple groups of Pe_Array_groups 610 . The number of Pe_Array_groups 610 is set according to the folding and spatial mapping method. Each group of Pe_Array_groups 610 is composed of n*n PE array units, where n is the convolution kernel size.

[0055] Each column of n PE array units is connected by a ROUTER 620, which is used to accumulate the output parts and results of the array units in one column.

[0056] Each ROUTER 620 includes a fifo for storing partial sum results. When a new partial sum result is transmitted, the value of the fifo is read and added to the new value before being stored in the fifo.

[0057] Among them, the addition tree module 700 includes an accumulator 710 and an output feature map buffer area 530. The accumulator 710 is used to accumulate the output partial sum of Pe_Array_group 610 to obtain the output feature map partial sum and write it into the output feature map buffer area 530.

[0058] like Figure 2 The following is a flowchart of the operation of a general CNN accelerator based on fixed row data flow. To better illustrate this solution, the following workflow is explained with a specific example. The example is a case where the input feature map size is 112, the number of input channels is 32, the padding is 1, the stride is 1, the convolution kernel size is 3, and the number of convolution kernels is 64.

[0059] 1) After the system is started, the network parameter register 210 part of the control register module 200 is configured through the AHB bus, including setting the convolution kernel size to 3, the number of convolution kernels to 64, the convolution kernel step size to 1, the input feature map size to 112, the number of input feature map channels to 32, and the padding to 1.

[0060] 2) Set the network parameter register 210 related to loading weight data, set the DMA read address to the starting address of the external DDR storage module 800 for storing weight data, set the address range to the size of the weight data, set the DMA read and write direction to read, set the DMA working enable, and DMA starts to move data from the external DDR storage module 800 to the weight cache area.

[0061] 3) Set the network parameter register 210 related to loading the feature map data, set the DMA read address to the starting address of the external DDR storage module 800 for storing the feature map data, set the address range to the size of the feature map data, set the DMA read and write direction to read, set the DMA working enable, and start DMA to move data from the external DDR storage module 800 to the input feature map buffer area.

[0062] 4) The PE array module 600 is folded and spatially mapped based on the deployed network parameters (the number of parallel processing convolution kernels and the convolution kernel size), and is configured into 8 groups of PE_Array_Groups 610. Each PE_Array_Group 610 contains 3x3 PE array units. Each PE array unit is assigned an ID in a diagonal arrangement (for example, when the stride is 1, the first column of the first row is 1, the second column of the first row and the first column of the second row is 2, the third column of the first row and the second column of the second row and the first column of the third row are 3, and so on). Each PE array unit is assigned a flag indicating whether it is the first row or the last row. The calculation results of the PE array units in the first row are directly written into the FIFO in the ROUTER 620. The PE array units in the middle rows read the values ​​in the FIFO in the ROUTER 620, add the values ​​to the calculation results obtained by the PE array units, and then write the values ​​into the FIFO. The PE array units in the last row read the values ​​in the FIFO in the ROUTER 620, add the values ​​to the calculation results, and directly send the values ​​to the adder tree module 700.

[0063] 5) Set the working register weight loading enable, and the weights begin to be transferred from the weight buffer area to the PE array module 600, where the parameters of a total of 8 convolution kernels are transferred at a time, each convolution kernel contains the parameters of 8 channels, and the 8 convolution kernels are respectively transferred to 8 PE_Array_groups, each PE_Array_group contains 3x3 PE array units, that is, 3 rows and 3 columns of PE array units, each row of PE array units shares the same weight, and the 3 rows of weight data of the convolution kernel are respectively transferred to the 3 rows of PE array units.

[0064] 6) Set the working register feature map loading enable, using a feature map multiplexing mode. The eight groups of PE_Array_group 610 share the same input feature map data, and the input feature map data will be broadcast to the PE array unit with the matching ID. When the feature map is loaded, the PE array unit also starts calculation. Taking the workflow of the column PE array unit as an example, the PE array unit in the first row will first obtain the calculation result and send the calculation result to ROUTER 620, write it into the fifo, and then the PE array unit in the second row will obtain the calculation result and send the calculated result to ROUTER 620. At the same time, the fifo will be read and the read value will be added to the calculation result sent by the PE array unit and then sent to the fifo. Then the PE array unit in the third row will obtain the calculation result and send the calculation result to ROUTER 620. At the same time, the fifo will be read and the read value will be added to the result sent by the PE array unit and sent to the addition tree module 700. Each column of PE array units outputs a row of partial sum results of 8 channels. The output combination of the three columns of PE array units at different times corresponds to the final output of the partial sum of all rows of the eight channels, and these results are sent to the output feature map buffer 530 of the adder tree module 700. After calculating the output results of all rows of the eight channels, the PE array module 600 will send a conv_channel_done feedback signal to the state controller module 300.

[0065] 7) When the conv_channel_done signal is detected, the state controller module 300 will reload the weight data and the feature map data into the PE array module 600, and update the 1~8 channel data of the weight data to the 9~16 channel data. The input feature map data is also updated from the 1~8 channel data to the 9~16 channel data. The PE array module 600 starts the calculation and transmits the final output results of all 8-channel rows to the addition tree module 700. At the same time, it reads the output result values ​​of all 8-channel rows of the last output feature map cache 530 and accumulates the output results of all 8-channel rows transmitted this time and puts them back into the output feature map cache 530.

[0066] 8) In the above process, because the number of output channels is 64, PE array module 600 processes the result values ​​of 8 channels at a time. Therefore, the loop repeats 8 times until the eighth round of data calculation for all channels is completed. At this time, the value in output feature map buffer 530 is the final output feature value. After calculating the output results for all channels, PE array module 600 sends a conv_group_done signal to state controller module 300.

[0067] 9) Set the DMA related parameter registers, set the DMA read and write direction to write, set the DMA operation to be enabled, and the DMA starts to move data from the output feature map buffer area 530 to the external DDR storage module 800.

[0068] 10) When the conv_group_done signal is detected, the process of loading the weight data and the input feature map data is reloaded, and the 1-8 convolution kernel data of the weight data are updated to the 9-16 convolution kernel data. The feature map data is read again, and the PE array module 600 starts calculating. Because there are 32 convolution kernel data, 8 convolution kernel data are processed at a time, and the convolution of one layer is calculated in 4 cycles.

[0069] In order to better describe the detailed calculation process of the PE array module 600 of this solution, a specific description will be given below with reference to the accompanying drawings.

[0070] Please refer to Figure 3 , Figure 3 The process of performing an atomic operation on the PE array module 600 of the present invention. The atomic operation is performed on a PE array module 600. A two-dimensional convolution can essentially be broken down into a combination of one-dimensional convolutions. Each PE array unit performs a one-dimensional convolution, performing an atomic operation that convolves a row of the input feature map with a row of the weights to obtain a partial sum of the output feature map.

[0071] ;

[0072] where width R The width of the sum of the output feature maps, width ifmap is the width of the input feature map row, width weight is the width of the weight row.

[0073] The number of multiplications performed by a PE array unit in an atomic operation is ;

[0074] The number of additions a PE array unit performs in one atomic operation is ;

[0075] Please refer to Figure 4 , Figure 4 This is the process of executing a stripe operation in the PE array module 600 of the present invention. The stripe operation is performed on a column of PE array elements. The number of PE array elements in a column depends on the size of the convolution kernel. If the convolution kernel size is N, then Pe_Array_group 610 has N PE array elements in a column. The output feature map portions and results obtained by executing the atomic operation on each PE array element in the column are transmitted to ROUTER 620. ROUTER 620 is responsible for summing all the output feature map portions and results of the column PE array elements to ultimately obtain the row result of the output feature map.

[0076] Please refer to Figure 5 , Figure 5 This is the process of executing a channel operation in the PE array module 600 of the present invention. This channel operation is performed within a PE_Array_group 610. A channel operation processes up to eight channels of data. The number of channels processed can be selected using the network parameter register 210, which has a value between 1 and 8. For the first convolutional layer, which often has three input channels in RGB format, this number can be set to 3. For subsequent convolutional layers, which often have multiples of 8, the maximum number of channels processed can be set to 8.

[0077] A PE_ARRAY_group has 3 columns of PE array units. After one cycle T, the output feature map of 3 rows and 8 channels will be obtained. The channel operation is essentially a combination of stripe operations in time. After N cycles T, the output feature map of 3N rows and 8 channels will be obtained. The cycle to complete a channel operation is ;

[0078] Where T stripe_operation The time to complete a stripe_operation, height ifmap COL is the height of the input feature map after padding. PE is the number of column PEs. For a 64-channel input, a channel operation processes data from a maximum of 8 channels, requiring 8 channel operations.

[0079] Please refer to Figure 6 , Figure 6 The process of executing a group operation for the PE array in the present invention. The number of PE_Array_groups depends on the arrangement of the PE array after folding and spatial mapping. The group operation is performed in N groups of PE_Array_group610. The group operation is essentially the spatial parallelism of the channel operation. A group operation can be considered as a combination of N groups of channel operations. By multiplexing a set of feature map data and convolving them with N convolution kernels respectively, the output feature map results of N channels are obtained. For the case where the output channel is 32, assuming there are 8 groups of PE_Array_group610, one group operation processes the data of 8 convolution kernels at most, and 4 group operations are required.

[0080] Specifically, the folding and spatial mapping process of the PE array module 600 is mainly performed according to the size of the convolution kernel. When the convolution kernel size is 5, the number of PE array units in a group of Pe_Array_group is 5x5; correspondingly, when the convolution kernel size is 3, the number of PE array units in a group of Pe_Array_group is 3x3, and so on. After the folding and spatial mapping of the PE array module 600, the number of Pe_Array_groups is ;

[0081] where Num PE is the total number of PE array units in the PE array module 600, Num pe_of_eachgroupis the number of PEs in each PE_Array_group (3x3, 5x5, depending on the convolution kernel size). This is explained below with a specific example. When the convolution kernel size is 3, the PE array module 600 is folded into 8 Pe_Array_groups 610, spatially mapped to 3x3 PE array elements. Each of the 8 Pe_Array_groups calculates 8 convolution kernels. One row of PE array elements in each Pe_Array_group shares one row of convolution kernel weights, and three rows of PE array elements correspond to three rows of convolution kernel weights. The input feature map is passed to the PE array unit in a diagonal form. For example: at time T1, the first row of data of the input feature map is passed to the PE array unit with ID 1 (first row and first column); at time T2, the second row of data of the input feature map is passed to the PE array unit with ID 2 (first row and second column, second row and first column); at time T3, the third row of data of the input feature map is passed to the PE array unit with ID 3 (first row and third column, second row and second column, third row and first column); at time T4, the fourth row of data of the input feature map is passed to the PE array units with IDs 4 and 1 (third row and second column, second row and third column, first row and first column); at time T5, the fifth row of data of the input feature map is passed to the PE array units with IDs 5 and 2 (third row and third column, first row and second column, second row and first column); and so on.

[0082] The state loop process in the state controller module 300 of the present invention includes: the state loop of the DMA data handling module 400 reading and writing data, the state loop of the PE array module 600 loading weight data, the state loop of the PE array module 600 folding and spatial mapping, and the state loop of the PE array module 600 loading input feature map data and performing convolution operations. Figure 7, the state cycle of the DMA data handling module 400 reading and writing data is described as follows: (1) IDLE: state machine idle state; (2) BUS_REQ: when the state command register enable signal dma_en is received and the hready signal on the AHB bus is high, it jumps to the BUS_REQ state, indicating a request to occupy the AHB bus; (3) WR_AHB: when the hgrant and hready signals sent by the AHB bus are high, and the direct signal (in the read-write bit register) in the control register module 200 is high, it jumps to the WR_AHB state, and at this time starts to write the data in the output feature map buffer area to the external DDR storage module 800; (4) RD_AHB: when the hgrant and hready signals sent by the AHB bus are high, and the direct signal in the control register module 200 is high (the read / write bit in the register) is low, it jumps to the RD_AHB state, at which time the data of the external DDR memory module 800 starts to be written into the cache area; (5) WR_LAST: In the WR_AHB state, there will be a counter counting how much data has been written. When the counter value is equal to transfer_size (transfer size register), addr_finish is pulled high, entering the WR_LAST state, waiting for the last data to be written, and the next state directly enters the IDLE state; (6) RD_LAST: When writing the RD_AHB state, there will be a counter counting how much data has been written. When the counter value is equal to transfer_size (transfer size register), addr_finish is pulled high, entering the RD_LAST state, waiting for the last data to be read, and the next state directly enters the IDLE state. Please refer to Figure 8 , the state cycle of the PE array module 600 loading weights is described as follows: (1) IDLE: state machine idle state; (2) WEIGHTLOADSTART: when the cmd_weightloadstart_ready signal in the control register 200 is high, it enters the state of pre-reading weight data, at this time, reads the relevant registers of the control register 200, obtains the address of the data to be loaded from the weight buffer area, and the number of weight data to be read; (3) WEIGHTLOAD: at this time, the weight data is loaded from the weight buffer area 510 to the PE array module 600, and the counter starts counting. When the value of the counter is equal to the number of required weight data, the weightload_finish signal is pulled high and returns to the IDLE state. Please refer to Figure 9, the state cycle of folding and spatial mapping of PE array module 600: (1) IDLE: state machine idle state; (2) CONFSET: when the cmd_confset_ready signal in the control register module 200 is high, enter the PE array module 600 configuration state, and perform the folding and spatial mapping process of PE array module 600 according to the convolution kernel size and step value in the parameter register. When the confset_finish signal is detected, return to IDLE state. Please refer to Figure 10 , the PE array module 600 loads the input feature map data and performs the state loop of the convolution operation: (1) IDLE: the idle state of the state machine; (2) CONVDAT: when the cmd_convstart_ready signal in the control register module 200 is high and the needpadding signal is low, it enters the state of transmitting the input feature map. At this time, the input feature map data is loaded from the input feature map buffer 520 to the PE array module 600, and the PE array module 600 performs calculations. When a row of input feature map data is transmitted, if padding is required, the convdat2padding signal is pulled high and the state returns to the CONVPADDING state; (3) CONVPADING: when the cmd_convstart_ready signal in the control register module 200 is high and the needpadding signal is low, it enters the state of padding the input feature map with zeros. At this time, data 0 is transmitted to the PE array module 600, and the PE array module 600 performs calculations. When the padding is completed with 0, padding2convdat is pulled high and returns to the CONVDAT state; (4) CONVDOING: When all the data of the input feature map is loaded, it will enter this state. At this time, the PE array module 600 still needs to calculate for a period of time and pass the calculation results to the addition tree module 700.

[0083] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A general CNN accelerator based on fixed-row data flow, characterized in that: include: AHB bus interface module, control register module, state controller module, DMA data handling module, on-chip BRAM cache module, PE array module, addition tree module and external DDR storage module; The AHB bus interface module is electrically connected to an external module for transmitting instructions and data transmitted from the CPU and an external storage module; The control register module is used to receive instructions and data transmitted from the AHB bus and generate parameter information and save it into the status command register and the network parameter register, send the parameter information to the PE array module for folding and space mapping, and send the parameter information to the status controller module so that the status controller module issues different commands; The state controller module is used to receive parameter information and generate corresponding state commands to the DMA data handling module and the PE array module, so that the DMA data handling module performs data transfer and the PE array module performs data calculation; The DMA data handling module is used to receive the corresponding command and start the read and write data transmission between the AHB bus and the on-chip BRAM buffer module. When the data transmission is completed, a signal is returned to the status command register, and the status command register generates the next command according to the completion signal; The PE array module is used to receive the corresponding command, start reading the data of the on-chip BRAM buffer module and perform calculations to generate the part and the calculation results, and return a completion signal to the state controller module after the calculation is completed; The on-chip BRAM buffer area module is used to store weight data, input feature map data, and output feature map data; The addition tree module is used to receive partial sum results from the PE array module, accumulate the partial sum results, obtain a final calculation result, and pass it to the on-chip BRAM cache module.

2. The universal CNN accelerator based on row fixed data stream according to claim 1, characterized in that The control register module includes a network parameter register and a status command register; The AHB bus interface module is used to configure the parameters of the network parameter registers, including feature map size, convolution kernel size, number of input channels, number of convolution kernels, padding, stride step size, external DDR memory module address, DMA data transfer module read and write direction, and number of parallel processing channels; The AHB bus interface module is used to configure the parameters of the status command register and control the working status of the accelerator, including: DMA operation enable, weight loading enable, PE array configuration enable, and feature map data loading convolution calculation start enable.

3. The universal CNN accelerator based on row fixed data stream according to claim 2, characterized in that The state controller module performs state cycle tasks by reading the network parameter register and the state command register.

4. The universal CNN accelerator based on row fixed data stream according to claim 3, characterized in that The four state loop tasks include: the state loop of the DMA data handling module reading and writing data, the state loop of the PE array module loading weight data, the state loop of the PE array module folding and spatial mapping, and the state loop of the PE array module loading input feature map data and performing convolution operations.

5. The universal CNN accelerator based on row fixed data stream according to claim 1, characterized in that The DMA data handling module is responsible for data handling between the external DDR storage module and the on-chip BRAM buffer module according to the AHB protocol. The data handled by the DMA data handling module includes weight data, input feature map data and output feature map data.

6. The universal CNN accelerator based on row fixed data stream according to claim 5, characterized in that The on-chip BRAM cache module includes three pseudo dual-port SRAM caches, namely a weight cache for caching weight data, an input feature map cache for caching input feature map data, and an output feature map cache for caching output feature map data.

7. The universal CNN accelerator based on row fixed data stream according to claim 1, wherein The PE array module is provided with multiple groups of Pe_Array_groups. The number of Pe_Array_groups is set according to the folding and spatial mapping mode of the PE array. Each group of Pe_Array_group consists of n*n PE array units, where n is the convolution kernel size.

8. The universal CNN accelerator based on row fixed data stream according to claim 7, wherein Each column of n PE array units is connected by a ROUTER, which is used to accumulate the output parts and results of a column of array units.

9. The universal CNN accelerator based on row fixed data stream according to claim 8, wherein Each ROUTER includes a fifo for storing parts and results. When a new part and result is transmitted, the value of the fifo is read and accumulated with the new value and then stored in the fifo.

10. The universal CNN accelerator based on row fixed data stream according to claim 9, characterized in that The addition tree module includes an accumulator and an output feature map buffer area. The accumulator is used to accumulate the output partial sum of Pe_Array_group to obtain the output feature map partial sum and write it into the output feature map buffer area.

Images